Expediting Web Testing with Puppeteer and Instant Test Automation Technology

The digital landscape has significantly advanced over the past 1-2 years. The new normal in web applications include a shift toward progressive web apps (PWAs), advanced responsive websites, and Flutter apps.

Ensuring both functional as well as API testing of such advance web applications is becoming harder and harder. While there are valid test automation frameworks like Selenium and Cypress, these are covering only a subset of what is needed to complete the entire quality aspects across multiple environments.

New Normal: Instant Test Automation

To address the testing challenge over time, across multiple environments, and with big and dynamic data, tools need to be more intelligent, and utilize AI/ML to really track trends, predict issues prior to them becoming more serious, and guide the developers and testers in real-time.

An instant test automation technology that can leverage a standard test automation framework such as Puppeteer to auto generate both functional as well as API tests on the fly and as things change is a game changer for app developers. This provides alerts and insights in real time that the team can act upon and move much faster.

The way sch solution works is through modeling the website under test, auto generate based on traffic and AI proper tests and runs them to provide the proper insights.

Setting up such solution is easy and needs to be done only once. Also, such tool fits systems that are built atop cloud-native, containers and Amazon ECS.

Benefits of Shifting into Instant and Intelligent Test Automation

With the need to shift left, release faster with confidence, and handle large and complex environment, teams can benefit from a self-generating testing solution that really covers more grounds that a typical code-based framework would. This does not mean that there are no benefits or need for such frameworks like Selenium and others, however, with tools like UP9, teams can complement the code related functional tests with an additional layer of quality validation that touches reliability, APIs, services, security and much more.

Case Study: Quick Tour of Puppeteer & UP9

I took the Google puppeteer framework and connected it with UP9 to try generating a nice model and test automation assets on my local machine.

After following the basic NPM installation instructions, it was easy to launch a local browser and record traffic using the tool.

From my terminal I would then run this command (‘my.new.workspace’ could be any name that the developer selects):

up9 tap:start my.new.workspace

Once I ran that command, using puppeteer, UP9 launches a chrome browser that I can start using to navigate to my website for recording all traffic, actions, API calls and other services. Upon closing the browser, the tool will process all of the flows and generate the model.

Here is the auto generated workspace that was created based on a quick tour I ran with UP9 on the Perforce and Perfecto websites:

The workspace consists of few views:

  1. The generic workspace with all the high-level services model (diagram),
  2. The contracts that were used in my recording,
  3. A detailed traffic analysis with the entire API calls,
  4. Tests that were created and mocks to be used across different environments.

These views were perceived and constructed based on service-traffic, following from the observers of UP9 that were assigned to my designated workspace.

As part of my journey, I landed on the Perfecto free-trial website, and the tool was able to generate the API calls (Request/Response) and allow me to validate that is was working as expected (see below)

If I wanted to also run my auto-generated tests, I have within the UP9 portal (fully SaaS and web-based) a tests area from where I could launch my tests as well.

Bottom Line

As the DevOps market mature towards more intelligent testing using AI and ML, and as the systems under tests (SUTs) are also progressing, there needs to be a realization within the teams that their processes could be improved dramatically using such technologies. Developers productivity together with consistent app quality were a CIO/IT manager top of mind for a long while, and with the recent advancements in the market, teams can really enjoy a new era of smarter ways of working.

The Future of Test Automation Must Be Intelligent

A decade of Agile, DevOps and Continuous Testing is behind us. I heard this week in a conference I was presenting the phrase “Agile is Dead” and other claims about testing being inefficient in many ways.

While many tools have evolved over the past years, and test automation practices and lessons learned have been communicated to practitioners, it is IMHO not enough to move the needle on the efficiency of test automation and continuous testing – especially in the demanding and ever changing digital space.

To have a better DevOps pipeline, and to minimize both the waste as well as the level of escaped defects, testing and especially test automation must become smarter.

In this post I would list the opportunities for test automation to be more intelligent.

WHAT?

Test automation intelligence should guide engineers on which test automation scenarios to execute, automate if missing, and continuously maintain. This is one of the most complex debates within each automation team – WHAT TO AUTOMATE?

Answering this question is never easy, requires risk management, tradeoffs, and often guessing. In addition, it does not always align with the sprint time frames, hence, being thrown out of the software iteration.

With more intelligent solutions that are based on production data, business driven decision making, code analysis, teams will be in a better position to make such decisions.

I love the below visual that Ingo Philipp and originally, Michal Bolton drew around testing is always sampling. This shows that test automation usually addresses what we know about risks and the product out of the entire ocean of risks and features that we either don’t know, cannot assess, or things that were addressed in the past and there isn’t sufficient data to advise whether we should continue using.

New tools that are rising such as Launchable, SeaLights and others aim to help answer some of the above questions, and to help form a continuously relevant regression suite that matches the most recent product code changes, history of defects and more.

WHEN?

The second question that needs to be answered in a more intelligent way is – When to automate and execute my test scenarios? Some say – everything , everyday, some say – everything shifts left, and some are breaking the tests based on personas, skills, and naïve objectives.

Scheduling and deciding which tests to run when must be smarter, and needs to be tied to the overall process, the feedback loops, and the ability of the developers to act upon such execution feedback. If testers throws on the developers to late or to early the results, it makes it unrealistic for them to prioritize, resolve and fix.

Intelligent systems, must be able to determine based on the above WHAT section, which test scenarios, types, and cadence, such should be executed. In an ideal world, an intelligent system would be able to split between CI, BAT (build acceptance testing), integration, regression, NFT and production synthetic monitoring the entire test suites.

As the above diagram suggest, the pipeline has a pre-defined set of quality gates, phases, and milestones to manage and mitigate risks. An intelligent testing system should effectively place all the right test scenarios –CONTINUOUSLY– in the right phases. Keep in mind, the right test cases change from one release to the next.

WHICH?

Now that WHAT and WHEN questions are addressed, it is important to have a smarter approach into test automation coverage – on WHICH platforms and environments should my test automation scenarios be running constantly? Which mobile devices, which desktop browsers? which backend configurations are important? which data-driven scenarios should be plugged into the test cases? Which load and KPIs should be applied so I get the proper user experience feedback from my testing? and more.

Determining the WHICH in test automation is an art, and require data analytics, market research, customers validation, production monitoring, AIOps systems and more, but it is an imperative for success.

WHY?

After determining few WHs questions in test automation above, it is time for the analysis part of test automation intelligence – We defined what to automate, when to run, and on which platforms and environments, but once executing the tests, can we efficiently determine WHY tests are failing? What is the root cause per each failure so we lower the time for resolution (MTTR)?

Here, intelligent reporting platforms are critical in minimizing the feedback loop, the future decision making on which tests are of higher value, which platforms are more flaky then others and much more.

Without a solid test automation analysis platform, a great chunk of teams investment in testing can be of waste.

It is not that complicated to realize why tests are constantly failing, and put them into buckets of failure reasons. Perfecto is doing that in an automated fashion through its smart reporting SDK, and allows teams to filter a lot of the test noise post a regression cycle completion.

HOW?

Test creation has always been one of the hardest and most challenging task for SDETs and business testers. It includes great technical skills, meeting short time windows for automation, and requires deep understanding of the product requirements to really automate what’s expected. While many test automation frameworks evolved over the past years, Including Selenium, Appium, Espresso, XCUITest, and Cypress, these are all code based frameworks that even when coming on top of a BDD platform, they are hard to maintain over time, and analyze when they result in a failure.

New and more intelligent test automation frameworks that are codeless or low-code have grown and are becoming a new reality of test engineers. Such tools can really help address few key challenges that are prominent. Time to automate, time to maintain, flakiness and skillset gap. Intelligent test creation that is plugged with a TIA can be the future of smarter test automation, and enable practitioners to create the most relevant test scenarios with minimal maintenance and shorter amount of time. As I am writing this blog, such tools are still growing and have a lot to prove to the world prior to being widely adopted, but this in my mind should be the future of test automation, and once this is in place, connecting these tools to the cloud for parallel and scaled testing will be the ultimate solution for continuous testing.

BOTTOM LINE

A lot of what is written in this blog already exists to some extent, some if being developed these days and will be available in 2021. It is important to start drawing within each DevOps team the future of test automation goals and objectives and learn how such solutions will fit the process from test automation creation, execution, maintenance, analysis, TIA, defect management, quality governance and more.

This year, I launched my 3rd book that focuses on “Accelerating Software Quality: AI and ML in the Age of DevOps“. This book looks at the modern DevOps and explores how using AI and ML and smarter algorithms, teams can maximize their productivity and deliver more quality functionality faster.

There is a lot to get excited for in 2021, and especially around test automation in DevOps – Happy New Year and Good Luck in your Future of Test Automation