Category: Cross-Browser Testing

Posted on

A Frontend Web Developer’s Guide To Testing – Book Summary

In April 2022, I have published my 4th book which is 100% focused on how frontend web application developer’s can leverage the wide plethora of test automation frameworks that continuously evolve and provide more and more capabilities.

The book is available on Packt website (my publisher) as well as on Amazon and other book stores globally. I got great feedback so far from the community both on the importance of this book to practitioners, as well as the specific content. The book was reviewed by Bruno Bosshard with the foreword written by Gleb Bahmutov who’s one of the core leaders of Cypress in the marketplace.

Book Structure

The book consists of 3 main sections:

Section 1: Fundamentals of Web App Testing

This sections has the following main chapters and it offers structured approach to building a solid testing strategy across all methodologies – Exploratory, Functional, Performance, API, Accessibility, and more.

  1. Cross-Browser Testing Methodologies
  2. Challenges Faced by Frontend Web Application Developers
  3. Top Web Test Automation Frameworks
  4. Matching Personas and Use Cases to Testing Frameworks
  5. Introducing the Leading Frontend Web Development Frameworks

Section 2: Continuous Testing Strategy for Web App Developer’s

This section provides an overview of the criteria that frontend web application developers should look for when choosing a test automation framework, and specifically look into test coverage strategy for web apps.

  1. Map the Pillars of a Dev Testing Strategy for Web Applications
  2. Core Capabilities of the Leading JavaScript Test Automation Frameworks
  3. Measuring Test Coverage of the Web Application

Section 3: Frontend JavaScript Web Test Automation Framework Guides

The final section of the book dives deeper into the features and differences across the 4 leading web application testing frameworks, and concludes the section with an overview of some low-code testing tools that are derived from some of these testing frameworks.

  1. Working with the Selenium Framework
  2. Working with the Cypress Framework
  3. Working with the Playwright Framework
  4. Working with the Puppeteer Framework
  5. Complementing Code-Based Testing with Low-Code Test Automation

Overview of What’s Changing within Web Application Testing?

While writing the book i already became aware of few main trends that in my opinion will shape the future of the testing practice over the next few years.

  • Leveraging CDP to enhance test automation coverage, auditing of performance, network traffic and accessibility.
    • Selenium 4 added support for this rich protocol
    • Playwright and Puppeteer are build on top of CDP
    • Cypress integrates with CDP to benefit from its core features
  • Introduction of the modern concept of Component Testing!
    • Cypress version 10 officially supports Component Testing that allows isolation of a web application component for more rigor and focused testing.
    • Latest Playwright release also starts referring to component testing
Cypress Component Testing of React App (source: Cypress Blog)
  • Built-in low-code abilities within code-based testing frameworks
    1. Selenium 4 introduces a revamped version of the Selenium IDE
    2. Playwright offers its Code-Gen test recorder
    3. Cypress integrates with the Chrome browser recorder
  • Community contributions and Plugins!
    • Open source software can only grow through its communities and the level of engagement and support that such tools receive. With all of the above frameworks we see tremendous communities that provide real-time support through slack, gitter, Discord, GIT code sample, and a lot of customized plugins. With these plugins, test frameworks like Cypress, selenium, playwright and puppeteer enhances their features to cover visual testing, integrate with CI servers, accessibility testing, code coverage, API testing, CDP integration, and much more.

Bottom line

I do hope that this book will provide value to any frontend web application developers and test automation engineers, and serve them for the coming years. The digital transformation continues to evolve with modern web apps like progressive web apps (PWAs), responsive web (RWD), Flutter, and others. With such mature testing tools, practitioners are in a great place today to cover many of the sophisticated use cases, and eliminate bugs earlier in their software iterations.

Happy Testing!

Posted on

Expediting Web Testing with Puppeteer and Instant Test Automation Technology

The digital landscape has significantly advanced over the past 1-2 years. The new normal in web applications include a shift toward progressive web apps (PWAs), advanced responsive websites, and Flutter apps.

Ensuring both functional as well as API testing of such advance web applications is becoming harder and harder. While there are valid test automation frameworks like Selenium and Cypress, these are covering only a subset of what is needed to complete the entire quality aspects across multiple environments.

New Normal: Instant Test Automation

To address the testing challenge over time, across multiple environments, and with big and dynamic data, tools need to be more intelligent, and utilize AI/ML to really track trends, predict issues prior to them becoming more serious, and guide the developers and testers in real-time.

An instant test automation technology that can leverage a standard test automation framework such as Puppeteer to auto generate both functional as well as API tests on the fly and as things change is a game changer for app developers. This provides alerts and insights in real time that the team can act upon and move much faster.

The way sch solution works is through modeling the website under test, auto generate based on traffic and AI proper tests and runs them to provide the proper insights.

Setting up such solution is easy and needs to be done only once. Also, such tool fits systems that are built atop cloud-native, containers and Amazon ECS.

Benefits of Shifting into Instant and Intelligent Test Automation

With the need to shift left, release faster with confidence, and handle large and complex environment, teams can benefit from a self-generating testing solution that really covers more grounds that a typical code-based framework would. This does not mean that there are no benefits or need for such frameworks like Selenium and others, however, with tools like UP9, teams can complement the code related functional tests with an additional layer of quality validation that touches reliability, APIs, services, security and much more.

Case Study: Quick Tour of Puppeteer & UP9

I took the Google puppeteer framework and connected it with UP9 to try generating a nice model and test automation assets on my local machine.

After following the basic NPM installation instructions, it was easy to launch a local browser and record traffic using the tool.

From my terminal I would then run this command (‘my.new.workspace’ could be any name that the developer selects):

up9 tap:start my.new.workspace

Once I ran that command, using puppeteer, UP9 launches a chrome browser that I can start using to navigate to my website for recording all traffic, actions, API calls and other services. Upon closing the browser, the tool will process all of the flows and generate the model.

Here is the auto generated workspace that was created based on a quick tour I ran with UP9 on the Perforce and Perfecto websites:

The workspace consists of few views:

  1. The generic workspace with all the high-level services model (diagram),
  2. The contracts that were used in my recording,
  3. A detailed traffic analysis with the entire API calls,
  4. Tests that were created and mocks to be used across different environments.

These views were perceived and constructed based on service-traffic, following from the observers of UP9 that were assigned to my designated workspace.

As part of my journey, I landed on the Perfecto free-trial website, and the tool was able to generate the API calls (Request/Response) and allow me to validate that is was working as expected (see below)

If I wanted to also run my auto-generated tests, I have within the UP9 portal (fully SaaS and web-based) a tests area from where I could launch my tests as well.

Bottom Line

As the DevOps market mature towards more intelligent testing using AI and ML, and as the systems under tests (SUTs) are also progressing, there needs to be a realization within the teams that their processes could be improved dramatically using such technologies. Developers productivity together with consistent app quality were a CIO/IT manager top of mind for a long while, and with the recent advancements in the market, teams can really enjoy a new era of smarter ways of working.

Posted on

Code-Based Test Automation vs. Codeless Automation

 

As more advanced technologies are entering the continuous testing landscape powered by AI/ML, organizations and especially, practitioners, are debating which is better, and why if any should they adopt codeless test authoring solutions?

In this blog, I will be providing the various considerations to switch, and/or combine between the 2 test automation methods.

TL,DR –> There isn’t a magic answer to this debate, and there isn’t a method that is good vs. bad.

Top Considerations

To better address the question on when and why to use either methods, here are the top items to consider, not listed by importance, since each team might relate to different objectives and priorities:

  • What are the application use cases and flows to automate?
  • Which persona is going to create and maintain these scenarios?
  • What are the skill-set within the team/individuals for the job?
  • Is the app under test Mobile/web/responsive/desktop?
  • What are the time constrains for the project, and what are the release cadence going forward (weekly/monthly)?
  • Is the test suite meant to be integrated to other tools (CI/CD/Frameworks)?
  • Are there any advanced automation scenarios (chatbots, IOT, location, audio, etc.)?
  • What are the cost boundaries (tools, project, lab, etc.)
  • Is the test suite meant to be executed at large scale?
  • Is the project a new one, or an additional layer of coverage on top of existing code-based suite? (Selenium/Appium, etc.)

Diving Deeper

Now that we’ve listed some important consideration, let’s explain them a bit deeper.

For teams that are already working on a project (web/Mobile), and have implemented a good working amount of code-based test scenario that are embedded into processes, CI/CD, and other triggers, such consideration should be heavily considered – what is the incentive to change? are there coverage gap in the code-based suite? is there too much noise and flakiness attached to the existing test code? etc. Only if there is a good incentive like the ones mentioned, teams should consider adding codeless test scenarios into their pipeline.

On the other hand, for teams that are just starting a new project, that’s a perfect timing to look into the entire team skills, and decide based on the technology the project is built on, what tools to use – if might make sense for example, for a newly created website to combine a Selenium framework that SDETs that are Java/JavaScript developers would lead together with business testers that can remove some of the load from them via ML-Driven codeless selenium tools.

After covering use cases, quality of existing test suite, new vs. existing projects, lets also consider the time-frames and budget allocated to the project. It is clear that recording codeless scripts takes on average 6-10 times faster than coding the same scenario in Java or other development language. It involves setting up the platform and test environment, coding, debugging, executing at scale, assertions, and more. This obviously also translate into $$ savings. On the other hand, not all test scenarios can be easily recorded, since, for some advanced flows, coding might be a better approach and an easier one to maintain over time. This is why, sometimes it is better to look at the job-to-be-done, prior to rushing into scripting.

Code and Codeless Creation is Key for High Test Automation Coverage

Next in the overall debate is the Eco-system and tools landscape within the organization. Including a new technology is not easy, often not well-accepted, and also not always justified. In today’s reality when squad teams are working together and consists of a variety of resources with varying skills, objectives, and preferences, integrating a new technology should be done with proper consideration, and with proper validation that it “plays” nice within the other existing tools. Codeless tools in that context, should fill in an important gap within the team, integrate well into existing CI/CD and other processes, and not cause duplication of effort, or additional overhead.

Lastly for this blog (not for the entire debate), I would touch the cost of maintenance of test automation scripts. This is perhaps one of the problematic items for any test automation team. Writing a script once, making it run across platforms over time is an easier said than done task. Applications are constantly changing, and so does the platforms under test (mobile devices/OS versions, Browsers), therefore, scripts needs to be properly maintained to ensure clean and noise-free pipeline. Codeless in many ways addresses such challenge through self-healing of elements, test steps, and more. It can be also achieved within code-based projects through advanced reporting and analysis, with automated root-cause analysis and other methods, but codeless does shine the most in such cases.

Perfecto/TestCraft Object Self-Healing and Weight Mechanism (ML)

Bottom Line

I tried to keep this blog short and to the point, and leave the practitioners the decision-making across the 2 methods. As written in this post, there are plenty of questions to address prior to adopting a codeless tool, and how to combine it within existing code-based suites. Combining both methods in my honest opinion is the way to go in the future, and the way to maximize the overall test automation coverage with greater efficiency across the teams. Make the right decision that fits the project now, and also in the long-run.

I will be running a live webinar covering the exact blog topic hosted by Shani Shoham from 21. Feel free to register here

Posted on

Getting Started With Headless Browser Testing

[Guest Blog by Uzi Eilon, CTO, Perfecto]

The “shift left” trend is actually happening, developers as part of the DevOps pipeline need to test more and add more automation testing in order to release faster.

In addition, those tests are almost the last barrier before production, because the traditional testing is going away.

In such case, the standard unit tests are not good enough, and the E2E tests are complicated and require a longer time of setup and prepare.

This is the reason both Google and Mozilla released new JS headless browsers to help their developers to execute automation tests.

The same happened in the mobile area where Apple and Google released the XCUItest and Espresso.

Headless browsers provide the following capabilities in order for the developers to use it:

  • Same language , same IDE , same working environment:
    Most of the web develops work with  JS so these browsers are JS platform , to add new test you should open new class and write standard JS code.
  • FAST Feedback & Execution
    These tests need to be executed fast (sometime every commit) , these browser reduce the UI and rendering “noise” connect to the element directly  and run very fase.
  • Easy to setup
    Developers time is expensive, and developers will not add complicated processes for test , the setup of the tools is a simple npm installation.
  • Access to all the DevTools capabilities
    Developers need more details , these tools give access to all the DevTools data includes accessibility, network, log , security and more.
    Smart tests can be very powerful and cover not only the functionally but also the efficiency

In order to understand more I played with Puppeteer and I’m happy share my thoughts with you.

Installation

Very simple

npm i –save puppeteer

Documentation

Not a lot of examples or discussions about specific issues but I did find the API documentation that contains everything I was looking for.

Objects identification

Intuitive – Same way I connected to my object via any JS  .

Example:

  • by id : page.type(‘#firstName’,‘Uzi’);·
  • by class page.type(‘.class,‘Uzi’);

Sync and waiting for elements

In this case, I have to admit I struggled with the standard wait for navigation command, it was not stable:

await page.waitForNavigation({waitUntil:‘load’})

at the end I used the following :

await page.waitForSelector(‘#firstName’,{visible:true}).then(()=>
{      // do the actions per page
page.screenshot({path: ‘then.png’,fullPage: false})
});

 

UI

As part of my test I tried to verify the screen by taking a screenshot, I liked the way I could change the browser UI capabilities and configure my page:

const fullScreen = {
deviceScaleFactor: 1,
hasTouch: false,
height:  2400 ,
isLandscape: false,
isMobile: false,
width: 1800,
fullPage: true
};
page.setViewport(fullScreen)

 

Other devOpts options:

it is very easy to use, for example I would like to see all my links in a frame

for (let child of frame.childFrames())
{
dumpFrameTree(child, indent + ‘  ‘);
}

 

Summary:

Using the headless browser like Puppeteer was very easy and intuitive, it felt natural to add it as part of my testing code.

In addition, setting up the headless browser environment and executing was very simple and fast.

On the less convenient point, what I found was that to get the results directly into the CI, one should add more scripting code or use other executions methods.

Lastly, this method still ramps up, hence has some small bugs in few features and also lacks documentation and more samples, however, for an early testing tool for white-box/unit testing, it is very promising and well-positioned to complement tools such as Selenium. AS a matter of fact, what I also saw, is that other browser vendors are taking the same approach and investing in headless browsers – Mozilla, Microsoft.

 

P.S: If you want to learn more about the growing technologies and trends in the market, I encourage you to follow My podcast with Uzi Eilon called Testium (Episode 6 is fully dedicated to this subject)

Posted on

The Rise of Progressive Web Apps and The Impact on Cross-Browser Testing

If we all thought we’ve figured out the digital market from an application type perspective, and have seen the rise of mobile, and the transformation of web to responsive web – now we should all start getting used to a whole new type of application that should change the entire user experience and offer new web functionality – Meet PWA’s.

Google.com is a very clear example of such app, and Apple is about to introduce PWA capabilities in its upcoming WebKit engine.

What are Progressive Web Apps?

If to refer to Google official website dedicated to PWA, Google defines PWA as “A new way to deliver amazing user experiences on the web”

David Rubinstein from SD Time, actually add even more insights into these new app types:

PWAs can use device features like cameras, data storage, GPS and motion sensors, face detection, Push notifications, and more. This will pave the way for AR and VR experiences, right on the web. Imagine being able to redecorate your home virtually using nothing but your phone and a PWA. Pan your camera around a room, then use tools on a website to change wall colors, try out furniture, hang new artwork, and more. It may feel like a futuristic fantasy, but it’s close to reality.

The key behind PWA apps is to provide a rich end-user alternative to native apps. These apps can be launched from the device home screen adding layers of performance, reliability, and functionality to a web application without the need to install anything from the app store. In addition, these apps that are still JavaScript based, but with additional specific API’s can work even when there’s no internet connection and that’s  a huge advantage.

PWA apps leverage 2 main architectural features:

  • Service Workers – give developers the ability to manually manage the caching of assets and control the experience when there is no network connectivity.
  • Web App Manifest – That’s the file within the PWA that describe the app, provide metadata specific to the app like icons, splash screens and more. See below an example Google offers for such a descriptor file (Json)
{
  "short_name": "AirHorner",
  "name": "Kinlan's AirHorner of Infamy",
  "icons": [
    {
      "src": "launcher-icon-1x.png",
      "type": "image/png",
      "sizes": "48x48"
    },
    {
      "src": "launcher-icon-2x.png",
      "type": "image/png",
      "sizes": "96x96"
    },
    {
      "src": "launcher-icon-4x.png",
      "type": "image/png",
      "sizes": "192x192"
    }
  ],
  "start_url": "index.html?launcher=true"
}

In order to check the correctness of your PWA checklist and the entire app, Google offers some tools as part of their documentation like this Progressive Web Apps checklist, and their chrome built-in DevTools (see below visual),

As deeply covered in this great Dzone article, good PWAs also implement the PRPL pattern recommended by Google to enhance performance.

What Are the Implications of PWAs for Cross-Browser sites and Mobile Apps?

To understand the implications, I recommend dividing the question into the impacted Personas.

  1. Developers
  2. Testers
  3. Business
  4. End-Users

Each of the above Personas will have different benefits and implications when adopting this kind of apps.

Developers

For existing web developers, this new app type should present a whole new world of innovative opportunities. Since PWAs are still JavaScript based apps, developers do not need to gain new skills, but rather learn the new APIs offered through the Service Workers and see how they can be leveraged by their websites.  Since the PWA app runs on a mobile device and can be launched without a network connection and without any installation, obviously it needs to be validated by developers through unit and integration tests.

Going forward, the market envisions these apps impacting the native apps architecture in a way that there will only be 1 type of app that can seamlessly run on both browsers and mobile devices with one single implementation – that will require a heavier lift and re-work.

 

Testers

For testers, as in every new implementation, new tests (manual, automated) needs to be developed, executed and fit into the overall pipeline.

PWAs, in particular, introduces some unique use cases such as

  • No network operation
  • High performance 
  • Sensors based functionality (Location, Camera for AR/VR and more)
  • Cross-device functionality (like in Responsive, the experience should be the same regardless of the screen size/HW etc.)
  • Adhering to the design and checklist required by Google and soon Apple
  • Accessibility is always a need
  • Security of these apps (with and without being connected to the network)

Business

For the business, the new app types shall help increase the end-user engagement with the business. When having a web application that is richer in functionality, performs fast, and can be “always on” through an easy launch from the customers’ device home screen, this by definition should increase usage and move the needle to the business. My assumption is that large enterprises are already looking into these type of apps as the next-gen RWD apps.

End Users

At the end of the day, all products are aiming to get greater engagements with the customers and beat the competition. Obviously, if the end users will understand the value in these apps, and can “feel” it in their day by day activities, this will be a clear Win-Win situation to both the organization as well as the customer.

To assure end-user experience as Google envisioned when first launching this technology 3 years ago (2015), Dev and Test teams should continue their continuous testing activities, and make sure they are covering sufficient platform, features and use cases between each release and each new release of a platform or device.

To conclude this blog, I highly recommend watching the short video and read the blog from Mozilla on how PWA live within Firefox and how different experience users get from such apps (see below Firefox Wego app within Firefox browser in the background and a PWA Wego app in the foreground)

Happy PWA Testing!