Posts

Posted on

A Frontend Web Developer’s Guide To Testing – Book Summary

In April 2022, I have published my 4th book which is 100% focused on how frontend web application developer’s can leverage the wide plethora of test automation frameworks that continuously evolve and provide more and more capabilities.

The book is available on Packt website (my publisher) as well as on Amazon and other book stores globally. I got great feedback so far from the community both on the importance of this book to practitioners, as well as the specific content. The book was reviewed by Bruno Bosshard with the foreword written by Gleb Bahmutov who’s one of the core leaders of Cypress in the marketplace.

Book Structure

The book consists of 3 main sections:

Section 1: Fundamentals of Web App Testing

This sections has the following main chapters and it offers structured approach to building a solid testing strategy across all methodologies – Exploratory, Functional, Performance, API, Accessibility, and more.

  1. Cross-Browser Testing Methodologies
  2. Challenges Faced by Frontend Web Application Developers
  3. Top Web Test Automation Frameworks
  4. Matching Personas and Use Cases to Testing Frameworks
  5. Introducing the Leading Frontend Web Development Frameworks

Section 2: Continuous Testing Strategy for Web App Developer’s

This section provides an overview of the criteria that frontend web application developers should look for when choosing a test automation framework, and specifically look into test coverage strategy for web apps.

  1. Map the Pillars of a Dev Testing Strategy for Web Applications
  2. Core Capabilities of the Leading JavaScript Test Automation Frameworks
  3. Measuring Test Coverage of the Web Application

Section 3: Frontend JavaScript Web Test Automation Framework Guides

The final section of the book dives deeper into the features and differences across the 4 leading web application testing frameworks, and concludes the section with an overview of some low-code testing tools that are derived from some of these testing frameworks.

  1. Working with the Selenium Framework
  2. Working with the Cypress Framework
  3. Working with the Playwright Framework
  4. Working with the Puppeteer Framework
  5. Complementing Code-Based Testing with Low-Code Test Automation

Overview of What’s Changing within Web Application Testing?

While writing the book i already became aware of few main trends that in my opinion will shape the future of the testing practice over the next few years.

  • Leveraging CDP to enhance test automation coverage, auditing of performance, network traffic and accessibility.
    • Selenium 4 added support for this rich protocol
    • Playwright and Puppeteer are build on top of CDP
    • Cypress integrates with CDP to benefit from its core features
  • Introduction of the modern concept of Component Testing!
    • Cypress version 10 officially supports Component Testing that allows isolation of a web application component for more rigor and focused testing.
    • Latest Playwright release also starts referring to component testing
Cypress Component Testing of React App (source: Cypress Blog)
  • Built-in low-code abilities within code-based testing frameworks
    1. Selenium 4 introduces a revamped version of the Selenium IDE
    2. Playwright offers its Code-Gen test recorder
    3. Cypress integrates with the Chrome browser recorder
  • Community contributions and Plugins!
    • Open source software can only grow through its communities and the level of engagement and support that such tools receive. With all of the above frameworks we see tremendous communities that provide real-time support through slack, gitter, Discord, GIT code sample, and a lot of customized plugins. With these plugins, test frameworks like Cypress, selenium, playwright and puppeteer enhances their features to cover visual testing, integrate with CI servers, accessibility testing, code coverage, API testing, CDP integration, and much more.

Bottom line

I do hope that this book will provide value to any frontend web application developers and test automation engineers, and serve them for the coming years. The digital transformation continues to evolve with modern web apps like progressive web apps (PWAs), responsive web (RWD), Flutter, and others. With such mature testing tools, practitioners are in a great place today to cover many of the sophisticated use cases, and eliminate bugs earlier in their software iterations.

Happy Testing!

Posted on

Codeless Web Application Testing with Cypress Studio and Playwright CodeGen

While the code-based web application testing framework had significantly advanced over the past years, and both the competition between the leading frameworks as well as the richness of offering these frameworks provide grew, we have also experienced another interesting trend in this landscape – codeless capabilities.

Out of many solutions, I have decided to look into the rising Cypress Studio (Beta) and Playwright CodeGen tools to review their low-code abilities. Note that there are more mature commercial solutions in the market like Perfecto Codeless and Testim, however, here we will focus on the 2 open-source frameworks.

Before diving into these tools, here are few of the capabilities that I would expect to find in a complete low code/no-code test automation solution

  • Coverage of most testing types including functional, performance, accessibility, APIs, and more.
  • Parallel and scalable testing abilities like we have with Selenium Grid within cloud providers or in the open-source.
  • Ability to import code module into the codeless frameworks and export codeless created script into code like JavaScript.
  • Test reporting and analysis with rich artifacts and with abilities to slide and dice test data
  • Integrability into DevOps and CI/CD tools like Jenkins, GitHub Actions, and more
  • Ease of use and quick ramp up with the tool
  • Strong community, support, and documentation

Since both Cypress and Playwright are emerging frameworks, both cannot cover all of the above capabilities, however, they can provide some impressive test creation features. Note that the 2 frameworks are different by definition.

Cypress is a JavaScript only framework that runs inside the browsers with rich debugging and functionalities. Playwright is a chrome debugger protocol (CDP) framework that is led by Microsoft and through its architecture can support more language bindings and more browsers including WebKit Safari that isn’t supported yet by Cypress.

Playwright CodeGen (Test Generator)

Once you install Playwright Node library following the below commands, by definition you get the CodeGen library.

npm i -D @playwright/test

By running the following command from your local Playwright installation folder within your IDE terminal (in my case VS Code), you launch the test generator tool and can record any action on your web application that in parallel generates JavaScript test code.

npx playwright codegen http://continuoustesting.dev

npx playwright install

As you can see in the above recorded test code, it navigates to my blog webpage and performs some basic navigation and clicks on different links and menus. The generated code can than be exported and saved in different language bindings including the native Playwright Test, Java, JavaScript, Python, and C#.

Another cool feature of the Playwright CodeGen tool is the Explore button. This capability enables developers to mouse hover the web application under tests and inspect the elements via the Playwright Selector. When selecting an element from the tool you can than copy it into your IDE test code and proceed with your test creation. In addition, you can record, than stop, and continue recording few times as long as the tool is still running. Lastly, you can do basic debugging of your recorded test from the Playwright Inspector CodeGen tool through stepping over, execution stop and more.

Cypress Studio (Beta)

Like with Playwright, the codeless recorder tool is built into Cypress, but needs to be enabled through a command added to the cypress.json configuration file. Note again, this tool is still in an experimental stage and in beta as this blog is being written.

To start using the tool, simply enable it by adding the following line to your cypress.json file:

{ "experimentalStudio": true  }

From this moment, the tool is enabled, and when you run Cypress in GUI mode through

npx cypress open

You can than edit an existing JavaScript test spec file and click on the red recording button to inject recorded test code into your script, or build a whole new project. Cypress documentation recommends cloning their built-in GitHub repository and using the following JavaScript test spec as a ‘getting started’ project (https://github.com/cypress-io/cypress-realworld-app/blob/develop/cypress/tests/demo/cypress-studio.spec.ts)

As you can see from the above screenshot, once the Cypress Studio is enabled, editing a test step triggers the Cypress Studio button where you can add new commands to you test. As of now, 5 main test commands are supported by the tool – Check, Click, Select, Type, and Uncheck.

As mentioned above, you can create a brand new test through the recorder or add new code into an existing spec file.

I ran the tool and added a simple exploratory test into the built-in test spec called todo.spec.js that comes with the Cypress open-source framework, below is the generated JavaScript code.

Key Takeaways

It is clear from my quick overview of both codeless tools that they are only in the beginning and that there is potential to grow and add more of the above core capabilities of a fully ready test automation framework. Since these two tools are provided for free and in early stages, we shouldn’t complain, but rather embrace, and provide feedback to the owners of these frameworks to improve.

Advantages of Playwright CodeGen:

  1. More language bindings to export the created code
  2. More browsers to run the tests on
  3. The Explore object spy is quite easy to use
  4. Tool is separated from the basic code-based GUI

Disadvantages of Playwright CodeGen:

  1. Hard to debug tests
  2. Limited test creation of the tool, no automated assertions
  3. No documentation and examples to get started with

Advantages of Cypress Studio

  1. Easy to use from the Cypress GUI tool
  2. Friendly debugging tool
  3. Richer test creation abilities
  4. Good documentation and sample project to help ramp up

Disadvantages of Cypress Studio

  1. Limited test creation and assertion within the tool
  2. Only exports to JavaScript
  3. No support for all browsers (Safari WebKit)

Bottom Line

The 2 tools are a great start and an optimistic horizon for codeless tools complementing code based tools. Both are still growing and there are many gaps to close before they can be self-served by front-end developers and SDETs, however, they provide a sing for the future of intelligent test automation and this is promising. We should continue following the evolution of these tools including the newly Selenium IDE within Selenium 4 and continuously provide feedback to the communities to grow and build these tools right.

Posted on

The Importance of Software Testing for Businesses

Guest Blog from: Maverick Jones
Maverick Jones is a full-time geek and tech enthusiast. He likes to share his bylines and loves to gain audience attention.

Table of Contents

  • Introduction
  • What is Software Testing?
  • Types of Software Testing
  • 7 Reasons of Software Testing Being Important for Businesses
  • Conclusion

In this digital world, software is the one thing that influences a far bigger audience these days. It enhances the business prospects, contributes to upscale life standards, offers services to large audiences in less time, and evolves a smart world. Besides this, the software offers an industrial impact and plays an important role around all the walkways of people. But for this, it requires a fail-proof and smooth functioning of the software. And the main reason behind it is that software failures can shatter the business. Therefore, testing any software is a must.

Software testing ensures that the software performs adequately and offers user satisfaction. Testers make sure that the performance of the software is not sacrificed and for that, they run various tests using both manual and automated testing approaches.

In this blog, we will learn more about software testing and see how this concept enables business software to work smoothly.

What is Software Testing?

Software testing is a very critical process that offers an analysis that enables the testers to identify and evaluate the application. It determines whether the newly developed app meets the Business Requirement Specifications. It is a continuous process and is popularly known as Software Testing Life Cycle. This process works along with the life cycle of software development.

Software testing verifies each functionality at every stage and validates the app performance with the need. Besides, when it comes to checking software perfectly and delivering bug-free software, the approach of software testing can help in improvising the functionalities and usability of apps. With different types of testing approaches, software development companies test applications and make them safer for their clients.

Types of Software Testing

Software applications can be validated and verified in two ways. By applying two different types of testing – Automation and Manual Testing.

Automation testing is an approach where the testers write different scripts and then they rely on external software to perform software tests. Besides, when the testers want to double-check the product’s quality, even the manually tested software has to go through the automation tools. And this is the type of testing that assures great accuracy level and cost-efficiency. Besides, it also saves a lot of time.

Besides, manual testing is quite different. And as the name suggests, it is a very simple process of testing the software manually. It doesn’t require any type of automated testing tool that enables the testers to check the software automatically. Instead, the tester can work on the software and perform the testing process. The manual tester can easily detect errors and bugs by following through with different levels.

7 Reasons of Software Testing Being Important for Businesses

Here are some of the top reasons that show the importance of software testing –

1.   Security

Security is one of the most important reasons for software testing. And to consider it can be the smartest choice of it all. Security is considered a very crucial, sensitive, and vulnerable part of software testing. Besides, there are situations in which all the important details and information of the company gets stolen away. Therefore, security is considered the most reliable reason to make people believe in the quality of the software. Security is one thing that ensures that the product is safer to use.

2.   Customer Satisfaction

The main objective of the business owners of the products is to provide the best services and better customer satisfaction. In order to offer quality and customer satisfaction, adopting a software testing process is essential as it provides the perfect user experience. As the software development company tries to develop the best software for their clients with a testing approach, they will earn a reputation for reliable clients.

Therefore, to get praise from the clients and offer them a system that enables long-term benefits, having a perfect software testing approach is very important. A properly tested and secured system earns the client’s trust easily and also becomes safer to make transactions.

3.   Product Quality

To ensure that some of the products come to life in a secured manner, following the requirements of the product is necessary. The product you offer to the clients must be such that it serves them in one or another way. Besides, it must always bring out value as promised. And to make sure that it functions correctly by offering an effective customer experience, testing the products before their launch is necessary.

Every testing team must check the compatibility of the device before launching the product or application. For example, when you are planning to launch a software, checking that it is compatible with different devices and operating systems is a must. This shows the quality of the software.

4.   Saves Money

Software testing comes with various benefits and one of them is cost-effectiveness. This is the main reason why organizations go for software testing services. The software testing comprises a bunch of projects. When the testers find bugs in the early stage, fixing those bugs doesn’t cost much. Therefore, it has become a prerequisite.

Every business owner hires a tester or quality analyst who has great experience and thorough knowledge of the projects.

5.   Determines Software Performance

When an application or software has low performance, it brings down the app’s reputation in the market. The users no more prefer to use that application and even stop trusting the company that offers such products. And eventually, the company’s reputation also suffers. Therefore, every application or software must never be introduced in the market without software testing.

Once the software is tested and its performance is perfect, it must be launched in the market. And even after the launch, to keep the reputation of the product intact, continuous testing becomes necessary. Basically, if the performance of the software goes down, everything goes to waste. Therefore, considering software testing is the best option in determining the software performance is a must.

6.   Easy to Add New Features

When the code is old and interconnected, it becomes more difficult to update. And this is why tests are performed that counteract this calcification tendency and enable the developers to add new features to the software. But when the software developer is new, changing older parts of the code becomes horrifying for him. And this is where test results come in to offer every detailed information about the broken parts of the code.

This process enables the business owners to make their products stand out in the market.

7.   Enhances Development Process

With the help of Quality Assurance, the software testing services providers can find an array of errors and scenarios. And once they get a hold of the information about bugs, the developers can easily help in fixing them in very little time. the same in no time. To have such a smooth development process, the software testers should work with the development team parallelly in order to accelerate the procedure.

Conclusion

As seen in this blog, the importance of software testing is of high value in the world of software development. Software testing is a fine whole process that ensures the perfect quality of the product. Besides, testing also offers better usability, enhanced functionality, and reduces maintenance costs. Therefore, whenever you hire a software development company for your project, make sure to hire a team that has expert software testers who can use the latest tools and techniques to secure your product.

This Blog was provided by: Maverick Jones
Maverick Jones is a full-time geek and tech enthusiast. He likes to share his bylines and loves to gain audience attention.

Posted on

Buy the Best Software for Your Business with These 5 Steps

Guest blog post by Gloria Martinez.

Too many entrepreneurs have purchased software after watching a slick demo only to experience buyer’s remorse soon after. As a business owner, you want to buy software that delivers the value it promises, whether you buy it off the shelf or on a subscription basis.

This article shows you five crucial steps to buying software that fits your budget and needs:

  1. Assess your needs
  2. Seek buy-in
  3. Compare multiple options
  4. Mind the cost
  5. Pick a solution

Continuous Testing Blog from Eran Kinsbruner invites you to read on to find out more.

Assess Your Needs

Buying decisions that take into account all the facts result in satisfactory software purchases. Assess your needs before you make a substantial investment in software for your business. For example, if you’re looking to register your business as an LLC, then you may want to work with an online formation service that can help you correctly file the paperwork. You can call the Zen Business phone number to learn more about the process.

It’s important to break down which software and/or services will best serve your business before you consider purchasing anything, especially when you’re operating on a very tight budget.

Get Buy-In From Stakeholders

As the owner of the business, you may have the authority to make a unilateral decision on what software to buy. Still, it’s important to get buy-in from your employees, managers, IT team, and customers. Carrying out a comprehensive needs assessment helps achieve buy-in from all these stakeholders.

Take into account how implementing new software or replacing your current system will affect your processes, habits, hardware, and work culture. Purchasing new software often leads to significant changes in your operations. Let your employees know how these changes will affect them so they’re well-prepared to adjust.

Compare Prospective Software Solutions

Once you get stakeholder buy-in, start reviewing prospective software solutions. Narrow down to a few top-tier software providers. Your exclusion process should rely on questions based on the needs you identified in your needs assessment.

Here are a few examples of questions to guide you:

  • What are your must-have and nice-to-have features?
  • How much data storage do you require?
  • How much customer support will you need?
  • Do you want an on-site or cloud-based solution?
  • How scalable is the software?

Your budget also determines the software solutions you consider. A feature such as live chat support may drive up the price but may be worth the investment.

Consider the Cost

Research all the costs involved in purchasing and implementing the software to ensure you’re prepared. Here are a few considerations to make.

  • Is the software available as a one-time purchase or SaaS solution?
  • For SaaS solutions, does the provider offer a discount for annual billing?
  • Does the contract, if any, include an early cancellation penalty?
  • How much will it cost to scale up?
  • Is support billed separately?

Getting answers to these and other salient questions ensures you purchase software that fits your needs and budget.

Choose a Solution

You’ve whittled down your shortlist to one or two suitable software solutions. Take a multi-pronged approach to ensure you get the solution all stakeholders will find satisfactory.

  • Use a trial period to test the software.
  • Research the vendor by reading customer reviews.
  • Ask your industry peers who use the software their thoughts on it.

Again, consult your needs assessment to determine what trade-offs you’re willing to make.

As a business owner, your bottom line is an essential KPI. You want to optimize the returns on your software investment. The steps outlined in this article will help you buy the best software for your business.

This is a guest blog post by Gloria Martinez.

Gloria loves sharing her business expertise and hopes to inspire other women to start their own businesses and seek promotions in the workplace. She created WomenLed.org to spotlight and celebrate women’s achievements

Posted on

Understanding The Role Of Software Testing In Cyber Security

Gust Blog by: Kanika Vatsyayan is Vice-President – Delivery and Operations at BugRaptors 

Over the years, Cyber Security has become a highly sensitive and significant aspect of the IT industry. And it is very natural with hackers leaning on some more advanced approaches to breach public and private networks inspecting the personal information of various organizations.   

Most importantly, these cyber-attacks have the potential to feed on business data from any small or large organization using high-end technology to attack digital systems. As per the reports by weforum, approximately $5.2 trillion worth of data would be put at stake because of cyber-attacks from 2019 to 2023.   

However, software testing has emerged to be a practice that has helped streamline the cybersecurity factor restricting hacking and diminishing the chances of any threat to data. In other words, software security testing services has turned to be a trusted shield against cyber-attacks and therefore has gained extensive popularity amongst the IT sector.   

Importance of Software Testing  

With past incidences like the WannaCry ransomware attack of 2017 and the Yahoo breach of 2013, organizations are at constant threat of cyber breaches, deploying a perfect software testing strategy should be an ideal agenda to keep things protected. However, the limited propagation has made security testing an even more complex process to understand.   

Above all, there exists a significant irony in the current approach towards software testing for cybersecurity as the present focus of the development enthusiasts is centered around the new development practices. This cuts the scope of improvement for already released technologies and makes them more vulnerable to hacking due to major security gaps.  

Thus, there is an immediate need to realize the current circumstances and make continuous software testing and DevOps an essential part of the organization’s development practices to improve the entire working propaganda. However, the process needs companies to immediately locate software testing professionals to help with planning the cybersecurity initiatives while driving quality throughout the project pipeline.   

Also, the adoption of security testing services would not only aid the security process against unexpected cyber-attacks to industries like healthcare, finance, banking, etc. while improving the entire development design. In a nutshell, the right approach towards software testing could help drive a mature security practice that will run through different stages of the development lifecycle while aligning with cybersecurity objectives.  

Penetration Testing & Its Importance For Cyber Security  

Over the years, different surveys have revealed that 75 percent of cyber-attacks happen due to insecure code that makes the application unstable. Also, the shift of all the operational processes, as well as day to day tasks to third party applications that have access to global user data, have become the prime spot for cyber attackers.   

Therefore, it is necessary that your developers, as well as security testing service provider in India, both aim at improving the source code quality for any tasks that involve the use of data. However, the process to overcome any such code issues require correct implementation of the techniques like penetration testing as it works on how the code connects with the application.  

Penetration testing is a technique that highly resembles a cyber-attack or actually an intentional cyber-attack made on the application to identify all the system vulnerabilities. Moreover, penetration testing is a highly effective practice that helps meet the goals related to web application security as it helps to improve the Web Application Firewall or WAF protection.  

Intelligent System Security Checks  

Penetration testing is more of a test design arrangement made to work on security against any attempt made on hacking. Even the tools used during penetration testing are made to replicate the cyber-attacks with the objective of getting over any system vulnerabilities.  

Underline Weak Segments  

Another significant aspect of using the penetration testing approach is to identify weak links within the software code. The practice improves the scope for protecting the personal user data, card details, or any transactional record that might be at threat. In short, penetration testing works by analyzing the security vulnerabilities of the system and use them in strengthening the system structure against possible cyber threats.  

Added Network Security  

Penetration testing is a powerful technique that could help developers and software testers to work on improving the high-speed internet like 5G network security as well as applications running on the network. When implemented through a dynamic approach, penetration testing could help the users to identify any potential vulnerabilities related to the system for preventing any chances of attack. Also, the security-related information can be used to work on compliance goals and plan the security response time to make system protection more definite.   

The Crux: Planning Cyber Security With Software Testing  

Though penetration testing is a highly efficient software testing practice that is made to simplify the test operations related to cybersecurity, running through other dimensions of testing could help make things better.   

For instance, pen testers could consider escalating their cybersecurity initiatives by using advanced cyber penetration techniques. This would not only help make the system more secure but have a complete idea of the likely cyber-attacks that could be planned in the near future. Such arrangement allows tracking of methods that can be imposed by attackers and therefore can be used to prevent any hacking incidents.  

All in all, working on the cybersecurity component of software testing is a sophisticated practice that needs testers to analyze all the aspects of the technical landscape. It is not only made to expose potential attacks before they occur but rather uses a similar set of tools and approaches that can be used by hackers. Furthermore, it cuts off the chances of any security compromises that may happen with devices running on public networks.   

In a nutshell, software testing has redefined the entire concept of cybersecurity, offering solid protection to the user data as well as development companies. Thus, it is crucial that development firms should start recognizing the importance of software security testing services in the organization when they need to put off any prospective threats as it will not only help users to experience the best of the web but development companies to enjoy sustainable growth.   

All the best!  

 Author Bio:  Kanika Vatsyayan is Vice-President – Delivery and Operations at BugRaptors who oversees all the quality control and assurance strategies for client engagements. She loves to share her knowledge with others through blogging. Being a voracious blogger, she published countless informative blogs to educate audience about automation and manual testing. 

Posted on

Expediting Web Testing with Puppeteer and Instant Test Automation Technology

The digital landscape has significantly advanced over the past 1-2 years. The new normal in web applications include a shift toward progressive web apps (PWAs), advanced responsive websites, and Flutter apps.

Ensuring both functional as well as API testing of such advance web applications is becoming harder and harder. While there are valid test automation frameworks like Selenium and Cypress, these are covering only a subset of what is needed to complete the entire quality aspects across multiple environments.

New Normal: Instant Test Automation

To address the testing challenge over time, across multiple environments, and with big and dynamic data, tools need to be more intelligent, and utilize AI/ML to really track trends, predict issues prior to them becoming more serious, and guide the developers and testers in real-time.

An instant test automation technology that can leverage a standard test automation framework such as Puppeteer to auto generate both functional as well as API tests on the fly and as things change is a game changer for app developers. This provides alerts and insights in real time that the team can act upon and move much faster.

The way sch solution works is through modeling the website under test, auto generate based on traffic and AI proper tests and runs them to provide the proper insights.

Setting up such solution is easy and needs to be done only once. Also, such tool fits systems that are built atop cloud-native, containers and Amazon ECS.

Benefits of Shifting into Instant and Intelligent Test Automation

With the need to shift left, release faster with confidence, and handle large and complex environment, teams can benefit from a self-generating testing solution that really covers more grounds that a typical code-based framework would. This does not mean that there are no benefits or need for such frameworks like Selenium and others, however, with tools like UP9, teams can complement the code related functional tests with an additional layer of quality validation that touches reliability, APIs, services, security and much more.

Case Study: Quick Tour of Puppeteer & UP9

I took the Google puppeteer framework and connected it with UP9 to try generating a nice model and test automation assets on my local machine.

After following the basic NPM installation instructions, it was easy to launch a local browser and record traffic using the tool.

From my terminal I would then run this command (‘my.new.workspace’ could be any name that the developer selects):

up9 tap:start my.new.workspace

Once I ran that command, using puppeteer, UP9 launches a chrome browser that I can start using to navigate to my website for recording all traffic, actions, API calls and other services. Upon closing the browser, the tool will process all of the flows and generate the model.

Here is the auto generated workspace that was created based on a quick tour I ran with UP9 on the Perforce and Perfecto websites:

The workspace consists of few views:

  1. The generic workspace with all the high-level services model (diagram),
  2. The contracts that were used in my recording,
  3. A detailed traffic analysis with the entire API calls,
  4. Tests that were created and mocks to be used across different environments.

These views were perceived and constructed based on service-traffic, following from the observers of UP9 that were assigned to my designated workspace.

As part of my journey, I landed on the Perfecto free-trial website, and the tool was able to generate the API calls (Request/Response) and allow me to validate that is was working as expected (see below)

If I wanted to also run my auto-generated tests, I have within the UP9 portal (fully SaaS and web-based) a tests area from where I could launch my tests as well.

Bottom Line

As the DevOps market mature towards more intelligent testing using AI and ML, and as the systems under tests (SUTs) are also progressing, there needs to be a realization within the teams that their processes could be improved dramatically using such technologies. Developers productivity together with consistent app quality were a CIO/IT manager top of mind for a long while, and with the recent advancements in the market, teams can really enjoy a new era of smarter ways of working.

Posted on

Defect Detection and Security Prevention: How does Shift-Left Adoption Helps?

Dzuy Tran
Guest Blog by Dzuy Tran, Klocwork and Helix QAC Senior Sales Engineer, Perforce

Essentially, Shift-Left is the practice finding defects and preventing them early in software delivery process. This is done by shifting the scanning of defects to improve code quality process to the left of Software Development Life Cycle (SDLC), which is usually composed of four phases of activities:  design, develop, test, and release.

Shift-Left also applies to software security testing (SAST). The aspect of shifting left on security is one of a set of capabilities that is the result of higher software delivery and organization performance.  According to the 2020 state of DevOps Report published by Puppet Research, high-performing teams spend about 50% less time remediating the security issues than low-performing teams.  By better integrating information security (InfoSec) objectives into daily work, teams can achieve higher levels of software delivery performance and build more secure systems with approved compliance standard library and compliance scannable SCA tools.

Production Defects are Costly.

How Does Shift-Left Help Reduce Product Costs?

Production defects are costly, as an identified defect costs about 30 times more than the same defect fixed in development and takes about 5 times longer to resolve. What is more, certain industries — such as embedded software, medical devices, and high-value transactions — experience even higher costs and reputation damage from bugs.

In addition, fixing defects in testing costs about 10 times more than development and takes only about twice the time it would have taken before checking in the code. In waterfall development, it would not present a substantial difference. However, in the continuous integration assembly line methodology, defects found in testing still break the build.

Shifting Left Effectiveness

Shifting left defect detection — from testing and production to developers — is not a new concept. Rather, it becomes more and more critical as software is integrated into more mission-critical systems, IoT devices, and backend enterprise operations. As a result, the cost and impact of production defects increases as well.

When defects are discovered, developers must then find the contributing factors and fix them. In complex production systems, it is not usually caused by a single issue. Instead, it is often a series of factors that interact with one another to cause a defect. This goes for the defects that are found involving security, performance, and availability — all of which are expensive and time-consuming to remedy, and often require some architectural changes. The time required to find the defect, develop a solution, and fully test the fixes are unpredictable — causing delivery dates to be pushed back.

How Effective is Shift Left for Defect Detection and Prevention
The shift left process, optimizes continuous delivery workflows by reducing build pipeline

breakage. It also allows developers to spend less of their time on diagnosing problems and more time on preventing them during development. 

In addition, shift left should help enforce more discipline and create awareness of software quality. However, there are also other tools and techniques that can shift-left the defect detection responsibility to the individual developer:

  • Desktop Static Code Analysis (SCA)

Static code analysis automatically finds potential defects or security vulnerabilities in the source code prior to running the application. SCA tools — such as Klocwork or Helix QAC — can be light and look primarily at code syntax, or more sophisticated and examine the complex application execution paths. Industries such as automotive, healthcare, and aerospace mandate the use of such tools in the testing and validation phase.

Integrating SCA at the build or CI process improves quality and security. True shift-left requires wide adoption of SCA at the developer’s desktop, scanning and cleaning the code prior to checking in rather than waiting for the build to fail.

  • Use Code Frameworks

By using code frameworks, code components, or libraries, both commercial and open source reduces the volume of custom code developed and therefore eliminate defects making their way into the build. Standard “plumbing” tasks – such as complex UI elements, math and analytics algorithms, data mapping, networking, and so on – can be handled by code libraries while developers focus on true business logic code.  Properly tested frameworks that are supported commercially or by a vibrant community, proactively shift-left problem detection by not introducing “plumbing” code defects to begin with.

  • Developer Side Application Performance Management (APM)

Application performance management solutions provide production performance and failure alerting, which includes analytics designed primarily for production. It’s unusual not to have comprehensive monitoring in place for production applications. Plenty of commercial and open-source solutions are available for different cloud and on-premises environments.

However, they are not built with developers in mind. Lightweight APM tools designed specifically for developers shift-left performance problems and error detection to the developer desktop. Desktop APM — such as XRebel APM for Java and Zend Server developer edition or ZendPHP for PHP – allows developers to proactively optimize code before it enters the integration phase.

  • Standardize Environments

Adopting a standard, automatically generated application stack and virtualization or containerization that matches the production environments is another shift-left practice. It shifts a class of errors to the developers’ responsibility by not introducing them to begin with. In the cloud or on-premises, standard application stacks reduce chances of configuration and environment mismatch issues from making their way into the build.

Shifting-Left for Security Prevention

Security should be everyone’s responsibility. So, by shifting the security review process to the “left” or earlier in the SDLC phases requires several changes from traditional information security methods with additionally scanning for security vulnerability. However, that is not a significant deviation from traditional software development methods on closer inspection:

  • Get InfoSec Involved in Software Design

The InfoSec team should get involved in the design phase for all projects. When a project design begins, a security review can be added as a gating factor for releasing the design to the development stage. This review process might represent a fundamental change in the development process. However, this change may require developer training.

  • Develop Security-Approved Tools

Providing developers with preapproved libraries and toolchains that include input from the InfoSec team can help standardize developer code, and the tools should include a SCA tool — such as Klocwork or Helix QAC — to scan for any security vulnerabilities in the code, such as tainted-data, cross-site scripting, etc.

Using standard code makes it easier for the InfoSec team to help with reviewing it. Standard code also allows automated testing to check that developer is using preapproved libraries. This can help scale the input and influence from InfoSec teams.

  • Develop Automated Testing

Building security tests into the automated testing process means that code can be continuously tested at scale without requiring a manual review. Automated testing can identify common security vulnerabilities, and it can be applied uniformly as a part of a continuous integration pipeline or build process. Automated testing does require you to design and develop automated security tests (pre- and post-software releases), both initially and as an on-going effort as new security tests are identified.

Blending Quality and Security to Create DevSecOps

The traditionally separate relationship of development and security is long overdue for evolution, which has culminated into a cultural shift known as DevSecOps. The name suggests a blending of development, security, and operations. The DevSecOps methodology is built upon the “shift-left” philosophy of integrating cyber-risk management into the architecture and development process from inception. Built-in, not bolted-on, as they say.

With DevSecOps, security is baked into the code from the start, during the early stages of development. Security is part of the architecture, and the application of automated testing throughout the development process drives a higher level of both product quality and DevOps security. Security issues present earlier, making life easier for developers and less costly for management.

This blog was Created by Dzuy Tran, Senior Sales Engineer, Perforce Software

Dzuy Tran has over 30 years of experience in designing and development of Hardware and Software Embedded Systems, RTOS, Mobile Applications and Enterprise Systems. He helps customers when they have technical questions, assists with Proof of Concepts, and conducts demos of the Static Code Analysis tools and help guided customers on DevOps implementation processes and Continuous Integration deployment. Dzuy holds a master’s degree in Computer Science and Computer Engineering from National Technological University.

Posted on

How to Achieve both Coding Standard and Security Coverage Together with Safety Compliance

Majority of organizations are already deep in their DevOps maturity. Most researches are showing over 40% that adopted the process, and are moving towards automated processes, shift-left, and fast delivery of value to customers.

With that in mind, these organizations that span across different verticals from automotive, financial industries, gaming industries, and many more, need to not only deliver high quality code, but also in many cases meet and comply with specific standards and regulations.

If to only focus on the above compliances, teams must bake into their CI/CD pipelines the scanning against such standards, and ensure that they keep up with modifications that are constantly happening across these.

In addition to the above, there are also very important and critical compliances like Autosar (automotive open system architecture), OWASP (top security vulnerabilities), CERT, CWE, PCI and others.

When dealing with smaller teams, managing the code safety and compliance might be easier, however, when you are 1 squad within a bigger DevOps organization, this requires better governance and control. Keeping up with code changes, merges into different branches, while running continuous testing suites, performing code reviews, and SCA (static code analysis) becomes quite challenging.

Building the Perfect Mix : Safety and Compliance together with Code Standards Adherence

To obtain the right mix, continuously, teams must strategically plan their pipeline in a way that democratizes both continuous functional testing, non-functional testing together with the entire sets of compliances and code standards quality assurance.

If to analyze the above famous DevOps lifecycle diagram, teams can put the proper activities in the right phases of their cycle to cover all of their required goals.

If to look into an example, using an open-source GPS tracing repository , this project has various modules, and quite a lot of Java classes. Ensuring that the code adheres to the proper Java coding standards, as well as does not violate any of the OWASP items, continuously and from CI is not an easy task.

In the below screenshots, you can see that by running a simple Maven SCA job within CI as a batch project upon each code change, can easily generate a comprehensive report (In my case, I am using Klocwork SCA tool):

“kwmaven clean install compile -Dmaven.test.skip=true”

As soon as I run from the project folder the above command, a full build and scan are being performed on the entire code base, using specific compliance modules that I predefined. The developer receives at the end of his Jenkins job a detailed report and can also login to the KW dashboard to review each and every issue or violation (below is a snapshot of the post build report).

In the above case, I was using Java, and was configuring the below sets of compliance coverage to use for the scanning. Obviously, if I was to cover an app from the automotive or other embedded software industry, I could have added an additional taxonomy/ies.

From a process perspective, the developer should follow the process of per each code change, run a build (either using the team CI trigger, or his own local CI. Once he receives a clear report without critical issues surrounding safety, security, and other code quality standards, he can pass his changes toward the next phase of integration testing, functional regression testing and pre-production activities.

It is clear that such code scanning and quality activities, must be filtered properly to avoid redundant noise, false negatives, etc. This is why, relying on SCA tools that can grant the developers the option to filter by severity, modules, configurations and compliances, allows getting the job done while not overwhelming them with irrelevant feedback.

In the above Klocwork zone within Eclipse (or IntelliJ), users could filter through the relevant columns the issues by Severity and more.

To summarize this post: Teams especially within Agile and DevOps practices can enjoy both types of quality gates by employing SCA tools together with coding standards under the same source base, and once these activities generated Green lights, they can allow testing teams to run their jobs with higher confidence.

In most organizations, testing teams are requiring as a pre-requisite to starting their regression an SCA audit report showing that no major issues were detected within the build cycle.

Keep in mind that the above process and tools is 100% automated, and runs within CI which means, ever for large code bases, this is a few minutes of scanning to get to the quality gate with a peace of mind.

Posted on

The Future of Test Automation Must Be Intelligent

A decade of Agile, DevOps and Continuous Testing is behind us. I heard this week in a conference I was presenting the phrase “Agile is Dead” and other claims about testing being inefficient in many ways.

While many tools have evolved over the past years, and test automation practices and lessons learned have been communicated to practitioners, it is IMHO not enough to move the needle on the efficiency of test automation and continuous testing – especially in the demanding and ever changing digital space.

To have a better DevOps pipeline, and to minimize both the waste as well as the level of escaped defects, testing and especially test automation must become smarter.

In this post I would list the opportunities for test automation to be more intelligent.

WHAT?

Test automation intelligence should guide engineers on which test automation scenarios to execute, automate if missing, and continuously maintain. This is one of the most complex debates within each automation team – WHAT TO AUTOMATE?

Answering this question is never easy, requires risk management, tradeoffs, and often guessing. In addition, it does not always align with the sprint time frames, hence, being thrown out of the software iteration.

With more intelligent solutions that are based on production data, business driven decision making, code analysis, teams will be in a better position to make such decisions.

I love the below visual that Ingo Philipp and originally, Michal Bolton drew around testing is always sampling. This shows that test automation usually addresses what we know about risks and the product out of the entire ocean of risks and features that we either don’t know, cannot assess, or things that were addressed in the past and there isn’t sufficient data to advise whether we should continue using.

New tools that are rising such as Launchable, SeaLights and others aim to help answer some of the above questions, and to help form a continuously relevant regression suite that matches the most recent product code changes, history of defects and more.

WHEN?

The second question that needs to be answered in a more intelligent way is – When to automate and execute my test scenarios? Some say – everything , everyday, some say – everything shifts left, and some are breaking the tests based on personas, skills, and naïve objectives.

Scheduling and deciding which tests to run when must be smarter, and needs to be tied to the overall process, the feedback loops, and the ability of the developers to act upon such execution feedback. If testers throws on the developers to late or to early the results, it makes it unrealistic for them to prioritize, resolve and fix.

Intelligent systems, must be able to determine based on the above WHAT section, which test scenarios, types, and cadence, such should be executed. In an ideal world, an intelligent system would be able to split between CI, BAT (build acceptance testing), integration, regression, NFT and production synthetic monitoring the entire test suites.

As the above diagram suggest, the pipeline has a pre-defined set of quality gates, phases, and milestones to manage and mitigate risks. An intelligent testing system should effectively place all the right test scenarios –CONTINUOUSLY– in the right phases. Keep in mind, the right test cases change from one release to the next.

WHICH?

Now that WHAT and WHEN questions are addressed, it is important to have a smarter approach into test automation coverage – on WHICH platforms and environments should my test automation scenarios be running constantly? Which mobile devices, which desktop browsers? which backend configurations are important? which data-driven scenarios should be plugged into the test cases? Which load and KPIs should be applied so I get the proper user experience feedback from my testing? and more.

Determining the WHICH in test automation is an art, and require data analytics, market research, customers validation, production monitoring, AIOps systems and more, but it is an imperative for success.

WHY?

After determining few WHs questions in test automation above, it is time for the analysis part of test automation intelligence – We defined what to automate, when to run, and on which platforms and environments, but once executing the tests, can we efficiently determine WHY tests are failing? What is the root cause per each failure so we lower the time for resolution (MTTR)?

Here, intelligent reporting platforms are critical in minimizing the feedback loop, the future decision making on which tests are of higher value, which platforms are more flaky then others and much more.

Without a solid test automation analysis platform, a great chunk of teams investment in testing can be of waste.

It is not that complicated to realize why tests are constantly failing, and put them into buckets of failure reasons. Perfecto is doing that in an automated fashion through its smart reporting SDK, and allows teams to filter a lot of the test noise post a regression cycle completion.

HOW?

Test creation has always been one of the hardest and most challenging task for SDETs and business testers. It includes great technical skills, meeting short time windows for automation, and requires deep understanding of the product requirements to really automate what’s expected. While many test automation frameworks evolved over the past years, Including Selenium, Appium, Espresso, XCUITest, and Cypress, these are all code based frameworks that even when coming on top of a BDD platform, they are hard to maintain over time, and analyze when they result in a failure.

New and more intelligent test automation frameworks that are codeless or low-code have grown and are becoming a new reality of test engineers. Such tools can really help address few key challenges that are prominent. Time to automate, time to maintain, flakiness and skillset gap. Intelligent test creation that is plugged with a TIA can be the future of smarter test automation, and enable practitioners to create the most relevant test scenarios with minimal maintenance and shorter amount of time. As I am writing this blog, such tools are still growing and have a lot to prove to the world prior to being widely adopted, but this in my mind should be the future of test automation, and once this is in place, connecting these tools to the cloud for parallel and scaled testing will be the ultimate solution for continuous testing.

BOTTOM LINE

A lot of what is written in this blog already exists to some extent, some if being developed these days and will be available in 2021. It is important to start drawing within each DevOps team the future of test automation goals and objectives and learn how such solutions will fit the process from test automation creation, execution, maintenance, analysis, TIA, defect management, quality governance and more.

This year, I launched my 3rd book that focuses on “Accelerating Software Quality: AI and ML in the Age of DevOps“. This book looks at the modern DevOps and explores how using AI and ML and smarter algorithms, teams can maximize their productivity and deliver more quality functionality faster.

There is a lot to get excited for in 2021, and especially around test automation in DevOps – Happy New Year and Good Luck in your Future of Test Automation

Posted on

Code-Based Test Automation vs. Codeless Automation

 

As more advanced technologies are entering the continuous testing landscape powered by AI/ML, organizations and especially, practitioners, are debating which is better, and why if any should they adopt codeless test authoring solutions?

In this blog, I will be providing the various considerations to switch, and/or combine between the 2 test automation methods.

TL,DR –> There isn’t a magic answer to this debate, and there isn’t a method that is good vs. bad.

Top Considerations

To better address the question on when and why to use either methods, here are the top items to consider, not listed by importance, since each team might relate to different objectives and priorities:

  • What are the application use cases and flows to automate?
  • Which persona is going to create and maintain these scenarios?
  • What are the skill-set within the team/individuals for the job?
  • Is the app under test Mobile/web/responsive/desktop?
  • What are the time constrains for the project, and what are the release cadence going forward (weekly/monthly)?
  • Is the test suite meant to be integrated to other tools (CI/CD/Frameworks)?
  • Are there any advanced automation scenarios (chatbots, IOT, location, audio, etc.)?
  • What are the cost boundaries (tools, project, lab, etc.)
  • Is the test suite meant to be executed at large scale?
  • Is the project a new one, or an additional layer of coverage on top of existing code-based suite? (Selenium/Appium, etc.)

Diving Deeper

Now that we’ve listed some important consideration, let’s explain them a bit deeper.

For teams that are already working on a project (web/Mobile), and have implemented a good working amount of code-based test scenario that are embedded into processes, CI/CD, and other triggers, such consideration should be heavily considered – what is the incentive to change? are there coverage gap in the code-based suite? is there too much noise and flakiness attached to the existing test code? etc. Only if there is a good incentive like the ones mentioned, teams should consider adding codeless test scenarios into their pipeline.

On the other hand, for teams that are just starting a new project, that’s a perfect timing to look into the entire team skills, and decide based on the technology the project is built on, what tools to use – if might make sense for example, for a newly created website to combine a Selenium framework that SDETs that are Java/JavaScript developers would lead together with business testers that can remove some of the load from them via ML-Driven codeless selenium tools.

After covering use cases, quality of existing test suite, new vs. existing projects, lets also consider the time-frames and budget allocated to the project. It is clear that recording codeless scripts takes on average 6-10 times faster than coding the same scenario in Java or other development language. It involves setting up the platform and test environment, coding, debugging, executing at scale, assertions, and more. This obviously also translate into $$ savings. On the other hand, not all test scenarios can be easily recorded, since, for some advanced flows, coding might be a better approach and an easier one to maintain over time. This is why, sometimes it is better to look at the job-to-be-done, prior to rushing into scripting.

Code and Codeless Creation is Key for High Test Automation Coverage

Next in the overall debate is the Eco-system and tools landscape within the organization. Including a new technology is not easy, often not well-accepted, and also not always justified. In today’s reality when squad teams are working together and consists of a variety of resources with varying skills, objectives, and preferences, integrating a new technology should be done with proper consideration, and with proper validation that it “plays” nice within the other existing tools. Codeless tools in that context, should fill in an important gap within the team, integrate well into existing CI/CD and other processes, and not cause duplication of effort, or additional overhead.

Lastly for this blog (not for the entire debate), I would touch the cost of maintenance of test automation scripts. This is perhaps one of the problematic items for any test automation team. Writing a script once, making it run across platforms over time is an easier said than done task. Applications are constantly changing, and so does the platforms under test (mobile devices/OS versions, Browsers), therefore, scripts needs to be properly maintained to ensure clean and noise-free pipeline. Codeless in many ways addresses such challenge through self-healing of elements, test steps, and more. It can be also achieved within code-based projects through advanced reporting and analysis, with automated root-cause analysis and other methods, but codeless does shine the most in such cases.

Perfecto/TestCraft Object Self-Healing and Weight Mechanism (ML)

Bottom Line

I tried to keep this blog short and to the point, and leave the practitioners the decision-making across the 2 methods. As written in this post, there are plenty of questions to address prior to adopting a codeless tool, and how to combine it within existing code-based suites. Combining both methods in my honest opinion is the way to go in the future, and the way to maximize the overall test automation coverage with greater efficiency across the teams. Make the right decision that fits the project now, and also in the long-run.

I will be running a live webinar covering the exact blog topic hosted by Shani Shoham from 21. Feel free to register here