Test Automation – Page 3 – Continuous Testing and DevOps Blog

October 9, 2016

3 Ways to Make Mobile Manual Testing Less Painful

With 60% of the industry still functioning at 30% mobile test automation it’s clear that manual testing is taking a major chunk of a testing team’s time. As we acknowledge the need for both manual and automation testing, and without drilling down into the caveats of manual testing, let’s understand how can teams can reduce the time it takes, and even transition to an automated approach to testing.

1. Manual and Automation Testing: Analyze Your Existing Test Suite

You and your testing team should be well-positioned to optimize your test suite in one of the following ways:
– Scope out irrelevant manual tests per specific test cycles (e.g. not all manual tests are required for each sanity or regression test)
– Eliminate tests that consistently pass and don’t add value
– Identify and consolidate duplicate tests
– Suggest manual tests that are better-suited for automation (e.g. data driven tests or tests that rely on timely setup of environments)

The result should be a mixture of both manual and automation testing approaches, with the goal of shifting more of the testing toward automation.

2. Consider a Smooth Transition to “Easier” Test Frameworks

In most cases, the blocker for increasing test automation lies inside the QA team, and is often related to a skills gap. Today’s common mobile test automation tools are open-source, and require medium to high-level development skills in languages such as Java, C#, Python and Java Script. These skills are hard to find within traditional testing teams. On the other hand, if QA teams utilize alternatives such as Behavior Driven Development (BDD) solutions like Cucumber, it creates an easier path toward automation by virtue of using a common language that is easy to get started and scale.

3. Shift More Test Automation Inside the Dev Cycle

When thinking about your existing test automation code and the level of code coverage it provides, there may be a functional coverage overlap between the automated and manual testing. If the automation scripts are shared across the SDLC and are also executed post-commit on every build, this can shrink some of the manual validation work the testing team needs to do. Also – and miraculously related! – by joining forces with your development and test automation teams and having them help with test automation, it will decrease workload and create shorter cycles, resulting in happier manual testers.

Bottom Line:

Most businesses will continue to have a mix of manual and automation testing. Manual testing will never go away and in some cases, it is even a product requirement. But as you optimize your overall testing strategy, investing in techniques like BDD can make things much easier for everyone involved with both manual and automation testing throughout the lifecycle.

August 5, 2016

4 Benefits of Using the Espresso Test Automation Tool

If you’re an Android developer, you’re probably familiar with Google’s Espresso test automation framework. As an open-source tool, it’s very easy for developers to use and extend within their working environment (Android Studio IDE).

But before discussing the benefits of Espresso, let’s understand the motivations and pains developers and test automation engineers face today while trying to validate their Android application (APK) throughout the build/dev/test workflow.

Each build needs to be validated after code changes are made.
Dependencies on remote servers and other workstations for testing slow down the process.
Unit and functional tests need to be easy to execute from both an IDE and continuous integration perspective.
Apps need to be tested using the latest Android OS APIs that support new platform features and OS versions.
Testing needs to occur on both emulators and real devices.

In light of these challenges, it’s clear why the adoption of the Espresso automation framework is high. Even though Espresso is an instrumentation-based test framework, it has many benefits to both developers and test automation engineers. It uses Junit underneath the hood, so Espresso is easy to use within leading IDEs and provides useful testing annotations and assertions. It’s also fully integrated within the leading Google Android IDE – Android Studio.

Here are four main benefits of using Espresso:

1. Espresso workflow is simple to use

The way Espresso works is by allowing developers to build a test suite as a stand-alone APK that can be installed on the target devices alongside the application under test and be executed very quickly.

2. Fast and reliable feedback to developers

As developers are trying to accelerate deployment, Espresso gives them fast feedback on their code changes so they can move on to the next feature or defect fix; having a robust and fast test framework plays a key role.

Espresso does not require any server (like Selenium Remote WebDriver) to communicate with; instead it runs side-by-side with the app and delivers very fast (minutes) test results to the developer.

3. Less mobile testing flakiness

Because Espresso offers a synchronized method of execution, the stability of the test cycle is very high. There’s a built-in mechanism in Espresso that, prior to moving to the next steps in the test, validates that the Element or Object is actually displayed on the screen. This eliminates test execution from breaking when confronted with “objects not detected” and other errors.

4. Developing Espresso test automation isn’t hard

Developing Espresso test automation is quite easy. It is based on Java and Junit, which is a core skillset for any Android app developer. Because Espresso works seamlessly within the Android Studio IDE, there’s no setup or ramping up and no “excuses” – to actually shift quality in the in-cycle stage of the app SDLC.

In addition to the above, there is of course the large community powered by Google that pushes the Espresso test automation framework and allow easy and fast ramp up for newcomers.

Learn more using the Espresso Cheat sheet below:

Espresso Test Automation Framework

Perfecto is offering support for both Android Studio IDE as well as the ability to install and launch an Espresso test suite (APK) on real devices in the cloud across various locations and user conditions. For more information, please refer to the Perfecto Community and search for “Android Studio” or “Espresso.”

July 11, 2016

Joe Colantonio’s Test Talk: Mobile Testing Coverage Optimization

How does a company nowadays put together a comprehensive test strategy for delivering high-quality experiences for their applications on any device? I think this is the question I get asked most frequently and it is the biggest challenge in today’s market, how to tackle mobile testing and responsive web testing. The solution can be the difference between an app rated 1 star or an app rated 5 stars.

Play Podcast

I had a lot of fun talking to Joe Colantonio from Test Talks about how to create a successful app starting with my Digital Test Coverage Optimizer. Listen to the full talk to hear my ideas on moving from manual testing to automation, tracking the mobile market, the difference between testing in simulators and emulators versus real devices and more.

https://joecolantonio.com/testtalks/110-mobile-testing-coverage-optimization-eran-kinsbruner/

May 7, 2016

7 Mobile Test Automation Best Practices

Developing a mobile test automation scenario isn’t that complicated. Developers and testers use a variety of commercial test automation frameworks or open source tools such as Selenium and Appium to do automation. However, when trying to execute these tests on real devices or integrate them into an Agile or CI (continuous integration) workflow, things get a little complicated.

The major challenges around mobile test automation

The essence of developing test automation is to be able to use and re-use scripts many times, across platforms and environments. Test automation should be as maintainable as possible, especially as new platforms and product features are released. Many organizations that develop test automation for their mobile apps face the following challenges:

Executing the tests against a variety of real mobile devices
Executing these tests in parallel
Leveraging existing test code (re-usability) for new tests
Including real end-user environments/conditions (changing network conditions, low battery) in the tests
Overcoming unexpected interruptions (incoming call, apps running in background)
Running these tests unattended — over night, as part of a Jenkins CI job

These are just few of the challenges organizations confront when trying to progress from older SDLC processes and meet faster releases and enhanced Dev–>Build–>Deploy–>Test–>Deploy cycles.

7 practical test automation tips

Overcoming these challenges starts with few changes in the overall mobile app dev and test processes.

Consider these seven recommendations for building sustainable unattended automation.

The key to mobile test automation is to start with a small number of test cases, automate them, and assure that they are robust enough and can be executed in parallel and unattended. Only then should you invest more and grow the test suite.

An important question to ask at the start is: What should I be automating? Organization often do not choose the right tests to automate, resulting in lost development time, weak ROI, and an over-reliance on manual testing.

To learn more about the 7 Ways to Overcome Test Automation Obstacles, please join us next week for a webinar hosted by myself, automation expert and author Daniel Knott, and Perfecto’s Director of Technology Uzi Eilon.

April 26, 2016

Responsive Web: Test for the Real User Experience

One of the great benefits of building a responsive web site (RWD) is it can give the user a consistent web experience across any digital device, in any location.

When it comes to RWD testing, it’s important to test the navigation and functionality on desktop web browsers and mobile devices, but that alone is not enough to guarantee a consistent user experience at all times. The end user is constantly moving between environments throughout the day, and these environments have various attributes, including:

Network conditions (Poor, good, no network)
Locations
App context based on platform and location
Background activities (apps running and consuming resources)
Ads and other popups that block your site content (see image below)

With so many real user environments to consider for both mobile and the desktop web, testing teams should include user conditions in their RWD test plan on top of the traditional testing for UI, navigation, functionality and client-side performance. It will give your DevTest team peace of mind and reduce quality risks significantly.

To learn more, check out our new Responsive Web Testing Guide.

March 24, 2016March 24, 2016

Responsive Web: Five Testing Considerations

With more and more consumers expecting to shop, bank, work and socialize across different devices, organizations are embracing responsive web design (RWD) as a tool to help them deliver a consistent digital experience on every screen.

Growth of cross-device transactions (Source: Criteo’s State of Mobile Commerce Report)

But due to the complexity of digital environments and user experiences — responsive web is easier said than done. Organizations that develop RWD sites often face challenges when testing to assure smooth website navigation and a great user experience across multiple devices and platforms.

For more information, read our new Comprehensive Guide to Building a Responsive Web Testing Strategy

To get there, we recommended including the following five building blocks as part of your RWD test plan.

Testing for these five areas will help achieve sufficient test coverage, a great user experience and higher traffic to your site.

To download the complete guide for testing RWD Site, go here

March 3, 2016

User Conditions Importance in Web App Testing

Usually when talking about Web testing, QA teams tend to stick with functional, layout and performance testing. In this blog we would highlight the importance of testing for web apps from the end-user perspective and assuring that the experience is also covered in the test plans.

In today’s competitive landscape many web app sites are exposed to a lot of interrupts whether from web add-ons or in the majority of cases adware popups which often interrupt the user flows within the web page or even cover existing and important content from your web page.

Screen-Shot-2015-09-09-at-11.54.14-AM-640x275

Source: Mashable

In such cases as mentioned above, Dev and Test team need to asses the impact of such pop ups from upper bar security and other add-ons interruptions together with larger ad pop ups covering your web content – as seen above, sometime even when blocking the ads would not solve the problems since the ad vendor will sense that they are on and will pop a different screen.

As a supporting fact to the pain ads are adding to web site owners, we see reports of more than 5.5 millions downloads of Adblock plus tool for FireFox browser just over the last 30 days.

As a quick tip in this blog – enhance your web test plan beyond functionality and cover the cases of environment conditions.

February 11, 2016

Tests to Include Within Automation Suite

When developing a mobile or desktop test automation plan organization often struggle with the right scope and coverage for the project.

In previous post, i covered the test coverage recommendations in a mobile project and now, i would like to also expand on the topic of which tests to automate.

Achieving release agility with high quality is fully dependent today more than ever on continuous testing which is gained through proper test automation, however automating every test scenario is not feasible and not necessary to meet this goal.

In the below table we can see some very practical examples of test cases with various parameters with a Y/N recommendation whether to automate or no.

As shown below, and as a rule for both Mobile, Web and other projects the key tests by definition which should be added to an automation suite (from ROI perspective and TTM) are the ones who are:

Required to be executed against various data sets
Tests which ought to run against multiple environments (Devices, Browsers, Locations)
Complex test scenario’s (these are time consuming and error prone when done manually)
Tedious and repetitive test cases are a must to automate
Tests which are dependent on various aspects (can be other tests, other environments etc.)

Bottom line: Automation is key in today’s digital world, but doing it right and wisely can shorten time to market, redundant resources and a lot of wasted R&D time chasing unimportant defects coming from irrelevant tests

Happy Testing!

February 11, 2016

Q1 2016 Calendar Overview

The year just started but as you can see, the market is already busy and we can see as a continuation to 2015, that Apple is much more active on its bug fix releases compared to Android.

Sign up for my quarterly Digital Test Coverage index report to stay up to date with the market trends, top devices, OS versions and desktop browsers.

Digital Test Coverage Download Page

Happy Testing!

February 4, 2016February 23, 2016

Few Best Practices Around Mobile Testing and Agility

When looking at some key blockers for Dev and Test team which are trying to either increase their existing test coverage, release more frequently without compromising quality we see some common pitfalls which with some planning in advance can be unblocked.

Let’s look first at the core mobile testing pillars:

The above boxes represent either a full or a subset of a mobile app testing plan. Some of the above can fit into a functional test cycle, some regression or unit and some can be pre-release acceptance tests.

The importance of planning the test coverage and the contents of each iteration in the cycle can be a critical task to the overall app life cycle velocity.

In order to meet both Quality and Velocity goals, Dev/test/QE teams ought to include portions of tests in a model which is based on tests stability.

Let me explain – When trying to include in a CI acceptance test cycle or a functional test cycle more tests than needed, without really debugging each of the tests on few devices, there is a high risks of few tests to fail due to unexpected pop ups, bugs in the tests, specific device issues etc. – such tests will obviously damage and block the entire test cycle.

In order to have a fluent CI/Automation cycle, the recommended practice is to start with a small but robust subset which was already executed few times in the past on more than 1 real device, and were debugged with high probability of not getting stuck etc. Only once this suite was “certified” as stable, it would make sense to increase with the right dependencies and validation points the scope of your cycle, and add more automation tests in order to increase the CI cycle scope.

Such a paced approach which may seem trivial does not happen in many organizations, therefore as soon as a new device is introduced, or a new test is added to cover new features or screen, or simply when a new device unexpected pop up comes up – the CI process breaks.

This results in slow down of the process, delays in release and development tasks and frustration.

To summarize:

Construct your CI and automation cycle and “certify” each test case and only once it is stable and can run unattended – add it to the acceptance test suite
continuously debug your entire relevant test suite whenever a new feature, OS, device are introduced to assure nothing breaks your process
- Also assess the tests efficiency in detecting bugs – the ones who keep running and doesn’t add value might be candidates for elimination, making room for newer and more efficient tests
Less == More –> Assess the most valuable tests which are candidates to identify more bugs than others and include them in the cycle, redundant tests just consume time, resources, and can put your entire cycle in danger
Make sure you can gain access to all of your devices under tests (DUT) at all time for development, debugging, and continuous testing
Include sufficient debugging artifacts in your test code either through Try’s and Catches, visual screen/scenario’s validation or other debugging logs, outputs, vitals.