Month: February 2016

Posted on

Tests to Include Within Automation Suite

When developing a mobile or desktop test automation plan organization often struggle with the right scope and coverage for the project.

In previous post, i covered the test coverage recommendations in a mobile project and now, i would like to also expand on the topic of which tests to automate.

Achieving release agility with high quality is fully dependent today more than ever on continuous testing which is gained through proper test automation, however automating every test scenario is not feasible and not necessary to meet this goal.

In the below table  we can see some very practical examples of test cases with various parameters with a Y/N recommendation whether to automate or no.

As shown below, and as a rule for both Mobile, Web and other projects the key tests by definition which should be added to an automation suite (from ROI perspective and TTM) are the ones who are:

  • Required to be executed against various data sets
  • Tests which ought to run against multiple environments (Devices, Browsers, Locations)
  • Complex test scenario’s (these are time consuming and error prone when done manually)
  • Tedious and repetitive test cases are a must to automate
  • Tests which are dependent on various aspects (can be other tests, other environments etc.)

Picture1

Bottom line: Automation is key in today’s digital world, but doing it right and wisely can shorten time to market, redundant resources and a lot of wasted R&D time chasing unimportant defects coming from irrelevant tests

Happy Testing!

 

 

Posted on

Q1 2016 Calendar Overview

Q12016

The year just started but as you can see, the market is already busy and we can see as a continuation to 2015, that Apple is much more active on its bug fix releases compared to Android.

Sign up for my quarterly Digital Test Coverage index report to stay up to date with the market trends, top devices, OS versions and desktop browsers.

Digital Test Coverage Download Page

Happy Testing!

Posted on

Few Best Practices Around Mobile Testing and Agility

When looking at some key blockers for Dev and Test team which are trying to either increase their existing test coverage, release more frequently without compromising quality we see some common pitfalls which with some planning in advance can be unblocked.

Let’s look first at the core mobile testing pillars:

MobileTestPillars-3

The above boxes represent either a full or a subset of a mobile app testing plan. Some of the above can fit into a functional test cycle, some regression or unit and some can be pre-release acceptance tests.

The importance of planning the test coverage and the contents of each iteration in the cycle can be a critical task to the overall app life cycle velocity.

In order to meet both Quality and Velocity goals, Dev/test/QE teams ought to include portions of tests in a model which is based on tests stability.

Let me explain – When trying to include in a CI acceptance test cycle or a functional test cycle more tests than needed, without really debugging each of the tests on few devices, there is a high risks of few tests to fail due to unexpected pop ups, bugs in the tests, specific device issues etc. – such tests will obviously damage and block the entire test cycle.

In order to have a fluent CI/Automation cycle, the recommended practice is to start with a small but robust subset which was already executed few times in the past on more than 1 real device, and were debugged with high probability of not getting stuck etc. Only once this suite was “certified” as stable, it would make sense to increase with the right dependencies and validation points the scope of your cycle, and add more automation tests in order to increase the CI cycle scope.

Such a paced approach which may seem trivial does not happen in many organizations, therefore as soon as a new device is introduced, or a new test is added to cover new features or screen, or simply when a new device unexpected pop up comes up – the CI process breaks.

This results in slow down of the process, delays in release and development tasks and frustration.

To summarize:

  • Construct your CI and automation cycle and “certify” each test case and only once it is stable and can run unattended – add it to the acceptance test suite
  • continuously debug your entire relevant test suite whenever a new feature, OS, device are introduced to assure nothing breaks your process
    • Also assess the tests efficiency in detecting bugs – the ones who keep running and doesn’t add value might be candidates for elimination, making room for newer and more efficient tests
  • Less == More –> Assess the most valuable tests which are candidates to identify more bugs than others and include them in the cycle, redundant tests just consume time, resources, and can put your entire cycle in danger
  • Make sure you can gain access to all of your devices under tests (DUT) at all time for development, debugging, and continuous testing
  • Include sufficient debugging artifacts in your test code either through Try’s and Catches, visual screen/scenario’s validation or other debugging logs, outputs, vitals.

Happy unattended testing!

Posted on

Selenium Is the New Testing Tool Standard

Seems like the debate in the world of test automation tools is over.

If few years back HP QTP/UFT (formerly WinRunner) was the standard and most commonly used tool for test automation in the QA space, those days are over.

The shift toward Agile, Devops and such trends together with the digital transformation which includes multi platform testing of Mobile, Web, IOT in a very short amount of times changed the tools landscape and the testing requirements.

See below a snapshot of the top required testing tools which show that the shift already started in 2011 where Selenium passed HP tools in the market adoption.

qtp vs selenium

Source: http://www.seleniumguide.com/

The requirements today are that testing is done as early as possible in the project life cycle (SDLC) and to enforce this process, developers ought to play a significant role – Testing is now being developed and executed by all Agile team members including developers, testers, ops people and others.

In order for the shift and the adoption to grow the tools need to be tightly integrated into the developers environment (IDE’s) which in the digital space might be Eclipse, Android Studio, Visual Studio, Xcode or other cross platform IDE’s like PhoneGap or Titanium.

The additional aspect of test framework adoption such as Selenium and Appium lies in the Open-Source nature of these tools. The flexibility of such open source tools to get extended by developers according to their needs is a great deal compared to closed testing tools such as UFT which are disconnected from the IDE and development environments.

We shall continue to monitor the tools space and movement, but seems like the open source tools is becoming standard for Agile, DevOps practitioners which find these tools suitable for their shift left activities, keeping up with the market dynamics and competition, as well as great enablers for quality and velocity maintainability.

To get some heads up into what is the future of Selenium, and how are the efforts moving on toward making the web browsers drivers (Chrome, Firefox, IE etc.) standard and managed by the browser vendors, refer to this great session (courtesy of Applitools)

http://testautomation.applitools.com/post/120437769417/the-future-of-selenium-andreas-tolfsen-video

Posted on

Planning Mobile Test Coverage

In any conversation i participate the topic of test coverage comes up – and it is indeed a great challenge for business, practitioners whether they are developers or testers (Agile, DevOps, Waterfall etc.)

Before we understand the how, let’s understand the objectives and coverage definition.

Coverage Aspects:

  1. Device coverage
  2. Market coverage
  3. Test case/use case coverage
  4. Environment conditions coverage

When we mention device coverage, we should try and include some relevant factor, not just the DUT (Device under test), because it is simply not enough.

Device Coverage

Proper device coverage shall include few important properties and the more permutations you’re going to include in your test lab the higher coverage you will reach. Some of the MUST properties which i would recommend to have as part of the mix are:

  • Screen size & Resolution
  • PPI (Pixel per inch)
  • OSV (OS Version)

To that mix you need to relate to leading market devices and also to legacy devices which are still popular by many users in various geo’s (e.g. Samsung Galaxy S3, iPad 2) in order to obtain both legacy OS and new OS coverage + the above device characteristics.

Market Coverage

Let’s understand Market coverage – This term relates to a combination of data sources to which some teams may have access to, and some won’t. Such coverage term will typically be a combination of leading market statistics and organizational web traffic or monitoring reports which would highlight information around most usage coming from which platform, browsers etc. When combining both Market and Org. data teams can best match their target audience and test against what’s right for their customers from current top usage perspective and in addition get market coverage around new and emerging devices/OSV to allow them to stay on top of market trends.

Another important aspect around coverage is of course the test cases themselves.

Test Case Coverage

Determining the right test cases to execute against each platform and in each test iteration throughout the SDLC (software development life cycle) is a crucial AGILE enabler and an efficiency driver. When there is a robust automation foundation within the organization teams can take advantage of this system and sometime “fail” by overloading it with either redundant test cases or inefficient test cases which does not add the right value. The key to increase test case coverage is to combine Manual & Automation testing (automate of course as much as possible) but only include the cross platform robust test automation and unit tests which are repeatable, valuable for a quick feedback loop between Dev and QA and leave the platform specific tests, corner cases, and such either to be done manually or as a separate JOB/cycle to assure flawless CI/Automation process.

Even with the above in mind, keep in mind that automation without ongoing maintenance, review of the test code will eventually fail especially around mobile due to constant platform specif changes, new features added or new unexpected popups which may block automation tests from running end to end.

Test Environment

Last, for a digital test coverage the user experience and the environment in which the user operates in is everything. Not covering the right environment would eventually waste testing and dev time since these efforts will be done against the wrong or “happy path only” environment. A real mobile environment takes into account the following:

  • Network conditions (2G, 3G, Wifi)
  • Background applications running as a “noise background” – consuming resources, taking over GPS resources or camera
  • incoming calls/popups
  • different screen orientation changes while app is in the foreground
  • Location of the app 
  • Locale & language

When taking all of the above under consideration, organizations can really build a test lab which provides sufficient coverage for their product and can easily adjust the lab based on market and product dynamics.

Happy Testing!