In any conversation i participate the topic of test coverage comes up – and it is indeed a great challenge for business, practitioners whether they are developers or testers (Agile, DevOps, Waterfall etc.)
Before we understand the how, let’s understand the objectives and coverage definition.
Coverage Aspects:
- Device coverage
- Market coverage
- Test case/use case coverage
- Environment conditions coverage
When we mention device coverage, we should try and include some relevant factor, not just the DUT (Device under test), because it is simply not enough.
Device Coverage
Proper device coverage shall include few important properties and the more permutations you’re going to include in your test lab the higher coverage you will reach. Some of the MUST properties which i would recommend to have as part of the mix are:
- Screen size & Resolution
- PPI (Pixel per inch)
- OSV (OS Version)
To that mix you need to relate to leading market devices and also to legacy devices which are still popular by many users in various geo’s (e.g. Samsung Galaxy S3, iPad 2) in order to obtain both legacy OS and new OS coverage + the above device characteristics.
Market Coverage
Let’s understand Market coverage – This term relates to a combination of data sources to which some teams may have access to, and some won’t. Such coverage term will typically be a combination of leading market statistics and organizational web traffic or monitoring reports which would highlight information around most usage coming from which platform, browsers etc. When combining both Market and Org. data teams can best match their target audience and test against what’s right for their customers from current top usage perspective and in addition get market coverage around new and emerging devices/OSV to allow them to stay on top of market trends.
Another important aspect around coverage is of course the test cases themselves.
Test Case Coverage
Determining the right test cases to execute against each platform and in each test iteration throughout the SDLC (software development life cycle) is a crucial AGILE enabler and an efficiency driver. When there is a robust automation foundation within the organization teams can take advantage of this system and sometime “fail” by overloading it with either redundant test cases or inefficient test cases which does not add the right value. The key to increase test case coverage is to combine Manual & Automation testing (automate of course as much as possible) but only include the cross platform robust test automation and unit tests which are repeatable, valuable for a quick feedback loop between Dev and QA and leave the platform specific tests, corner cases, and such either to be done manually or as a separate JOB/cycle to assure flawless CI/Automation process.
Even with the above in mind, keep in mind that automation without ongoing maintenance, review of the test code will eventually fail especially around mobile due to constant platform specif changes, new features added or new unexpected popups which may block automation tests from running end to end.
Test Environment
Last, for a digital test coverage the user experience and the environment in which the user operates in is everything. Not covering the right environment would eventually waste testing and dev time since these efforts will be done against the wrong or “happy path only” environment. A real mobile environment takes into account the following:
- Network conditions (2G, 3G, Wifi)
- Background applications running as a “noise background” – consuming resources, taking over GPS resources or camera
- incoming calls/popups
- different screen orientation changes while app is in the foreground
- Location of the app
- Locale & language
When taking all of the above under consideration, organizations can really build a test lab which provides sufficient coverage for their product and can easily adjust the lab based on market and product dynamics.
Happy Testing!
Continuous Testing and DevOps Blog 