When looking at some key blockers for Dev and Test team which are trying to either increase their existing test coverage, release more frequently without compromising quality we see some common pitfalls which with some planning in advance can be unblocked.
Let’s look first at the core mobile testing pillars:
The above boxes represent either a full or a subset of a mobile app testing plan. Some of the above can fit into a functional test cycle, some regression or unit and some can be pre-release acceptance tests.
The importance of planning the test coverage and the contents of each iteration in the cycle can be a critical task to the overall app life cycle velocity.
In order to meet both Quality and Velocity goals, Dev/test/QE teams ought to include portions of tests in a model which is based on tests stability.
Let me explain – When trying to include in a CI acceptance test cycle or a functional test cycle more tests than needed, without really debugging each of the tests on few devices, there is a high risks of few tests to fail due to unexpected pop ups, bugs in the tests, specific device issues etc. – such tests will obviously damage and block the entire test cycle.
In order to have a fluent CI/Automation cycle, the recommended practice is to start with a small but robust subset which was already executed few times in the past on more than 1 real device, and were debugged with high probability of not getting stuck etc. Only once this suite was “certified” as stable, it would make sense to increase with the right dependencies and validation points the scope of your cycle, and add more automation tests in order to increase the CI cycle scope.
Such a paced approach which may seem trivial does not happen in many organizations, therefore as soon as a new device is introduced, or a new test is added to cover new features or screen, or simply when a new device unexpected pop up comes up – the CI process breaks.
This results in slow down of the process, delays in release and development tasks and frustration.
To summarize:
- Construct your CI and automation cycle and “certify” each test case and only once it is stable and can run unattended – add it to the acceptance test suite
- continuously debug your entire relevant test suite whenever a new feature, OS, device are introduced to assure nothing breaks your process
- Also assess the tests efficiency in detecting bugs – the ones who keep running and doesn’t add value might be candidates for elimination, making room for newer and more efficient tests
- Less == More –> Assess the most valuable tests which are candidates to identify more bugs than others and include them in the cycle, redundant tests just consume time, resources, and can put your entire cycle in danger
- Make sure you can gain access to all of your devices under tests (DUT) at all time for development, debugging, and continuous testing
- Include sufficient debugging artifacts in your test code either through Try’s and Catches, visual screen/scenario’s validation or other debugging logs, outputs, vitals.
Happy unattended testing!
Continuous Testing and DevOps Blog 