In the world of mobile app testing, a developer’s claim “Well it works on my device” is not acceptable. Mobile is by far more fragmented than Desktop/Web platforms due to the variety of device types, OS versions, and the environment on which it operates (network impact, roaming, battery stretch etc.).
It is not sufficient to test on one or even a few devices, but instead test apps on several devices that adequately cover your end-users’ device pool. How can you keep up with testing apps on so many devices, you ask? Easy. Build a working mobile test automation strategy that runs the same script across all of your devices, making your Dev/QA teams confident that they are delivering the best app to end-users.
From the experience Perfecto Mobile gathers everyday, we came up with the must-haves for building a continuous, unattended mobile test automation strategy that provides immediate feedback to its Dev/QA teams.
Relevant Device Selection
It is one of the toughest dilemmas in today’s market to identify the right device X OS mix on which teams should develop and test their mobile app against. Base your selection on the right criterias such as:
- Target market and competition/End-Users –> On which devices and OS versions your end-users actually use the most (based on organizational analytic data)
- Generic market statistics –> Top sold devices (smartphones and Tablets) in the relevant regions. Such data can be retrieved from analyst research such as IDC, Flurry, ComScore and others.
- Change rate and market dynamics –> As we all see, the mobile market is in constant change and your teams need to “listen” to the trends and changes- adopt them and change the strategy accordingly
Unattended mobile app test automation which supports continuous integration and fast feedback:
- Mandates mobile OS system level control – If the script cannot initiate events outside of the AUT (App under test) it cannot really test the app as the end-user will use it
- Incoming alerts and error handling are also mandating system level control out side of the AUT
- Hybrid objects support – automation framework should support both OS level objects (DOM for web and hybrid apps) and Visual objects for full use case support
- Device logs, rich media reports (Video and Visual) and vitals (Memory status, Battery etc.) are critical evidence to Dev/DevOps teams as feedback to act upon
Device Elasticity
As your mobile app project grows/shrinks based on your SDLC phase, your automation solution needs to be flexible enough to accommodate necessary changes and provide the needed support to your team (even more critical in cases of distributed teams).
Integration is Key for Cross Team Collaboration
Whether your Agile/Dev/QA teams are using ALM tools (HP, Microsoft, etc.) or OSS (Selenium, Robotium, JUnit) tools, your automation framework solution should be able to connect to these tools and bridge any gaps in the daily continuous work flow. Use what your team knows – this is the best way to release mobile apps faster without compromising quality.
Real Devices
It is imperative to understand that testing on something other than real devices in your critical automation scripts and as your mobile app evolves is a huge risk. Only real devices running formal OEM OS flavors against real networks reveal an app’s true functionality in the same way your end-user will experience it.
To conclude, see below (Fig 1), a check list of mobile testing pillars which need to be part of your Mobile App Testing Plan.
Fig 1: The pillars of mobile app testing
Good Luck!
Continuous Testing and DevOps Blog 