January 2016 – Continuous Testing and DevOps Blog

It’s an exciting time to be a digital company. Your customers are engaging with your products on various screens, moving between desktop web browsers to apps on mobile devices. But in the effort to guarantee quality web and mobile experiences, organizations are struggling to find the right testing mix.

It’s true that mobile is far more complex and fragmented than the web, but with so many web browser/OS permutations out there (i.e. Chrome OS 47 running on Windows 7, 8.1, 10, Mac Yosemite, etc.) precise testing becomes a challenge.

To help DevTest teams test more precisely, Perfecto recently published the “Digital Test Coverage Index – Edition 3”, a quarterly report that provides a prescriptive way to build a digital test lab that covers 30%, 50%, or 80% of mobile device and web browser markets in various geographies. The report — intended for organizations just starting their digital journey or trying to move to the next stage — is based on market share data and analysis of enterprise customer usage in Perfecto’s cloud testing lab.

Using the above 30%-50%-80% coverage model featured in the Index report, teams can more accurately define their desired testing parameters and allocate the recommended devices and virtual machines running the relevant desktop browsers. Teams that are developing a responsive web application (RWD) can refer to the Index and then test the app in their lab on the recommended smartphones and tablets alongside the recommended desktop browser/OS permutations.

On the subject of web browser/OS mixes: According to our latest Index, of the top 30% of desktop browsers in the U.S. market, Chrome OS 46 (version 47 was just released and is well-adopted already) is by far the leading browser on Windows 7, followed by FireFox OS version 42 on Windows 7, and Safari OS 9 on Mac OS El Capitan. The Index report includes the complete 30%-50%-80% matrix for web/OS and mobile device/OS combinations.

It’s also worth noting that in the browser testing landscape, the Windows 10 platform is gaining momentum and will soon become the second most popular desktop OS in most of the geographies, according to market share numbers.

It will take more than looking at a list of smartphones and web browsers to ensure full digital test coverage for native and hybrid mobile apps, mobile web browsers and RWD. So organizations need to combine their existing customer analytics with a regularly updated test coverage index that reflects market adoption rates in various geographies. Another important metric to monitor is the status of legacy platforms that are still relevant enough to test against. For example, the Samsung Galaxy S3 is a leading legacy smartphone in most markets in the same way that we still see many Windows 7 machines even though Windows 8 and Windows 10 are widely available.

For more details on how to test for the full digital experience, download the free Digital Test Coverage Index.

As we wrap up another year, we thought it would be fun and informative to take a look back at the mobile devices, operating system updates and trends that hit the market this year.

Let’s start with this bird’s-eye view graphic of the important releases that made 2015 such an innovative year in mobile.

Mobile Market 2015 Retrospective Calendar, Source (Perfecto's Digital Test Coverage Index Report) — Mobile Market 2015 Retrospective Calendar (Source: Perfecto’s Digital Test Coverage Index report)

There were more than 30 significant smartphones and tablets released this year. Many of these devices such as the iPhone 6S and 6S Plus, Samsung Galaxy S6, the Galaxy Note 5 and the LG Nexus 5X quickly became popular on a global scale. When putting together our quarterly Digital Test Coverage Index we noticed that these newly-minted devices ranked high up in the indexes in both the U.S. and Europe.

But a device is nothing if it’s not running on an updated operating system — and this year brought 15 OS releases (major and minor), from Apple iOS 8.2 to 9.2, and Android 5.1.1 to Android 6.0 Marshmallow. Microsoft released what it hopes will be a mobile game-changer, Windows 10. All of these OS releases will have an impact on the already fragmented mobile space, keeping DevTest teams busy with more re-testing.

This year we also saw disruptive technologies take center stage, such as Apple Force Touch, Touch ID authentication, Mobile Payments, Voice commands and contextual awareness apps based on location.

Looking ahead, it’s clear that 2016 is going to be just as hectic, fragmented, and EXCITING. In the coming year, enterprises can expect mobile and web user engagement to be key business drivers, forcing many organizations to do rigorous testing of new app features on mobile devices, OSes and web browsers to deliver a memorable user experience to customers.

As a sneak in preview in our latest Digital Test Coverage Index, we’ve added a 2016 calendar and we already see new trends emerging such as increasing tablet screen sizes that aim to replace today’s laptops (i.e. the Samsung Galaxy View tablet with its unique screen size of 18.4”, and Apple’s iPad Pro with its 12.9” screen size).

For more details on important mobile and web test coverage trends, download the Digital Test Coverage Index.

Happy holidays to all. Here’s to successful digital test coverage in 2016!