I presented last night at a local Boston meetup hosted by BlazeMeter a session together with my colleague Amir Rozenberg.
The subject was the shift from legacy to open-source frameworks, the motivations behind and also the challenges of adopting open-source without a clear strategy especially in the digital space that includes 3 layers:
Open source connectivity to a Lab
Open-source and its test coverage capabilities (e.g. Can open-source framework support system level, visual analysis, real environment settings and more)
open-source reporting and analysis capabilities.
During the session, Amir also presented an open-source BDD/Cucumber based test framework called Quantum (http://projectquantom.io)
As we kick-off 2017, I am thrilled to release the most updated 6th edition of the Digital Test Coverage Index report, a guide to help you decide how to build your test lab. 2016 was an exciting year in the Digital space, and as usual, Q4 market movement is sure to impact 2017 development and testing plans. And it doesn’t appear that the market is slowing down, with continued innovation expected this year. In this post, I will summarize the key insights we saw last quarter, as well as few important things that are projected for 2017 that should be applied when building your test lab.
Key Takeaways
Beta OS versions remain an important aspect of your test coverage strategy. With Apple releasing 5 different minor versions of iOS 10 since it’s release in September 2016, iPhone/iOS 10 beta are a “must-include in your test lab” device/OS combination. On the browser side, Chrome and Firefox beta versions are also critical test targets for sustaining the quality of your mobile web/responsive websites.
The Android fragmentation trend is changing, with Google putting pressure on device manufacturers to keep pace with the latest OS versions. As evidence, we already see that Android 6.x has the greatest market share as of Q42016, with roughly 27%, followed by Android Lollipop. With Google releasing its first Android Pixel devices, the market is already starting to see a boost in Android 7 Nougat adoption which is expected to grow within Q12017 to between 2-5% market share.
Galaxy S7 and S7 Edge were a turning point for Samsung: Over the last year, Samsung has seen a revenue slowdown due, in part, to competition from both Apple and emerging Android manufacturers OnePlus, Xiaomi, and Huawei. With the launch of Samsung S7 & S7 Edge, the company is regaining its position. We can see in this edition of the Index (and the previous one,) that Samsung is the leading brand in many countries, which should impact the test coverage plans in Brazil, India, Netherlands, UK, Germany and U.S.
The mobile app engagement methods are evolving, with various enterprises counting on the mobile platform to drive more revenues and attract more users. We are seeing greater adoption of external application integration either through dedicated OS-level applications like the iOS iMessage or through other solutions like the Google app shortcuts that were recently introduced as part of Android 7.1. These changes represent a challenge from a testing perspective, since there is now additional outside-of-app dependencies that the Dev and QA teams need to manage.
Test Lab size is expected to slightly grow YoY as the market matures: Looking at the annual growth projection below, we see a slight growth in the need for a 10, 25 and 32 device lab, based on new the devices that are being introduced into the market faster than old devices are retired. What we see is an annual introduction of around 15 leading devices per year with an average retirement of 5-7 per year (due to decreased usage, terminated support by vendor etc.). Integrating these numbers into the 30%-80% model would bring the annual growth as demonstrated in the following graph.
2017 Trends
As this is the first Index for 2017, here are the most important market events that will impact both Dev and QA teams in the digital space, in the categories of Mobile, Web or both.
New Players
The most significant player to joins the mobile space in 2017 is Nokia. After struggling for many years to become a relevant vendor, and being unsuccessful under the Windows Phone brand, Nokia is now back in the game with a new series of Android-based devices that are supposed to be introduced during MWC 2017. A second player that is going to penetrate the mobile market is Microsoft who is supposed to introduce the first Microsoft Surface Phone during H1 2017.
Innovative Technologies
During 2017 we will definitely continue to see more IoT devices, smartwatches, and additional features coming from both Google and Apple, in the mobile, automotive and smart home markets. In addition, we might see the first foldabletouchsmartphone released to the market by Samsung under the name “Samsung X”. In addition, we should see a growing trend of external App interfaces in various forms such as bots, iMessages, App Shortcuts and Voice based features. The market refers to these trends as result of “App Fatigue” which is causing organizations to innovate and change the way their end-users are interacting with the apps and consuming data. From a testing perspective, this is obviously a change from existing methods and will require re-thinking and new development of test cases. In a recent blog, I addressed the above – feel free to read more about it here.
Key Device Launches to Consider for an Updated Test Lab
Most of the below can be seen in the market calendar for 2017, but the highlights are listed here as well:
Samsung S8/S8 Edge flagship devices from Samsung are due by February 2017 and should be the successors of the highly successful S7/S7 Edge devices
iPhone 8/iPhone 8 Plus together with iOS 11 launch in MID-September 2017 will mark the 10th anniversary for the Apple iPhone series. This launch is expected to be a groundbreaking one for iOS users.
Huawei Mate 9/Mate 9 Pro, and in general, the Huawei smartphone portfolio is continuing its global growth. 2017 should continue the growth trend both in China and India, but also as seen in this Index report in many European countries where we are already seeing devices like Huawei P8, P9, and others in use.
From a web perspective, we are not going to see any major surprises from the leading browsers like Chrome, FireFox, and Safari. However, from MicrosoftEdge browser, we expect a significant market share uptick as more and more users adopt Windows 10 and abandon legacy Windows OS machines.
In the Index report, you may find all the information necessary to better plan for 2017, as well as market calendars for both mobile and the web, plus a rich collection of insights and takeaways. DOWNLOAD HERE.
As a mobile evangelist at Perfecto, i foresee the entire mobile and web space for the past 10+ years, following major trends both in the device/hardware front as well as the platform/OS (operating System) front.
I was an Apple user for the past 2 years, using an iPhone 6 Plus device both for my personal as well as my work daily activities. Last month i decided it’s time for a change and i replaced my iPhone with a Google Nexus 6P phablet.
Let me explain some of my reasons to that switch:
Quality and Innovation
Platform Restrictions
Future Looking
Quality and Innovation
In the front of quality and/vs. innovation i found out that as a 2 year trend, Apple’s iOS was constantly straggling with quality that mostly came on top of innovative features and end user -experience. For the past 2 years Apple released 10 versions of iOS 8 stopping at a stable GA of iOS 8.4.1, while for iOS 9 Apple released 10+ versionsstopping at a recent 9.3.5 GA release that addresses security issues. To compare this trend to Android platform – Android5.0 Lollipop released in November 2014 and was enhanced till latest version of 5.1.1 (~5 versions in 2 years). Android Marshmallow6.0 was released in October 2015 and since than only had an additional version of 6.0.1 release. Last month (August 22nd) Google released its new Nougat 7.0 release that is available to users (like me) that hold a Nexus device. iOS 10 is just around the corner with the iPhone 7 devices, but based on the current trend and enormous public Beta versions, it seems like no major changes are expected in the quality/release cadence.
In the Android history we see some major enhancements around sensor based capabilities for payment, logging in as well as UX (user experience) features such as multi window support (see below image), android Doze (battery saving capability). In iOS we also see enhancements around sensors like force-touch, apple pay however these features IMO come in short compared to the platform stability over the past 24 months and the platform constrains which i’ll highlight in the next section.
Platform Restrictions
From an ens user perspective, some of the important platform features involves the ability to customize his UX and look and feel of his personal device. Also having the ability to easily manage his media files such as photos and music with a reasonable storage availability. Apple flagship device with massive market share across regions is the iPhone 6/6S with a default storage (un-expandable) of 16GB – I hardly know a person who has this device/storage size that is happy with that, and does not need to constantly delete files, cancel auto savings of WhatsApp media files and alike. In addition, continuously working with iTunes software as a dependency to media/songs sync is a pain and often i found myself losing my favorite music files or getting them duplicated by simply having to switch from 1 PC to another (people do that, and there are procedures that might have prevented this outcome but still). Compared to the above, most Android devices that are not coming with an external storage option are by default coming with a 64 GB internal memory, and in addition working with music file system is a simple and straight forward task to do.
Switching from my iPhone and iTunes to a Nexus device while having my Gmail account was a very simple thing to do, my music, photos and apps easily “followed” me to the Android device that is already running Android 7 in a stable way.
iOS is not all bad, don’t get me wrong – from an adoption perspective, and device/OS fragmentation this is by far a much better managed platform compared to Android that rolls out its latest GA version in a 4-6 months delay to a non-Nexus device (example: Samsung). In addition the iOS tablets are still a leader in that front with 4-6 years old tablets like iPad Air, iPad 2 that are the most commonly used tablets in the market that can still run iOS 9 OS versions. It is not the case when it comes to Android tablets that tend to be replaced by their end-users in a shorter period of time that iPads.
Future Looking
From a future looking perspective, my opinion is that Google is still going to have a global market share advantage over Apple and will continue to innovate with less frequent releases due to quality than Apple. 2017 is going to show us a continuous battle between Android 7 and iOS 10 in a market that becomes more and more digital and mobile dependent, and with this in mind – the challenge of quality, innovation and less restrictions will be even more critical to independent users as well as large enterprises who are already today fully digital.
As an end-user, i would look at both Google and Apple and examine how their overall digital strategy will transform and enable easier connectivity with smart devices like watches etc., as well as less limited storage and device/OS customization. From a Dev and Test perspective i would assume we will continue to see growing adoption of open-source tools such as Espresso, XCTest UI, Appium etc. as a method of keeping up with the OS platform vendors – Only such open-source frameworks can easily and dynamically grow and support new features and functionalities compared to legacy/commercial tools which are slower to introduce new API’s and new capabilities into their solutions.
How does a company nowadays put together a comprehensive test strategy for delivering high-quality experiences for their applications on any device? I think this is the question I get asked most frequently and it is the biggest challenge in today’s market, how to tackle mobile testing and responsive web testing. The solution can be the difference between an app rated 1 star or an app rated 5 stars.
I had a lot of fun talking to Joe Colantonio from Test Talks about how to create a successful app starting with my Digital Test Coverage Optimizer. Listen to the full talk to hear my ideas on moving from manual testing to automation, tracking the mobile market, the difference between testing in simulators and emulators versus real devices and more.
When it comes to RWD testing, it’s important to test the navigation and functionality on desktop web browsers and mobile devices, but that alone is not enough to guarantee a consistent user experience at all times. The end user is constantly moving between environments throughout the day, and these environments have various attributes, including:
Network conditions (Poor, good, no network)
Locations
App context based on platform and location
Background activities (apps running and consuming resources)
Ads and other popups that block your site content (see image below)
With so many real user environments to consider for both mobile and the desktop web, testing teams should include user conditions in their RWD test plan on top of the traditional testing for UI, navigation, functionality and client-side performance. It will give your DevTest team peace of mind and reduce quality risks significantly.
To learn more, check out our new Responsive Web Testing Guide.
With more and more consumers expecting to shop, bank, work and socialize across different devices, organizations are embracing responsive web design (RWD) as a tool to help them deliver a consistent digital experience on every screen.
Growth of cross-device transactions (Source: Criteo’s State of Mobile Commerce Report)
But due to the complexity of digital environments and user experiences — responsive web is easier said than done. Organizations that develop RWD sites often face challenges when testing to assure smooth website navigation and a great user experience across multiple devices and platforms.
For more information, read our new Comprehensive Guide to Building a Responsive Web Testing Strategy
To get there, we recommended including the following five building blocks as part of your RWD test plan.
Testing for these five areas will help achieve sufficient test coverage, a great user experience and higher traffic to your site.
To download the complete guide for testing RWD Site, go here
Usually when talking about Web testing, QA teams tend to stick with functional, layout and performance testing. In this blog we would highlight the importance of testing for web apps from the end-user perspective and assuring that the experience is also covered in the test plans.
In today’s competitive landscape many web app sites are exposed to a lot of interrupts whether from web add-ons or in the majority of cases adware popups which often interrupt the user flows within the web page or even cover existing and important content from your web page.
In such cases as mentioned above, Dev and Test team need to asses the impact of such pop ups from upper bar security and other add-ons interruptions together with larger ad pop ups covering your web content – as seen above, sometime even when blocking the ads would not solve the problems since the ad vendor will sense that they are on and will pop a different screen.
As a supporting fact to the pain ads are adding to web site owners, we see reports of more than 5.5 millions downloads of Adblock plus tool for FireFox browser just over the last 30 days.
As a quick tip in this blog – enhance your web test plan beyond functionality and cover the cases of environment conditions.
The year just started but as you can see, the market is already busy and we can see as a continuation to 2015, that Apple is much more active on its bug fix releases compared to Android.
Sign up for my quarterly Digital Test Coverage index report to stay up to date with the market trends, top devices, OS versions and desktop browsers.
When looking at some key blockers for Dev and Test team which are trying to either increase their existing test coverage, release more frequently without compromising quality we see some common pitfalls which with some planning in advance can be unblocked.
Let’s look first at the core mobile testing pillars:
The above boxes represent either a full or a subset of a mobile app testing plan. Some of the above can fit into a functional test cycle, some regression or unit and some can be pre-release acceptance tests.
The importance of planning the test coverage and the contents of each iteration in the cycle can be a critical task to the overall app life cycle velocity.
In order to meet both Quality and Velocity goals, Dev/test/QE teams ought to include portions of tests in a model which is based on tests stability.
Let me explain – When trying to include in a CI acceptance test cycle or a functional test cycle more tests than needed, without really debugging each of the tests on few devices, there is a high risks of few tests to fail due to unexpected pop ups, bugs in the tests, specific device issues etc. – such tests will obviously damage and block the entire test cycle.
In order to have a fluent CI/Automation cycle, the recommended practice is to start with a small but robust subset which was already executed few times in the past on more than 1 real device, and were debugged with high probability of not getting stuck etc. Only once this suite was “certified” as stable, it would make sense to increase with the right dependencies and validation points the scope of your cycle, and add more automation tests in order to increase the CI cycle scope.
Such a paced approach which may seem trivial does not happen in many organizations, therefore as soon as a new device is introduced, or a new test is added to cover new features or screen, or simply when a new device unexpected pop up comes up – the CI process breaks.
This results in slow down of the process, delays in release and development tasks and frustration.
To summarize:
Construct your CI and automation cycle and “certify” each test case and only once it is stable and can run unattended – add it to the acceptance test suite
continuously debug your entire relevant test suite whenever a new feature, OS, device are introduced to assure nothing breaks your process
Also assess the tests efficiency in detecting bugs – the ones who keep running and doesn’t add value might be candidates for elimination, making room for newer and more efficient tests
Less == More –> Assess the most valuable tests which are candidates to identify more bugs than others and include them in the cycle, redundant tests just consume time, resources, and can put your entire cycle in danger
Make sure you can gain access to all of your devices under tests (DUT) at all time for development, debugging, and continuous testing
Include sufficient debugging artifacts in your test code either through Try’s and Catches, visual screen/scenario’s validation or other debugging logs, outputs, vitals.
It’s an exciting time to be a digital company. Your customers are engaging with your products on various screens, moving between desktop web browsers to apps on mobile devices. But in the effort to guarantee quality web and mobile experiences, organizations are struggling to find the right testing mix.
It’s true that mobile is far more complex and fragmented than the web, but with so many web browser/OS permutations out there (i.e. Chrome OS 47 running on Windows 7, 8.1, 10, Mac Yosemite, etc.) precise testing becomes a challenge.
To help DevTest teams test more precisely, Perfecto recently published the “Digital Test Coverage Index – Edition 3”, a quarterly report that provides a prescriptive way to build a digital test lab that covers 30%, 50%, or 80% of mobile device and web browser markets in various geographies. The report — intended for organizations just starting their digital journey or trying to move to the next stage — is based on market share data and analysis of enterprise customer usage in Perfecto’s cloud testing lab.
Using the above 30%-50%-80% coverage model featured in the Index report, teams can more accurately define their desired testing parameters and allocate the recommended devices and virtual machines running the relevant desktop browsers. Teams that are developing a responsive web application (RWD) can refer to the Index and then test the app in their lab on the recommended smartphones and tablets alongside the recommended desktop browser/OS permutations.
On the subject of web browser/OS mixes: According to our latest Index, of the top 30% of desktop browsers in the U.S. market, Chrome OS 46 (version 47 was just released and is well-adopted already) is by far the leading browser on Windows 7, followed by FireFox OS version 42 on Windows 7, and Safari OS 9 on Mac OS El Capitan. The Index report includes the complete 30%-50%-80% matrix for web/OS and mobile device/OS combinations.
It’s also worth noting that in the browser testing landscape, the Windows 10 platform is gaining momentum and will soon become the second most popular desktop OS in most of the geographies, according to market share numbers.
It will take more than looking at a list of smartphones and web browsers to ensure full digital test coverage for native and hybrid mobile apps, mobile web browsers and RWD. So organizations need to combine their existing customer analytics with a regularly updated test coverage index that reflects market adoption rates in various geographies. Another important metric to monitor is the status of legacy platforms that are still relevant enough to test against. For example, the Samsung Galaxy S3 is a leading legacy smartphone in most markets in the same way that we still see many Windows 7 machines even though Windows 8 and Windows 10 are widely available.
For more details on how to test for the full digital experience, download the free Digital Test Coverage Index.
Continuous Testing and DevOps Blog 