September 2016 – Continuous Testing and DevOps Blog

Why is this a problem for developers and testers?

Let’s examine iOS 9 adoption. Based on the data below from Mixpanel, two weeks after iOS 9 was launched on September 16, 2015, 40% of users already upgraded to iOS 9. Today, iOS 9 adoption is close to 90%.

Now, here’s a quick look at market share numbers for iOS devices.

Based on this data from Localytics and the recent Perfecto Digital Test Coverage Index report it’s clear that at least iPad 2, iPhone 4S and iPad mini are among the most-used devices in various markets, including the U.S. (see below)

Implications for mobile testing plans

The information above means one thing. Dev and test teams will need to support the new iPhone 7/7 Plus along with other Apple devices that can run iOS 10. Additionally, they need to reserve a portion of testing for iOS 9 for the older devices mentioned above. Unfortunately, this will create latency in testing activities. It will also require test automation so tests can be easily executed on devices running iOS 9 and iOS 10.

Another side effect may be lower device adoption for iOS 10 because large groups of users will simply be stuck on iPad 2, iPad Mini and iPhone 4S. We also see a clear market trend of older iPads continuing to be the most popular, as iPad users take longer to upgrade to new tablets than Android tablet users.

The end result is we’re going to see growing iOS fragmentation in various markets and complex testing ahead.

Open source test framework implications

iOS 10 is not just disrupting the mobile device and OS landscape, it’s also impacting open-source test frameworks such as Appium.

As you can read in the threads below, iOS 10 is “breaking” Appium test framework functionality, impacting installations and the launching of IPA files. It is also causing issues working with XCUITest and the iOS WebKit.

It’s worth noting that vendors such as Perfecto are able to overcome this challenge as they continue to support Appium test automation with iOS 10 on iOS devices for all beta versions and the GA version from the first day it was publicly available.

As a mobile evangelist at Perfecto, i foresee the entire mobile and web space for the past 10+ years, following major trends both in the device/hardware front as well as the platform/OS (operating System) front.

I was an Apple user for the past 2 years, using an iPhone 6 Plus device both for my personal as well as my work daily activities. Last month i decided it’s time for a change and i replaced my iPhone with a Google Nexus 6P phablet.

Let me explain some of my reasons to that switch:

Quality and Innovation
Platform Restrictions
Future Looking

Quality and Innovation

In the front of quality and/vs. innovation i found out that as a 2 year trend, Apple’s iOS was constantly straggling with quality that mostly came on top of innovative features and end user -experience. For the past 2 years Apple released 10 versions of iOS 8 stopping at a stable GA of iOS 8.4.1, while for iOS 9 Apple released 10+ versions stopping at a recent 9.3.5 GA release that addresses security issues. To compare this trend to Android platform – Android 5.0 Lollipop released in November 2014 and was enhanced till latest version of 5.1.1 (~5 versions in 2 years). Android Marshmallow 6.0 was released in October 2015 and since than only had an additional version of 6.0.1 release. Last month (August 22nd) Google released its new Nougat 7.0 release that is available to users (like me) that hold a Nexus device. iOS 10 is just around the corner with the iPhone 7 devices, but based on the current trend and enormous public Beta versions, it seems like no major changes are expected in the quality/release cadence.

In the Android history we see some major enhancements around sensor based capabilities for payment, logging in as well as UX (user experience) features such as multi window support (see below image), android Doze (battery saving capability). In iOS we also see enhancements around sensors like force-touch, apple pay however these features IMO come in short compared to the platform stability over the past 24 months and the platform constrains which i’ll highlight in the next section.

Platform Restrictions

From an ens user perspective, some of the important platform features involves the ability to customize his UX and look and feel of his personal device. Also having the ability to easily manage his media files such as photos and music with a reasonable storage availability. Apple flagship device with massive market share across regions is the iPhone 6/6S with a default storage (un-expandable) of 16GB – I hardly know a person who has this device/storage size that is happy with that, and does not need to constantly delete files, cancel auto savings of WhatsApp media files and alike. In addition, continuously working with iTunes software as a dependency to media/songs sync is a pain and often i found myself losing my favorite music files or getting them duplicated by simply having to switch from 1 PC to another (people do that, and there are procedures that might have prevented this outcome but still). Compared to the above, most Android devices that are not coming with an external storage option are by default coming with a 64 GB internal memory, and in addition working with music file system is a simple and straight forward task to do.

Switching from my iPhone and iTunes to a Nexus device while having my Gmail account was a very simple thing to do, my music, photos and apps easily “followed” me to the Android device that is already running Android 7 in a stable way.

iOS is not all bad, don’t get me wrong – from an adoption perspective, and device/OS fragmentation this is by far a much better managed platform compared to Android that rolls out its latest GA version in a 4-6 months delay to a non-Nexus device (example: Samsung). In addition the iOS tablets are still a leader in that front with 4-6 years old tablets like iPad Air, iPad 2 that are the most commonly used tablets in the market that can still run iOS 9 OS versions. It is not the case when it comes to Android tablets that tend to be replaced by their end-users in a shorter period of time that iPads.

Future Looking

From a future looking perspective, my opinion is that Google is still going to have a global market share advantage over Apple and will continue to innovate with less frequent releases due to quality than Apple. 2017 is going to show us a continuous battle between Android 7 and iOS 10 in a market that becomes more and more digital and mobile dependent, and with this in mind – the challenge of quality, innovation and less restrictions will be even more critical to independent users as well as large enterprises who are already today fully digital.

As an end-user, i would look at both Google and Apple and examine how their overall digital strategy will transform and enable easier connectivity with smart devices like watches etc., as well as less limited storage and device/OS customization. From a Dev and Test perspective i would assume we will continue to see growing adoption of open-source tools such as Espresso, XCTest UI, Appium etc. as a method of keeping up with the OS platform vendors – Only such open-source frameworks can easily and dynamically grow and support new features and functionalities compared to legacy/commercial tools which are slower to introduce new API’s and new capabilities into their solutions.