iOS 10 Is About to Disrupt Mobile Testing Plans

For background: Apple has decided with iOS 10 to make certain iPads, iPods and iPhones obsolete by stopping the OS they can run at iOS 9.x.

The list of devices that will not be able to upgrade to iOS 10 and above are:

iPad 2
iPad 3rd gen
iPad mini
iPhone 4S
iPod touch 5th gen

Why is this a problem for developers and testers?

Let’s examine iOS 9 adoption. Based on the data below from Mixpanel, two weeks after iOS 9 was launched on September 16, 2015, 40% of users already upgraded to iOS 9. Today, iOS 9 adoption is close to 90%.

Now, here’s a quick look at market share numbers for iOS devices.

Based on this data from Localytics and the recent Perfecto Digital Test Coverage Index report it’s clear that at least iPad 2, iPhone 4S and iPad mini are among the most-used devices in various markets, including the U.S. (see below)

Implications for mobile testing plans

The information above means one thing. Dev and test teams will need to support the new iPhone 7/7 Plus along with other Apple devices that can run iOS 10. Additionally, they need to reserve a portion of testing for iOS 9 for the older devices mentioned above. Unfortunately, this will create latency in testing activities. It will also require test automation so tests can be easily executed on devices running iOS 9 and iOS 10.

Another side effect may be lower device adoption for iOS 10 because large groups of users will simply be stuck on iPad 2, iPad Mini and iPhone 4S. We also see a clear market trend of older iPads continuing to be the most popular, as iPad users take longer to upgrade to new tablets than Android tablet users.

The end result is we’re going to see growing iOS fragmentation in various markets and complex testing ahead.

Open source test framework implications

iOS 10 is not just disrupting the mobile device and OS landscape, it’s also impacting open-source test frameworks such as Appium.

As you can read in the threads below, iOS 10 is “breaking” Appium test framework functionality, impacting installations and the launching of IPA files. It is also causing issues working with XCUITest and the iOS WebKit.

It’s worth noting that vendors such as Perfecto are able to overcome this challenge as they continue to support Appium test automation with iOS 10 on iOS devices for all beta versions and the GA version from the first day it was publicly available.

2 Replies to “iOS 10 Is About to Disrupt Mobile Testing Plans”