Mobile project planning

Hi

Up until now i covered in the various posts the aspects of Mobile testing, trends, tools for automation, mobile platforms etc.

All of the above does not help much if the mobile project is not well planned in advanced with SMART milestones, efficient choices of devices (lead/gold devices), right tools and Agile ALM (Application life-cycle management).

In this short post i will advise the most important project milestones and how to define them.

First milestone:

Product definition and feature list –> In this milestone the product management ought to chose the supported platforms and devices for the project (based on thorough market research world wide and based on the projected release date which should be relevant for the selected devices), the project scope (the feature set should be well defined, the supported languages/localizations should be defined, the required certifications (Google CTS, Google Market, Apple App Store certification and more) should be clear, the target customers and of course the projected release date.

Not to forget the UI/UE aspect which is a critical aspect in any mobile project (Not only because the OS requirements but also for the end users).

Second Milestone:

– Core team build up and preliminary work plan implementation, staffing etc.

– Purchasing/ordering of equipment as needed, tools for development and automation should be brought up for approval

– High level Test Plan and Product specification should be developed during that milestone and approve by the core team

Third Milestone:

– Product development

– Unit test development

– Test development on the lead/gold devices per platform (Manual and Automated)

– Documentation creation (As needed and as defined by product)

– Sanity testing on supported devices, emulators, and cloud based handsets.

fourth Milestone:

– Feature complete

– Test development complete

– Full cycle execution on lead devices

– Porting kick off –> Building of subset devices (Either from the lead devices family or new families) versions for testing (Only sanity testing is required for devices within the lead devices family, while full cycles on new family members)

** Family can be defined by a group of devices from the same OEM with the same OS and Screen resolution (e.g. Samsung Galaxy S and Samsung galaxy S Plus are related to the same family –> http://www.gsmarena.com/samsung_i9001_galaxy_s_plus-3908.php ; http://www.gsmarena.com/samsung_i9000_galaxy_s-3115.php).

Fifth Milestone:

– In this stage the project should be well tested, debugged, ported to at least several devices and families, as well as documentation should be ready, certification status should be GREEN, and the bug trends for the project should also be minimal, quality criteria is mostly met.

– Project release to the various markets for approval is being done

– Based on approval and if needed pre-customer approval the project is released!!

The above is just in a nutshell list of items which i feel are a must in a common project plan for a mobile project, i might have missed few items, or the order can be different in few companies, but the above should bring you to a safe release if followed properly.

Thanks,

Eran Kinsbruner

Free iOS automation testing tool – Test Studio for iOS

Hi

I will not write too much in this post, just wanted to provide a link for a free mobile automation tool for iOS web/GUI applications which is available in the App Store

http://www.telerik.com/automated-testing-tools/ios-testing.aspx?utm_source=twitter&utm_medium=sm&utm_content=FRDTNCKNWJ&utm_campaign=ad

The tool provides a simple record/replay mechanism on the iOS native platform and fits basic and not too complex web and native applications.

Soon it should allow sync of the test results and more in the cloud.

Test Studio for iOS does not use image based element detection. It relies on object based recording instead.

I recommend to give it a try

Regards,

Eran Kinsbruner

Integrating a new Android AVD (Emulator) into Android SDK

Hi

In this post i would simply show the quick and simple steps on how to add a new OEM Emulator into your Android SDK and run a sample Android application (APK) on it or on a specific Emulator.

The assumption for this post is that you’ve already downloaded and installed on your Windows/Linux machine the latest Android SDK.
If you have not yet done so, you may download the SDK from: http://developer.android.com/sdk/index.html

** Please note that the SDK requires a JDK (Java Development Kit) to be installed on your PC.

Once you have the SDK installed, you will have a basic skin/AVD (Android Virtual Device) to work with, which comes with the SDK.

As application developer/tester you know that 1 emulator will not give you enough confidence about your application functionality and behavior (without getting into the debate of the use of real device – which i tend to recommend to start ASAP and in the worse case, no later than Beta milestone), so you need to get as much coverage as possible using as many emulators as possible.

Most OEM’s develop and provide for free many emulators which you can simply add to your sdk (We have Samsung, Motorola, LG and HTC as well as Sony Ericsson who provide Tablets and Smartphones emulators which you can download and deploy in the SDK).

From your SDK installation path, simply paste the ZIP with the AVD files into the folder:

{SDK_HOME}/platforms/android-xx/skins –> When android-xx stands for the version of Android on which you wish you run your application on (e.g. – Android-10 stands for Android 2.3.3 Ginger Bread, while android-15 stands for Android ICS 4.0.3).

See example of a folder into which i created few simulators (or AVD’s):

The problems which people often ran into or ask about are, what is the method to install and run an application through the ADB command line on a specific emulator while i have 2 or more running?

The answer is simple:

From the command line assuming you have few AVD’s running, run the command “adb devices”

You will get an output in the below form:

“C:\android-sdk_r18-windows\android-sdk-windows\platform-tools>adb devices
List of devices attached
emulator-5560   device
emulator-5564   device”

If you than want to install an APK on 1 of the above emulators (e.g. emulator-5560), simply run this command:

“adb -s emulator-5560 install “demo.apk”

 

That’s it

I know that it is a quick and short post, if you have any problems or questions about the above – feel free to ask

Regards,

Eran Kinsbruner

JAMO Solutions – Mobile test automation tool

Hi,

I am sure that most mobile testers heard about JAMO solutions, but if not than i will briefly introduce this solution.

In past blogs i mentioned few powerful automation tools such as PerfectoMobile, Keynote Device anywhere, SeeTest by ExperiTest and FoneMonkey (MonkeyTalk).

Except of the Monkey Talk – all others are commercial tools and not free.

Jamo is also a commercial and not free solutions which comes in various licensing models from a company based in Belgium  (http://www.jamosolutions.com/editions.html).

The solutions is based on a plug in to various of IDE’s such as QTP (HP), Eclipse, and Visual Studio, which than allows connectivity over USB or WiFi through a Device manager to the mobile platform (iOS, Android, BlackBerry and Windows Mobile) on which an agent is also installed to provide the full control and automation via record/replay, easy to input text utils etc.

For Android OS device automation using eclipse, you can see a short demo of the tool and its methodology:

In the same manner as the video above users can automate their scenario’s through QTP or Visual studio for the various OS platforms.

For more information or inquiries please contact Jamo through their web site:

http://www.jamosolutions.com/home.html

Regards,

Eran

Mobile testing using Benchmarks

Hi,

We know that many devices vary between each other in their HW capabilities, meaning – the fact that 2 devices by 2 different or similar OEM’s, running the same OS/different OS version will score differently in a benchmark run or a performance test.

This is usually due to hardware differences, defects related to memory allocation, and others.

The use of mobile benchmarks had become quite significant and useful and can allow the application developers and testers to do objective testing on many different devices using a benchmark tool and get unique scores.

By comparing the overall score among the devices under test they can reach a conclusion which device has the higher performance and decide to either use it as “Gold” device for its porting platform, or if the score is extremely low, use it for testing when that device serves as the low performer in the list allowing the team to gain some prerequisite information on how their application would run on the device/s.

The benchmarks are usually downloaded for free from the Android or App store markets, and allows at the end of the quick test run to compare the device score with other devices which ran in the past and see the overall position of the device under test in compare to the market.

Usually the tests involve mathematical calculations (such as finding the prime number from a list), string sorting or number sorting, 2D/3D gaming with FPS calculations, reading and writing to a memory of the device (SD Card etc.), and some other CPU tests.

I strongly recommend to use such tools as part of the testing and as part of the decision for porting a new/existing device.

Information on commonly used benchmarks is available in the web, however i have good experience with the following ones:

– For Android – PassMark and Antutu (Which also provides Battery saver tool and CPU master to control the CPU speed)

http://www.androidbenchmark.net/passmark_chart.html

http://www.antutulabs.com/AnTuTu-Benchmark

 

– For iOS – PassMark and iBench

http://itunes.apple.com/us/app/ibench/id385703033?mt=8

Good Luck

Regards,

Eran Kinsbruner

Latest Mobile world news (July 2012)

Hi,

I’ve collected few topics which are “hot” these days in the mobile world and which i find interesting and relevant to the global audience.

  • Apple states that NFC support  (Near Field communication) might be included in the next iPhone 5 device

http://www.businessinsider.com/iphone-5-may-include-nfc-chip-for-mobile-payments-2012-6

  • RIM (research in motion) is considering selling its business to Facebook or Amazon (Treat this as a rumor rather than a fact)

http://www.businessinsider.com/rim-considers-selling-its-handset-business-2012-6

  • Time spent on mobile apps vs. web browsing is growing year by year – see below latest trends (Minutes per day):

  • Microsoft announces on the new Windows 8 OS with Internet Explorer 10 release with new features, enhancements and more, as well as new Metro customizable UI

http://windows.microsoft.com/en-US/windows-8/release-preview

  • Check out the list of 20 best mobile phones in the world according to techradar site:

http://www.techradar.com/news/phone-and-communications/mobile-phones/20-best-mobile-phones-in-the-world-today-645440?artc_pg=22

Enjoy 🙂

Regards,

Eran Kinsbruner

Porting mobile apps – few guidelines and insights

Hi

When we talk about mobile world complexities, it is important to understand the meaning of Porting.

We know that the mobile platform is dynamic, often changes and being influenced by many aspects (Calls, SMS’s and other interrupts, but also from diversity of mobile OS’s, new devices entering the market and more)

Porting of mobile apps, means – To take an existing working mobile application (Game, utility, web service or other) which was proved to work fine on few “lead”/”gold” devices which were (hopefully) well-defined by the project manager, and make a port of the application to many other handsets, tablets and so on.

You can imagine, that any generic core bug in one of the “lead” devices will obviously drill down to the ported handsets, so at first it is important to pick the right “lead” device which is new enough in the market, solid, and represent a “family” of existing devices (e.g. Samsung Galaxy SII for the Android OS).

Second thing to keep in mind, is that the time to develop and release a product (Mobile related) is ~3-6 months so the device needs to still be “relevant” in the market and be considered a “lead” in the market at the time of release.

Once we are certain that we picked the right lead devices and we have a Beta version with all features implemented and functioning, we than enter the porting phase in which we start to introduce new devices (hopefully from the same families of the lead devices which we ported) and install the application on them to perform sanity.

If the application was developed and designed  “for portability” – we should be able to produce various version per new handsets which are built from the same core base with the distinguish of icons, resources and minor differences – in that case fixing bugs should also be easy.

If the design was not right than we can enter a “nightmare” of fixing bugs backwards in the core source, and maintaining many old and new devices for a long time.

The right model is to have a solid and generic core source base from which the release manager generates using build targets the right handsets by using property files which matches the right device “family” and profile.

Porting bugs can vary between handsets and can be caused by many reasons which are not always related to our application (e.g – an application which uses GPS or Camera and these functions are buggy on the device may impact the application stability – this is something which porting should handle, issues related to screen resolution, rotating from landscape to portrait and more).

In our days when we have the Android OS, iOS and Windows Phone and we wish to develop similar application to these 3 or more platforms, porting becomes even more complex – For that we can consider the above insights, and also integrate tools such as Code Name One, PhoneGap or other tools which allows to use one code base and deploy it to various mobile platforms and languages.

In future blogs i will try and refer to the method of developing a master test plan for the lead devices and a subset/sanity test plan for the ported devices which are derived from the lead ones.

For questions and more details, you may contact me as usual: eran.kinsbrunner@gmail.com

Regards,

Eran Kinsbruner

What should we expect in the upcoming Android 4.1 – JellyBeans

Hi,

With quite low % of penetration (lower than expected) of the ICS to the Android world, we start to hear more and more about the upcoming release of Android which should be called Jelly Beans (Android 4.1)

It is not yet official, but rumors state that the following features (At least part of them) will find their way into this new release.

One juicy but perhaps unlikely rumour is that Jelly Bean will run Windows 8 — in a dual-boot scenario that would allow Droid lovers to switch between Android and Windows – I really doubt this rumor!

http://crave.cnet.co.uk/mobiles/android-5-0-jelly-bean-could-run-windows-8-50006981/

Another mooted update to Android involves adding more gestures to the interface. Last year Google filed a patent that used a series of letter gestures to act as short-cuts — drawing the letter W, for instance, could be used as a quick way to fire up Wikipedia.

A more widespread prediction for Jelly Bean is that Google’s Chrome for Android browser will be integrated into the 4.1 OS– available now to ICS users — becomes the standard offering in the OS, rather than an additional download via Google Play.

Integrated built in File Manager app into the Android 4.1 is also expected.

Android 4.1 is also expected to introduce an improved KeyBoard with more ease of use features (e.g. – long key press on a key to switch to alternate characters and numbers.

Android 4.1 states to also be a “Better Battery, Power management and Faster OS” – Will wait and see.

The important thing which Google will try to improve is the timely releases of its OS updates, it has been quite a while since a release was made, in opposed to the 6 month release track which Google was hoping to offer.

Another thing is that Google will seek to compete with Apple’s Siri voice assistant with its own mouth interface addition – the new ‘Google Now’ should try and serve as a voice assistant starting Jelly Beans. And many Android users are also anticipating some kind of file manager application, to help manage what are becoming increasingly complex media machines, along with (hoped for) improvements on the battery and power management front, and a speedier way to get the latest version of Android.

P.S: The Next Next release of Android post 4.1 should be called Key Lime Pie (KLP) 🙂

Thanks and Regards,

Eran Kinsbruner

Complexities in automating tests on mobile platforms

Hi,

In previous blogs i shared some of the difficulties in running a mobile project, and few leading cloud based solutions which can be useful in dealing with few of the difficulties i raised.

In this blog i would like to highlight few of the problems which an automation team which develops tests for mobile should be aware of in advance.

From one hand, we want to isolate when developing automation our test environment and make it free of interrupts and noises, this in order to have a consistent and flawless test run without the need to complicate the test scripts etc., however – from the other hand when developing automation (especially for mobile) – we want to actually introduce as many interrupts as possible during the test run since these are most probably some of the issues which a real user will face when running the application we tend to test etc.

So – what is the proposed solution?

IMHO, the way to develop robust automation for mobile requires to have a controlled test environment with the right number of mobile handsets per platform, while adding to the “formula” automation which will trigger possible interferences (such as incoming call, incoming SMS/Mail, low battery events, lost of communication, screen orientation change etc.).

We have the ability these days to use e.g. Android Emulator in order to simulate an incoming call/SMS, or Roaming, or change of Network as well as change of location (by using the Android DDMs tool with the Location configuration tool), which we can use in our test automation environment. We also have tools which act as a private cloud within a closed network (e.g. Seetest by Experitest) which can allow us to test on multiple devices at the same time using parallel scenario’s etc.

The bottom line is: We know that automating tests on mobile which is a hard thing on its own (need to handle detection of unique objects, dynamic events and many more), however the tests we develop, should be relevant to the mobile world and combine mobile world interrupts such as i mentioned above, otherwise we can assure that the application runs OK on the device, but when interrupts starts to show up – we do not cover this at all since we isolated the test platforms from the “world”.

Another complexity when doing automation for mobile (on top of the variety of platforms to support – iOS, Android etc.), is the “portability” of the developed tests across mobile handsets. We know already that solutions such as “PerfectoMobile” uses ScriptOnce keyword based development language to overcome this, and allow a develop once, run everywhere solution, and we also see other tools today such as PhoneGap, CodeNameOne, MagicSoftware  uniPaaS tool which aim to cover development and automation in a cross-platform environment.

I will extend my blogs in the future around each of the above cross-platform solutions and explain how they work and what are their advantages.

I wrote a while ago a presentation about mobile test automation which is still relevant and can be complimentary to this blog – see Link: http://www.slideshare.net/ek121268/mobile-automated-testingerank

Regards,

Eran Kinsbruner

Mobile testing in the cloud

Hi,

In this short blog, I will try to give a small taste of the mobile testing cloud leading solutions these days (I am sure that there are many more, but these are the one i ran across and can point you to).

When talking about mobile testing, we need to understand that there are plenty of complexities which mobile testing manager need to deal with.

Few examples:

– Plenty of mobile handsets (Each few weeks/months new phones show up) – support of existing applications/solutions/services on new handsets is a challenge (e.g.: A working and deployed app which runs ok on Samsung Galaxy SII – need to be tested on the newly launched Samsung Galaxy S III)

– Testing different locales on different operators, gaining high network’s coverage

– porting activities testing (during mature development life cycle)

– expensive logistics in acquiring many devices which will serve R&D teams

The above are just few examples of difficulties around mobile testing which the cloud based solutions provide a solution for.

Testing in the cloud saves money for the companies and prevent them for acquiring expensive handsets for rather short testing cycles. The cloud provides coverage in various operators across the world, and quick TTM.

Few of the leading tools which are out there are listed below:

 PerfectoMobile – “Mobile Cloud Interactive” –> A cloud based solution which provides access to hundreds of real handsets which are spread across the world with real SIM inside connected to real operators (Vodafone, Verizon etc.)

The solution is based on a device rack (DDM) which can hold up to 4 different/similar devices which are connected to a server, and against each of their screen there is a video camera pointed and provide liver streaming (through web browser) of whatever is being performed/displayed on the handset. The flow of each handset is captured and saved as a video file (in FLV format) allowing the testers/developers an easy way to reproduce and report bugs etc.

On top of the above cloud service which has a pricing model (pay by the hour of use) the company also provide an automation robust solution (which i will report in future blog/s) which plugs in to commonly used automation solutions (such as HP QTP). For more info on the Mobile Cloud solution – http://www.perfectomobile.com/portal/cms/services/web_access_to_real_handsets.html

KeyNote Device anywhere – “Test Center Enterprise Interactive” – In a similar model as stated above in PerfectoMobile, we have the older solution in the market which provides as well device remote access in a cloud based solution across the world (different pricing model). This solution requires user registration and a windows based utility download from which one can access the mobile cloud, run tests on “ordered” handsets, and get detailed reports.

The above as mentioned earlier, provides increased coverage on more devices, networks and locales and increases the level of confidence in the quality of the vendors deliverables.

http://www.keynotedeviceanywhere.com/mobile-application-enterprise-testing.html

– SeeTest by Experitest – Company which provides unique test automation solution which plugs into various IDE’s (Jsystem, QTP and more), and also in a unique model provides a local mobile cloud in the company home network. A company who owns its devices and wants to run existing and/or new test plans on them in one single run from any PC in the company – he can get remote connectivity to each of the handsets connected via USB (or USB Hub) to any of the machines in the network and launch the test scripts (Great solution when trying to build continuous integration solution inside a company R&D)

More details: http://experitest.com/cloud/

– Last – We have SOASTA CloudTest solution – This is a load/performance testing solution within a cloud based product.

The company holds access to thousands of PC”s which can be configured to access simultaneously to any web url/server and introduce load. See a recent press release which introduces the cooperation of SOASTA and HP around load testing:

“Cloud-based test service delivers automated performance testing across 47 locations with access to over 350,000 servers and 17 different APIs” ….. http://www.soasta.com/company/news/press-releases/soasta-brings-cloudtest-to-hp-cloud-services/

 The mobile test automation which is also provided by the company allows a tester a unthethered agent to be installed on his target device from which he can test against the SOASTA IDE his application (through WiFi/3G connection).

For more information on these solutions: http://www.soasta.com/?gclid=CL72odjo_a8CFQpd3wodNFfMUw

Hope you find these high level information useful.

For further details contact me at Eran.Kinsbrunner@gmail.com

Regards,

Eran Kinsbruner