Mobile project planning

Hi

Up until now i covered in the various posts the aspects of Mobile testing, trends, tools for automation, mobile platforms etc.

All of the above does not help much if the mobile project is not well planned in advanced with SMART milestones, efficient choices of devices (lead/gold devices), right tools and Agile ALM (Application life-cycle management).

In this short post i will advise the most important project milestones and how to define them.

First milestone:

Product definition and feature list –> In this milestone the product management ought to chose the supported platforms and devices for the project (based on thorough market research world wide and based on the projected release date which should be relevant for the selected devices), the project scope (the feature set should be well defined, the supported languages/localizations should be defined, the required certifications (Google CTS, Google Market, Apple App Store certification and more) should be clear, the target customers and of course the projected release date.

Not to forget the UI/UE aspect which is a critical aspect in any mobile project (Not only because the OS requirements but also for the end users).

Second Milestone:

– Core team build up and preliminary work plan implementation, staffing etc.

– Purchasing/ordering of equipment as needed, tools for development and automation should be brought up for approval

– High level Test Plan and Product specification should be developed during that milestone and approve by the core team

Third Milestone:

– Product development

– Unit test development

– Test development on the lead/gold devices per platform (Manual and Automated)

– Documentation creation (As needed and as defined by product)

– Sanity testing on supported devices, emulators, and cloud based handsets.

fourth Milestone:

– Feature complete

– Test development complete

– Full cycle execution on lead devices

– Porting kick off –> Building of subset devices (Either from the lead devices family or new families) versions for testing (Only sanity testing is required for devices within the lead devices family, while full cycles on new family members)

** Family can be defined by a group of devices from the same OEM with the same OS and Screen resolution (e.g. Samsung Galaxy S and Samsung galaxy S Plus are related to the same family –> http://www.gsmarena.com/samsung_i9001_galaxy_s_plus-3908.php ; http://www.gsmarena.com/samsung_i9000_galaxy_s-3115.php).

Fifth Milestone:

– In this stage the project should be well tested, debugged, ported to at least several devices and families, as well as documentation should be ready, certification status should be GREEN, and the bug trends for the project should also be minimal, quality criteria is mostly met.

– Project release to the various markets for approval is being done

– Based on approval and if needed pre-customer approval the project is released!!

The above is just in a nutshell list of items which i feel are a must in a common project plan for a mobile project, i might have missed few items, or the order can be different in few companies, but the above should bring you to a safe release if followed properly.

Thanks,

Eran Kinsbruner

Free iOS automation testing tool – Test Studio for iOS

Hi

I will not write too much in this post, just wanted to provide a link for a free mobile automation tool for iOS web/GUI applications which is available in the App Store

http://www.telerik.com/automated-testing-tools/ios-testing.aspx?utm_source=twitter&utm_medium=sm&utm_content=FRDTNCKNWJ&utm_campaign=ad

The tool provides a simple record/replay mechanism on the iOS native platform and fits basic and not too complex web and native applications.

Soon it should allow sync of the test results and more in the cloud.

Test Studio for iOS does not use image based element detection. It relies on object based recording instead.

I recommend to give it a try

Regards,

Eran Kinsbruner

Integrating a new Android AVD (Emulator) into Android SDK

Hi

In this post i would simply show the quick and simple steps on how to add a new OEM Emulator into your Android SDK and run a sample Android application (APK) on it or on a specific Emulator.

The assumption for this post is that you’ve already downloaded and installed on your Windows/Linux machine the latest Android SDK.
If you have not yet done so, you may download the SDK from: http://developer.android.com/sdk/index.html

** Please note that the SDK requires a JDK (Java Development Kit) to be installed on your PC.

Once you have the SDK installed, you will have a basic skin/AVD (Android Virtual Device) to work with, which comes with the SDK.

As application developer/tester you know that 1 emulator will not give you enough confidence about your application functionality and behavior (without getting into the debate of the use of real device – which i tend to recommend to start ASAP and in the worse case, no later than Beta milestone), so you need to get as much coverage as possible using as many emulators as possible.

Most OEM’s develop and provide for free many emulators which you can simply add to your sdk (We have Samsung, Motorola, LG and HTC as well as Sony Ericsson who provide Tablets and Smartphones emulators which you can download and deploy in the SDK).

From your SDK installation path, simply paste the ZIP with the AVD files into the folder:

{SDK_HOME}/platforms/android-xx/skins –> When android-xx stands for the version of Android on which you wish you run your application on (e.g. – Android-10 stands for Android 2.3.3 Ginger Bread, while android-15 stands for Android ICS 4.0.3).

See example of a folder into which i created few simulators (or AVD’s):

The problems which people often ran into or ask about are, what is the method to install and run an application through the ADB command line on a specific emulator while i have 2 or more running?

The answer is simple:

From the command line assuming you have few AVD’s running, run the command “adb devices”

You will get an output in the below form:

“C:\android-sdk_r18-windows\android-sdk-windows\platform-tools>adb devices
List of devices attached
emulator-5560   device
emulator-5564   device”

If you than want to install an APK on 1 of the above emulators (e.g. emulator-5560), simply run this command:

“adb -s emulator-5560 install “demo.apk”

 

That’s it

I know that it is a quick and short post, if you have any problems or questions about the above – feel free to ask

Regards,

Eran Kinsbruner

JAMO Solutions – Mobile test automation tool

Hi,

I am sure that most mobile testers heard about JAMO solutions, but if not than i will briefly introduce this solution.

In past blogs i mentioned few powerful automation tools such as PerfectoMobile, Keynote Device anywhere, SeeTest by ExperiTest and FoneMonkey (MonkeyTalk).

Except of the Monkey Talk – all others are commercial tools and not free.

Jamo is also a commercial and not free solutions which comes in various licensing models from a company based in Belgium  (http://www.jamosolutions.com/editions.html).

The solutions is based on a plug in to various of IDE’s such as QTP (HP), Eclipse, and Visual Studio, which than allows connectivity over USB or WiFi through a Device manager to the mobile platform (iOS, Android, BlackBerry and Windows Mobile) on which an agent is also installed to provide the full control and automation via record/replay, easy to input text utils etc.

For Android OS device automation using eclipse, you can see a short demo of the tool and its methodology:

In the same manner as the video above users can automate their scenario’s through QTP or Visual studio for the various OS platforms.

For more information or inquiries please contact Jamo through their web site:

http://www.jamosolutions.com/home.html

Regards,

Eran

Mobile testing using Benchmarks

Hi,

We know that many devices vary between each other in their HW capabilities, meaning – the fact that 2 devices by 2 different or similar OEM’s, running the same OS/different OS version will score differently in a benchmark run or a performance test.

This is usually due to hardware differences, defects related to memory allocation, and others.

The use of mobile benchmarks had become quite significant and useful and can allow the application developers and testers to do objective testing on many different devices using a benchmark tool and get unique scores.

By comparing the overall score among the devices under test they can reach a conclusion which device has the higher performance and decide to either use it as “Gold” device for its porting platform, or if the score is extremely low, use it for testing when that device serves as the low performer in the list allowing the team to gain some prerequisite information on how their application would run on the device/s.

The benchmarks are usually downloaded for free from the Android or App store markets, and allows at the end of the quick test run to compare the device score with other devices which ran in the past and see the overall position of the device under test in compare to the market.

Usually the tests involve mathematical calculations (such as finding the prime number from a list), string sorting or number sorting, 2D/3D gaming with FPS calculations, reading and writing to a memory of the device (SD Card etc.), and some other CPU tests.

I strongly recommend to use such tools as part of the testing and as part of the decision for porting a new/existing device.

Information on commonly used benchmarks is available in the web, however i have good experience with the following ones:

– For Android – PassMark and Antutu (Which also provides Battery saver tool and CPU master to control the CPU speed)

http://www.androidbenchmark.net/passmark_chart.html

http://www.antutulabs.com/AnTuTu-Benchmark

 

– For iOS – PassMark and iBench

http://itunes.apple.com/us/app/ibench/id385703033?mt=8

Good Luck

Regards,

Eran Kinsbruner

Mobile Device Management and Security (MDM)

Hi,

We already covered the high penetration of mobile devices into the organizations, and work places.

We also mentioned the trend of B.Y.O.D (Bring your own device) which may introduce risks to the companies, since when the personal mobile is plugged into the work PC and becomes a storage drive, it can bring into the network many kinds of vulnerabilities, viruses and so on.

For such threats and in order to allow big/small organizations to manage their mobile assets, data, security, install from remote new ROM images, block stolen devices, restore data to devices and more, products which are called MDM’s were developed and are now commonly used by IT managers.

In this post i will briefly state few players in this field, to give a sample of where this field is heading to and to demonstrate the importance of MDM now and most important in the future, in which we will see more and more smartphones/tablets being used inside any organization.

In the market today we have products such as: Maas360 (by Fiberlink), AirWatch (Mobisec),  Nutech Systems, and more.

Most tools gives the below advantages to the IT manager:

  • Increase scale of mobile deployments
  • Enable BYOD and shared device programs
  • Centralize management of mobile platforms
  • Gain real-time visibility into mobile environment
  • Administer consistent policies across devices
  • Enforce enterprise security and compliance
  • Secure data transmitted to and from devices
  • Complete enterprise Data Loss Prevention (DLP)
  • Analyze and report critical device information

Maas360 (http://www.maas360.com/) provides MDM solution for any enterprize organization for cross-platform mobile handsets and Tablets  (Android, iOS, BlackBerry, and others). allows the administrator to manage its devices, add/remove/modify/fully wipe device and more functions, as well get specific information on each handsets in his pull (storage left on the device, OS version and more).

See sample video to learn more on the solution:

– AirWatch(http://www.air-watch.com/solutions/mobile-device-management?gclid=CKymv7zw3LACFUUhtAodESoPzQ) – Also provides MDM solution with many similar options such as the Maas360, and also provides monitoring capabilities, and dashboards of the assets, alerts when needed and more.

Has support to most commonly used mobile platforms, and has remote views, remote control to handsets and other options and configurations.

– MobiSec (http://www.mobisec.co.il/index.php/english) – Israeli company which deploys the above AirWatch MDM solution and integrates it into the organizations.

Regards,

Eran Kinsbruner

Complexities in automating tests on mobile platforms

Hi,

In previous blogs i shared some of the difficulties in running a mobile project, and few leading cloud based solutions which can be useful in dealing with few of the difficulties i raised.

In this blog i would like to highlight few of the problems which an automation team which develops tests for mobile should be aware of in advance.

From one hand, we want to isolate when developing automation our test environment and make it free of interrupts and noises, this in order to have a consistent and flawless test run without the need to complicate the test scripts etc., however – from the other hand when developing automation (especially for mobile) – we want to actually introduce as many interrupts as possible during the test run since these are most probably some of the issues which a real user will face when running the application we tend to test etc.

So – what is the proposed solution?

IMHO, the way to develop robust automation for mobile requires to have a controlled test environment with the right number of mobile handsets per platform, while adding to the “formula” automation which will trigger possible interferences (such as incoming call, incoming SMS/Mail, low battery events, lost of communication, screen orientation change etc.).

We have the ability these days to use e.g. Android Emulator in order to simulate an incoming call/SMS, or Roaming, or change of Network as well as change of location (by using the Android DDMs tool with the Location configuration tool), which we can use in our test automation environment. We also have tools which act as a private cloud within a closed network (e.g. Seetest by Experitest) which can allow us to test on multiple devices at the same time using parallel scenario’s etc.

The bottom line is: We know that automating tests on mobile which is a hard thing on its own (need to handle detection of unique objects, dynamic events and many more), however the tests we develop, should be relevant to the mobile world and combine mobile world interrupts such as i mentioned above, otherwise we can assure that the application runs OK on the device, but when interrupts starts to show up – we do not cover this at all since we isolated the test platforms from the “world”.

Another complexity when doing automation for mobile (on top of the variety of platforms to support – iOS, Android etc.), is the “portability” of the developed tests across mobile handsets. We know already that solutions such as “PerfectoMobile” uses ScriptOnce keyword based development language to overcome this, and allow a develop once, run everywhere solution, and we also see other tools today such as PhoneGap, CodeNameOne, MagicSoftware  uniPaaS tool which aim to cover development and automation in a cross-platform environment.

I will extend my blogs in the future around each of the above cross-platform solutions and explain how they work and what are their advantages.

I wrote a while ago a presentation about mobile test automation which is still relevant and can be complimentary to this blog – see Link: http://www.slideshare.net/ek121268/mobile-automated-testingerank

Regards,

Eran Kinsbruner

Mobile testing in the cloud

Hi,

In this short blog, I will try to give a small taste of the mobile testing cloud leading solutions these days (I am sure that there are many more, but these are the one i ran across and can point you to).

When talking about mobile testing, we need to understand that there are plenty of complexities which mobile testing manager need to deal with.

Few examples:

– Plenty of mobile handsets (Each few weeks/months new phones show up) – support of existing applications/solutions/services on new handsets is a challenge (e.g.: A working and deployed app which runs ok on Samsung Galaxy SII – need to be tested on the newly launched Samsung Galaxy S III)

– Testing different locales on different operators, gaining high network’s coverage

– porting activities testing (during mature development life cycle)

– expensive logistics in acquiring many devices which will serve R&D teams

The above are just few examples of difficulties around mobile testing which the cloud based solutions provide a solution for.

Testing in the cloud saves money for the companies and prevent them for acquiring expensive handsets for rather short testing cycles. The cloud provides coverage in various operators across the world, and quick TTM.

Few of the leading tools which are out there are listed below:

 PerfectoMobile – “Mobile Cloud Interactive” –> A cloud based solution which provides access to hundreds of real handsets which are spread across the world with real SIM inside connected to real operators (Vodafone, Verizon etc.)

The solution is based on a device rack (DDM) which can hold up to 4 different/similar devices which are connected to a server, and against each of their screen there is a video camera pointed and provide liver streaming (through web browser) of whatever is being performed/displayed on the handset. The flow of each handset is captured and saved as a video file (in FLV format) allowing the testers/developers an easy way to reproduce and report bugs etc.

On top of the above cloud service which has a pricing model (pay by the hour of use) the company also provide an automation robust solution (which i will report in future blog/s) which plugs in to commonly used automation solutions (such as HP QTP). For more info on the Mobile Cloud solution – http://www.perfectomobile.com/portal/cms/services/web_access_to_real_handsets.html

KeyNote Device anywhere – “Test Center Enterprise Interactive” – In a similar model as stated above in PerfectoMobile, we have the older solution in the market which provides as well device remote access in a cloud based solution across the world (different pricing model). This solution requires user registration and a windows based utility download from which one can access the mobile cloud, run tests on “ordered” handsets, and get detailed reports.

The above as mentioned earlier, provides increased coverage on more devices, networks and locales and increases the level of confidence in the quality of the vendors deliverables.

http://www.keynotedeviceanywhere.com/mobile-application-enterprise-testing.html

– SeeTest by Experitest – Company which provides unique test automation solution which plugs into various IDE’s (Jsystem, QTP and more), and also in a unique model provides a local mobile cloud in the company home network. A company who owns its devices and wants to run existing and/or new test plans on them in one single run from any PC in the company – he can get remote connectivity to each of the handsets connected via USB (or USB Hub) to any of the machines in the network and launch the test scripts (Great solution when trying to build continuous integration solution inside a company R&D)

More details: http://experitest.com/cloud/

– Last – We have SOASTA CloudTest solution – This is a load/performance testing solution within a cloud based product.

The company holds access to thousands of PC”s which can be configured to access simultaneously to any web url/server and introduce load. See a recent press release which introduces the cooperation of SOASTA and HP around load testing:

“Cloud-based test service delivers automated performance testing across 47 locations with access to over 350,000 servers and 17 different APIs” ….. http://www.soasta.com/company/news/press-releases/soasta-brings-cloudtest-to-hp-cloud-services/

 The mobile test automation which is also provided by the company allows a tester a unthethered agent to be installed on his target device from which he can test against the SOASTA IDE his application (through WiFi/3G connection).

For more information on these solutions: http://www.soasta.com/?gclid=CL72odjo_a8CFQpd3wodNFfMUw

Hope you find these high level information useful.

For further details contact me at Eran.Kinsbrunner@gmail.com

Regards,

Eran Kinsbruner