Category: Agile

Posted on

Expediting Web Testing with Puppeteer and Instant Test Automation Technology

The digital landscape has significantly advanced over the past 1-2 years. The new normal in web applications include a shift toward progressive web apps (PWAs), advanced responsive websites, and Flutter apps.

Ensuring both functional as well as API testing of such advance web applications is becoming harder and harder. While there are valid test automation frameworks like Selenium and Cypress, these are covering only a subset of what is needed to complete the entire quality aspects across multiple environments.

New Normal: Instant Test Automation

To address the testing challenge over time, across multiple environments, and with big and dynamic data, tools need to be more intelligent, and utilize AI/ML to really track trends, predict issues prior to them becoming more serious, and guide the developers and testers in real-time.

An instant test automation technology that can leverage a standard test automation framework such as Puppeteer to auto generate both functional as well as API tests on the fly and as things change is a game changer for app developers. This provides alerts and insights in real time that the team can act upon and move much faster.

The way sch solution works is through modeling the website under test, auto generate based on traffic and AI proper tests and runs them to provide the proper insights.

Setting up such solution is easy and needs to be done only once. Also, such tool fits systems that are built atop cloud-native, containers and Amazon ECS.

Benefits of Shifting into Instant and Intelligent Test Automation

With the need to shift left, release faster with confidence, and handle large and complex environment, teams can benefit from a self-generating testing solution that really covers more grounds that a typical code-based framework would. This does not mean that there are no benefits or need for such frameworks like Selenium and others, however, with tools like UP9, teams can complement the code related functional tests with an additional layer of quality validation that touches reliability, APIs, services, security and much more.

Case Study: Quick Tour of Puppeteer & UP9

I took the Google puppeteer framework and connected it with UP9 to try generating a nice model and test automation assets on my local machine.

After following the basic NPM installation instructions, it was easy to launch a local browser and record traffic using the tool.

From my terminal I would then run this command (‘my.new.workspace’ could be any name that the developer selects):

up9 tap:start my.new.workspace

Once I ran that command, using puppeteer, UP9 launches a chrome browser that I can start using to navigate to my website for recording all traffic, actions, API calls and other services. Upon closing the browser, the tool will process all of the flows and generate the model.

Here is the auto generated workspace that was created based on a quick tour I ran with UP9 on the Perforce and Perfecto websites:

The workspace consists of few views:

  1. The generic workspace with all the high-level services model (diagram),
  2. The contracts that were used in my recording,
  3. A detailed traffic analysis with the entire API calls,
  4. Tests that were created and mocks to be used across different environments.

These views were perceived and constructed based on service-traffic, following from the observers of UP9 that were assigned to my designated workspace.

As part of my journey, I landed on the Perfecto free-trial website, and the tool was able to generate the API calls (Request/Response) and allow me to validate that is was working as expected (see below)

If I wanted to also run my auto-generated tests, I have within the UP9 portal (fully SaaS and web-based) a tests area from where I could launch my tests as well.

Bottom Line

As the DevOps market mature towards more intelligent testing using AI and ML, and as the systems under tests (SUTs) are also progressing, there needs to be a realization within the teams that their processes could be improved dramatically using such technologies. Developers productivity together with consistent app quality were a CIO/IT manager top of mind for a long while, and with the recent advancements in the market, teams can really enjoy a new era of smarter ways of working.

Posted on

Most Recommended Software Testing Books to Read in 2020 and Beyond

As we kick of a new decade of software development and testing, and as digital continuous to challenge test automation engineers that are trying to fit the testing within shorter then ever cycles, here are the top recommended software testing books to consider – the order isn’t the priority, they are all awesome books and equally recommended:

#1 Agile Testing – Lisa Crispin and Janet Gregory

Readers will come away from this book understanding

  • How to get testers engaged in agile development
  • Where testers and QA managers fit on an agile team
  • What to look for when hiring an agile tester
  • How to transition from a traditional cycle to agile development
  • How to complete testing activities in short iterations
  • How to use tests to successfully guide development
  • How to overcome barriers to test automation

#2 More Agile Testing – Lisa Crispin and Janet Gregory

Main learning objectives from the book:

  • How to clarify testing activities within the team
  • Ways to collaborate with business experts to identify valuable features and deliver the right capabilities
  • How to design automated tests for superior reliability and easier maintenance
  • How agile team members can improve and expand their testing skills
  • How to plan “just enough,” balancing small increments with larger feature sets and the entire system
  • How to use testing to identify and mitigate risks associated with your current agile processes and to prevent defects
  • How to address challenges within your product or organizational context
  • How to perform exploratory testing using “personas” and “tours”
  • Exploratory testing approaches that engage the whole team, using test charters with session- and thread-based techniques
  • How to bring new agile testers up to speed quickly–without overwhelming them

#3 Hands on Mobile App Testing – Daniel Knott

Readers will learn how to

  • Establish your optimal mobile test and launch strategy
  • Create tests that reflect your customers, data networks, devices, and business models
  • Choose and implement the best Android and iOS testing tools
  • Automate testing while ensuring comprehensive coverage
  • Master both functional and nonfunctional approaches to testing
  • Address mobile’s rapid release cycles
  • Test on emulators, simulators, and actual devices
  • Test native, hybrid, and Web mobile apps
  • Gain value from crowd and cloud testing (and understand their limitations)
  • Test database access and local storage
  • Drive value from testing throughout your app life-cycle
  • Start testing wearables, connected homes/cars, and Internet of Things devices

#4 Enterprise Continuous Testing – Wolfgang Platz

Enterprise Continuous Testing: Transforming Testing for Agile and DevOps introduces a Continuous Testing strategy that helps enterprises accelerate and prioritize testing to meet the needs of fast-paced Agile and DevOps initiatives. Software testing has traditionally been the enemy of speed and innovation—a slow, costly process that delays releases while delivering questionable business value. This new strategy helps you test smarter, so testing provides rapid insight into what matters most to the business.

#5 Continuous Testing for DevOps Professionals – Eran Kinsbruner and Leading Market Experts

Continuous Testing for DevOps Professionals is the definitive guide for DevOps teams and covers the best practices required to excel at Continuous Testing (CT) at each step of the DevOps pipeline. It was developed in collaboration with top industry experts from across the DevOps domain from leading companies such as CloudBees, Tricentis, Testim.io, Test.ai, Perfecto, and many more.The book is aimed at all DevOps practitioners, including software developers, testers, operations managers, and IT/business executives

#6 Accelerate –The Science of Lean Software and DevOps, Nicole Forsgren

How can we apply technology to drive business value? For years, we’ve been told that the performance of software delivery teams doesn’t matter―that it can’t provide a competitive advantage to our companies. Through four years of groundbreaking research to include data collected from the State of DevOps reports conducted with Puppet, Dr. Nicole Forsgren, Jez Humble, and Gene Kim set out to find a way to measure software delivery performance―and what drives it―using rigorous statistical methods. This book presents both the findings and the science behind that research, making the information accessible for readers to apply in their own organizations.

#7 Complete Guide to Test Automation, Arnon Axelrod

  • Know the real value to be expected from test automation
  • Discover the key traits that will make your test automation project succeed
  • Be aware of the different considerations to take into account when planning automated tests vs. manual tests
  • Determine who should implement the tests and the implications of this decision
  • Architect the test project and fit it to the architecture of the tested application
  • Design and implement highly reliable automated tests
  • Begin gaining value from test automation earlier
  • Integrate test automation into the business processes of the development team
  • Leverage test automation to improve your organization’s performance and quality, even without formal authority
  • Understand how different types of automated tests will fit into your testing strategy, including unit testing, load and performance testing, visual testing, and more

#8 The DevOps Handbook, Gene Kim, Jez Humble, Patrick Debois, John Willis

More than ever, the effective management of technology is critical for business competitiveness. For decades, technology leaders have struggled to balance agility, reliability, and security. The consequences of failure have never been greater―whether it’s the healthcare.gov debacle, cardholder data breaches, or missing the boat with Big Data in the cloud. And yet, high performers using DevOps principles, such as Google, Amazon, Facebook, Etsy, and Netflix, are routinely and reliably deploying code into production hundreds, or even thousands, of times per day. Following in the footsteps of The Phoenix Project, The DevOps Handbook shows leaders how to replicate these incredible outcomes, by showing how to integrate Product Management, Development, QA, IT Operations, and Information Security to elevate your company and win in the marketplace

#9 The Digital Quality Handbook – Eran Kinsbruner and Industry Experts

As mobile and web technologies continue to expand and basically drives large organizational business in virtually every vertical or industry, it is critical to understand how to take existing release practices for mobile and web apps to the next level, including software development life cycle (SDLC), tools, quality, etc. Organizations which are already enjoying the power of digital are still struggling with various challenges that can be related to many factors, such as:

  • SDLC and processes maturity
  • Expanding test coverage to include more non-functional testing, user condition testing, etc.
  • Coping with existing limitations of open source tools and frameworks
  • Sustaining correctly sized and up-to-date mobile test labs
  • Getting proper quality insights upon each test cycle prior and post production
  • Branching wisely cross-platform and cross-feature test suites

#10 Specification by Example , How Successful Teams Deliver the Right Software, Gojko Adzic

Gojko has a list of great books, this is one of the great ones, but check out other of his books as well.

Specification by Example is an emerging practice for creating software based on realistic examples, bridging the communication gap between business stakeholders and the dev teams building the software. In this book, author Gojko Adzic distills interviews with successful teams worldwide, sharing how they specify, develop, and deliver software, without defects, in short iterative delivery cycles.

#11 Clean Code,  A Handbook of Agile Software Craftsmanship, Robert C. Martin

Readers will come away from this book understanding

  • How to tell the difference between good and bad code
  • How to write good code and how to transform bad code into good code
  • How to create good names, good functions, good objects, and good classes
  • How to format code for maximum readability
  • How to implement complete error handling without obscuring code logic
  • How to unit test and practice test-driven development

#12 Continuous Delivery,  Reliable Software Releases through Build, Test, and Deployment Automation, Jez Humble and David Farley

The authors introduce state-of-the-art techniques, including automated infrastructure management and data migration, and the use of virtualization. For each, they review key issues, identify best practices, and demonstrate how to mitigate risks. Coverage includes

  • Automating all facets of building, integrating, testing, and deploying software
  • Implementing deployment pipelines at team and organizational levels
  • Improving collaboration between developers, testers, and operations
  • Developing features incrementally on large and distributed teams
  • Implementing an effective configuration management strategy
  • Automating acceptance testing, from analysis to implementation
  • Testing capacity and other non-functional requirements
  • Implementing continuous deployment and zero-downtime releases
  • Managing infrastructure, data, components and dependencies
  • Navigating risk management, compliance, and auditing

#13 The Guide to Software Testability, Ash Winter and Rob Meaney

Learn practical insights on how testability can help bring teams together to observe, control and understand the systems they build. Enabling them to better meet customer needs, achieve a transparent level of quality and predictability of delivery.

#14 A Practical Guide to Testing in DevOps, Katrina Clokie

A Practical Guide to Testing in DevOps offers direction and advice to anyone involved in testing in a DevOps environment

#15 Test Automation University, Angie Jones and Applitools

While not a book, a great shout out to my colleague, friend and co-author in one of my above mentioned books Angie Jones for launching, leading and building the Test Automation University. This is obviously an additional online resource that complements books and other written material.

Summary

I am sure that there are plenty of other great and practical books, but i went with this list as a start. If you feel that there must be an additional up to date book, reach out to me, and I will be more than happy to add to this list.

Happy Reading!

 

 

Posted on

The Essentials of iOS App Testing For iPhone X

48 hours ago, Apple revealed its new and futuristic iPhone X. Regardless of its design, and debatable price tag, this device also introduced a whole set of functionality, display, and engagement with the end-user.

iOS11 is turning to be quite different from previous releases from both user adoption which is still low (~30%) and also from a quality perspective – 4 patch releases in 1.5 months is a lot.

Most of the changes are already proving to cause issues for existing apps that work fine on iOS11.x and former iPhones like iPhone 8, 7 and others.

In this post, I’d highlight some pitfalls that testers as well as developers ought to be doing immediately if they haven’t done so already to make sure their apps are compliant with the latest Apple mobile portfolio.

The post will be divided into 2 areas: Mobile testing recommendations and App Development recommendations.

Mobile Testing for iPhone X/iOS 11

  • Test across all supported platforms as a general statement. iOS11 isn’t for every device, and apps are stuck on iOS10 that has different functionality than the iOS11. Test your apps across iOS9.3.5, iOS 10.3.3, and the latest iOS11.x
  • iPhone X comes from the factory with iOS11.0.1, requesting for an update to iOS1.1 – that means, this device will never get the intermediate iOS11.0.2/iOS11.0.3 – if customers haven’t yet updated to iOS1.1, you may want to have 1 device like iPhone 8/7 still on iOS11.0.3 so you have coverage for iOS11.Latest-1
  • Display and Screen Size for iPhone X specifically changed, and this device has a 5.8” screen size that is different for all other iPhones. Testing UI elements, Responsive apps layouts and other graphics on this new device is obviously a must (below is an example taken from CVS native app showing UI issues already found by me while playing with the device). This device is also full screen similar to the Samsung S8/Note 8 devices. A lot of tables, text field, and other UI elements need to be iOS11/iPhopne X ready by the developers.

Image title

  • New gestures and engagement flow impact usability as well as test automation scripts. In iPhone X, unlike previous iPhones, the user has no HOME Button to work with. That means that in order for him to launch the task manager  (see below) and switch or kill a background running app, he needs to follow a different flow. What that means is that at first, the app testing teams need to make sure that this new flow is covered in testing, and more important, if these flows are part of a test automation scenario, the code needs to be adapted to match the new flow.

Image title

In addition to the removal of the Home button that causes the new way of engaging with background apps, the way for the user to return to the Home screen has also changed. Getting back to the Home screen is a common step in every test automation, therefore these steps need to account for the changes, and replace the button press with a Swipe Up gesture.

  • Authentication and payment scenarios also changed with the elimination of the Touch ID option, that was replaced with the Face ID. While iPhone X introduced an innovative digital engagement with the Face recognition technology, the de-facto today to log in into apps, make payments and more, is still the Fingerprint authentication. Testing both methods is now a quality and dev requirement. From a scan that I ran through the leading apps in the market (see examples below), there is a clear unpreparedness for iPhone X. Most apps will either show on their UI the option to log in via Touch ID or if they support Face ID, they will allow users to use it, while still showing on the UI and in the app settings the unsupported option.

Image title

  • Testing mobile web and responsive web apps in both landscape and portrait mode with the unique iPhone X display is also a clear and immediate requirement. I also found issues mostly around text truncation and wrong leverage of the entire screen to display the web content.

Image title

 

In addition, trying to work with Hulu.com website proved to also be a challenge. Most menu content is being thrown to the bottom of the screen under the user control, making it simply inaccessible. Obviously, the site is not ready for iPhone X/Safari Browser.

Mobile Apps Development

  • Optimize existing iOS apps from both UI as well as authentication perspective. As spotted above, there are clear compatibility issues around the removal of the Touch ID option, that needs to be modified on the UI side of the apps when launched on iPhone X. In addition, scaling UI elements on the new screen whether for RWD apps or mobile apps needs refactoring as well. Apple is offering app developers a ui guidlines to help make the changes fast.Image title
  • Leverage advanced capabilities in iOS11 that best suit the new chipset (AI11 Bionic) and the camera sensors, to introduce digital engagement capabilities around augmented reality (ARKit API’s) and others. Retail apps and games are surely the 1st most suitable segments to jump on these innovative capabilities and enrich their end users’ experiences.

Image title

Bottom Line

The new iPhone X might be paving the way together with the Android Note 8 for a new era of innovation that offers App developers new opportunities to better engage and increase business values. If quality will not be aligned with these innovative opportunities, as shown above, that transformation will be quite challenging, slow and frustrating for the end users’

It is highly recommended for iOS app vendors across verticals to get hands-on experience with the new platform, assess the gaps in quality and functionality, and make the required changes so they are not “left behind” when the innovative train moves on.

Posted on

Optimizing Mobile Test Automation Across The Pipeline

With the massive innovation that drives the digital market these days, organizations are continuing to develop features, as well as new test code to cover these features.

What I’ve learned is that often, the test code developers would not always stop and look back into their existing test suites and validate whether the new tests that are being developed are somehow a superset to existing ones. In addition, legacy tests are a continuous load and overhead on your SDLC cycles length if they are not being maintained over time.

Oil Transport

Many Owners To The Same Problem

Since we live in an agile/DevQAOps world, test code development is not a QA only problem, but rather everyone3s. Tests are being executed throughout the pipeline from Dev to integration and pre/post production testing.

Use of smart tagging mechanism for your test scenarios (login), suites (App A) and types unit, regression) can be a good step towards gaining control over your tests.

Without some context, discipline, and continuous structured validation of the tests, it will become harder as you progress your SDLC to debug, analyze and solve defects (would be like finding the key in the below visual mess)

Find the Key in the Picture

Recommended Practices

  • Develop the tests with context, tags and proper annotations that would make sense to you and your team even 12 months from the development day. Make sure that in your execution reports you then have a way to filter using these annotations to only get the view of a given functional area, platform etc.
  • Match your device under tests capabilities to the test code and application under test. Make sure that you focus e.g. your fingerprint based tests only on the devices that support it (API XX and above).
  • Perform test code review every agreed upon time – in such review, group your feature specific test suites and try to optimize, merge, eliminate flakiness, identify missing coverage areas etc. It is harder to do it as the time progresses, so depending on your release cadence and test development maturity, set the right goals – more reviews would be better than less – it will also be shorter and more efficient that way since the delta between such review will be smaller.
  • Drive joint Dev, Test, Product, Marketing decisions based on data – When you have the ability to get quality analysis from your entire test suites, it is recommended to gather all counter parts and brainstorm on the findings. Which tests are most effective, can we shrink based on the data the release cycles, are we missing tests for specific areas, are there platforms that are more buggies than others, which tests takes longer than others to finish etc.
  • Optimize your CI and build-acceptance testing – based on the above intelligence, teams can reach data driven decision about what to include in their CI as well. Testing in the build cycle via CI should be fast, reliable with zero false positives. With quality insights on your tests, you can decide and certify the most valuable and fastest tests to get into this CI testing, and by that to shrink the overall process without risking coverage aspect.

CI_Dash1.png

Bottom Line

A test is code, and like you refactor, maintain, retire and improve your code, you should do the same to your tests. Make sure to always be in control over your tests, and by that, gain control over your quality of your app in a continuous manner.

Happy Testing!

Posted on

Introducing Reporting Test Driven Development (RTDD)

In the era of “[.. ] Driven Development” trends like BDD, TDD, and ATDD it is also important to realize the end goal of testing, and that’s the quality analysis phase.

In many of my engagements with customers, and also from my personal practitioner experience I constantly hear the following pains:

  1. Test executions are not contextually broken, therefore are too long to analyze and triage
  2. Planning test executions based on trends, experience, and insights is a challenge – e.g. which tests are finding more bugs than the other?
  3. Dealing with flaky tests is an ongoing pain especially around mobile apps and platforms
  4. On-Demand quality dashboards that reflect the app quality per CI Job, Per app build, Per functionality tested area etc.

 

Introducing Reporting Test Driven Development (RTDD)

As an aim to address the above pains, that I’m sure are not the only related ones, I came to an understanding, that if Agile/DevOps teams start thinking about their test authoring and implementation with the end-in-mind (that is the Test Reports) they can collect the value at the end of each test cycle as well as prior during the test planning phase.

When teams can leverage a test design pattern that assigns their tests with custom Contextual Tags that wrap an entire test execution or a single test scenario with annotations like “Regression“, “Login“, “Search” and so forth – suddenly the test suites are better structured, easily maintained and can be either included/excluded and filtered through at the end of execution.

In addition, when the entire suite is customized by tags and annotations, management teams can easily retrieve on-demand quality dashboard and be up to date with any given software iteration.

Finally, developers that get the defect reports post executions, can easily filter and drill down into the root cause in an easier and more efficient manner.

If you think about the above, the use of annotations as a method to manage test execution and filter them is not a new concept.

TestNG Annotations with Selenium Example (source: Guru99)

As seen above, there are supported ways to tag specific tests by their priority, it is just a matter of thinking about such tags from the beginning.

Doing reverse engineering to a large test suite is painful, hard to justify and most often too late since the product by then is already out there and the teams are left to struggle with the 4 mentioned consequences from above.

RTDD is all about putting structure, governance, and advanced capabilities into your test automation factory.

If we examine the following table that divides various tags by 3 levels, it can serve as 1 reference that can be immediately used either through built-in tagging and annotation coming from TestNG or other reporting solutions.

As can be seen in the above table, think about an existing test suite that you recently developed. Now, think about the exact test suite that is tag-based according to the above 3 categories:

  1. Execution level tags
    1. This tag can encapsulate the entire build or CI JOB-related testing activities, or it can differentiate the tests by the test framework in which you developed the scripts. That’s the highest classification level of tags that you would use.
  2. Test suite level tags
    1. This is where you start breaking your test factory according to more specific identifiers like your mobile environment, the high-level functionality under test etc.
  3. Logical test level tags
    1. These are the most granular test tags identifiers that you would want to define per each of your test logical steps to make it easy to filter upon, triage failures, and plan ongoing regressions based on code changes.

As a reference implementation for an RTDD solution in addition to the basic TestNG implementation that can be very powerful if being used correctly with its listeners, pre-defined tags and more,  I would like to refer you to an open-source reporting SDK that enables you to do exactly what is mentioned in the above post.

When using such SDK with your mobile or responsive web test suites, you achieve both, the dashboards as seen below as well as a fast defect resolution that drills down by both Test case and Platform under test

Code Sample: Using Geico RWD Site with Reporting TDD SDK [Source: My Personal GIT)

 

Digital Dashboard Example With Predefined ContextTags (source: Perfecto)

 

Bottom Line

What I have documented above, should allow both managers, test automation engineers, and developers of UI/Unit and other CI related tests to extend either a legacy test report, a testNG report or other – to a more customizable test report that, as I’ve demonstrated above, can allow them to achieve the following outcomes:

  • Better structured test scenarios and test suites
  • Use tagging from early test authoring as a method for faster triaging and prioritizing fixes
  • Shift tag based tests into planned test activities (CI, Regression, Specific functional area testing, etc.)
  • Easily filter big test data and drill down into specific failures per test, per platform, per test result or through groups.
  • Eliminate flaky tests through high-quality visibility into failures

The result of the above is a facilitation of a methodological-based RTDD workflow that can be maintained much easier than before.

Happy Testing (as always)!

Posted on

Google Mobile Friendly With Perfecto and Quantum

Guest Blog Post by Amir Rozenberg, Senior Director of Product Management, Perfecto

resize

Google recently announced “Mobile First Indexing”, from Google:

To make our results more useful, we’ve begun experiments to make our index mobile-first. Although our search index will continue to be a single index of websites and apps, our algorithms will eventually primarily use the mobile version of a site’s content to rank pages from that site, to understand structured data, and to show snippets from those pages in our results (Source).

screen-shot-2017-02-13-at-5-33-26-pm

More recently they made the Google Mobile-Friendly tool and guidelines available. A very nice interactive version is available here, and images at the bottom of the thread, while there’s also an API (which, thanks to Google, can allow users to exercise first before they code). Google also offers code snippets in several languages.

Notes:

  • Google takes a URL and renders it. If you run multiple executions in parallel there’s no point in sending the same URL from every execution because the result would be the same
  • Google returns basically “MOBILE_FRIENDLY” or not. Suggest to set the assert on that
  • The current API differs from the UI such that it only provides the results for Mobile friendly (and the UI gives also mobile and web page speed). Hopefully, Google adds that to the response 😉
  • This will probably not work for internal pages as Google probably doesn’t have a site-to-site secure connection with your network.

 

For developers and testers who do not have time, testing mobile friendliness repeatedly probably will simply not happen. That’s why I integrated Google Mobile-Friendly API into Quantum:

  • Added 2 Gherkin commands
// If you navigate directly to this page
Then I check mobileFriendly URL "http://www.nfl.com"
// If you got to this page through clicks
Then I check mobileFriendly current URL
  • Added the Gherkin command support (GoogleMobileFriendlyStepsDefs.java)
  • And the script example is pretty simple:
@Web
Feature: NFL validate

  @SimpleValidation
  Scenario: Validate NFL
    Given I open browser to webpage "http://www.nfl.com"
    Then I check mobileFriendly current URL
    Then I check mobileFriendly URL "http://www.nfl.com"
    Then I wait "5" seconds to see the text "video"

 

That’s it. Next steps:

 

Ideas for future improvement:

  • You can automate the validation such that every click would trigger a check with Google behind the scenes.

Just for fun, some more screenshots for detailed analysis for NFL.com:

 

screen-shot-2017-02-13-at-5-33-48-pm

 

screen-shot-2017-02-13-at-5-34-09-pm

screen-shot-2017-02-13-at-5-34-23-pm

 

 

Posted on

Criteria’s for Choosing The Right Open-Source Test Automation Tools

I presented last night at a local Boston meetup hosted by BlazeMeter a session together with my colleague Amir Rozenberg.

The subject was the shift from legacy to open-source frameworks, the motivations behind and also the challenges of adopting open-source without a clear strategy especially in the digital space that includes 3 layers:

  1. Open source connectivity to a Lab
  2. Open-source and its test coverage capabilities (e.g. Can open-source framework support system level, visual analysis, real environment settings and more)
  3. open-source reporting and analysis capabilities.

During the session, Amir also presented an open-source BDD/Cucumber based test framework called Quantum (http://projectquantom.io)

Full presentation slides can be found here:

Happy Reading

Eran & Amir

Posted on

Shifting Mobile App Quality Into the Dev Build Cycles

It’s no doubt that quality is becoming a joint feature team responsibility, and with that in mind – it is not enough for traditional QA engineers to develop and execute test automation post a successful build, but actually the growing expectations now are from the Dev team to also take part and include as many tests as they can in their build cycles per each code commit.

Tests can be either unit, functional, UI or even small scale performance tests.

With that in mind, Dev team need a convenient environment that allows them to perform these quality related activities so they deliver better code faster!

Developers today are specifically challenged with the following:

  1. Solving issues that come from production or from their QA teams that require a specific device or/and environment that’s usually not available for the dev team
  2. Validation of newly developed apps or features within apps across different environments and devices as part of their dev process
  3. Lack of shared assets for the entire dev team
  4. Ability to get a “long USB cable” that enables full remote device capabilities & debugging

Perfecto just made available as part of its continuous quality lab in the cloud a set of new tools and capabilities that addresses these requirements and enable Dev team to accomplish their goals.

Perfecto’s DevTunnel solution for Android that is part of the recent 9.4 release is the 1st significant step toward helping developers accomplish more tests as part of the build cycle.

dt

With the above challenges and requirements in mind, Perfecto has developed a unique solution called the “DevTunnel” which enables developers to get enhanced remote access to mobile devices in the cloud and perform any operation that they could have done with these devices if they were locally connected – things like debugging, running unit tests, testing UI at scale from within the IDE and more.

espredebug

In addition, when referring to Android Dev activities, it’s clear that Android Studio & IntelliJ IDEA are the leading IDE’s to operate in. For that, Perfecto invested in developing a robust plugin that integrates nicely into the development workflow.

Espresso Framework

It’s no doubt that Espresso test automation framework is becoming more and more adopted across the developers for various reasons like:

  1. Embedded into Android Studio play an important role for Android developers.
  2. It’s very fast and easy to execute and receive feedback on Android devices

Espresso can be used within the Perfecto lab today in the following 2 modes

  • Locally – Execution through DevTunnel (see below)
  • Via Continuous Integration (CI) – using a command for Espresso test execution through Jenkins server

In the community series targeted to Dev Tunnel, you can learn more about the capabilities, use cases and get samples to get you started with the new capability.

To see this also in action, please refer to the video playlist that demonstrates how to get started and install DevTunnel, use Perfecto Lab within Android Studio with Espresso for testing and debugging purposes and more.

 

Good Luck!

Posted on

7 Mobile Test Automation Best Practices

Developing a mobile test automation scenario isn’t that complicated. Developers and testers use a variety of commercial test automation frameworks or open source tools such as Selenium and Appium to do automation. However, when trying to execute these tests on real devices or integrate them into an Agile or CI (continuous integration) workflow, things get a little complicated.

The major challenges around mobile test automation

The essence of developing test automation is to be able to use and re-use scripts many times, across platforms and environments. Test automation should be as maintainable as possible, especially as new platforms and product features are released. Many organizations that develop test automation for their mobile apps face the following challenges:

  1. Executing the tests against a variety of real mobile devices
  2. Executing these tests in parallel
  3. Leveraging existing test code (re-usability) for new tests
  4. Including real end-user environments/conditions (changing network conditions, low battery) in the tests
  5. Overcoming unexpected interruptions (incoming call, apps running in background)
  6. Running these tests unattended — over night, as part of a Jenkins CI job

These are just few of the challenges organizations confront when trying to progress from older SDLC processes and meet faster releases and enhanced Dev–>Build–>Deploy–>Test–>Deploy cycles.

7 practical test automation tips

Overcoming these challenges starts with few changes in the overall mobile app dev and test processes.

Consider these seven recommendations for building sustainable unattended automation.

Test automation

The key to mobile test automation is to start with a small number of test cases, automate them, and assure that they are robust enough and can be executed in parallel and unattended. Only then should you invest more and grow the test suite.

An important question to ask at the start is: What should I be automating? Organization often do not choose the right tests to automate, resulting in lost development time, weak ROI, and an over-reliance on manual testing.

To learn more about the 7 Ways to Overcome Test Automation Obstacles, please join us next week for a webinar hosted by myself, automation expert and author Daniel Knott, and Perfecto’s Director of Technology Uzi Eilon.

Posted on

Tests to Include Within Automation Suite

When developing a mobile or desktop test automation plan organization often struggle with the right scope and coverage for the project.

In previous post, i covered the test coverage recommendations in a mobile project and now, i would like to also expand on the topic of which tests to automate.

Achieving release agility with high quality is fully dependent today more than ever on continuous testing which is gained through proper test automation, however automating every test scenario is not feasible and not necessary to meet this goal.

In the below table  we can see some very practical examples of test cases with various parameters with a Y/N recommendation whether to automate or no.

As shown below, and as a rule for both Mobile, Web and other projects the key tests by definition which should be added to an automation suite (from ROI perspective and TTM) are the ones who are:

  • Required to be executed against various data sets
  • Tests which ought to run against multiple environments (Devices, Browsers, Locations)
  • Complex test scenario’s (these are time consuming and error prone when done manually)
  • Tedious and repetitive test cases are a must to automate
  • Tests which are dependent on various aspects (can be other tests, other environments etc.)

Picture1

Bottom line: Automation is key in today’s digital world, but doing it right and wisely can shorten time to market, redundant resources and a lot of wasted R&D time chasing unimportant defects coming from irrelevant tests

Happy Testing!