Mobile & Web Test Coverage Index for How To Build Smarter Cross-Platform Testing

If you make your living building or testing applications, you know the market is constantly changing. And there’s one question you can’t avoid: Which platforms do I need to test against to make sure I’m not missing something important?

Mobile & Web Test Coverage Index for How To Build Smarter Cross-Platform Testing

That’s exactly why we made the Mobile & Web Test Coverage Index. It breaks down the most used phones and top web browsers around the world — so you can create a test coverage strategy that mitigates risk and targets the most popular devices among your audience.

In this article, you’ll find a proven test coverage methodology for cross-platform testing with our latest insights.

  • Top 4 DevOps trends for 2019.
  • Mobile and web browser testing coverage release calendar for 2019.
  • Get proven methodology to align cross-platform testing into the development cycle.
  • Learn the most used phones and browsers your audiences use for highly targeted testing.

Content Summary

Why We Created the Index
How We Built the Index
How to Use the Index
Test Coverage Considerations
Insights for 2019

Why We Created the Index

Our customers often ask us a simple question: On what devices and browsers should I be testing my mobile and web applications?

As the leader in continuous testing for mobile and web apps, Perfecto is uniquely positioned to give the most accurate answer.

In this article, we’ve merged our exclusive insights with current market data to provide a benchmark of devices, web browsers, and user conditions to test on — including location changes and apps in the background — to help you test smarter.

This is the culmination of what we’ve learned from our most-valued customers. We’re excited to pass these insights on to you.

How We Built the Index

No single data source tells the full story, so for these indexes we’ve combined exclusive Perfecto data and mobile market usage data across a variety of countries into our own analysis.

The indexes offer up-to-date benchmarks that list the top smartphones, tablets, and operating systems based on usage. These are the device/OS combinations on which you should develop and test your mobile, web, and responsive web apps.

For desktop browsers, we include one index that can be applied to all 13 countries in the report.

Mobile Device Coverage:

  • Data from testing on 4,000+ devices
  • Market share data (400,000 users)
  • Our own research and analysis
  • 3rd party data

How to Use the Index

Each country featured in the report has its own digital coverage profile. It includes testing recommendations for mobile devices and operating system versions to help create a holistic view of what’s required for digital quality testing.

To identify your optimal test coverage, you’ll first need to determine your device coverage goal.

Devices that Matter to You
Our list of 32 devices per index for each country is narrowed down from hundreds of possible options to give you the maximum coverage possible.

Each index is divided into three groups —Essential, Enhanced, and Extended.

Each group includes the previous group’s devices. If your testing covers all 32 devices across the three groups, you’ll have the most complete recommended test coverage with the least risk.

Select your target country and then refer to the corresponding device list. Make note of the “recommended OS version” column to see the recommended testing mix of devices and platforms.

We include an up/down arrow or equals sign to indicate usage changes since last quarter. Devices that are new to the index or are reference devices running the latest OS are marked accordingly.

  • Essential Coverage: Top 10 “must test” devices based on usage
  • Enhanced Coverage: Top 25 devices, including legacy and trending devices and different screen sizes
  • Extended Coverage: Top 32 devices, includes niche, legacy and brand new devices to represent the “long tail”

Testing Web Browsers
In our standalone Web Test Coverage Index (pg. 11), we list the browser/OS combinations — categorized by latest, previous, and newest beta — that organizations in every country should test on to achieve desired market coverage. Given that browser versions update quickly and often automatically without users knowing, it is important to ensure that browser quality aligns with a variety of OS versions.

Finally, we’ve included updated mobile market and web calendars (pgs 25 and 26) showing what device models, OS versions, and browser/OS combinations to look out for this year.

Factors is an objective reflection of selected markets based on mobile device usage of specific operating system versions. Some of the device and operating system combinations that are recommended in this report may no longer be available from OEMs or wireless network providers due to today’s ever changing market dynamics.

Test Coverage Considerations For Mobile Test Automation Execution

If we break down the pipeline into testing types and triggers, it would look as follows (Fig 1.)

Figure 1: Platform Coverage
Figure 1: Platform Coverage

To assess size, capacity, and the right platform coverage, teams need to calculate their test lab size based on unique inputs.

Figure 2: Sizing Example
Figure 2: Sizing Example

As indicated in this report, there are a few mobile coverage buckets ranging from Essential, through Enhanced, and Extended (top 10, 25, and 32 mobile platforms permutations). Teams ought to decide based on analytics and risk-based decisions which bucket they believe is the right one for their first step.

Next, they need to bundle in their test data parameters like the number of tests, the test duration, and the required execution time. These inputs will provide the actual time a full cycle or subset may take based on the lab configuration. Assuming it is outside the boundaries of the testing cycle time, more parallelization and platforms may be required.

As an example, Figure 2 took 150 regression tests each running for three minutes against one platform. Once multiplying each test against the test coverage bucket, we receive the overall number of hours such a cycle will take.

To include the cycle in the time window, we need to run more in parallel. Cost avoidance in this example assumes an average $100K annual salary and 2080 working hours a year. Each organization can include their own averages to get their outputs as close as possible.

As identified below, by using parallel test execution against the right platforms with the right test cases with high value, teams can get back a lot of time in return, reduce costs, and expedite quality feedback and visibility.

Insights for 2019

As we get deeper into 2019, emerging trends are reshaping the digital landscape for both mobile and desktop web apps. And while these developments are exciting, DevOps teams will need to prepare for the opportunities and challenges they present. In this edition, we’ll focus on the following four trends:

  • Foldable smartphones.
  • Android Q and iOS 13 releases.
  • The transition from 4G LTE to 5G networks.
  • The rise of AI and machine learning (ML).

Keep reading to see how these four trends are already disrupting and changing the way apps are being developed and consumed.

Foldable smartphones
Up until recently, developing apps for regular smartphones and tablets was a complex task. But this grew exponentially more complex following the 2019 MWC Barcelona (formerly Mobile World Congress), where Samsung, LG, and Huawei announced their newly developed foldable smartphones.

Samsung announced its Galaxy Fold. LG, with a dedicated accessory, announced its LG V50 ThinQ foldable device. And Huawei did the same with its Mate X device.

Multi-Window Feature Support
Multi-Window Feature Support

To properly support these innovative products, developers and testers need to learn the new capabilities of these devices, make proper adjustments to the applications, and perform extensive testing.

Foldable smartphones can support up to three apps that run in parallel in the foreground using the on-resume functionality. This is a new behavior for smartphones. Before this technology existed, devices put all background apps on pause mode while the one in the foreground was in use.

We recommend mobile app developers and testers ensure they have strong coverage for the various mobile OS and smartphone families that support all possible Android app continuity. That includes multi-window, multi-view, or multi-resume capabilities. Teams also need to make sure that the app performs well, displays correctly from a UI/UX perspective, and functions properly across these families.

Android Q and iOS 13 releases
Each year we see the launch of new major mobile OS platforms.

In March, Google launched its first Android Q (10) beta to developers to start exploring, testing, and preparing their apps for the Android Q GA coming in August of 2019. The new platform is only supported on emulators on Pixel/Pixel XL real devices.

We recommend teams start leveraging the beta version to ensure your test automation scripts run and that you don’t experience any regressions. In addition, there are important changes to the OS like privacy, security, UI settings, connectivity, and more that may impact your app. Because of this, we recommend you follow the upcoming beta releases until GA. This will allow you to be on top of any changes that will be deployed in the final release.

Similarly, iOS 13 will be released as beta most likely in June of 2019 with GA coming in September. This release will come with a set of new iPhone 11 smartphones and many new features that require preparation by both the app developers and testers.

Recently Apple launched two new iPads and announced a new Apple Card that may pave the road towards financial user engagement. This will require third party dependent apps and banking apps to rethink how they compete or leverage this new technology.

The transition from 4G LTE to 5G networks
This technology is just one example of the progression and growing complexity of the digital landscape — which is set to grow even more complex with the emergence of the fifthgeneration wireless (5G) network.

These days, we are experiencing the early deployments of 5G in many geographies including the United States, Australia, Japan, and some countries in Europe, such as the U.K.

New 5G technology will provide greater mobile and IoT network reliability and ultra-speed traffic. It also means that more innovative technologies and platforms will be able to better connect and work together.

Smart homes and cities, critical apps within healthcare, augmented reality, IoT, wearables, 3D videos, and advanced voice apps will all experience better and more reliable communication between these digital platforms thanks to 5G. But with opportunity comes challenge: to make all of these interoperable and seamless.

5G Usage Scenarios
5G Usage Scenarios

The rise of AI and machine learning (ML)
More and more segments and domains in the digital markets are starting to invest and leverage artificial intelligence (AI) and machine learning (ML) capabilities.

With the growing complexity of developing, testing, and navigating big data, combined with the need for faster response to market events, organizations are looking at smarter engines that use AI and ML. These technologies can help optimize some of the time and complex tasks that are slowing them down.

We’ve seen both big data analysis — as well as codeless test automation — being introduced recently using AI and ML capabilities. In the future, we will continue seeing more of these technologies evolving to help address the pace of innovation and the complexities that they bring to enterprises.

Bottom Line
2019 is a pivotal time for app developers to make the necessary changes to their products to support emerging capabilities, as well as ensure great quality and user-experience to their customers.

New mobile platforms, foldable devices, and faster networks will all play a part in altering the digital landscape. Are you and your team positioned to keep up?

Evolution of Testing
Evolution of Testing

Source: Perfecto