The world of mobile is filled with opportunity, but, to make an impact, your digital property needs to be used by thousands – if not millions – of users around the world. These users are on different devices, operating systems, networks and more. To keep them happy and loyal, your users must be delighted and satisfied every time they use your app, no matter where they are.
This article offers tips and insights on:
- How to overcome common QA challenges
- How to test for the mobile web versus native apps
- How to expand your testing coverage at scale
Read on this article to learn the constants in mobile app quality and learn how to take advantage of the mobile matrix.
Content Summary
Introduction
Mobile Boom
Mobile Challenges
Common Functional Issues
Functional Testing
Common Usability Issues
Usability Studies
Common Security Issues
Security Testing
Common Localization Issues
Localization Testing
Device Fragmentation
Operating System Fragmentation
Carriers and Connectivity
Native Apps
The Rise of Mobile
Conquering The Mobile Matrix
Emulators and Simulators
Conclusion
Introduction
You developed a mobile app … now what?
By the time you finish reading this eBook, the mobile app landscape will change. New OS versions will be released. A bunch of new devices will hit the market. Cool new apps will go viral, gaining millions of users and setting new standards in user expectations. And mobile app testing will become much more complex and challenging. Discouraged? Don’t be. The world of mobile is filled with excitement and opportunity and anyone can have the next hit app! But to make a splash, your digital property needs to stand up to use by thousands – if not millions – of users around the world with different devices, operating systems, networks, connections and more. To keep those users – your users – delighted, your app needs to be high quality and thoroughly tested. Despite the ever-changing mobile landscape, there are some constants in the realm of mobile app quality that organizations of any size – from small startups to major global enterprises – can study, understand and take advantage of. In the following pages, we’ll discuss the mobile landscape, dispel some common myths and outline proven strategies to help you deal with the exhaustive mobile matrix. Ready to learn more? Let’s get started …
Mobile Boom
By now, the mobile boom is obvious and it’s clear that mobile apps are here to stay. Smartphones and tablets are rapidly becoming the primary method of interaction for consumers and businesses worldwide. Smartphone penetration in the US reached 81% in 2016, with global penetration hitting 66%. But mobile goes beyond smartphones and tablets. Apps are now being incorporated into cars, wearable technology and home appliances (which are considerably less “mobile” but still important in terms of applications.)
As mobile adoption increases, apps will become even more critical to the success of businesses. Companies that recognize this trend and take actions to make the quality, security and usability of their digital properties top priority will find themselves with a tremendous advantage
75% of American smartphone owners download a mobile app every month. – Tune
The explosion of apps can be seen in just about every industry, from consumer use to B2B tools:
- Retail: Location-aware mobile commerce, omnichannel offerings, in-store checkout and more
- Media: Magazines and newspapers are going digital and spreading to readers on every operating system
- Travel: Mobile bookings, check-ins, maps and on-demand services
- Education: Tablets and apps are increasingly being used in classrooms and for on-site work training
- Healthcare & Fitness: From digital medical records and in-home care apps to fitness trackers and diet apps, the health, wellness and fitness world is going mobile
- Finance: Apps do everything from on-the-go banking to real-time trading and portfolio analysis
- Social: Social networks and geolocation meet-up apps need to follow users wherever they go
- Gaming: From serious console games to casual mobile games, gaming is a top app category that spans user demographics
- Business apps: CRM, ERP and HR systems are developing mobile access points
- Productivity: Docs, spreadsheets and presentations are increasingly moving to tablets
- Collaboration: Workers are ditching laptops and need to access email, IM and other important features via mobile
- Smart TVs: Media, social, news, weather, sports and gaming apps are in demand on smart TVs
- Emerging tech: Smartwatches, in-car apps, lifestyle and fitness trackers, smart appliances, etc. need optimized and well-tested digital experiences
By 2022 there will be 6.8 billion smartphones in circulation, overtaking the number of active fixed line subscriptions worldwide. – Ericsson
Mobile Challenges
The majority of users expect your app to load within three seconds. If your app doesn’t launch fast enough, users will find an alternative and many report that they will never return.
Worse, a poorly performing digital property can have a negative impact on a user’s overall opinion of your brand. A study by SOASTA found that “almost nine out of ten Americans associate negative feelings with brands that have poorly performing websites and mobile apps (88%).”
Right or wrong, users have incredibly high expectations when it comes to digital quality.
Other problem areas:
- OS & device: If you’re aiming for widespread adoption, then you need to provide a seamless user experience regardless of device or OS.
- Security & privacy: Companies that violate user privacy run the risk of making headlines for all the wrong reasons. Users expect their personal data to remain private and digital properties to be 100% free of threats.
- Connection speed: Does your digital property run the same on 3G, LTE and Wi-Fi connections? If the answer is no, expect upset users. In some instances, a poor connection can be the difference between a five-star and onestar review.
- Data usage: Users are wising up to the fact that some apps drain battery faster than others – and they’re abandoning those offending apps in favor of more economical ones. When developing an app, stay mindful of how it functions in all situations.
Why failing hurts
Back in the day, one bad customer experience could spread via word-of-mouth and reach a few prospective customers. Thanks to social media, that same word-of mouth now travels at the speed of light and reaches hundreds, thousands or even more users.
The other major difference is app store ratings. Imagine a poor review stamped directly on your product – that’s what happens with bad app store ratings.
“You can launch a beautifully designed native application, but if it crashes, then it will receive a poor rating and users will go elsewhere,” said Michael Croghan, Mobile Solutions Architect at USA TODAY. “Our goal is to launch nothing short of a 4.5 star app every time – no exceptions. Anyone can read your app store rating. There’s no way to hide poor quality in the world of mobile.”
Much as we love freedom and choice, we also love things that just work, reliably and seamlessly. – Chris Anderson, Wired
Common Functional Issues
Like with all software, you need to make sure every facet of your digital property works as intended. Pay special attention to these key areas.
Sign-up and Login: This may seem like a no-brainer, but if users cannot easily access your app, all your efforts will be wasted. If your app requires a password and user name, pay close attention to the fields and make sure it’s easy for users to enter their information.
Menu Options: Menu options can often be difficult to access or decipher. Make sure menu items like Help, About, etc. are easy to find, navigate and select.
Actions: Any problems related to scrolling, selection, the back button, etc. are bound to lead to trouble.
Connection Speed and Carrier: Testing on only one carrier hasn’t been an option for a long while. Make sure your basic tests cover the most likely carriers, as performance can vary greatly from one carrier to another.
Screen Size: Screen size discrepancies should be a top consideration while testing your app. Does everything look good and work correctly on a range of screen sizes?
Interruptions: How does your app behave when the device battery is at full, medium and low strength? Your app doesn’t operate in a vacuum, so test how it works when interruptions like battery power, SMS, MMS and video calls occur.
Error Messages: Your error messages should be clear, concise and actionable in the eyes of the everyday user.
Functional Testing
When it comes to app functionality, remind all departments from design to development to QA that making sure the product works is a shared responsibility. Test early and test often – combining automated and manual testing to cover your digital property from every angle.
It is exceptionally important to test the functionality of your app on a range of devices. If an app doesn’t render correctly, it can mean navigation or other actions are entirely impossible – making your app essentially useless. And don’t forget regression testing any time you change, remove or add a feature. Use the app like you expect a new user to, revealing potential problems.
Common Usability Issues
The goal of usability studies is to ensure a user can complete the tasks they are expected to complete. More importantly, they must be able to do so easily and without becoming frustrated. A good user experience can be thwarted by a number issues.
Functionality: A mobile application must present the user with the appropriate functionality. If the functionality of an application is perceived as incomplete or inadequate, customers may be lost.
Layout and Design: Good layout and design allow a user to easily complete tasks. Don’t offer so many features or content that your digital property is difficult to navigate. Everything should be easy to see and interact with on a small screen.
Interaction: The flow of a digital experience must be natural and interactions should be easy. For example, if a button is placed in the perceived ‘wrong’ location, users can become frustrated and abandon the application. If links are too close together, they’ll be hard to select.
Usability Studies
Not all users are created equal. If you want to get a true sense of your mobile app’s usability, be sure to select participants that closely mirror your target audience.
Usability is highly subjective, so take care not to ask leading questions when crafting a usability survey and pay special attention when looking at the survey data. It’s important for you to separate minor feedback from recurring issues that hinder the app’s usability for a large portion of users.
If you need help navigating the usability landscape, consider working with a professional usability expert. Experts can craft surveys, analyze the raw data for meaningful takeaways and give you recommendations based on an understanding of user expectations and current mobile standards.
Common Security Issues
The explosion of mobile presents an entirely new set of security challenges. While most of the security tools and practices used for traditional web and desktop applications are equally applicable to mobile, there are some unique concerns to keep in mind, including lost or stolen devices, mobile malware, data leaks, encryption and more.
Ask yourself about:
- Confidentiality: Does your app keep private data private?
- Integrity: Can the data from your app be trusted and verified?
- Authentication: Does your app verify that users are who they say they are?
- Authorization: Does your application properly limit user privileges?
- Availability: Can an attacker take your app offline?
- Non-Repudiation: Does your app keep records of events?
Security Testing
With the rise of bring-your-own-device (BYOD) programs, making sure your app isn’t the gateway for a hacker is crucial for you, your reputation, your users and your company.
Security testing is a practiced skill, not necessarily one a general QA team can adequately accomplish. Likewise, security testing varies depending on what you’re testing. Are you testing a mobile website, a web API, a backend server, all the above? If no one on your QA team is willing to dig into security testing on a deep level, find an expert who can fill the gap.
Another crucial aspect that falls under security testing is privacy and transparency. Companies who access unnecessary device features or use data in a way users aren’t aware of will suffer an angry backlash when the truth comes out. When testing, make sure your privacy policy and information about handling user data is easy for your users to find and understand.
Software can be correct without being secure. Indeed, software can meet every requirement and perform every specified action flawlessly yet still be exploited by a malicious user. This is because security bugs are different from traditional bugs. In order to locate security bugs, testers have to think differently too. – James Whittaker, Testing Expert
Common Localization Issues
The worldwide proliferation of mobile apps requires your products to be accessible and contextual for users in a variety of markets and regions. But without proper localization, companies can exclude, confuse or offend entire subsets of users without even knowing it.
Relying on translation software or outsourcing firms is not the same thing as proper localization. If your app isn’t being reviewed by native speakers or people who know the local area, you could be missing major mistakes. Apps that fail to fully account for different cultural understandings will not survive and thrive.
Localization Testing
Localization testing should consider factors such as:
- Content: Static and dynamic content like catalogs, search results, metadata
- Dates: Is the date January 1 or 1 January?
- Characters: Différent länguages have ðifferent set∫ θf characters
- Postal codes: In some countries, postal codes contain letters
- Phone numbers: Different formats for different markets
- Direction: Some languages are written left to right, others are right to left
- Currency conversion: Especially important for retailers
- Tax calculation: VAT, sales tax and others vary from country to country
- Cultural context: Words and phrases may technically be translated correctly, but they could have a different implied meaning in another culture
Our global customers have different demands. we want products to ‘feel local’ and to support features that may be unique to specific markets. as we specialize our products for certain markets, it introduces more challenges for testing – like requiring special cultural knowledge. – Patrick Copeland, Sr. Engineering Director, Google
Device Fragmentation
Perhaps the most difficult aspect of the mobile testing matrix is device fragmentation. Though the iOS device matrix is growing more than ever, fragmentation is especially an issue for the Android operating system.
Take, for example, the image to the right. This is a data chart by OpenSignal of over 24,000 separate Android device models encountered in August 2015 (For comparison, the same chart from 2012 found nearly 4,000 devices.)
If you’re concerned with quality on cross-platform apps, you’ll encounter a similarly complicated matrix.
Device Fragmentation
Operating System Fragmentation
Although less daunting than the hardware matrix, the variety of mobile operating systems also poses a challenge for dev and engineering teams whose goal is to provide a consistent user experience across platforms.
Even within a single operating system, developers can encounter fragmentation issues. Mobile users are not always quick updaters – and some users who want the latest OS version may not have access to it right away. Developers need to ensure apps work on the newest OS version and several past versions that are likely still in use.
Android: Even though Google is into version 6 of their OS, a large portion of users are still on 4.x and some even on 2.3.x.
iOS: iOS users are traditionally quick updaters, but developers should still support at least one (if not two) back version of Apple’s mobile OS.
In-the-wild testing: In-the-wild testing is the practice of moving part of your QA out of the lab and into the real world. Testing in real-world conditions provides a better perspective on how apps will work in the hands of your real users. It also gives you access to a greater range of devices, operating systems, versions.
Carriers and Connectivity
If you’ve ever experienced unexplained inconsistencies with the performance of your mobile app, there’s a good chance that carriers and connections are the culprit. Perhaps more than any other criteria, issues associated with carriers are almost always found outside of the lab environment.
Without moving a portion of your testing out of the lab and into the real world, there’s no way to ensure quality across carriers and locations. These situations are impossible to recreate accurately in a lab.
Native Apps
Native apps have a higher use rate and remain the only option if your app needs to access specific device APIs (such as the camera or address book). Native apps also offer a highly controllable custom experience because they are developed specifically for a designated operating system – so you can control exactly how the app looks on each device. The built-in exposure of being included in an app store is another draw for native apps.
Since a unique native app needs to be created for each operating system, your quality efforts also need to expand. Native apps should always be tested on a range of devices and carriers associated with that operating system, as well as on different versions of the OS.
Users of different operating systems have different expectations, so listen to what they’re saying after you launch your app. With the ability of users to rate and review your app in the public app stores, there’s little tolerance for poor quality.
The Rise of Mobile
Despite the rise and continued dominance of native apps, mobile web is still extremely – and increasingly – important. It’s also sometimes the better option.
Mobile web is attractive for several reasons. It is often cheaper and easier than creating and maintaining native applications because developers can create one source code that will function across platforms.
By adopting responsive web design, teams can create one cohesive experience that responds automatically to whichever size device a user is on, from large screen desktops to the smallest smartphones and every tablet in between. This is especially important in today’s world of growing device fragmentation and rising mobile web use. In fact, many of today’s mobile users feel that companies that don’t have sites that work well on mobile don’t care about quality.
Conquering The Mobile Matrix
Historically, when companies wanted to improve their digital quality, they did so within the sterile environment of a lab – far removed from where their users live, work and play. The evolution of mobile eliminates that option as an effective solution on its own. If testing for mobile apps is conducted exclusively in a central location, teams can’t really be sure their apps will work in the hands of their actual users in the real world.
So what’s a tech leader to do? Hire testers in key geographic markets? Construct an elaborate QA lab? Spend more on simulators and emulators? Declare their digital quality “good enough?” The answer is simple – move a portion of your testing out of the lab and into the wild.
Professional in-the-wild testing gives companies access to experienced testers around the world who will test your app in the same scenarios your users will actually experience it. Testers use real hardware, with real software, on imperfect connections. In short, they test under real-world conditions.
Emulators and Simulators
Mobile emulators and simulators are still an important testing tool in that they enable developers to verify general functionality and perform regular regression testing.
However, the very nature of emulators and simulators means testing is occurring in an environment far removed from the real world, with a series of actions performed by a mouse and keyboard, not fingers on a touch screen.
The convenience of simulators and emulators can lull teams into a false sense of security. But the advantages of such tools are limited in scope, and should never be considered a substitute for real-world, on-device testing performed by live testers. After all, emulators and simulators aren’t dealing with rush hour in LA, testing in the fields of Iowa or holding a device in one hand and a hot cup of coffee in the other … while on a bus. Using emulators and simulators in tandem with in-the wild testing will give you the best results.
Extra Tip: Some companies offer labs filled with real devices for testing. These tests still rely on computer generated actions and reports. Just because an app didn’t crash doesn’t mean all the graphics loaded correctly, there were no GUI bugs and links were easy to click with fingers. For that, you need real users on a range of devices.
In-the-wild testing truly can be an extension of your QA, and not a replacement. We talk about it in terms of a ‘force multiplier.’ It allows our QA team to project a much broader coverage path. It becomes an extension of what they’re doing. – Brad Schneider, Director of Application Development, The Container Store
Conclusion
There was a time when the mobile matrix was overwhelmingly complex and difficult to tackle. But thanks to the evolution of in-the-wild testing, that’s changed.
As the mobile market continues to grow, those brands that emphasize quality and pay special attention to real-world test coverage and feedback will enjoy increased market share, profitability and – above all – user loyalty and delight. Those who neglect quality will struggle to keep up in a world filled with app options.
The future of mobile remains bright. Once reserved for a tech-savvy niche, the use of mobile apps is now firmly entrenched in the mainstream. There’s no turning back, and that means mobile app quality must rapidly advance to keep up with user demand and expectations.
Source: Applause
