Overcome the challenges in Continuous Testing to optimize DevOps

Businesses around the world are focusing on delivering a high-quality product in less time. There is a need to continuously develop and test the product before deployment. Continuous testing is helpful to accelerate the delivery but the developers and testers might face some issues in continuous testing. This white paper explains the challenges in continuous testing and how to overcome those challenges.

Overcome the challenges in Continuous Testing to optimize DevOps
Overcome the challenges in Continuous Testing to optimize DevOps

Content Summary

The Core of Continuous Testing
Challenges Faced while Implementing Continuous Testing
Test Design Techniques
Deficient Requirements and User Stories
Test Data Distribution
Test Automation
System Constraints
Performance Testing Remains ad hoc, In-house and Incomplete
Access to Dependent Systems and Test Environments
An Inability to React to Change
Harmonize Continuous Testing Using a Cloud-Based Platform

Businesses around the world are focusing on delivering a high-quality product in less time. There is a need to continuously develop and test the product before deployment. When it comes to mobile apps, there is a lot at stake as the app must be user-friendly or else the users will immediately uninstall the app. Whether it’s an eCommerce app, banking app, health app, games, entertainment, etc., it is very crucial to test the app thoroughly before releasing it.

More than 60% of the delays happen in the testing stage of the SDLC and the main reason is that generally around 70% of the testing is done manually. Organizations are no longer following the traditional waterfall approach to soware delivery. The agile development methodology is more common and it requires continuous integrations and delivery. It accelerates release cycles, improves the quality of apps, and lead to a better overall experience for users. So let’s understand what is the basic principle of continuous testing.

The Core of Continuous Testing

Continuous testing is the process of executing automated tests to provide fast and continuous feedback to the developers negating the business risks associated with the soware. Tests are executed alongside continuous integration every day. Teams practicing continuous delivery run tests many times a day whenever the app is updated into the version control system. The developers, testers, and operations team need to collaborate and align the process with the business priorities.

The Core of Continuous Testing
The Core of Continuous Testing

Challenges Faced while Implementing Continuous Testing

Testing is one of the most important aspects but it’s usually responsible for delays in the CI/CD pipeline. Traditional testing methods are slow and unsystematic to be continuous. There can be multiple defects that can slip through production, which will harm the quality and will add to the cost.

Test Design Techniques

Test cases have to be created manually from written requirements and flat diagrams. This time-consuming activity oen produces ineffective tests and it is a barrier in continuous testing. The time required to manually create tests in a linear manner is neither scalable nor feasible.

Test cases derived manually typically cover whatever tests come in the mind of a tester. Even the experienced testers will struggle to cover a fraction of the millions of possible paths. The incomplete documentation and complex applications add to the pain. Therefore tests are repetitively testing the expected functionality while negating the scenarios that can cause a collapse.

Deficient Requirements and User Stories

Due to unspecified requirements, the majority of defects occur before a single line of code has even been written. If these defects go undetected for a long time then there will be more damage to the quality and more time it will take. Most of the requirements are stored in different file formats, with almost no dependency mapping. This happens even with the agile methodology and application lifecycle management tools.

App management tools provide a centralized repository for requirements and user stories, but the way in which the requirements are gathered is not very efficient. Most of the time users enter chunks of text into open fields and change requests are not compiled into the monolithic documents that stood at the start of traditional waterfall projects. The logical steps do not contain the normal language in a system that needs to be developed and tested.

The logical steps do not contain the normal language in a system that needs to be developed and tested. It tends to lead to ambiguity which increases the likelihood that the desired functionality will misunderstand. Defects come up in the code when this happens and the test cases might not be accurate enough to detect them. Research shows that just 4% of the average project’s investment went into establishing clear business objectives, due to which around 19% of projects failed and 52% were deemed “challenged”.

Test Data Distribution

Aer defining the test cases, testers need data to execute the cases. Test data for many organizations equate to a copy of production data, which contains only a fraction of the possible scenarios that need to be tested. This data is always out of date and exclude unexpected results as it is sampled from past scenarios in production.

If the testers had tests to cover all the features and functions, they probably won’t have the data needed to execute them. Data provisioning is oen performed in an unsystematic manner that creates cross-team dependencies with little reporting or tracking. Testers usually get data through emails or by filling out request forms in spreadsheets.

Typically a specific team is responsible for locating and provisioning the data into test environments. The central team finds the data set to fulfill a given set of test cases from among complex production sources. The process is documented in an unstructured manner and sometimes teams have to deliver tens of thousands of records a week.

Re-usability of test cases is necessary to make test automation effective. But sometimes the time spent waiting for a data refresh can be longer than the planned iteration. Testers oen rely on common data sources, once the data has been provisioned. They are therefore le frustrated when another team overwrites their data or it is lost during a refresh. A team might have to wait for another team to finish using their preferred set of data. A code change made by another team might cause their tests to fail even when there is no error in the code.

Test Automation

Executing each and every regression test manually will be a slow process and it is not feasible while releasing soware in short iterations. However, automation usually requires the effort of repetitious, complex script definition or keyword selection and it does not increase test coverage. It might even take more time then the time saved on test execution. Apart from UI, API testing must be included and it should be more data-driven. This is why automated testing must start earlier in the development lifecycle.

System Constraints

Testers need production-like systems to execute tests and they rely on a limited number of shared environments, which require costly preproduction infrastructure and take weeks to configure. Sometimes testers also require access to the unfinished, back-end or third-party components in a composite application. Testers might have to wait for them as well to become available. Some organizations use virtualization to simulate constrained components, but there is a need for realistic and consistently defined virtual data.

Defining the complex request-response pairs accurately can create significant overheads. Record and playback can be used, but this is only possible when the recorded data will not cover the future scenarios needed for rigorous testing. Scripting offers an alternative to defining complex scenarios manually, but it is slow and complex. Although, there is oen no guarantee that testers will have environments to execute test needed for rigorous testing. This approach to provisioning environments is not in line with the speed and quality expected in continuous testing and also, the ultimate user experience suffers when defects are detected late.

In large enterprises, almost a third of IT executives (32%) stated that their IT departments had ‘fully embraced Continuous Testing’. However, 58% of the enterprises surveyed deploy a new build daily (and 26% at least hourly).

Performance Testing Remains ad hoc, In-house and Incomplete

It is important for organizations to design and build applications with a view to minimal or no degradation at peak load from day one. Performance testing cannot be performed early or rigorously if legacy tools and techniques are in use. These ways of testing cannot scale to the millions of users required to test peak stress performance or to reflect actual user behavior. As a result, performance testing does not guarantee that an app and its third-party integrations can work quickly for experience-driven users.

Access to Dependent Systems and Test Environments

A further challenge is executing the automated tests. This brings you back to the issue of having available systems to execute against and depends on the ability to quickly release a system to test environments. The issue is that multiple teams use different technologies to contribute to soware delivery. This includes file continuous integration tools, management systems, databases, version control, and more. All the additional test databases, infrastructure, and virtual services need to be deployed into the test environment before moving the code.

Complex manual scripting is oen used to connect the disparate tools, but this requires constant maintenance and takes a lot of time. A lack of standardization further risks errors in deployment and more defects making it to production. Testers sometimes use an out-of-date version or build when they cannot wait for deployment. Although, this should be avoided as any tests written since the last build was deployed will either not find defects or will show test failures even if the latest code is fine.

An Inability to React to Change

Continuous testing depends on the ability to build seamlessly on the effort of past iterations when something changes. Change requests today are rarely captured formally, but rather through a continuous barrage of email requests and fragmented user stories. Developers and testers must first identify how it will affect the numerous, interdependent components of a system when a change request comes in. Lack of dependency mapping and static requirements mean that this is usually a manual, error-prone task. Any developer or tester can have subject matter expertise of the relationships between every component in a heterogeneous system. An innocuous change might then have a huge system-wide impact, requiring weeks or months of rework.

Then there is the time required to update existing test and development assets, before creating any new ones needed to implement the change. As test assets have been derived manually from requirements and are stored in formats different from the requirements, they are not traceable back to the requirements. The data and virtual services must then be updated or created for the new test pack before a new system is deployed to the test environment. In some instances, testers let all their existing tests pile up in an effort to retain coverage, but this leads to wasteful over-testing with invalid tests causing test failures. If testing is going to keep up with the rate at which user needs change, existing assets need to become more reactive to change. The effort of previous iterations must be built upon, rather than repeated, focusing on innovation.

Harmonize Continuous Testing Using a Cloud-Based Platform

One way to overcome the challenges in continuous testing and achieve quality at speed is to use advanced testing tools. There are tools for test data management and error tracking which makes it easier to organize the resources. Then there are test automation tools like Appium which makes automation easier for the testers. Another way is to use a cloud-based testing platform where you get hundreds of real mobile devices with different specifications to test the app.

This is where pCloudy comes into the picture. It is a cloud-based testing platform loaded with futuristic features to support continuous testing. Automated tests can be executed on these devices in parallel to save time and maintain continuity. The added advantage is that there is no need to spend a huge amount of money to set up a device lab as all the devices will be available on the cloud 24/7. It is integrated into popular testing tools and frameworks which helps in speeding up the automation process and hence contributing to continuous testing.

Source: pCloudy