Summer 2019 Guide to Automated Testing and Facilitating Continuous Delivery

Do you know where your team stacks up against your peers when it comes to test automation? Perfecto partnered with DZone to bring you The 2019 Guide to Automated Testing to give you insight to where you fit in.

Summer 2019 Guide to Automated Testing and Facilitating Continuous Delivery

Read on this article and you’ll learn:

  • Which types of tests teams decide to automate and why.
  • Where teams are at in their transition to DevOps.
  • Signs your test automation is failing you.
  • Trends in testing strategies and technology.

Content Summary

Demographics
Automated and Manual Testing: A Continued Shift Left
The Evolving Role Of Devops
Testing Strategies
Technology Adoption

Key Research Findings

Demographics

For this year’s Automated Testing Survey, we polled 498 IT professionals, with a survey completion rating of 73%. Below is some of the basic demographic information based on the responses of these survey takers.

  • On average, respondents have 17 years of experience in the field.
  • Respondents work for organizations based in two main geographic regions
    — the USA (41%) and Europe (32%)
    — while they live in three main geographic regions
    — Europe (34%), the USA (27%), and South Central Asia (14%).
  • Respondents tend to work for enterprise-level organizations:
    – 23% work for orgs sized 1,000-9,999
    – 22% work for organizations sized 10,000+
    – 20% work for organization sized 100-499
    – 60% of respondents work on teams of 10 people or fewer.
  • Respondents are engaged in developing three main types of software:
    – 82% develop web applications
    – 46% are working on enterprise business applications
    – 25% develop native mobile apps
  • Survey takers tend to perform one of three main roles:
    – 34% are developers/engineers
    – 20% are developer team leads
    – 16% work as architects
  • Three main programming language ecosystems are used by respondents’ organizations:
    – 78% use Java
    – 69% use client-side JavaScript
    – 40% use Python
  • The ratio of respondents who work at an org with a dedicated DevOps team is 50:50.

Automated and Manual Testing: A Continued Shift Left

Even though the shift-left mentality and methodology has been around for several years, it continues to gain momentum. In 2018, when we asked respondents at what stage in the SDLC do they typically begin automated testing, 54% reported development and 31% said staging/ QA/testing. In this year’s survey, 66% of respondents told us that they begin automated testing on their software during development, and 22% begin in staging/QA/testing. Interestingly, while automated testing continues its rather seismic shift to the left, manual testing methodologies saw little to no change over last year’s survey. In 2018, 51% reported beginning manual testing in development, while 40% reported beginning in staging/QA/testing. This year, 49% reported performing manual testing in development and 44% claimed to do so in staging/QA/testing. But, again, neither of these year-over-year changes are statistically significant enough to signify a large shift in manual testing’s place in the SDLC.

At which stage does your organization usually begin automatically testing software?
At which stage does your organization usually begin automatically testing software?
At which stage does your organization usually begin manually testing software?
At which stage does your organization usually begin manually testing software?

While the above highlights the usage patterns of those developers who have adopted either automated or manual testing, we also gathered data around why certain respondents’ organizations opted against one or the other of these testing strategies. Among those survey takers whose organizations/teams do not perform automated testing, the most common reason for doing so, reported by 28%, is that it is not considered a critical priority. In 2018, 33% reported not considering automated testing a critical priority. This matches up nicely with the continued adoption and leftward shifting of automated testing discussed above. When it comes to those whose organizations do not perform manual testing, one statistic jumps of the page. In 2018, 7% of respondents reported the main reason for not adopting manual testing was the lack of a QA/testing team. In 2019, 43% told us they had no dedicated QA/testing team. This seems a troubling stat. One possible answer to this large year-over-year change is that, as testing is continuously shifted left into the development phase of the SDLC, developers are becoming de facto testers on top of their roles of software engineers.

Now that we’ve covered where automated and manual tests occur in the SDLC at a high-level, let’s drill down a bit to examine the types of tests that are automated and performed manually. Among those respondents who perform automated testing, four main tests were reported: integration, component, performance, and story-level. Comparing the data we received around these tests to our historical data from 2018, we find that the percentage of those using automated integration tests stayed stable (77% in 2018 vs. 78% in 2019), while automated story-level tests (36% in 2018 vs. 46% in 2019) and component tests (52% in 2018 vs. 59% in 2019) both grew in popularity. Automated performance tests, however, fell slightly year-over-year, dropping from a 51% adoption rate in 2018 to 46% in 2019. Among those performing manual tests, four types of tests proved predominant: user acceptance tests, usability tests, story-level tests, and postdeployment tests. Interestingly, despite the seemingly stagnant state of manual testing outlined above, we found that all four of these manual tests grew year-over-year in terms of adoption rates. Here’s a quick breakdown of this historical data:

  • User Acceptance Tests
    – 2018: 60%
    – 2019: 67%
  • Usability tests
    – 2018: 50%
    – 2019: 58%
  • Story-level tests
    – 2018: 41%
    – 2019: 47%
  • Post-deployment tests
    – 2018: 36%
    – 2019: 40%

The Role Of Devops In Automated Testing

The Evolving Role Of Devops

As noted in the Demographics section at the beginning of this report, the ratio of respondents’ organizations who have and don’t have a dedicated DevOps team is an even 50-50 split, which didn’t change significantly from last year (in 2018 48% of respondents reported having a DevOps team). But for those who do have dedicated DevOps teams, the role of these teams is gravitating more to a cultural role than a technological one. In our 2018 Automated Testing survey, 67% of survey takers reported that the main goal of their DevOps team was to help the organization adopt the best Continuous Delivery tools; in 2019, this rose to 74%. Additionally, last year 46% of respondents told us that their DevOps team worked to increase the collaboration and break down silos between Dev and Ops; this year, that figure rose to 53%. All the other, more technical, DevOps team functions either did not grow or shrank in importance. The most shocking, however, was the decreased impetus to develop and deliver software across the entire stack. In 2018, 52% reported this a main goal of their DevOps team; this year, 42% said this type of software delivery was a main goal for their DevOps team. Also, improving frequency of deployments (62% in 2018 vs. 63% in 2019) and introducing automation across the SDLC (56% in both 2018 and 2019) remained of equal importance year-over-year.

Despite the growing cultural role of DevOps in enterprise software organizations, it still has a large technological part to play. If we compare the data on those respondents whose organizations have dedicated DevOps teams to the stats we gathered on respondents’ software build processes, DevOps’s continued role in automation becomes clear. Of the 58% of respondents who told us they break their builds up into stages, 34% have a dedicated DevOps team, and of the 46% who include automatic checks to proceed, 21% have a DevOps team. Additionally, while a total of 34% of respondents have security issue detection built into their SDLC, 21% of those respondents have a DevOps team in their organizations create agility and automation throughout their build process and implement proper testing and security practices.

Testing Strategies

Much like the role of DevOps, the role of testing within the SDLC seems to be undergoing an evolution. In 2018, 28% of respondents reported to practice test-driven development (TDD) across all their development efforts. In 2019, this number fell to 22%, while the percentage of respondents who do not practice TDD at all remained stable (36% in 2018 vs. 34% in 2019). The number of respondents using TDD selectively, however, saw a year-over-over increase. In 2018, 36% of survey takers told us they use TDD “only in certain teams.” In this year’s survey, that number rose to 45%. If we compare these numbers to our data on organizations with DevOps teams, we find that those orgs who have dedicated DevOps teams are more likely to practice TDD across the board than those that don’t. 15% of those respondents who use TDD on all their projects also have a dedicated DevOps team, versus 7% who have no DevOps team, but still adhere to TDD for all their development efforts. Interestingly, TDD adoption among “only certain teams” proved equally popular among organizations with and without DevOps teams.

Despite this year-over-year shift in the frequency of test-driven development efforts, the benefits that respondents reported to derive from TDD remained the same. Just as in our 2018 survey, the top four benefits of TDD were: improved quality of code (80%); less time spend debugging (59%); easily maintainable code (49%); the ability to use tests as documentation (45%). Intriguingly, after running these numbers against our stats on DevOps adoption, only one of these benefits seems to occur more frequently in organizations that had dedicated DevOps teams. Among those who reported seeing improved code quality due to TDD, 54% work for organizations with DevOps teams, while 46% do not.

Which tests in your organization's pipeline(s) are currently automated?
Which tests in your organization’s pipeline(s) are currently automated?
Does your organization practice test-driven-development (TDD)?
Does your organization practice test-driven-development (TDD)?

Moving from TDD to the continuous testing paradigm, however, we see DevOps have a more considerable effect. When we asked survey takers which of the four continuous testing pillars they saw as the most challenging, we saw the following results (number one receiving the most votes for “most challenging,” with number four receiving the least votes for “most challenging”):

  1. Maintaining a stable and up-to-date test lab
  2. Test automation creation
  3. Test execution and orchestration
  4. Test report analysis

Among those respondents who reported to work in an organization with a dedicated DevOps team, we see these results turned almost upside down. If we look at the four pillars of continuous testing through this DevOps lens (again, with number one receiving the most votes for “most challenging,” with number four receiving the least votes for “most challenging”), we see the following:

  1. Test execution and orchestration
  2. Test report and analysis
  3. Maintaining a stable and up-to-date test lab
  4. Test automation creation

Tellingly, those survey takers who work for companies with a DevOps team found test automation the least challenging of the four pillars.

Which of the four containers testing pillars is the most challenging for you?
Which of the four containers testing pillars is the most challenging for you?

Technology Adoption

In order to continue to make strides in the areas delineated above, organizations have begun a shift toward the use of microservices and containers. And while these technologies do not necessitate the use of a DevOps paradigm, DevOps teams have found them a boon to their efforts. In terms of microservices adoption, among the general survey population 35% reported adopting a microservice architecture for some projects. But, of even more consequence, we found that 20% of survey takers have both had a dedicated DevOps team and had adopted microservices for some projects. Thus, well over half of respondents who have adopted microservices on certain projects also work in a DevOps development environment. When we look to containers, we see much the same pattern play out. Among the general population, 54% reported to have adopted container technology, while 32% of all respondents both had adopted containers and had a dedicated DevOps team. Again, over half of the users of this technology work in a DevOps-enabled organization. Additionally, when we asked respondents in what environments they use containers, 49% reported to use containers in a DevOps environment.

Do you have an officially designated DevOps team in your organization?
Do you have an officially designated DevOps team in your organization?

Source: DZone and Perfecto