Challenges, Opportunities, and Outcomes for Different Penetration Testing Models

Penetration tests are an integral part of compliance initiatives like PCI-DSS and SOC 2. However, with 56% of consumers still dissatisfied with results, the issue may be less the pen-testers, as much as the “on staff.”

Check out this article for a look into the challenges, opportunities, and outcomes for the different testing models — and what you should consider before launching your next pen test.

Read on to learn:

• Why compliance is no longer the #1 reason for security testing, and what other factors play a role.
• Eight major issues with traditional penetration testing, and what they mean for security teams.
• What the results from a recent survey of security leaders tell us about the current state of penetration testing.

Content Summary

Introduction
Why Penetration Test?
Why Traditional Penetration Tests Aren’t Fit for Purpose
Penetration Testing: What are the Options?
The 2020 State of Penetration Testing
What’s Next for Security Testing?
The End of an Era

Introduction

Pentesting started in the ’90s as an adversary simulation. Its job wasn’t to find everything, it was to define the things a malicious attacker was most likely to do, and actually could do in a given system. As a result, the practice was absorbed by a variety of compliance initiatives aimed at assuring regulators, and other stakeholders that a given organization took seriously ever-evolving active threats in the wild.

In 2006 PCI-DSS mandated pen-testing and vulnerability scanning. In 2007 they published a special interest group paper that defined the relationship between the two as “a pentest is a vulnerability scan with manual confirmation of exploitability.” This created an explosion of scanner-assisted pen tests, which did increasingly less to surface new, complex threats, at increasingly high margins.

While the practical value of attack simulation itself hasn’t waned, many security leaders view the current model for resourcing and deploying pen tests to be a ‘necessary evil’. They know serious vulnerabilities are often missed during testing — but they also know that penetration testing addresses an established business need, like attainment of PCI-DSS, HIPAA, SOC 2, and other compliance initiatives.

When risk overtakes compliance

Over the last 15 years, cybersecurity has inched closer to the boardroom table. In 2020, it finally has a seat. Where previously many executives viewed the function purely as a cost center, cybersecurity’s relatively recent ability to influence the financial decisions of customers, partners, and investors has greatly elevated its status amongst the C-suite.

What changed? High-profile breaches like those of Target, Equifax, and Marriott made security tangible to even the least tech-savvy consumer. And at the same time, increasingly large fines from regulators made it clear that simply achieving compliance with industry frameworks wasn’t enough to keep an organization safe from cyber attacks.

Pulling on that thread puts traditional penetration testing in the cross-hairs. Where previously these programs were considered essential for a strong security program, the current method for resourcing and deploying them has failed to keep up with the evolution of the modern attack surface. After all, how could 1-2 penetration testers accurately mimic the activity of the entire global cybercriminal community in just a couple of weeks?

This report examines

• Why compliance is no longer the #1 reason for security testing, and what other factors play a role.
• Eight major issues with traditional penetration testing, and what they mean for security teams.
• The testing options available to modern organizations, plus their pros and cons.
• What the results of a recent Bugcrowd survey tell us about the state of penetration testing in 2020.
• How the next generation of penetration testing is addressing the shortcomings of traditional services.

Why Penetration Test?

In the past, many organizations used penetration testing primarily as a tool to achieve compliance.

However, as cybersecurity programs have evolved, they have become more risk-based. Most industry professionals understand compliance is just one of many key security objectives. They also know that satisfying a compliance framework, while essential, does little to ensure the security of the organization.

In 2020, there are six primary reasons why organizations continue to invest in penetration testing:

• Protect the organization and its assets: Cyberattacks pose a serious — even existential — threat, and any digital asset is a potential target. Penetration testing is used to identify vulnerabilities in websites, applications, and other digital systems before they can be exploited by an attacker.
• Protect customer data: Customer data is among the most important assets an organization has. Its possession is heavily regulated. Any breach of customer data is potentially devastating, as it can lead to heavy fines from industry regulators — not to mention a loss of customer trust. Penetration testing is used to find and close vulnerabilities that could otherwise be used to gain unauthorized access to customer data.
• Reduce cyber risk: Once a vague concept, cyber risk is now a calculable factor. Using tools like the Threat Category Risk framework, it can be clearly articulated as a dollar value. For organizations with a mature cybersecurity function, managing cyber risk is the #1 priority in cyber defense, and penetration testing a critical component.
• Save the organization money: Cybersecurity is no longer considered a cost center by cyber mature organizations, and for some, it has a measurable ROI. Security testing expenses can be tied directly to a reduction in the cost of incident response, remediation, and regulatory fines.
• Satisfy stakeholder requirements: Customers, suppliers, shareholders, and other stakeholders have a huge influence on the decisions an organization makes. As concepts like supply chain risk have become more widely understood, key stakeholders have increasingly demanded close attention to cyber risk management. Penetration testing plays a crucial role in this area.
• Preserve the organization’s image and reputation: Cyber incidents can fundamentally harm an organization’s ability to operate by undermining customer trust in its products, services, and brands. A major motivation for investment in penetration testing is to preserve customer trust by avoiding high-profile incidents.

Why Traditional Penetration Tests Aren’t Fit for Purpose

In November 2018, a Bugcrowd survey of 200 cybersecurity leaders1 found 56% were dissatisfied with their current penetration tests. Since then, the traditional penetration testing model has only become less effective as a tool for promoting security and managing cyber risk.

To be clear, there is a huge distinction between penetration testers — the experts who use their skills to identify security vulnerabilities — and the model through which they are deployed.

Penetration testers are an incredible resource. Bugcrowd’s ‘Crowd’ of security experts and ethical hackers includes thousands of penetration testers. If they could, every organization would have dozens of penetration testers working full-time to identify vulnerabilities in its digital assets. However, this approach is cost-prohibitive and logistically impossible. There simply aren’t enough penetration testers in the industry.

As important as penetration testers are, however, the traditional penetration testing service model no longer meets the needs of modern organizations. Here’s why:

• Scheduling delays: Organizations are frequently forced to accept long wait times (up to months) for each testing period. As penetration testing providers seek to reduce time on the ‘bench’ for salaried employees, getting resources where and when needed is a perpetual challenge.
• Incompatible incentives: The provider’s need to reduce overhead can result in the assignment of those not suited for the engagement at hand. Unfortunately, the only thing protecting customers from the caveat emptor nature of this model is the provider’s desire to win the renewal next year.
• Speed of results: With a standard penetration test, the customer doesn’t receive results until the engagement is concluded, often 14-24 days after testing begins. This leaves tested assets vulnerable for an unnecessarily long time.
• Questionable skill fit: A typical penetration test is carried out by 1-2 testers over two weeks. Regardless of how experienced the testers are, they can’t be versed in every possible attack technique, and their skill sets may not be appropriate to the asset being tested. Equally, customers don’t have the option to select which testers are assigned to their projects.
• Checklist focused: Most penetration tests are checklist-based, with minimal time or incentive for testers to use their initiative or ‘dig deeper’ to find complex vulnerabilities.
• Point-in-time testing: Most digital assets are penetration tested a maximum of 1-2 times per year. With modern, agile development lifecycles, new codebase versions are released much more frequently. While an asset may be secure immediately following a test, new code releases could leave it vulnerable to attack until the next scheduled test.
• Lack of incentive: Traditional penetration testing providers operate a ‘pay for time’ business model, where customers pay for a certain number of hours, and the assigned tester is only required to finish the methodology at that time. The number and severity of vulnerabilities surfaced during this time is irrelevant to the tester’s final pay.
• Lack of SDLC integration: Traditional penetration tests aren’t constructed in a way that actively integrates security and development teams. Developers must manually migrate vulnerabilities to their preferred workspace (e.g., JIRA or ServiceNow) before ‘sifting through’ a long report lacking context, priority, and guidance on how to safely resolve.
• Poor results: A typical penetration test finds just eight high-value, unknown vulnerabilities on average. These valid findings are interspersed with false positives and no-risk issues, making them hard to identify and resolve. Worse, many genuine high-risk vulnerabilities are simply not identified.

Due to poor results, high cost, and time delays, traditional penetration testing services are not cost-effective security control. Worse, because skill fit for a project is likely sub-optimal and testers aren’t incentivized to ‘go deep,’ it’s likely that genuine, high-risk vulnerabilities will be missed.

Given this, the traditional penetration testing model is simply ineffective for Cyber Risk Management.

Penetration Testing: What are the Options?

While security testing has become synonymous with traditional penetration testing services, there are four primary testing options.

Traditional Penetration Testing

Despite the issues raised in the last section, many organizations still rely on traditional penetration testing services. The ‘traditional’ model consists of one or two testers working against a set methodology for a defined period, usually anywhere from three days to two weeks. This format is a mainstay of the security industry, and at this point, executives and business leaders are pre-sold on the need for it.

Pros:

• Established budget line item
• A known quantity
• Best for targets that require physical presence to access/test

Cons:

• Delays to scheduling and results
• Inflexible with questionable skill fit
• Not optimized to incentivize true risk reduction

Crowdsourced Security Penetration Testing

The crowdsourced security penetration test is a comparatively new method of testing. Crowdsourced options utilize a large pool of pay-per-project testers that work remotely. Often combined with an additionally incentivized ‘pay for results’ approach to billing, crowdsourced testing is becoming the go-to choice for security-conscious organizations.

Pros:

• Rapid setup and time to value
• Real-time results and SDLC integration
• Option to ‘pay for results’ instead of time

Cons:

• Not optimized for highly sensitive or physical targets too big to ship
• ‘Bounty’ approach may not fit buying cycles
• The new business case may be required

Internal Security Testing

While often not feasible for smaller organizations, some enterprises prefer to build and maintain in-house teams of security testers. This approach allows the organization to set its testing schedule and may reduce barriers in some areas, e.g., provision of credentials.

Pros:

• Best for extremely sensitive work (Secret, NOFORN)
• Tests can be run as frequently as needed
• The little marginal cost of testing

Cons:

• Labor-intensive to set up and maintain
• Impossible to retain all possible testing skills
• Hard to acquire new skills when needed

A Mixed Testing Approach

Some organizations use a combination of traditional, crowdsourced, and internal testing to meet the specific needs of each project.

Pros:

• Includes the best aspects of each method
• Potential for thorough security coverage
• Testing depth is as-needed for each project

Cons:

• Includes the worst aspects of each method
• Complex to arrange and maintain
• (Potentially) extremely high-cost

The 2020 State of Penetration Testing

In March 2020, we surveyed 129 cybersecurity engineers, managers, and CISOs to find out how they conduct their penetration testing. All of our respondents influenced their organization’s security testing budget, methodology, and scope. Here’s what we learned:

Compliance is no longer the #1 reason for testing

While 55% of respondents cited compliance as one of their reasons for testing, only 16% test purely for compliance purposes. Meanwhile, 61% of respondents cited best practice as a reason for testing, and 38% cited stakeholder requirements.

Traditional penetration testing services are still #1… just.

In the past, traditional penetration testing was a dominant force in security. However, recently, other approaches have gained popularity. Our survey shows that in 2020, across all industries and organization sizes, traditional penetration testing services account for just 35% of security testing.

Crowdsourced testing has jumped into second place at 25%, despite being around for a comparatively short time. 24% of organizations complete most of their testing internally, while 15% use a mixture of testing approaches.

Larger organizations are moving away from traditional penetration testing services.

While other options are catching up, traditional penetration testing is the most common testing method among small organizations with under 1,000 employees. For larger organizations — where increased budgets open up more options — things are less clear.

Traditional penetration testing and crowdsourced testing are both utilized by just under a third of organizations with 1-10k employees. Considering that crowdsourced testing is a far more recent option, this highlights a rapid movement away from traditional penetration testing services.

At the enterprise level (10K + employees) the percentage of organizations relying on traditional penetration testing services is barely half the rate we see in small organizations (21% vs. 39%). It’s also dead equal with the percentage of enterprises using crowdsourced testing. Meanwhile, more than half (57%) of enterprises rely primarily on internal security testing.

Crowdsourced testing finds more, higher-value vulnerabilities.

When it comes to results, crowdsourced testing is the clear winner. 76% of crowdsourced testers received at least 10 vulnerabilities per two-week test, compared to 57% of traditional penetration testing services.

The quality of results was also higher. Only a small fraction (13%) of crowdsourced testers received less than 5% high-value vulnerabilities, while traditional penetration testing services were twice as likely to deliver a poor result. Meanwhile, crowdsourced testing was 60% more likely than traditional penetration testing services to deliver a large proportion (26%+) of high-value vulnerabilities.

Internal testing programs performed extremely poorly on both the quality and quantity of results, despite their popularity with enterprises.

Traditional penetration testing is favored by infrequent testers.

66% of organizations that use traditional penetration testing services test very infrequently — once per year or less. By contrast, over half (52%) of organizations that use crowdsourced testing test at least quarterly. Organizations that test internally are the most frequent testers, with 60% testing at least quarterly.

Cost is comparable, but ROI isn’t

Our respondents placed traditional penetration testing neck-and-neck with crowdsourced testing on the total cost. However, since crowdsourced delivers more, higher-quality results, it’s a clear winner for ROI.

While the cost of maintaining an internal testing capability varies, there’s no question that it falls beyond what most organizations can afford. And, given its poor results, the ROI is questionable at best.

What can we learn from this?

Larger organizations recognize the issues with traditional penetration testing services… but haven’t chosen the best alternative. Internal testing performs poorly on the number and quality of vulnerabilities found.

Crowdsourced testing finds more and higher-value vulnerabilities than traditional penetration testing services, internal security testing, and mixed programs.

Crowdsourced testing offers higher ROI than other methods, as costs remain comparable while results are consistently better.

What’s Next for Security Testing?

Traditional penetration tests don’t meet the needs of modern organizations. A different solution is needed.

Crowdsourced testing approaches such as bug bounty programs have addressed many of the shortcomings of traditional penetration tests. By operating on a pay-for-findings model, these programs harness the power of the global hacking community to provide on-demand access to the expertise needed for each engagement.

However, bug bounty programs haven’t fully replaced the need for standardized testing. Compliance is still a crucial part of security, and most frameworks demand that testing follows a recognized methodology.

The next-generation penetration test

While many organizations share a need to achieve compliance milestones, not all have the same testing requirements or capacity. Some seek continuous coverage, to match increasingly rapid development cycles. Others need shorter testing windows throughout the year, as dictated by engineering workflows or budgetary and procurement cycles. Equally, an organization’s appetite for tester incentivization may be shaped by its bandwidth to process more vulnerabilities, as well as flexibility in maintaining an elastic pool of monetary rewards.

To address these varied needs, Bugcrowd has launched the next generation of penetration testing. One that taps into the diverse expertise of the global hacking community, while providing methodology-based coverage and essential compliance reporting. And vitally, one where the customer chooses the terms.

Crowd-powered Penetration Testing

Next-generation Penetration Test

• Continuous coverage and on-demand methodology-driven testing, with re-testing included
• Testers incentivized by reward for valid vulnerabilities
• Options for Premium SLAs and Coverage Analysis
• Cost: platform + incentive pool

‘Classic’ Penetration Test

• On-demand methodology-driven testing over a defined period based on project scope
• Options for re-testing and expedited reporting
• Cost: Per-day, no incentive pool

Both

• QSA-assessed compliance report: Meet PCI-DSS, NIST 800-53 rev4, ISO 27001 and more
• Set up in <72 hours on average: Avoid lengthy scheduling delays and receive results faster
• Streaming results: Receive vulnerabilities upon discovery and validation
• SDLC integrations: Push vulnerabilities to the places your developers live like GitHub and ServiceNow
• Remediation advice: Help development fix quickly with prescriptive instructions based on vulnerability type
• CrowdMatchTM: Draw from the largest pool of talent and ensure skills and experience match project needs.
• On-demand in-platform reporting: Monitor vulnerability status and program activity
• Fully managed: Bugcrowd handles pentester matching, activation, and remuneration, as well as vulnerability triage and prioritization

Harnessing the Crowd

Harnessing the power of the global hacking community requires structure, process, and deep experience in human-to-human interaction. All Bugcrowd’s crowd-powered penetration tests utilize the Crowdcontrol™ platform which includes dedicated program management. This combination of technology-enabled expertise enables us to provide:

• Thorough vetting and expert skills-matching of every crowdsourced penetration tester.
• Rapid triage, validation, and risk-ranking of all discovered vulnerabilities.
• Several software Development integrations for faster remediation.
• Rapid time to value as results are streamed immediately post-validation (not at program end).
• Full program onboarding, clearly defined SLAs, and dispute resolution.
• Full, real-time visibility into team activity, program outcomes, and costs.

Combined, these factors enable crowd-powered penetration tests to identify on average 7X more high-priority vulnerabilities than traditional penetration tests.

The End of an Era

Once the gold standard for cybersecurity, traditional penetration testing now falls far short of what’s needed by a modern organization. From cost to time, to quality of results, these services simply are not an effective tool for improving security outcomes or managing cyber risk. With the emergence of alternative methods, a compliance report alone no longer justifies the opportunity cost of a test that fails to deliver real results.

Penetration testing providers must evolve to provide both compliance assurance, as well as deep security insights. Modern, agile development lifecycles have highlighted the inability of traditional providers to adapt to this new world in a way that is both functional and cost-effective. A new model is required. By leveraging an elastic network of fully managed premium testing talent, Crowdsourced security platforms offer organizations a faster path to compliance without sacrificing the critical insights that help keep products and customers safe.

In 2020, crowdsourced security testing has already caught up with traditional penetration testing services in the enterprise market and is rapidly closing the gap with smaller organizations. In the coming years, this trend will only continue as more organizations recognize the shortcomings of traditional penetration testing services and begin to evaluate their options.

Key Takeaways

• Security testing is no longer purely for compliance: Modern organizations have to balance compliance with other needs, including customer and stakeholder requirements, financial concerns, and cyber risk management.
• Organizations have taken to mixing methods for penetration testing: To fill the gaps left by traditional testing services, modern organizations have begun incorporating other methods where appropriate, for example, crowdsourced testing, internal testing, and hybrid testing programs.
• Crowdsourced testing delivers more and higher quality vulnerabilities: Users of crowdsourced security programs report a greater volume of higher-quality vulnerabilities than traditional penetration testing services to provide.
• Traditional penetration testing services are losing popularity: Traditional services are just barely holding the top spot, while organizations are increasingly incorporating or switching to crowdsourced methods.
• Crowdsourced penetration testing is gaining traction with organizations of all sizes: Crowdsourced programs now account for between 20 – 30% of all security testing, depending on organization size.

Source: Bugcrowd: #1 Crowdsourced Cybersecurity Platform