Penetration Testing

The Penetration Testing practice involves standard outside-in testing of the sort carried out by security specialists. Penetration testing focuses on vulnerabilities in the final configuration and provides direct feeds to defect management and mitigation.

Penetration Testing Level 1

[PT1.1: 105] Use external penetration testers to find problems.

Many organizations aren’t willing to address software security until there’s unmistakable evidence that the organization isn’t somehow magically immune to the problem. If security has not been a priority, external penetration testers can demonstrate that the organization’s code needs help. Penetration testers could be brought in to break a high-profile application to make the point. Over time, the focus of penetration testing moves from “I told you our stuff was broken” to a smoke test and sanity check done before shipping. External penetration testers bring a new set of eyes to the problem.

[PT1.2: 89] Feed results to the defect management and mitigation system.

Penetration testing results are fed back to development through established defect management or mitigation channels, and development responds via a defect management and release process. Emailing them around doesn’t count. Properly done, the exercise demonstrates the organization’s ability to improve the state of security, and many firms are beginning to emphasize the critical importance of not just identifying but actually fixing security problems. One way to ensure attention is to add a security flag to the bug-tracking and defect management system. Evolving DevOps and integrated team structures do not eliminate the need for formalized defect management systems.

[PT1.3: 74] Use penetration testing tools internally.

The organization creates an internal penetration testing capability that uses tools. This capability can be part of the SSG or part of a specialized team elsewhere in the organization, with the tools improving the efficiency and repeatability of the testing process (and a frequently necessary part of CI/CD environments). Tools can include off-the-shelf products, standard-issue network penetration tools that understand the application layer, and handwritten scripts. Free-time or crisis-driven efforts do not constitute an internal capability.  

Penetration Testing Level 2

[PT2.2: 26] Provide penetration testers with all available information.

Penetration testers, whether internal or external, can do deeper analysis and find more interesting problems after they receive source code, design documents, architecture analysis results, and code review results. Penetration testers need everything that is created throughout the SSDL. If your penetration tester doesn’t ask for the code, you need a new penetration tester.

[PT2.3: 21] Schedule periodic penetration tests for application coverage.

The SSG periodically tests all applications in its purview according to an established schedule, which could be tied to a calendar or a release cycle. High-profile applications might get a penetration test at least once a year. This testing serves as a sanity check and helps ensure that yesterday’s software isn’t vulnerable to today’s attacks; it also helps maintain the security of software configurations and environments, especially containers and components in the cloud. One important aspect of periodic testing is to make sure that the problems identified are actually fixed and don’t creep back into the build. New automation created for CI/CD deserves penetration testing as well.  

Penetration Testing Level 3

[PT3.1: 10] Use external penetration testers to perform deep-dive analysis.

The organization uses external penetration testers to do deep-dive analysis for critical projects and to introduce fresh thinking into the SSG. These testers are experts and specialists who keep the organization up to speed with the latest version of the attacker’s perspective and have a track record for breaking the type of software being tested. Skilled penetration testers will always break a system, but the question is whether they demonstrate new kinds of thinking about attacks that can be useful when designing, implementing, and hardening new systems. Creating new types of attacks from threat intelligence and abuse cases prevents checklist-driven approaches that only look for known types of problems; it’s pretty much essential when it comes to new technology.

[PT3.2: 7] Have the SSG customize penetration testing tools and scripts.

The SSG either creates penetration testing tools or adapts publicly available ones to more efficiently and comprehensively attack the organization’s systems. Tools improve the efficiency of the penetration testing process without sacrificing the depth of problems that the SSG can identify. Automation can be particularly valuable under agile methodologies because it helps teams go faster. Tools that can be tailored are always preferable to generic tools. This activity considers both the depth of tests and their scope.