Software Security Testing

The Security Testing practice is concerned with prerelease testing, including integrating security into standard quality assurance processes. The practice includes use of black-box security tools (including fuzz testing) as a smoke test in QA, risk-driven white-box testing, application of the attack model, and code coverage analysis. Security testing focuses on vulnerabilities in construction.

Security Testing Level 1

[ST1.1: 100] Ensure QA supports edge/boundary value condition testing.

The QA team goes beyond functional testing to perform basic adversarial tests and probe simple edge cases and boundary conditions, no attacker skills required. When QA understands the value of pushing past standard functional testing using acceptable input, it begins to move slowly toward thinking like an adversary. A discussion of boundary value testing leads naturally to the notion of an attacker probing the edges on purpose. What happens when you enter the wrong password over and over?

[ST1.3: 88] Drive tests with security requirements and security features.

Testers target declarative security mechanisms with tests derived from requirements and security features. A tester could try to access administrative functionality as an unprivileged user, for example, or verify that a user account becomes locked after some number of failed authentication attempts. For the most part, security features can be tested in a fashion similar to other software features; security mechanisms based on requirements such as account lockout, transaction limitations, entitlements, and so on are also tested. Of course, software security is not security software, but getting started with features is easy. New deployment models, such as cloud, might require novel test approaches.

Security Testing Level 2

[ST2.1: 30] Integrate black-box security tools into the QA process.

The organization uses one or more black-box security testing tools as part of the QA process. Such tools are valuable because they encapsulate an attacker’s perspective, albeit generically; tools such as IBM Security AppScan or Fortify WebInspect are relevant for web applications, and fuzzing frameworks such as Synopsys Defensics are applicable for most network protocols. In some situations, other groups might collaborate with the SSG to apply the tools, for example, a testing team could run the tool but come to the SSG for help interpreting the results. Because of the way testing is integrated into agile development approaches, black-box tools might be used directly by engineering. Regardless of who runs the black-box tool, the testing should be properly integrated into the QA cycle of the SSDL.

[ST2.4: 14] Share security results with QA.

The SSG routinely shares results from security reviews with the QA department. Using security results to inform and evolve particular testing patterns can be a powerful mechanism leading to better security testing. CI/CD makes this easier because of the way testing is integrated in a cross-functional team. Over time, QA engineers learn the security mindset, and this activity benefits from an engineering-focused QA function that is highly technical.

[ST2.5: 12] Include security tests in QA automation.

Security tests run alongside functional tests as part of automated regression testing. In fact, the same automation framework houses both, and security testing is part of the routine. Security tests can be driven from abuse cases identified earlier in the lifecycle or tests derived from creative tweaks of functional tests.

[ST2.6: 13] Perform fuzz testing customized to application APIs.

Test automation engineers or agile team members customize a fuzzing framework to the organization’s APIs. They could begin from scratch or use an existing fuzzing toolkit, but customization goes beyond creating custom protocol descriptions or file format templates. The fuzzing framework has a built-in understanding of the application interfaces it calls into. Test harnesses developed explicitly for particular applications make good places to integrate fuzz testing.

Security Testing Level 3

[ST3.3: 4] Drive tests with risk analysis results.

Testers use architecture analysis results to direct their work. If the architecture analysis concludes that “the security of the system hinges on the transactions being atomic and not being interrupted partway through,” for example, then torn transactions will become a primary target in adversarial testing. Adversarial tests like these can be developed according to risk profile, with high-risk flaws at the top of the list.

[ST3.4: 3] Leverage coverage analysis.

Testers measure the code coverage of their security tests (see [ST2.5 Include security tests in QA automation]) to identify code that isn’t being exercised. Code coverage analysis drives increased security testing depth. Standard-issue black-box testing tools achieve exceptionally low coverage, leaving a majority of the software under test unexplored, which is not a testing best practice. Using standard measurements for coverage such as function coverage, line coverage, or multiple condition coverage is fine.

[ST3.5: 3] Begin to build and apply adversarial security tests (abuse cases).

Testing begins to incorporate test cases based on abuse cases (see [AM2.1 Build attack patterns and abuse cases tied to potential attackers]), and testers move beyond verifying functionality and take on the attacker’s perspective. One way to do this is to systematically attempt to replicate incidents from the organization’s history. Abuse and misuse cases based on the attacker’s perspective can also be driven from security policies, attack intelligence, and standards. This turns the corner from testing features to attempting to break the software under test.