Security Strategy & Metrics

Strategy & Metrics Level 1

The Strategy & Metrics practice encompasses planning, assigning roles and responsibilities, identifying software security goals, determining budgets, and identifying metrics and gates.  

[SM1.1: 71] Publish process (roles, responsibilities, plan), evolve as necessary.

The process for addressing software security is broadcast to all stakeholders so that everyone knows the plan. Goals, roles, responsibilities, and activities are explicitly defined. Most organizations pick and choose from a published methodology, such as the Microsoft SDL or the Synopsys Touchpoints, and then tailor the methodology to their needs. An SSDL process must be adapted to the specifics of the development process it governs (e.g., waterfall, agile, CI/CD, DevOps, etc.) because it will evolve with both the organization and the security landscape. A process must be published to count. In many cases, the methodology is controlled by the SSG and published only internally. The SSDL does not need to be publicly promoted outside of the firm to have the desired impact.

[SM1.2: 66] Create evangelism role and perform internal marketing.

In order to build support for software security throughout the organization, someone in the SSG must play an evangelism role. This internal marketing function helps keep executives and all other stakeholders current on the magnitude of the software security problem and the elements of its solution. An agile coach familiar with security, for example, could help teams adopt better software security practices as they transform to an agile methodology. Evangelists typically give talks for internal groups including executives, extend invitations to outside speakers, author white papers for internal consumption, or create a collection of papers, books, and other resources on an internal website and promote its use. A canonical example of such an evangelist was Michael Howard’s role at Microsoft just after the Gates memo.

[SM1.3: 67] Educate executives.

Executives are periodically shown the consequences of inadequate software security and the negative business impact that poor security can have. They’re also shown what other organizations are doing to attain software security, including dealing with the risks of adopting “flavor of the day” engineering methodologies with no oversight. By understanding both the problem and its proper resolution, executives can support the SSI as a risk management necessity. In its most dangerous form, such education arrives courtesy of malicious hackers or public data exposure incidents. Preferably, the SSG will demonstrate a worst-case scenario in a controlled environment with the permission of all involved (e.g., actually showing working exploits and their business impact). In some cases, presentation to the Board can help garner resources for an ongoing SSI. Bringing in an outside guru is often helpful when seeking to bolster executive attention. Tying education to specific development areas, such as mobile or cloud services, or particular methodologies, such as CI/CD and DevOps, can help convince leadership to accept SSG recommendations where they might otherwise be ignored in favor of faster release dates or other priorities.

[SM1.4: 101] Identify gate locations, gather necessary artifacts.

The software security process includes release gates/checkpoints/milestones at one or more points in an SDLC or, more likely, multiple SDLCs. The first two steps toward establishing security-specific release gates are to identify gate locations that are compatible with existing development practices and to begin gathering the input necessary for making a go/no-go decision. Importantly, the gates are not enforced at this stage. For example, the SSG can collect security testing results for each project prior to release but stop short of passing judgment on what constitutes sufficient testing or acceptable test results. Shorter release cycles, as are seen in organizations practicing CI/CD, often require creative approaches to collecting the right evidence and rely heavily on lightweight, super-fast automation. The idea of identifying gates first and enforcing them later on is extremely helpful in moving development toward software security without major pain. Socialize the gates and then turn them on once most projects already know how to succeed. This gradual approach serves to motivate good behavior without requiring it.

Strategy & Metrics Level 2

[SM2.1: 47] Publish data about software security internally.

The SSG publishes data internally about the state of software security within the organization to facilitate improvement. This information might come in the form of a dashboard with metrics for executives and software development management. Sometimes, these published data are not shared with everyone in the firm but with the relevant executives only. In such cases, publishing the information to executives who then drive change in the organization is necessary. In other cases, open book management and data published to all stakeholders helps everyone know what’s going on, with the philosophy that sunlight is the best disinfectant. If the organization’s culture promotes internal competition between groups, this information adds a security dimension to the game. The time compression associated with CI/CD calls for measurements that can be taken quickly and accurately, focusing less on historical trends (e.g., bugs per release) and more on speed (e.g.,  time to fix).

[SM2.2: 42] Enforce gates with measurements and track exceptions.

SDLC security gates are enforced for every software project: to pass a gate, a project must either meet an established measure or obtain a waiver. Even recalcitrant project teams must now play along. The SSG tracks exceptions. A gate could require a project to undergo code review and remediate any critical findings before release. In some cases, gates are directly associated with controls required by regulations, contractual agreements, and other business obligations, and exceptions are tracked as required by statutory or regulatory drivers. In other cases, gate measures yield key performance indicators that are used to govern the process. A revolving door or a rubber stamp exception process does not count. If some projects are automatically passed, that defeats the purpose of enforcing gates. Even seemingly innocuous development projects, such as a new mobile client for an existing back-end or an application ported to a cloud environment from an internal data center, must successfully pass the prescribed security gates in order to progress. Similarly, APIs, frameworks, libraries, COTS, microservices, container configurations, and so on are all software that must traverse the security gates.

[SM2.3: 44] Create or grow a satellite.

A satellite begins as a collection of people scattered across the organization who show an above-average level of security interest or skill. Identifying this group is a step toward creating a social network that speeds the adoption of security into software development. One way to begin is to track the people who stand out during introductory training courses (see [T3.6 Identify satellite through training]). Another way is to ask for volunteers. In a more top-down approach, initial satellite membership is assigned to ensure complete coverage of all development/product groups. Ongoing membership should be based on actual performance. A strong satellite is a good sign of a mature SSI. In new or fast-moving technology areas such as mobile development, or in development paradigms such as DevOps, satellite members can help combine software security skills with domain knowledge that might be underrepresented in the SSG. Agile coaches make particularly useful satellite members.

[SM2.6: 39] Require security sign-off.

The organization has an initiative-wide process for accepting security risk and documenting accountability. A risk acceptor signs off on the state of all software prior to release. For example, the sign-off policy might require the head of the business unit to sign off on critical vulnerabilities that have not been mitigated or SSDL steps that have been skipped. The policy must apply to outsourced projects, such as a boutique mobile application, and to projects that will be deployed in external environments, such as the cloud. Informal or uninformed risk acceptance alone does not count as security sign-off because the act of accepting risk is more effective when it is formalized (e.g., with a signature, form submission, or something similar) and captured for future reference. Similarly, simply stating that certain projects never need a sign-off does not achieve the desired results.

Strategy & Metrics Level 3

[SM3.1: 15] Use an internal tracking application with portfolio view.

The SSG uses a centralized tracking application to chart the progress of every piece of software in its purview, regardless of development methodology. The application records the security activities scheduled, in progress, and completed, incorporating results from activities such as architecture analysis, code review, and security testing even when they happen in a tight loop. The SSG uses the tracking application to generate portfolio reports for many of the metrics it uses. A combined inventory and risk posture view is fundamental. In many cases, these data are published at least among executives. Depending on the culture, this can cause interesting effects via internal competition. As an initiative matures and activities become more distributed, the SSG uses the centralized reporting system to keep track of all the moving parts.

[SM3.2: 7] Run an external marketing program.

The SSG helps the firm market the SSI outside to build external support. Software security grows beyond being a risk reduction exercise and instead becomes a competitive advantage or market differentiator. The SSG might write papers or books about its SSDL or have a public blog. It might also participate in external conferences or trade shows. In some cases, a complete SSDL methodology can be published and promoted externally. Mobile and cloud security projects can make great software security case studies. Sharing details externally and inviting critique can bring new perspectives into the firm.

[SM3.3: 18] Identify metrics and use them to drive budgets.

The SSG and its management choose the metrics that define and measure SSI progress. These metrics will drive the initiative’s budget and resource allocations, so simple counts and statistics won’t suffice. Metrics also allow the SSG to explain its goals and progress in quantitative terms. One such metric could be security defect density, a reduction in which could be used to show a decreasing cost of remediation over time. Recall that, in agile methodologies, metrics are best collected early and often in a lightweight manner. The key here is to tie technical results to business objectives in a clear and obvious fashion in order to justify funding. Because the concept of security is already tenuous to many business people, making an explicit tie-in can be helpful.