Strategy & Metrics

Strategy & Metrics Level 1

The Strategy & Metrics practice encompasses planning, assigning roles and responsibilities, identifying software security goals, determining budgets, and identifying metrics and software release conditions.

[SM1.1: 94] Publish process and evolve as necessary.

The process for addressing software security is published and broadcast to all stakeholders so that everyone knows the plan. Goals, roles, responsibilities, and activities are explicitly defined. Most organizations pick an existing methodology, such as the Microsoft SDL or the Synopsys Touchpoints, and then tailor it to meet their needs. Security activities, such as those grouped into an SSDL process, are adapted to software lifecycle processes (e.g., waterfall, agile, CI/CD, DevOps) so activities will evolve with both the organization and the security landscape. In many cases, the process is defined by the SSG and published only internally; it doesn’t need to be publicly promoted outside the firm to have the desired impact. In addition to publishing the written process, some firms also encode it into an application lifecycle management (ALM) tool as software-defined workflow.

[SM1.2: 70] Create evangelism role and perform internal marketing.

The SSG builds support for software security throughout the organization through ongoing evangelism. This internal marketing function, often performed by a variety of stakeholder roles, keeps executives and others up to date on the magnitude of the software security problem and the elements of its solution. A scrum master familiar with security, for example, could help teams adopt better software security practices as they transform to an agile methodology. Evangelists can increase understanding and build credibility by giving talks to internal groups (including executives), publishing roadmaps, authoring white papers for internal consumption, or creating a collection of papers, books, and other resources on an internal website and promoting its use.

[SM1.3: 75 Educate executives.

Executives are regularly shown the ways malicious actors attack software and the negative business impacts they can have on the organization. They’re also shown what other organizations are doing to mature software security, including how they deal with the risks of adopting emerging engineering methodologies with no oversight. By understanding both the problems and their proper resolutions, executives can support the SSI as a risk management necessity. In its most dangerous form, security education arrives courtesy of malicious hackers or public data exposure incidents. Preferably, the SSG will demonstrate a worst-case scenario in a controlled environment with the permission of all involved (e.g., by showing working exploits and their business impact). In some cases, presentation to the Board can help garner resources for a new or ongoing SSI efforts. Bringing in an outside guru is often helpful when seeking to bolster executive attention. Tying education to specific development areas, such as DevOps groups using cloud-native technologies, can likewise help convince leadership to accept SSG recommendations when they might otherwise be ignored in favor of faster release dates or other priorities.

[SM1.4: 117] Implement lifecycle governance.

The software security process includes conditions for release (such as gates, checkpoints, guardrails, milestones, etc.) at one or more points in a software lifecycle. The first two steps toward establishing security-specific release conditions are to identify locations that are compatible with existing development practices and to then begin gathering the input necessary to make a go/no-go decision, such as risk ranking thresholds or defect data. Importantly, the conditions might not be verified at this stage. For example, the SSG can collect security testing results for each project prior to release, then provide their informed opinion on what constitutes sufficient testing or acceptable test results without trying to stop a project from moving forward. In CI/CD environments, shorter release cycles often require creative approaches to collecting the right evidence and rely heavily on automation. The idea of defining governance checks in the process first and enforcing them later is extremely helpful in moving development toward software security without major pain (see [SM2.2 Verify release conditions with measurements and track exceptions]). Socializing the conditions and then verifying them once most projects already know how to succeed is a gradual approach that can motivate good behavior without requiring it.

Strategy & Metrics Level 2

[SM2.1: 64] Publish data about software security internally.

To facilitate improvement, data are published internally about the state of software security within the organization. This information might come in the form of a dashboard with metrics for executives and software development management. Sometimes, these published data won’t be shared with everyone in the firm but only with relevant executives who then drive change in the organization. In other cases, open book management and data published to all stakeholders help everyone know what’s going on, the philosophy being that sunlight is the best disinfectant. If the organization’s culture promotes internal competition between groups, this information can add a security dimension. Increasingly, security telemetry is used to gather measurements quickly and accurately, and might initially focus less on historical trends (e.g., bugs per release) and more on speed (e.g., time to fix) and quality (e.g., defect density). Some SSIs might publish these data primarily in engineering groups within pipeline platform dashboards, democratizing measurement for developer self-improvement.

[SM2.2: 61] Verify release conditions with measurements and track exceptions.

Security release conditions (or gates, checkpoints, guardrails, milestones, etc.) are verified for every project, so each project must either meet an established measure or obtain a waiver in order to move forward normally, and the SSG tracks exceptions. In some cases, measures are directly associated with regulations, contractual agreements, and other obligations, with exceptions tracked as required by statutory or regulatory drivers. In other cases, measures yield some manner of KPIs that are used to govern the process. Allowing any projects to automatically pass or granting waivers automatically without due consideration defeats the purpose of verifying conditions. Even seemingly innocuous software projects must successfully satisfy the prescribed security conditions in order to progress to or remain in production. Similarly, APIs, frameworks, libraries, bespoke code, microservices, container configurations, and so on are all software that must satisfy security release conditions. It’s possible, and often very useful, to have verified the conditions both before and after the development process itself. In modern development environments, the measurement process for conditions will increasingly become automated (see [SM3.4 Integrate software-defined lifecycle governance]).

[SM2.3: 55] Create or grow a satellite.

Create a collection of people scattered across the organization who show an above-average level of security interest or skill—a satellite. Forming this group of advocates, sometimes referred to as champions, is a step toward scaling security by creating a social network that speeds the adoption of security into software engineering. One way to build the initial group is to track the people who stand out during introductory training courses; see [T3.6 Identify new satellite members through observation]. Another way is to ask for volunteers. In a more top-down approach, initial satellite membership is assigned to ensure good coverage of development groups, but ongoing membership is based on actual performance. A strong satellite is a good sign of a mature SSI. The satellite can act as a sounding Board for new SSG projects and, in new or fast-moving technology areas, can help combine software security skills with domain knowledge that might be under-represented in the SSG or engineering teams. Agile coaches, scrum masters, and DevOps engineers can make particularly useful satellite members, especially for detecting and removing process friction. In some agile environments, satellite-led efforts are being replaced by automation.

[SM2.6: 65] Require security sign-off.

The organization has an initiative-wide process for accepting security risk and documenting accountability, with a risk owner signing off on the state of all software prior to release. The sign-off policy might require the head of a business unit to sign-off on critical vulnerabilities that have not been mitigated or on SSDL steps that have been skipped, for example. The sign-off policy must apply both to outsourced projects and to projects that will be deployed in external environments, such as the cloud. Informal or uninformed risk acceptance alone isn’t a security sign-off because the act of accepting risk is more effective when it’s formalized (e.g., with a signature, a form submission, or something similar) and captured for future reference. Similarly, simply stating that certain projects don’t need sign-off at all won’t achieve the desired risk management results. In some cases, however, the risk owner can provide the sign-off on a particular set of software project acceptance criteria, which are then implemented in automation to provide governance-as-code, but there must be an ongoing verification that the criteria remain accurate and the automation is actually working.

Strategy & Metrics Level 3

[SM3.1: 27] Use a software asset tracking application with portfolio view.

The SSG uses centralized tracking automation to chart the progress of every piece of software and deployable artifact (e.g., container registries) in its purview, regardless of development methodology. The automation records the security activities scheduled, in progress, and completed, incorporating results from SSDL activities even when they happen in a tight loop or during deployment. The combined inventory and security posture view enables timely decision-making. The SSG uses the automation to generate portfolio reports for multiple metrics and, in many cases, publishes these data at least among executives. Depending on the culture, this can cause interesting effects via internal competition. As an initiative matures and activities become more distributed, the SSG uses the centralized reporting system to keep track of all the moving parts.

 

[SM3.2: 4] Run an external marketing program.

To build external awareness, the SSG helps market the SSI beyond internal teams. In this way, software security can grow risk reduction exercises into a competitive advantage or market differentiator. The SSG might publish papers or books about its software security capabilities or have a public blog. It might provide details at external conferences or trade shows. In some cases, a complete SSDL methodology can be published and promoted outside the firm, and governance-as-code concepts can make interesting case studies. Regardless of venue, the process of sharing details externally and inviting critique is used to bring new perspectives into the firm.

[SM3.3: 18] Identify metrics and use them to drive resourcing.

The SSG and its management choose the metrics that define and measure SSI progress in quantitative terms. These metrics are reviewed on a regular basis and drive the initiative’s budgeting and resource allocations, so simple counts and out-of-context measurements won’t suffice here. On the technical side, one such metric could be defect density, a reduction of which could be used to show a decreasing cost of remediation over time, assuming of course that testing depth has kept pace with software changes. Recall that, in agile methodologies, metrics are best collected early and often using event-driven processes with telemetry rather than point-in-time data collection. The key is to tie security results to business objectives in a clear and obvious fashion in order to justify funding. Because the concept of security is already tenuous to many business people, making an explicit tie-in can be helpful. 

[SM3.4: 1] Integrate software-defined lifecycle governance.

Organizations begin replacing traditional document, presentation, and spreadsheet-based lifecycle management with software-based delivery platforms. Humans, sometimes aided by tools, are no longer the primary drivers of progression from each lifecycle phase to the next. Instead, organizations rely on automation to drive the management and delivery process with ALM/ADLM software, such as Spinnaker or pipeline platform software like GitHub, and humans participate asynchronously (and often optionally), like services. Automation often extends beyond the scope of CI/CD to include functional and nonfunctional aspects of delivery, including health checks, cut-over on failure, rollback to known-good software, defect discovery and management, compliance verification, and a way to ensure adherence to policies and standards. Some organizations are also evolving their lifecycle management approach by integrating their compliance and defect discovery data to begin moving from a series of point-in-time go/no-go decisions (e.g., release conditions) to a future state of continuous accumulation of assurance data (e.g., output from sensors embedded in development and production). Lifecycle governance extends beyond defect discovery, and often includes incorporation of intelligence feeds and third-party security research, vulnerability disclosure and patching processes, as well as other activities.