Using leading indicators to measure software security maturity

If one thing about cyber security marketing irritates me, it’s the value claims.

I’m thinking of the measures that vendors place on products that are meant to validate the effectiveness or worth of this particular magic box. Whether it’s “stops 93% of vulnerabilities” or “reduces cyber attacks by 75%”, these numbers don’t help in any meaningful way.

Sadly, this shot-in-the-dark approach to measuring security effectiveness enters our businesses. We share numbers that represent at best, a historical view of our actions or at worst, a work of chaotic fiction.

When measuring the effectiveness of our application security programs, we need to think differently.

What you did to get here won’t get you there.

Traditionally, we’ve measured how poor security has affected us in the past — common approaches include tracking how many incidents we had, how many vulnerabilities we found, or how many bugs were reported. 

Let’s look closer at incidents as an example — if you reported on the number of incidents per quarter, you’d most likely be looking for a downward trend over time.

However, we need to remember that this approach to measurement is based on some assumptions.

Assumption #1

You know about all the cyber security incidents your company has faced.

Assumption #2

Your operations, systems, and operating environments are not changing.

Assumption #3

Cyber attackers are not evolving.

When you track something like the number of incidents over time, what you’re essentially learning is, “For the last period, given the circumstances we operated under, this is what happened.” 

When you think about it like that, wouldn’t it be risky to assume that these metrics are a good predictor of what’s to come?

Learning to predict the appsec future

For all of us without crystal balls, what are our options for measuring how effective our security initiatives are?

The answer lies in leading indicators.

Leading indicators are a way of measuring the likelihood of success by looking at the processes that lead to success, rather than looking at the event itself.

Let’s go back to our “number of incidents” example and look at how we could translate it into leading indicators.

Historical measures

• Number of incidents last month

Leading indicators

• Percentage of systems with logging and monitoring enabled 

• Average time to alert response

As you can see, we aren’t using metrics to say “we had fewer security incidents”.

We’re using them to say that we’re consistently applying the controls we know increase the likelihood of an incident being identified, and we’re also reducing the time to respond should something happen.

Applying leading indicators to application security

If we focus specifically on the application security space, what are some leading indicators we could use?

Percentage of code scanned by static analysis tools on every pull request

Regardless of what specific solution you use, measuring how much of your application code is scanned (and how frequently) is a great way of ensuring that if you have the right tools in place, they’re being used, and they have the highest chance of being effective.

Read more about common security testing methodologies here.

Number of people taking part in security initiatives and events

Cyber security is a team sport. If your application security program relies on one very motivated person to do everything, you’re destined to have security issues in the future. No single person can do everything, and everyone needs a break or change of scenery from time to time. Supporting a growing number of people to be involved in security reduces the likelihood of key person risk and ensures that more eyes and hands are helping with security.

What's next?

In future posts we'll dig further into leading indicators of application security success. We’ll also look at concepts like measuring security velocity. 

For now, though, remember that measuring effectiveness is not only good for motivation and communicating with stakeholders, but also for clearly understanding where your gaps are.

If you take away one thing from this post,  let it be this: instead of using the past as a measure of your security preparedness, use leading indicators to understand and monitor the actions and processes that predict your likely future security effectiveness.