Measuring security success in DevOps environments can be tricky. For one, integrating security in DevOps is as much about changing cultural mindsets as it is a practice. That’s not always easy to quantify. Obviously, numbers are important, but all metrics aren’t created equal. For example, the overall number of hours worked or number of scans conducted, while helpful, aren’t necessarily key indicators of progress. What matters more is whether security has been improved and if the overall attack surface been reduced. Framed another way, the best numbers are those that show their business impact and costs.
It helps to remember the objective of integrating security in DevOps is to deliver secure software inside of processes resilient enough to recover from inevitable vulnerabilities and attacks. DevOps means speed and delivery, which is not the natural end goal of security. Measuring success means the goals of each need to be considered. Organizations who want to measure their DevOps security growth should pay attention to:
Top Vuln Types – both New and Recurring:
Vulnerability trends help determine what areas need more focus, improvement, and prioritization. Ultimately, these top line indicators are important to see how effective your security practices are when applied to real world scenarios.
Reduction in Hours Spent Resolving Security Issues: One key area organizations should focus on is not necessarily the quantity of defects but instead how quickly those defects are resolved by the team. Time to resolution is as key as it gets, and should always be decreasing.
Number of Security Tickets: Are the number of security related tickets going down? As security becomes more integrated, they should. The overall percentage of tickets that are due to security events should also decrease.
Delivery Issues:
Is security impacting the speed of application delivery? Is security causing failed builds? If so, security likely still needs to be included earlier in the process.
False Positives: One of the results of integrating security into DevOps is that the number of false positives will decrease because testing focuses developer attention on real issues, not ‘maybes.’ If the number of false positives is not going down, then security is still being bolted on, not brushed in.
Compliance:
The time to achieve compliance for each new application should show a continual drop, while the percentage of audits passed should be continuously improving until it hits 100%.
Centralization:
Different types of testing return different results. As well, it can be extremely difficult for security testers to maintain consistency in both methodology and results across multiple tests and ranges. If your scan repository is not centralized and correlated, your program should strive to move towards it.
Integration:
There is a point in a program’s growth when security becomes elevated to its proper level of importance. But one true indication that security is being built in, and not brushed on (or jammed in) is when it becomes so fundamentally integrated that security is simply an inherent part of the process. Organizations who call it DevSecOps should ask why security needs its own pillar when it should be as fundamental as ‘coding’ or ‘compiling.’ Put simply, security must become an ingrained part of the culture to be successful.