Your security testers are sad.
You brought them in under NDA, gave them a copy of the code and access to a test environment, and let them loose to do a penetration test. A week later they came back to you with a report of vulnerabilities they found. They listed each bug and how to recreate it, what the impact could be, and how likely it is to be exploited.
As the project manager, getting those vulnerabilities closed may be the only thing standing between you and go-live.
You put the high severity fixes straight into the current sprint and the rest go on the backlog, or get written up in a risk register. Job done! The team has a launch party and gets started on the next project.
Six months later…
You engage the same security testers on your new project and a week later they hand you an eerily familiar report. The vulnerabilities are superficially different but the tune is the same. Your poor tester must feel like they are stuck in a time loop, trapped finding variations on the exact same bugs, day after day, month after month.
Are these same vulnerabilities inevitable?
Why is the new report so often similar to all the previous reports?
You may not be getting all the value from the report that you can. While a list of vulnerabilities in a report will allow you to fix the system now, it might also point to ways your team could eliminate a kind of vulnerability for good.
Example: Misconfigured Server
Let’s say a tester spots that your search server was misconfigured to allow remote code execution. The immediate fix for this is often a one-liner in a config file somewhere. The real question, though, is why the server was misconfigured? Here are some possible contributing factors:
- The search server hardening guide didn’t include this step.
- The team doesn’t have a hardening guide for search servers.
- No one is reviewing configuration work.
- Someone has to make the change on the server by hand rather than through a config-as-code system.
- The team’s vulnerability scanning didn’t cover this server, or this vulnerability.
Note that none of these possible causes ends with “human error”. The humans in your team will make mistakes and it’s up to the team to develop processes that protect themselves from this inevitability. This can include team training to ensure everyone has the skills they need to write secure code.
Example: Insecure Code
Often, several things have to go wrong for a vulnerability to appear:
- A SQL injection vulnerability might suggest that the team aren’t using the development framework safely and that the team’s code review process has flaws.
- A vulnerable library might indicate that the team hasn’t been given the time to update the application’s dependencies and that they don’t have mechanisms that notify them of critical patches.
Vulnerabilities emerge in systems with multiple points of weakness and your team should try and understand them all.
How do you get the most out of a testing report?
As you go through the report, pretend that each vulnerability was part of a security incident and adopt the perspective of an investigator leading a post-incident review. Perform a small blameless retrospective. Ask the five whys. Look for contributing factors and patterns.
Capture insights
Identify lessons that your team can learn from this and then feed these lessons into your feedback system. This is the most critical step of the process. If you are in an agile team your feedback system might be the sprint retro and the team’s backlog. Lessons that you don’t feed back into what you do in the future are lessons you will be forced to relearn.
You may find that some issues are out of your team’s direct control and comes from the organisational environment. Make sure that you record environmental challenges and include them in reporting to management. After all, what is not known cannot be remedied.
Learn lessons early
An external security test is an expensive piece of work and as a result your company might only schedule them once a year, or on a per-project basis. This can make it hard to integrate with your regular cadence of work. Look for ways to build similar tests in to your day-to-day work.
Could any of the following have caught the report’s bugs earlier?
- Including requirements from the OWASP Application Security Verification Standards when you write your user stories.
- Training your testers to use tools like OWASP ZAP to find vulnerabilities early.
- Including static analysis tools like SonarQube in your build pipeline to identify common coding issues.
- Introducing processes to learn about important changes to your dependencies (e.g. using Libraries.io) and schedule regular maintenance tasks to keep them up to date.
Like all bugs, security vulnerabilities are cheaper the earlier you find them. None of the above ideas will catch every possible security vulnerability, but every one they catch will be one less vulnerability that makes it into preproduction and needs remediation.
Share the knowledge
Companies can be inclined to restrict a security report on a need-to-know basis. This is a solid instinct as a list of known unresolved vulnerabilities could give an attacker a head-start. One group that the company should not exclude are the teams responsible for building and maintaining the affected systems. As trusted employees they already have access to sensitive information and privileges; withholding a report only serves to keep these teams from the information they need to protect your assets.
It is not enough to put the bare action item on their backlog. A well-written security report contains a wealth of detail about each vulnerability and how to prevent its recurrence. Your teams need to be able to read these and understand the risk each vulnerability represents. Give them the information they need to do their jobs as well as they can. This includes junior team members!
Once a vulnerability is fixed there isn’t much benefit in keeping it a secret. Encourage your teams to share what they’ve learned with each other. An interesting vulnerability or preventative solution makes a great lightning talk or knowledge base article.
Your Next Report
Next time your team receives a security report, sit with the team and identify as many lessons as you can about what the testers found and how you might prevent the same issue emerging in the future. Capture those ideas in your work tracker and team documents. Find ways to confirm that you are applying your learning to new systems by building tests for vulnerabilities into your process.
Aim to delight your security testers the next time you engage them by demonstrating that you learned from their hard work and have improved how your team builds and maintains their systems. They will be grateful for it and their report may have refreshingly new lessons to learn!
