In this blog post, we cover ideas that were first discussed by our SafeStack Academy Secure Development community at our online members’ seminar on detecting attacks in software products, hosted by Christian Frichot and Erica Anderson.
As part of every Secure Development membership, our learners are invited to monthly online seminars hosted by our expert team. These seminars connect you with our community of learners and create a safe space to share challenges and approaches with, all while staying up to date with relevant topics and emerging threats.
If you enjoy what we share below and would like to be involved in these conversations as they happen, grab a free 14-day trial and join us at the next one!
We all know cyber security is an important aspect of building high-quality software, and it’s a balancing act between having security controls that help us prevent and detect events and working at speed.
Sometimes we can prevent a security event from happening by making conscious decisions in how we design, develop, and manage our software. And sometimes we just need to be ready to detect and handle an attack. In this post, we go through a few ways you and your team can get started with detecting attacks in your software.
Preparing for an “assume breach” approach
While we all want to develop software that’s secure as soon as it leaves our brains, finding its way to a GitHub repo via our keyboards, history has shown us this isn’t possible.
Software libraries change over time, potentially introducing security issues. New hacking techniques pop up that may shatter previously held assumptions. A SaaS vendor you integrate with has suffered a breach.
It’s no one’s fault. Writing software is complicated, let alone writing secure software and maintaining that level of security over time.
There was a point in time when cyber security teams would strive to build impenetrable castles around their precious IT assets. They’d invest in security programmes that took years to plan and implement, including putting layers of expensive security hardware and software products in place, educating users, and implementing security processes to try and identify issues before releasing software through a gate into production.
While this defence-in-depth technique still stands the test of time, another security paradigm is becoming very popular: assuming you’re breached or compromised.
How software teams can address security in an assumed breach approach is varied. From a process perspective, performing threat models and tabletop incident response exercises are a fantastic start. Investigating how to mature your architectures to embrace zero trust paradigms can also help from a technology perspective.
While these measures may reduce the impact of an incident, pre- and post-breach, they can also miss an important capability already central to many software and cloud environments: logging, monitoring, and observability.
Where attacks start
When architecting your logging and monitoring capabilities, start where the attacks start. Your software will likely be exposed to untrusted networks in one way or another. Looking at your trust boundaries and understanding how data flows between these boundaries is where you want to begin your logging efforts.
One of the benefits of constructing data flow diagrams and writing up threat models is that you can start to capture which components of your software solutions are more exposed to untrusted networks.
While a zero trust approach will steer us towards not trusting anything outside of our services, there’s value in understanding if your API is directly exposed to the internet or not.
In threat modelling speak, this is known as an attack surface. But don’t just stop at your internet exposed systems; you need to consider your administrative tools and interfaces, too. These interfaces include cloud provider portals, third party integrations, and other deployment tools. It’s also important to consider any end-user devices, like laptops and phones.
Another benefit of using threat modelling to refine and focus your logging and monitoring efforts is that it helps you understand potential threat actors.
Attempts to compromise your software may come from numerous sources, ranging from low-skill, low-resource attackers leveraging automated tools and looking for low-hanging fruit, all the way up to motivated, skilled attackers.
Attacks from these sources may be able to leverage more resources to either bypass, evade, or overload your logging systems.
A similar methodology that can help ensure your software emits the correct data and event logs is to consider leveraging the MITRE ATT&CK framework. This framework breaks down several tactics with subsequent techniques that an attacker may use to abuse your system.
For example, cherry-picking some of the data from the Cloud Matrix, in the Initial Access tactic, these techniques apply to your software solutions:
And then further along the chain, under the Persistence tactic, these techniques may also apply to your software:
This summary isn’t extensive, as ATT&CK includes much more data — but these sources give you an idea of where to prioritise logging and monitoring in your software.
How we can detect attacks
Now that we know where to start with logging and monitoring, how do we move on to detecting actual attacks?
Different attacks will require different events to be logged and captured, and it also depends on what normal baseline behaviour and events look like.
At SafeStack Academy, we find it helpful to look into attack events by event source. This is because the type of information recorded by these sources will vary and the level of trust you have in them may vary as well, depending on how well-equipped or configured that source is.
Here are the event sources we use:
- Specific monitoring tools
- Application and network logs
Your users are often the canary in the coal mine, meaning they’ll be the first to tell you when they’re experiencing a negative impact with your software.
While your users aren’t the source of detecting attacks you want to rely on, it’s important that you’re prepared for them to do so.
Make sure your products, particularly those operating in a Software-as-a-Service (SaaS) format, have appropriate customer support processes in place — options like live chat and a dedicated email address for support requests are industry-standard ways of making sure users have an easy way to flag issues. Set these processes up in a way that makes them available whether users are logged in or not, as they still need to be able to contact you if their account has been compromised.
Specific monitoring tools are also important and will become even more so as your environment and organisation grow. These tools include enterprise Security Incident and Event Monitoring (SIEM) tools, Intrusion Detection Systems (IDS), Web Application Firewalls (WAF), and so on.
For more advanced technology, using canaries, honeypots, or honey tokens can also provide excellent bang-for-buck in knowing if you’re under attack.
The event source we’re going to dig into now is application and network logs. These are valuable and multi-purpose — not only will you use these logs to monitor performance and perform troubleshooting, but they‘re also key for detecting attacks.
Getting the best out of your application logs
Now that we know where attacks may come from and some potential pathways we could see an attacker cross, let’s narrow down how we can build our software to provide high-value logs and data to detect these issues. Often, the first question is “how do we generate or emit logs?”
Using existing logging libraries
Ideally, we want to avoid reinventing the wheel. Chances are your programming language or framework has several existing or popular logging libraries. Take advantage of these libraries where you can, but make sure to perform some due diligence of your own.
To get you started, here’s a summary of popular logging libraries sorted by programming language.
- Apache Log4j 2 (Make sure you use the most up to date version to mitigate CVE-2021-44832)
Developer beware: While we stand by the above advice, there are risks you need to navigate. The idea of “leveraging a popular logging library” backfired aggressively with the Log4j issues that emerged in December 2021. This risk is present in many third-party libraries and highlights the importance of maintaining a software bill of materials and continuously scanning your software and dependency chains for vulnerabilities. Combining this with continuously deploying updates can also help manage this risk!
Using request or correlation IDs
In distributed or microservices architectures, request IDs can help correlate logs in the case of an event traversing multiple systems. These are unique identifiers tagged onto a request at the start of a flow through your application. You might also know them as correlation IDs or distributed tracing. If you don’t have this capability in your surrounding infrastructure, consider adding this as a custom HTTP header element at the first touchpoint in your handling or routing of HTTP requests.
Standardising log format
The next aspect you should consider and then standardise on in your environment is the format of your application logs. These could be single-line CSVs, XML, binary formats, or JSON. JSON formatted logs are growing in popularity as they offer a means to emit data that’s readable by a human and parsable by a computer. Regardless of the format, trying to standardise can help you collate data and respond to future issues.
Writing or emitting logs
Finally, you’ll want to think about where to emit or write these logs. Your application may write them to a file, or perhaps emit them to STDOUT (standard output). You may also want to leverage the underlying operating system formats available to you, like Syslog or Windows Events. This decision is more related to how you implement centralised logging and monitoring, as you’ll need to make sure that wherever these logs go, you’re able to feed them into your overarching platforms.
How other tools can help
Other products and tools — particularly Application Performance Monitoring technology — can help you monitor your software, not just for cyber security issues but also to help improve overall monitoring and observability. These products often integrate directly into your software and communicate with central SaaS dashboards, offering you and your operations team insight into what’s happening in your software. Some of these products are starting to include native security capabilities, too — sometimes referred to as Real-time Application Security Protection (or Runtime Application Self-Protection).
Which application log events should you focus on?
So you’ve selected your logging library, you’ve tweaked the logs format, and you’re sending them off to a centralised platform.
But what actual events should we be capturing to make sure we can detect potential indicators of attack?
Authentication is the obvious one, particularly monitoring and emitting failed login attempts for potential brute-force attacks or password spraying. Logging and trending successful logins are also important if you want to implement any “unusual travel” style rules in your products. The benefit of having data collected about login events is that you may also eventually want to surface some of this to your users as well, similar to how Google Mail will show you that you’re currently logged in from several places.
Your authentication platform may integrate with a third party, like Auth0. If it is, these third parties should also be monitored in similar ways wherever possible.
User profile or account changes
While updating your contact information (including email, address, or phone number) can be a completely legitimate thing to do, it can also be something an attacker does as soon as they’ve taken over an account. Other profile changes like adjusting payment options are also ripe for abuse — so be sure to keep an eye on these kinds of changes.
Viewing or exporting sensitive profile attributes
Sometimes, viewing or validating some information may be slightly unusual. Many platforms ask you to re-authenticate or perform a second-factor step-up if you try to view personal details, like your address. Someone attempting to see these fields and not completing a step-up may indicate a potential attack.
Misuse of communication or file sharing features
A common motivator for an attacker to compromise a user’s account is sending malicious messages to other users. This feature is widespread on social networking platforms but could be abused if your software allows users to message or share content.
Staying on top of threat models and incident response
Threat modelling is a great way to identify the higher risk processes in your software products that may require more verbose logging or data emitting.
Lastly, keep refining and maturing your entire incident response process. Logging and monitoring are great ways to keep an eye on what’s happening, but without also having incident response runbooks in place, you won’t be any better off when bad things happen. Keep training that muscle by running tabletop exercises where you can!
Keen to join us at our next seminar?
We hope you’ve enjoyed this post on some of what we covered at our community seminar on detecting software attacks.
One of the great things about being part of our Secure Development community is that you’re invited to these seminars every month. There’s something for everyone, whether you want to actively take part or just watch and listen along, and you can replay them any time. While they’re hosted by our team, they’re designed to help you learn from other learners too, giving you a safe space to share how you’re approaching these topics in your organisation and hear what’s worked (or not) for others.
- All our past seminars as well as the next one coming up
- Our full course catalogue
- Hands-on labs where you can explore concepts and test your knowledge
We’d love to see you at our next seminar, so why not get started today?
We love to hear from you
If you enjoyed reading this blog post or if something sparked an interest, please share it with us. Drop us a line at email@example.com and let us know what you think.