Detecting attacks in your software
In this blog post, we cover ideas that were first discussed by our SafeStack Academy Secure Development community at our online members' seminar on detecting attacks in software products hosted by Christian Frichot and Erica Anderson.
As part of every Secure Development membership, our learners are invited to monthly online seminars hosted by our expert team. These seminars connect you with our community of learners and create a safe space to share challenges and approaches with you, all while staying up to date with relevant topics and emerging threats.
If you enjoy what we share below and would like to be involved in these conversations as they happen, grab a free 14-day trial and join us at the next one!
We all know cyber security is an important aspect of building high-quality software, and it’s a balancing act between having security controls that help us prevent and detect events and working at speed.
Sometimes, we can prevent a security event by making conscious decisions in how we design, develop, and manage our software. And sometimes, we need to be ready to detect and handle an attack. In this post, we go through a few ways you and your team can get started detecting attacks in your software.
Preparing for an "assume breach" approach
While we all want to develop secure software as soon as it leaves our brains, finding its way to a GitHub repo via our keyboards, history has shown us this isn’t possible.
Software libraries change over time, potentially introducing security issues. New hacking techniques pop up that may shatter previously held assumptions. A SaaS vendor you integrate with has suffered a breach.
It’s no one’s fault. Writing software is complicated, let alone writing secure software and maintaining that level of security over time.
At a point in time, cyber security teams would strive to build impenetrable castles around their precious IT assets. They’d invest in security programs that took years to plan and implement, including putting layers of expensive security hardware and software products in place, educating users, and implementing security processes to try and identify issues before releasing software through a gate into production.
While this defence-in-depth technique still stands the test of time, another security paradigm is becoming very popular: assuming you’re breached or compromised.
How software teams can address security in an assumed breach approach is varied. Performing threat models and tabletop incident response exercises are a fantastic start from a process perspective. Investigating how to mature your architectures to embrace zero trust paradigms can also help from a technology perspective. While these measures may reduce the impact of an incident, pre- and post-breach, they can also miss an important capability already central to many software and cloud environments: logging, monitoring, and observability.
Where attacks start
When architecting your logging and monitoring capabilities, start where the attacks start. Your software will likely be exposed to untrusted networks in one way or another. Looking at your trust boundaries and understanding how data flows between these boundaries is where you want to begin your logging efforts.
One of the benefits of constructing data flow diagrams and writing up threat models is that you can start to capture which components of your software solutions are more exposed to untrusted networks.
While a zero-trust approach will steer us towards not trusting anything outside of our services, there’s value in understanding if your API is directly exposed to the internet or not.
In threat modeling speak, this is known as an attack surface. But don’t just stop at your internet-exposed systems; you must also consider your administrative tools and interfaces. These interfaces include cloud provider portals, third-party integrations, and other deployment tools. It’s also important to consider any end-user devices, like laptops and phones.
Another benefit of using threat modeling to refine and focus your logging and monitoring efforts is that it helps you understand potential threat actors.
Attempts to compromise your software may come from numerous sources, ranging from low-skill, low-resource attackers leveraging automated tools and looking for low-hanging fruit, all the way up to motivated, skilled attackers.
Attacks from these sources may be able to leverage more resources to either bypass, evade, or overload your logging systems.
A similar methodology that can help ensure your software emits the correct data and event logs is to consider leveraging the MITRE ATT&CK framework. This framework breaks down several tactics with subsequent techniques that an attacker may use to abuse your system.
For example, cherry-picking some of the data from the Cloud Matrix, in the Initial Access tactic, these techniques apply to your software solutions:
And then further along the chain, under the Persistence tactic, these techniques may also apply to your software:
This summary isn’t extensive, as ATT&CK includes much more data — but these sources give you an idea of where to prioritize logging and monitoring in your software.
How we can detect attacks
Now that we know where to start logging and monitoring, how do we move on to detecting actual attacks?
Different attacks will require different events to be logged and captured, depending on what normal baseline behavior and events look like.
At SafeStack, we find it helpful to look into attack events by event source. This is because the type of information recorded by these sources will vary, and your trust level may vary, depending on how well-equipped or configured that source is.
Here are the event sources we use:
Users
Specific monitoring tools
Application and network logs
Your users are often the canary in the coal mine, meaning they’ll be the first to tell you when they’re experiencing a negative impact with your software.
While your users aren’t the source of detecting attacks you want to rely on, it’s important that you're prepared for them to do so.
Make sure your products, particularly those operating in a Software-as-a-Service (SaaS) format, have appropriate customer support processes in place — options like live chat and a dedicated email address for support requests are industry-standard ways of making sure users have an easy way to flag issues. Set these processes up in a way that makes them available whether users are logged in or not, as they still need to be able to contact you if their account has been compromised.
Specific monitoring tools are also essential and will increase as your environment and organization grow. These tools include enterprise Security Incident and Event Monitoring (SIEM) tools, Intrusion Detection Systems (IDS), Web Application Firewalls (WAF), and so on. For more advanced technology, using canaries, honeypots, or honey tokens can also provide excellent bang-for-buck information about knowing if you’re under attack.
The event source we will dig into now is application and network logs. These are valuable and multi-purpose — not only will you use these logs to monitor performance and perform troubleshooting, but they‘re also key for detecting attacks.
Getting the best out of your application logs
Now that we know where attacks may come from and some potential pathways we could see an attacker cross, let’s narrow down how we can build our software to provide high-value logs and data to detect these issues. Often, the first question is, “How do we generate or emit logs?”
Using existing logging libraries
Ideally, we want to avoid reinventing the wheel. Chances are your programming language or framework has several existing or popular logging libraries. Take advantage of these libraries where you can, but make sure to perform some due diligence of your own. Here's a summary of popular logging libraries sorted by programming language to get you started.
JavaScript
Thanks to Geshan Manandhar for his post on Node.js logging libraries, which we’ve drawn from here.
Python
Java
Apache Log4j 2 (Make sure you use the most up-to-date version to mitigate CVE-2021-44832)
TypeScript
C#
PHP
Developer beware: While we stand by the above advice, there are risks you need to navigate. The idea of “leveraging a popular logging library” backfired aggressively with the Log4j issues that emerged in December 2021. This risk is present in many third-party libraries and highlights the importance of maintaining a software bill of materials and continuously scanning your software and dependency chains for vulnerabilities. Combining this with continuously deploying updates can also help manage this risk!
Using request or correlation IDs
In distributed or microservices architectures, request IDs can help correlate logs in the case of an event traversing multiple systems. These are unique identifiers tagged onto a request at the start of a flow through your application. You might also know them as correlation IDs or distributed tracing. If you don’t have this capability in your surrounding infrastructure, consider adding this as a custom HTTP header element at the first touchpoint in handling or routing HTTP requests.
Standardizing log formats
The next aspect you should consider and then standardize on in your environment is the format of your application logs. These could be single-line CSVs, XML, binary formats, or JSON. JSON formatted logs are growing in popularity as they offer a means to emit data that’s readable by a human and parsable by a computer. Regardless of the format, trying to standardize can help you collate data and respond to future issues.
Writing or emitting logs
Finally, you’ll want to consider where to emit or write these logs. Your application may write them to a file or perhaps emit them to STDOUT (standard output). You may also want to leverage the underlying operating system formats, like Syslog or Windows Events. This decision is more related to how you implement centralized logging and monitoring, as you’ll need to ensure that wherever these logs go, you can feed them into your overarching platforms.
How other tools can help
Other products and tools — particularly Application Performance Monitoring technology — can help you monitor your software for cyber security issues but also help improve overall monitoring and observability.
These products often integrate directly into your software and communicate with central SaaS dashboards, offering you and your operations team insight into what’s happening in your software.
Some of these products are starting to include native security capabilities, too — sometimes called Real-time Application Security Protection (or Runtime Application Self-Protection).
Which application log events should you focus on?
So you’ve selected your logging library, tweaked the logs format, and sent them to a centralized platform. But what actual events should we capture to ensure we can detect potential indicators of attack?
Authentication events
Authentication is obvious, particularly monitoring and emitting failed login attempts for potential brute-force attacks or password spraying. Logging and trending successful logins are also important if you want to implement any “unusual travel” style rules in your products.
The benefit of collecting data about login events is that you may eventually want to surface some of this to your users as well, similar to how Google Mail will show you that you’re currently logged in from several places. Your authentication platform may integrate with a third party, like Auth0. If it is, these third parties should also be monitored similarly wherever possible.
User profile or account changes
While updating your contact information (including email, address, or phone number) can be a completely legitimate thing to do, it can also be something an attacker does as soon as they’ve taken over an account. Other profile changes, like adjusting payment options, are also ripe for abuse, so monitor them.
Viewing or exporting sensitive profile attributes
Sometimes, viewing or validating some information may be slightly unusual. Many platforms ask you to re-authenticate or perform a second-factor step-up if you try to view personal details, like your address. Someone attempting to see these fields without completing a step-up may indicate a potential attack.
Misuse of communication or file-sharing features
A common motivator for an attacker to compromise a user’s account is sending malicious messages to other users. This feature is widespread on social networking platforms but could be abused if your software allows users to message or share content.
Staying on top of threat models and incident response
Threat modeling is a great way to identify the higher-risk processes in your software products that may require more verbose logging or data emitting. Lastly, keep refining and maturing your entire incident response process. Logging and monitoring are great ways to keep an eye on what’s happening, but without incident response runbooks in place, you won’t be any better off when bad things happen. Keep training that muscle by running tabletop exercises where you can!
Keen to join us at our next seminar?
We hope you’ve enjoyed this post on what we covered at our community seminar on detecting software attacks. One of the great things about being part of our Secure Development community is that you’re invited to these seminars every month. There’s something for everyone, whether you want to take part or watch and listen along actively, and you can replay them any time.
While our team hosts them, they’re designed to help you learn from other learners, giving you a safe space to share how you approach these topics in your organization and hear what’s worked (or not) for others.
Our free 14-day trial gives you full access to our Secure Development program, including:
All our past seminars, as well as the next one coming up
Our full course catalogue
Hands-on labs where you can explore concepts and test your knowledge
We'd love to see you at our next seminar, so why not get started today?