Introduction
It’s fundamental for enterprises and organizations of any size to have established practices of logging and monitoring at the application level. The risk of an internal data breach, or even the chance of a spoofing attack, is always possible. These attacks have been proven time and time again to be detrimental to the brand of the organization who’s systems have been compromised.
While logging at the application level helps you oversee what your systems are doing, many do not have audit logging, human logging. What are your engineers doing in these systems? What exactly are they doing in production? Visibility should be at its highest when overlooking what is happening in production—unfortunately for most, it’s foggy.
So what’s the difference between audit logging and application logging?
Application logging will record every action as it happens. An application log is best put as a record of an action. For example, an application log will look something like this:
[12:30:06 PM UTC] Engineer A accessed and pulled data from Database C.
It tells you what a user did. It gives you a high level overview of what happened.
Audit logging will go further into the details of what happened during specific actions. Audit logs are a security-relevant chronological record of events and changes. They record relevant security information for designated actions and answer the questions such as who performed the action, what the action was, and most importantly, how the system responded. It helps prevent/detect malicious use. Adding to the original example for application logging, an example of an audit log of that same action will look something like this.
[12:30:06 PM UTC] Engineer A accessed Database C.
[12:31:05 PM UTC] Engineer A ran “X code” to pull data from Database C.
[12:32:10 PM UTC] Command ran successfully, sent data back to engineer A.
[12:33:10 PM UTC] Engineer exited Database C.
Quick disclaimer for the examples, the logs won’t really look exactly like this. This was just a high-level example.
Let’s take a deep dive into how audit trails actually work, the benefits of having them in your organization, and how to go about designing them.
Audit Trails
Audit trails capture events and show all the relevant information in a human-readable JSON format. Audit events are defined by the organization/user. They represent a certain action. For example, audit events can be created off of certain actions such as user.invite, user.delete, group.update_user, etc.
Benefits from audit logging
Staying compliant. Audit trails are compatible with common regulatory frameworks such as SOC 2, HIPAA, SOX and ISO-27001 so your organization can stay compliant with industry standards.
Audit trails are immutable. They reflect what actually happened due to their tamper proof nature. This helps with reconstruction of events, problem analysis, and gaining insight into your production workflows.
Audit trails are scalable. Audit trail software is designed to scale with your organization. The best products can handle billions of audit trail events per month, while ensuring no events are lost.
Audit trails are secure. Data can and should be encrypted in transit and at rest.
Audit trials provide intrusion detection for individual accountability. Any attempts of parties trying to log on to production systems they're not supposed to will all be recorded.
Best practices for designing audit trails
Audit trails should answer the who, what, when, where, and why. It’s important to consider these guidelines when designing your audit trail events. Here are some best practices in doing so.
Include all associated information
A good rule of thumb is to include all information necessary in your event so that your future self will be able to understand exactly what happened at a particular point in time, in the past, without any additional context other than the audit event itself.
Here is an example of an event that answers who, what, when, where, and why:
Good
{
"action": "audit_trail.create",
"actor": "ben@cased.com",
"actor_id": "user_1dQpY8bBgEwdpmdpVrrtDzMX4fH",
"location": "123.123.123.123",
"audit_trail": "security",
"audit_trail_id": "audit_trail_1dQpY8bBgEwdpmdpVrrtDzMX4fH",
"organization": "cased",
"organization_id": "org_1dQpY1jKB48kBd3418PjAotmEwA",
"environment": "live",
"environment_id": "env_1dQpXqQwXxsQs9sohN9HrzRAV6y",
"http_method": "POST",
"http_url": "<https://app.cased.com/audit-trails>",
"http_request_id": "740386ac-490c-43f0-96b2-48853629b04f",
"http_user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.106 Safari/537.36",
"timestamp": "2020-05-01T10:22:43.836593Z"
}
With this event we understand that ben@cased.com created the security audit trail that belongs to the cased organization and it happened on May 1st, 2020 at 10:22 UTC. Other information is included to provide additional context such as the HTTP request information and the environment the audit trail belonged to at that point in time.
Bad
{
"action": "audit_trail.create",
"audit_trail": "cased",
"timestamp": "2020-05-01T00:00:00.000000Z"
}
With this event we answer what happened and when it happened, but do not know who created the audit trail, where it took place, and why it occurred.
Human and machine readable information
Your first priority should be making your events readable for humans. For each piece of machine-readable information (like an id) you should have at least one piece of human-readable information (like a name).
Good
{
"action": "user.invite",
"actor": "ted@cased.com",
"actor_id": "user_1dQpY1fUFHzENWhTVMvjCilAKp9",
"user": "ben@cased.com",
"user_id": "user_1dQpXyWHS8A1P2bT0cWKu6fVYMk",
"organization": "cased",
"organization_id": "org_1dQpY1jKB48kBd3418PjAotmEwA",
"http_method": "POST",
"http_url": "<https://app.cased.com/users/invitations>",
"http_request_id": "1763522b-9dd8-4a43-8930-18b65380b140",
"http_user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.106 Safari/537.36",
"timestamp": "2015-06-02T06:22:10.289555Z"
}
Remember that the names of an actor, user, or organization may change in the future, making it difficult to follow changes over time. By including the human-readable information alongside the machine-readable version, you know what a particular value was at a point in time.
Bad
{
"action": "user.invite",
"actor_id": "user_1dQpY1fUFHzENWhTVMvjCilAKp9",
"user_id": "user_1dQpXyWHS8A1P2bT0cWKu6fVYMk",
"organization_id": "org_1dQpY1jKB48kBd3418PjAotmEwA",
"http_method": "POST",
"http_url": "<https://app.cased.com/users/invitations>",
"http_request_id": "1763522b-9dd8-4a43-8930-18b65380b140",
"http_user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.106 Safari/537.36",
"timestamp": "2015-06-02T06:22:10.289555Z"
}
For this example entry, if the actor or user ids were deleted from your systems between June 2nd, 2015 and when you view the audit event, you would not have the enough information to know who performed the action.
Publish an event for every PUT, PATCH, POST, DELETE HTTP request
PUT, PATCH, POST, and DELETE HTTP requests modify data. So it's important that you publish at least one event to your audit trail for any request that uses those HTTP methods.
High signal, less noise
It's useful to compare audit trails with general system logs. System logs contain events generated by machines from both human and automated actions, while audit trails are solely for actions performed by humans. As a result, audit trails contain high-signal events, while system logs are much noisier.
Every web request could results in dozens of log lines across machines in your infrastructure. Meanwhile you typically will see an audit event only when someone modifies data which happens at a much lower rate.
There are exceptions to this guideline if you need to publish an event to keep track of someone viewing a sensitive web page.
It's easier to stop publishing an event than having not published it at all
You may wonder whether a particular event is useful or too noisy to publish. It's better to err on the side of publishing more events, and tune as you go. You cannot recreate audit events that took place in the past, but you can stop publishing events in the future.
Answer the question
When designing an audit event, consider what future questions may be asked that result in the event being retrieved. The prompt may be a support ticket, a data request, a subpoena, or investigating a security incident.
For example, if you have a user permission system and are required to publish an event when a user's permission changes, make sure you include the new permission and the previous permission in the same event.
Good
{
"action": "group.update_user",
"actor": "ted@cased.com",
"actor_id": "user_1dQpY1fUFHzENWhTVMvjCilAKp9",
"user": "ben@cased.com",
"user_id": "user_1dQpXyWHS8A1P2bT0cWKu6fVYMk",
"permission": "admin",
"permission_was": "member",
"group": "Legal",
"group_id": "group_1dQpXyWHS8A1P2bT0cWKu6fVYMk",
"organization": "cased",
"organization_id": "org_1dQpY1jKB48kBd3418PjAotmEwA",
"http_method": "POST",
"http_url": "<https://app.cased.com/groups/legal/members>",
"http_request_id": "dd4547f9-1317-43cf-affd-20ec230972b8",
"http_user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.106 Safari/537.36",
"timestamp": "2015-06-02T06:22:10.289555Z"
}
This example contains the user's new permission of admin and the previous permission of member. We can now answer "what was was the user's previous permission and what was the permission the user obtained" by just looking at a single event.
Bad
{
"action": "group.update_user",
"actor": "ted@cased.com",
"actor_id": "user_1dQpY1fUFHzENWhTVMvjCilAKp9",
"user": "ben@cased.com",
"user_id": "user_1dQpXyWHS8A1P2bT0cWKu6fVYMk",
"group": "Legal",
"group_id": "group_1dQpXyWHS8A1P2bT0cWKu6fVYMk",
"organization": "cased",
"organization_id": "org_1dQpY1jKB48kBd3418PjAotmEwA",
"http_method": "POST",
"http_url": "<https://app.cased.com/groups/legal/members>",
"http_request_id": "dd4547f9-1317-43cf-affd-20ec230972b8",
"http_user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.106 Safari/537.36",
"timestamp": "2015-06-02T06:22:10.289555Z"
}
This example does not contain any information to answer what changed about a particular user in a group. Now we are left with more questions than we started with.
Protecting Sensitive data
Safeguarding and cataloguing sensitive data, such as Personally Identifiable Information (PII) and Protected Health Information (PHI), is an important part of compliance.
Examples
Audit trail software allows you to specify what data we should consider as sensitive.
Let's consider an event of a user changing their mailing address). For
this event, you would want location, email, and address fields to be
safeguarded as:
{
"action": "user.change_mailing_address",
"address": "1600 Pennsylvania Ave NW, Washington, DC 20500",
"user": "president@whitehouse.gov",
"location": "124.240.195.167",
"device": "browser",
"user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_3) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.5 Safari/605.1.15",
"timestamp": "2020-03-26T19:49:31.534665Z"
}
The software would encrypt, tokenize, and replace the sensitive data with random reference tokens.
These reference tokens are safe to store in your system if necessary.
{
"action": "user.change_mailing_address",
"address": "pii_0dd96d0845d0cbe6fba752719d77f771",
"user": "pii_842eba08a2895aab696f64ecaba67d3e",
"location": "124.240.195.167",
"device": "browser",
"user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_3) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.5 Safari/605.1.15",
"timestamp": "2020-03-26T19:49:31.534665Z"
}
Accessing Sensitive Data
All access to sensitive data results in audit trail events recorded and viewable in your audit trail software alongside the audit trail event the sensitive data originated from. Your audit trail software should always record who accessed this data, where it was accessed from, whether or not it was authorized, the reason it was accessed, and when it was accessed.
Conclusion
We hope this in-depth overview of audit trails encourages your organization to implement some form of audit logging. Enhance visibility into your production systems, stay secure and compliant.
If you're interested and want to learn more schedule a demo or visit our documentation.
