We’ve all heard about data breaches happening at companies. They put out a press report saying data has been leaked, there’s some outrage for a few days, and you never hear about it again.

But things are not so straightforward behind the scene.

If you’re a company whose data is breached, you will likely have to pay regulatory fines and legal fees to protect yourself in court. Not to mention the millions of dollars in lost sales & long-lasting brand damage.

 In 2022, the average cost of a single data breach was $4.35 million.

What is the most common cause of data breaches? Leaked Credentials!

But that’s not the worst thing that can happen to such companies. The trouble continues to the painful process of locating the leaked credentials, identifying all the services that access it, and then rotating the credential value itself. Most likely, nobody in the organization would know the footprint of these credentials, so you’ll inevitably set up day-long war rooms.

This can take from weeks to several months for organizations with legacy systems. So you could continue to suffer more data breaches while trying to fix the leaked credential.

Try explaining that to your customers and the regulators!

So what’s the best way to prevent credentials from leaking? There’s obviously no golden bullet, but frequently auditing the usage is one of the best practices to adopt.

What is Usage Auditing?

Usage audit involves tracking and monitoring the use of an organization’s secrets, such as passwords, API keys, SSH keys, API tokens, etc.

More specifically, it means that you always have to do the following:

Setup Access Management

Secrets should be just that. Secrets. Your secret stack should be able to define who has access to which secret and when they can access it.

Ideally, this should be based on the Principle of Least Privilege, so users and services can only get access to the credentials they need to complete their tasks. Brownie points for giving access only for the short time required to complete the job.

This is the first step towards making sure that your secrets are safe.

What most companies get wrong is they stop here, thinking this is all that they need to do!

Monitor Audit Trails

You’ve now set up access management for all your secrets. But do you know how well it is working? Or are you just assuming it works?

What makes an access management system truly effective is robust audit logging. Like how you use product analytics to measure if a product is working well, you need audit logging to ensure your Access Management system works well.

It should keep a detailed record of all the activities related to your secrets. This includes when and by whom a secret was accessed, modified, or deleted and even when a new one was created.

Once set up, this can save you hours of effort in the future. It can help you with:

1. Compliance reporting: Most compliance certifications (SOC-2, ISO 27001, and such) have strict requirements around access to sensitive data. Proper Audit trails help demonstrate that you have a robust Access Management system.
2. Periodic Audits: It’s hygiene to do before big events and spikes (for example, a sale day in an e-commerce company). It’s lucrative for hackers to attempt data theft before big events because it would have an outsized effect on business metrics.
3. Anomaly Detection: Getting alerts whenever someone tries to access a system without the proper permission or maybe at the wrong time can help detect data breaches at their earliest stages. Simply the shift in usage trends can highlight an impending potential risk!
4. Forensic Investigation: Something may still go wrong despite all your precautions, and your secrets leak. Proper audit trails will help you investigate and determine the leak’s source.
5. Access Relevance: Do your services still need access to those credentials? Very often, services might have broad access to secrets via overly permissive policies, or services might have access to secrets they no longer need. Constant monitoring of how principals use their access will help your organization reduce the surface area for potential attacks.

Prevent Secret Drift

When setting up your secrets infrastructure for the first time, you create an ideal workflow with well-defined processes. But over time, new people join, people get promoted, while some leave your company; you add new services, and you deprecate a few.

Suddenly your setup looks very different from when it started.

The thing about secret drift is that it happens slowly when you think your secrets are being managed well, and then over time, you realize that the problem has become big and expensive to solve.

So ideally, a great usage auditing system should constantly monitor for drift and help you:

1. Detect old secrets: Most companies usually define a secret rotation policy, but usually, because of other priorities and the sheer implementation difficulty, this rotation doesn’t happen. Over time, this results in old credentials being used throughout your system.
2. Detect drift in access: When you set up your access controls, everything is clean and according to spec. But as your company grows and changes, more employees and vendors are onboarded, resulting in you losing control over who has access to and what. The worst part? You can’t change this without causing disruptions to service.
3. Detect Unintended Use: By monitoring the usage of secrets, you can detect when they are being used in ways they shouldn’t be. For example, if a secret that’s supposed to be used by a particular service is being accessed from an unexpected location, that could indicate drift.
4. Understand the Scope of Drift: If secrets are being accessed inappropriately, it might suggest systemic issues with managing them.

At first look, all of this looks pretty straightforward to implement.

But our technical architecture has become extremely complex with multiple integration points and teams, and it is tough to implement all of this without making life difficult for developers. Once that happens, developers often find workarounds, eventually increasing vulnerabilities toward data breaches.

So the ideal way to solve it is to implement it as a part of a Modern Secret Stack, which is built for the complexity of modern architecture and lets developers do what they like — build things!