The Lost Art of Keeping a Secret

When we’re working with product development teams, one of our foundational principles is that product security isn’t just a quality of what’s being delivered, it’s in the quality of your ways of working, too. There are many ways that how you’re delivering products can cause security issues. One of these ways is how you’re handling the secrets that are part of your product development (credentials, API keys, private certificates, etc.). A screw-up here can very quickly cause a Very Bad Day™, so what should we be doing to keep our secrets, well, secret?

The Reading Material

I could give you a big list of do’s, don’ts and Best Practices here, but that’s been done already, many times. Don’t bother googling, you’ll just get blogspam from vendors of secrets management tools—here’s OWASP’s Secret Management Cheat Sheet. It’s quite long and thorough. Weirdly, it doesn’t say “Don’t log your secrets in plain text to a log file then leave those log files laying around all over the place” but I guess the authors needed to assume some level of responsibility on the reader’s part to start with, so I’ll say it. Don’t do that. And to the enormous amount of teams out there currently doing that, please stop doing that.

Fortunately, as of this year, the very good Top 10 CI/CD Security Risks from Cider Security pretty clearly tells you not to do that. Specifically, CICD-SEC-6 Insufficient Credential Hygiene. So, you don’t need a big reference list of things from me. What I do want to give you is the fundamentals—some core concepts that will help you to look at how you’re working and come up with your own solutions to your own potential ways-of-working security issues.

Back to Basics

Security people like to talk about “Controls”. A lot. Product development teams don’t. This is unfortunate, because there are some concepts here that will help us get the fundamentals of managing secrets right. Firstly, what the heck are “Controls”?

Wikipedia to the rescue!

Security controls are safeguards or countermeasures to avoid, detect, counteract, or minimize security risks to physical property, information, computer systems, or other assets.

Ok, that’s pretty straightforward. A fence is a security control. We definitely want to “avoid, detect, counteract, or minimise” the risks of problems with our secrets so we need controls. Enough controls so that we’re feeling happy we’re going to avoid problems or catch them before they get too big, but not so many controls that they’re having a negative impact on our day-to-day delivery work.

A collection of security controls, grouped by theme, is called a Control Framework. A collection of Control Frameworks, with a large, bureaucratic organisational function to keep track of them and impose them on everyone else is called Governance¹.

There are some pretty standard ways of classifying types of security control. Firstly, we can classify them by how they work:

• Procedural controls (e.g. ways of working, policies, standards)
• Technical controls (e.g. passwords, firewalls, encryption)
• Physical controls (e.g. fences again, doors, cool beepy numberpads that get you into your office floor)

Secondly, by when they work:

• Preventative controls—intended to prevent an incident occurring
• Detective controls—intended to identify and characterise an incident in progress
• Corrective controls—intended to limit the extent of any damage caused by the incident

Which is all very by-the-book. We’d want to lean pretty heavily towards preventative controls as far as secrets management goes but need some detective and corrective controls for when the worst happens. And we’re going to get most mileage out of procedural and technical controls, although don’t rule out locking Steven’s keyboard in a metal box as a physical control to prevent him from pushing secrets to a public GitHub repository again².

Something that should particularly concern us, as product development teams at (currently or aspiring to be) high-performing technology organisations³, is how scalable the control is. I’ll propose a further classification, in order of preference:

1. Fully automated control
2. Partially automated control with a human response
3. Controls that all delivery teams can do for themselves
4. Controls that a central team must do for them
5. Controls that only one individual in the organisation can do

Now we have a good way to assess the effectiveness of any controls we’re thinking of adopting to make our secrets management safer. Let’s run some examples through the assessment.

How good are we?

Control 1: Peer reviewing changes

We could use our peer review process to try to catch any mistakenly or deliberately committed secrets to our codebases. Add a review checklist item to look for anything that’s a credential or other secret and have the original author use the correct handling methods instead of hard-coding secrets.

This control is:

• Procedural, it’s a way of working so we’re going to have to constantly ensure it gets done.
• Preventative, but probably a bit late if it’s all we’re going to do. The secret could be considered compromised if its hit any branch in the DVCS.
• A control that all delivery teams can do for themseves. There’s a natural scale to these controls—as your organisation grows and adds people/teams, you have more people to perform the activity.

Not bad. It’s a control that can scale but it has a couple of drawbacks.

Control 2: A pre-commit git hook

We could ensure everyone with commit access to the code runs a pre-commit hook on their local machine that checks for secrets in commits and blocks them if one is present. This prevents any detected secrets from making it into the repo at all. As the OWASP Secrets Management Cheat Sheet covers—the Yelp Detect Secrets project can give you a good start on this.

This control is:

• Technical, it’s a technical control that, once implemented, works reliably and consitently. You need to enforce everyone uses it in some way, however.
• Preventative, and preventative early enough to catch any problems before they become, well, problems.
• A fully automated control, our “gold standard” for scalability. No humans needed.

Much better. We definitely want this one in place, potentially adding the checklist item to our peer review process to catch anyone not using the pre-commit hook and get them to configure it locally.

Control 3: A repo scanner for existing secrets in code

We could automate a scan of all our source repos with a secrets detection tool. This would find any secrets that were already committed to our codebases, or even committed but then removed without purging the commit history. If we’re using GitHub or GitLab then we’ve got this built in to the tool.

This control is:

• Technical, a good, reliable control. Once configured, no need for continuous human intervention.
• Detective, we’ve already caused a problem by the time this control is effective. But it will definitely help to catch the problem early and, hopefully, before any serious damage is caused.
• A Partially automated control with a human response, in this case we’re only having to manage by exception so we have a pretty good scaling factor.

Another good control, we should have this in place since it adds the capability to detect historic secrets compromises as well as new ones.

Control 4: Panicking a bit and Googling how to expunge git commit history because we can never remember the commands

Let’s also add, “Invalidating the compromised secret and generating a new one”, to this control. This is your response to a detected problem.

This control is:

• Procedural, you should probably write a bit of a Playbook for what to do for this control, to prevent mistakes in your response.
• Corrective, we’re trying to resolve the incident once its happened quickly to limit any damage done
• Ideally this is a control that every delivery team can do for themselves, but possibly it needs a central security team to rotate keys/regenerate credentials. Worst case it could be a single person who has the ability to do this. That wouldn’t be good.

So, really, we’ve established that once a secret has been compromised it’s pretty hard to automate a complete response and we’re going to have to get people involved. Possibly specialist people. We should definitely do everything we can to avoid this control being needed at all, ever.

Hopefully, this has helped give you a bit of a taxonomy for security controls that will mean you can cast a critical eye over any plans to avoid secrets getting out there into the wild. Possibly you’re reading this after one already has and you want to make sure you don’t repeat that painful experience again. So, toughen up your ways of working, implement some technical controls and perhaps it won’t!

If you’re responsible for delivering digital products, services, software systems, applications, smart connected devices, pretty much anything technological and you’d like some help in making sure that everything is as secure as it needs to be then use our contact form below and we’ll get in touch for a chat.