Benjamin Franklin said that three people can keep a secret only if two of them are dead. If you want to keep your teammates around, let's look at safely handling secrets.
When we’re working with product development teams, one of our foundational principles is that product security isn’t just a quality of what’s being delivered, it’s in the quality of your ways of working, too. There are many ways that how you’re delivering products can cause security issues. One of these ways is how you’re handling the secrets that are part of your product development (credentials, API keys, private certificates, etc.). A screw-up here can very quickly cause a Very Bad Day™, so what should we be doing to keep our secrets, well, secret?
I could give you a big list of do’s, don’ts and Best Practices here, but that’s been done already, many times. Don’t bother googling, you’ll just get blogspam from vendors of secrets management tools—here’s OWASP’s Secret Management Cheat Sheet. It’s quite long and thorough. Weirdly, it doesn’t say “Don’t log your secrets in plain text to a log file then leave those log files laying around all over the place” but I guess the authors needed to assume some level of responsibility on the reader’s part to start with, so I’ll say it. Don’t do that. And to the enormous amount of teams out there currently doing that, please stop doing that.
Fortunately, as of this year, the very good Top 10 CI/CD Security Risks from Cider Security pretty clearly tells you not to do that. Specifically, CICD-SEC-6 Insufficient Credential Hygiene. So, you don’t need a big reference list of things from me. What I do want to give you is the fundamentals—some core concepts that will help you to look at how you’re working and come up with your own solutions to your own potential ways-of-working security issues.
Security people like to talk about “Controls”. A lot. Product development teams don’t. This is unfortunate, because there are some concepts here that will help us get the fundamentals of managing secrets right. Firstly, what the heck are “Controls”?
Wikipedia to the rescue!
Security controls are safeguards or countermeasures to avoid, detect, counteract, or minimize security risks to physical property, information, computer systems, or other assets.
Ok, that’s pretty straightforward. A fence is a security control. We definitely want to “avoid, detect, counteract, or minimise” the risks of problems with our secrets so we need controls. Enough controls so that we’re feeling happy we’re going to avoid problems or catch them before they get too big, but not so many controls that they’re having a negative impact on our day-to-day delivery work.
A collection of security controls, grouped by theme, is called a Control Framework. A collection of Control Frameworks, with a large, bureaucratic organisational function to keep track of them and impose them on everyone else is called Governance1.
There are some pretty standard ways of classifying types of security control. Firstly, we can classify them by how they work:
Secondly, by when they work:
Which is all very by-the-book. We’d want to lean pretty heavily towards preventative controls as far as secrets management goes but need some detective and corrective controls for when the worst happens. And we’re going to get most mileage out of procedural and technical controls, although don’t rule out locking Steven’s keyboard in a metal box as a physical control to prevent him from pushing secrets to a public GitHub repository again2.
Something that should particularly concern us, as product development teams at (currently or aspiring to be) high-performing technology organisations3, is how scalable the control is. I’ll propose a further classification, in order of preference:
Now we have a good way to assess the effectiveness of any controls we’re thinking of adopting to make our secrets management safer. Let’s run some examples through the assessment.
We could use our peer review process to try to catch any mistakenly or deliberately committed secrets to our codebases. Add a review checklist item to look for anything that’s a credential or other secret and have the original author use the correct handling methods instead of hard-coding secrets.
This control is:
Not bad. It’s a control that can scale but it has a couple of drawbacks.
We could ensure everyone with commit access to the code runs a pre-commit hook on their local machine that checks for secrets in commits and blocks them if one is present. This prevents any detected secrets from making it into the repo at all. As the OWASP Secrets Management Cheat Sheet covers—the Yelp Detect Secrets project can give you a good start on this.
This control is:
Much better. We definitely want this one in place, potentially adding the checklist item to our peer review process to catch anyone not using the pre-commit hook and get them to configure it locally.
We could automate a scan of all our source repos with a secrets detection tool. This would find any secrets that were already committed to our codebases, or even committed but then removed without purging the commit history. If we’re using GitHub or GitLab then we’ve got this built in to the tool.
This control is:
Another good control, we should have this in place since it adds the capability to detect historic secrets compromises as well as new ones.
Let’s also add, “Invalidating the compromised secret and generating a new one”, to this control. This is your response to a detected problem.
This control is:
So, really, we’ve established that once a secret has been compromised it’s pretty hard to automate a complete response and we’re going to have to get people involved. Possibly specialist people. We should definitely do everything we can to avoid this control being needed at all, ever.
Hopefully, this has helped give you a bit of a taxonomy for security controls that will mean you can cast a critical eye over any plans to avoid secrets getting out there into the wild. Possibly you’re reading this after one already has and you want to make sure you don’t repeat that painful experience again. So, toughen up your ways of working, implement some technical controls and perhaps it won’t!
If you’re responsible for delivering digital products, services, software systems, applications, smart connected devices, pretty much anything technological and you’d like some help in making sure that everything is as secure as it needs to be then use our contact form below and we’ll get in touch for a chat.