Keeping secrets in Puppet

Security is always about making trade offs, and credential storage is one such case where you need to juggle usability against security. Take our server configurations at work, for example. So we can sleep at night, they're all handled by Puppet. That left us with a dilemma: if everything a server needs to start up is in Puppet, that also means it needs to know about secret stuff.

all our eggs in one basket

Storing credentials in our Puppet git repository is great in the event of a host dying at 3am, as it can be seamlessly replaced without having to scramble around trying to find details. On the downside, there's then a single place that houses all our secrets, ripe for the picking from wherever the git repository may be.

We used to run Puppet in a decentralised way, without Puppet masters, but as Adam mentioned, we moved to the agent/server model for a few reasons. Thankfully, this also gave us a solution we're happy with for storing secrets in our puppet configuration in the form of hiera-gpg. As the name might suggest it combines the power of hiera and the encryption of GPG to allow secret and not secret configuration to be combined when compiling a host's configuration. This wouldn't have worked in the decentralised world, as all hosts would have needed decryption keys and hence had access to all secrets. With puppet masters, however, decryption happens on the master and only the secrets a client needs are given to it.

HOWTO

This tutorial guides you through the process of configuring the GPG keys and hiera. As I said, security is all about trade offs of usability, and we've tried to make it a little more user-friendly with the addition of a couple of shell scripts to help out when dealing with encrypted files.

YAML files are encrypted with a number of public keys: those of some engineers as well as those used by the puppet master hosts. The most used script is one to edit encrypted YAML files. Your editor of choice will be invoked for you to make changes, the file validated and re-encrypted.

We have another that will re-encrypt all files: this is used when we rotate or change public keys. Both are in a private repository that I can't make public, but I've put them here and here if you're interested in using them. There are a few tweaks I'd like to make, such as trapping exits to enforce the tidy-up of plaintext files if they exist, but you're welcome to use them as a starting point.

what next?

This solution is great if you're encrypting short string values, but we'd also like to be able to encrypt entire files and use the same on-the-fly decryption. I may well be finding myself getting my hands dirty and writing my first ruby gem soon... unless you've already found a way of doing it. In which case, do let me know!

_{This originally appeared on our company blog. Picture credit: Sameer Vasta}