Secrets Manager for Homelab

For a few years, I’ve been managing the configuration of a bunch of self-hosted services using Ansible Playbooks. Each playbook needed at least one secret — the sudo password. Many of them needed to manage more (e.g. SMTP credentials for email notifications). Because I’ve always been paranoid about security, I stored most of those secrets in Ansible Vault, the password for which is stored in only one location — my memory. Therefore, each time I ran any of those playbooks, I’d have to enter two passwords interactively: the sudo password and the Ansible Vault password.

Aside from that, I wasn’t able to automate the execution of those playbooks via any CI/CD — I really didn’t want to expose secrets by entering these passwords as sensitive environment variables in my GitLab pipelines. Putting all those secrets in many locations makes tracking them extremely difficult over time.

Finally, I decided it’s time to centralize secrets storage by using a Password Manager. I was considering two of them: Hashicorp Vault, which I’ve had some limited experience with (only as a user, though — never operated an instance myself) and Bitwarden’s new Secrets Manager service. I was really biased towards Hashicorp Vault because of my previous experience. But I was a bit afraid of a steep learning curve in terms of managing its instance myself.

Let me describe the core differences between the two that made me choose one over the other.

Bitwarden Secrets Manager

Bitwarden’s Secrets Manager is a simple key-value store, and in my view, this is the feature I needed. I could store Ansible Vault passwords, sudo passwords, SMTP credentials, and more in the vault. At the moment, Bitwarden does not offer a self-hosted Secrets Manager service. The only option is their SaaS.

In order to solve my Ansible Playbook running problem, I’d have to do the following:

1. Create a Project within my Bitwarden Secrets Manager account.
2. Create all the secrets that I’d use in my playbook.
3. Create Machine Account(s) and associate it/them with the Project.
4. For each machine account, generate an access token.
5. When running Playbook locally via the CLI, I’d only need to store the generated Access Token (e.g., in a CLI password manager like pass).
6. When running Playbook via CI/CD, I’d have to store the generated Access Token as an environment variable.

The killer feature of Bitwarden Secrets Manager, though, is that it’s end-to-end encrypted. That means that Bitwarden doesn’t know what the values stored in your secrets are. The generated access tokens, used either by you directly via the CLI or by your CI/CD pipelines, act as decryption keys for your secrets’ values. Those are decrypted locally, where they are accessed (your CLI or CI/CD pipeline). That is important because Bitwarden has a gigantic target on its back — many APTs would like to hack it and get their hands on its customers’ secrets. Well, hacking Bitwarden’s back-end services would not be enough. They’d also need to obtain the customers’ access tokens.

Hashicorp Vault

HashiCorp Vault also provides key-value secret storage. I could use it the same way as Bitwarden Secrets Manager.

However, Hasicorp Vault is so much more than just a simple key-value store. In essence, it offers cryptography-as-a-service functionality, and that is also a killer feature.

To illustrate it, consider the following scenario. You’re implementing an API that needs to store data encrypted with the key you’re managing. You can ask Vault to create the encryption key. That key will never leave the Vault instance and will therefore never be known to anyone else (including your API or you). When your API needs to store data, it sends it to Vault, which encrypts it. Vault, as requested, will encrypt the data internally and send the corresponding ciphertext back to the API. Now your API can store encrypted data. Perfect. Now the other way around — the API needs to decrypt the data. First, it reads the saved ciphertext and sends it to Vault for decryption. Vault decrypts the ciphertext and returns the decrypted data to the API.

There are important implications of that kind of design. For example, if your API is compromised (hacked), attackers would only be able to request that Vault encrypt/decrypt data. They would not be able to do so themselves — they wouldn’t know the encryption key. In other words, the API got compromised, but the encryption key has not.

The alternative would be to design your API so it handles encryption itself. The API would generate the encryption key, store it in Vault, and whenever encryption/decryption is needed, fetch the key from Vault and perform the cryptographic operation itself. With Bitwarden, this would be the only option.

The implications of this design are that when the API is compromised, the attackers gain access to the encryption key. That’s much more dangerous in most scenarios, because if they had access to any ciphertext previously encrypted with that key, they’d be able to decrypt it. The only option to recover going forward is to rotate the encryption key. That’s a crucial observation, because in some cases, rotating the key is impossible (for example, when the key is baked into hardware devices sold to gazillion customers).

That encryption-as-a-service is really awesome. But that means the Vault must have access to the keys at all times to perform the encryption, so it simply cannot be end-to-end encrypted. If the Vault itself is hacked, the attackers get a huge present…

Conclusion

For my own threat model, end-to-end encryption won. Therefore, I’ve chosen Bitwarden Secrets Manager.

Using Bitwarden Secrets Manager is really easy, and there’s no operational overhead in the form of managing the Secrets Manager instance. Hashicorp Vault would be a viable option for me only as a self-hosted instance. Well, let’s face it — Hashicorp Vault’s Cloud offering does have a huge target on its back. My personal self-hosted instance for my home lab… would not. :) However, as I’ve said above, learning all the ins and outs of self-hosting would simply be too much for my limited spare time.

Anyway, going back to my Ansible Playbooks — now I store the Ansible Vault passphrase in Bitwarden Secrets Manager. And all the other secrets (including sudo passwords) in Ansible Vault are stored with my code on my self-hosted GitLab (accessible only on my home lab network). Best of both worlds. :)