Modern cloud platforms move at speed. The rate of change required to stay competitive in any market requires development teams to ship new features at a high velocity. When releasing new features rapidly, it is often difficult to maintain and control the configuration and secrets that are required to make your application work.
Part of this problem has been solved by moving configuration to code repositories therefore leveraging all of the advantages of git. However, secrets are often a pain point. Secrets can be difficult to maintain in a controlled and secure manner. Who should see the secret before it is stored? Where do you store the secret securely? How can you retrieve the secret for use in CI/CD?
All of these can be cumbersome issues to resolve. Mozilla SOPS can help.
It’s true. We can store all of our secrets in our git repositories. By doing so, we get all of the many benefits of git:
to mention a few. But how do we do this in a secure manner? We use an open source tool called Mozilla SOPS. SOPS stands for Secret OPerationS. It is is an editor of encrypted files that supports YAML, JSON, ENV, INI and BINARY formats and encrypts with AWS KMS, GCP KMS, Azure Key Vault, age, and PGP.
Now that we can store our secrets in git, we can store them alongside the config for our components. An example could look like this:
Here, we have the config and the secrets for our workload stored and versioned together. The workload here will read the secrets in from a .env file which is one of the formats supported by SOPS.
The secret file looks like this:
apiKey=ENC[AES256_GCM,data:6Hgj20Spy1O5lOoREApC6LeqPC8=,iv:F4WbIz0buEfDGdr83UsfCkbkrAJW+MxqU5ZwhbXKiU8=,tag:aKA8jql1JL64peE8un+32Q==,type:str]
sops_azure_kv__list_0__map_vault_url=https://demo-dev-uks-config-kv.vault.azure.net
sops_azure_kv__list_0__map_name=sops
sops_azure_kv__list_0__map_version=8773d4dd9cb1426fa9569dbd6cf33598
sops_azure_kv__list_0__map_created_at=2023-02-22T08:58:32Z
sops_unencrypted_suffix=_unencrypted
sops_azure_kv__list_0__map_enc=aWUkzZuGYmJNTc45QhRzbIkWc8Zy7k86gjPDdfItWCQm_kZJNDdtoTAWV0uJAzChI7Yor1ZNHN2c1Oyg45ugTtIjWpcyOvvMNo91VQpcif0cNVPWc9OIqPmadQeI7J1HW51lJ0S6ILPhlZBCsi7TrLH7PGAaRHIMXsyHDI6ZP9A4BL9AmLRoeJrOttzwy2zB1G3NrQaP-snVFUh2kiR8DSapnOHlKE-XrJGEMXrVVej2WEPSny9jnSr6GlqcH3uf_-Qm3dALCZZuVBLl00wcDTbMuXc3mNzzYK5K02cuSRYU9EED4Uq8-tEkT6QdWbV9kVQg4ohunOs-1VQmnpeb1euK-xv2sg1kW-pj4p_Nv6pKw6nc8REtYmmf5FAeXcS3Ks8-JGhcAnsr4bgO2cCTh0yXcNwg_OEEBYfKR4eALHb-11BVL0yhkG-OgwS04BIPzCtw5PbKjPa6fOugfZyQN_8guzcdZrN2JKO8guPE37y44piv1_3d7kGMyfKSKPg54gSNQtgPwO63bIIw694mhscvBedU5EJkkeUNdjmYeIGbV2jDzasbOrcbMNs-13Sad6xRfuDmw0s69n1HfsdmbJ1ZHTmlDU7fyYYsglnFHXEZ5YQEfC2_dpQQC6FX9lmbRfcra_nVecF4SQgGIKOwcC7dt0ZOTqQYXzN3sbRHdyg
sops_version=3.7.3
sops_lastmodified=2023-02-22T08:58:34Z
sops_mac=ENC[AES256_GCM,data:dpktA7ZStWnS4ewB0bKY0SL7nF6RxmnpwvF8tnv3odgpxc8zsLrsOEY74xaG4YElDM+S6w8fB/oG0oBRP48MhZxjQqcGURp2EBRc3iJIhHPq/HI5v/xDcPB16Nun83rFa5DUz+Snozv2eYPiMeuc/uKn/SS/c5s8LisVHXYQbxo=,iv:azxr2zXIVBF9XQB3KoLpT09G86vLycYf7jlDA6pRuM4=,tag:gXbu/jK4661OdhmWYLBe9A==,type:str]
We can see that the key:value pair is apiKey and the value is completely obfuscated. There is also metadata in the file that SOPS uses to understand how to decrypt the value. In this example, SOPS is backed off to an Azure Key Vault the details of which are stored in the metadata.
Now that we have our secret stored in a repository, we get all of the benefits of git for free. This screenshot shows what an audit trail can look like just by viewing the commit history:
We can see who committed a change to the file and when it was carried out. Also, we can leverage branch protection policies to ensure that proper review is carried out before merging to add/delete/update secrets.
What makes this approach secure and easily consumable is the difference between encryption and decryption. Anyone with the public key can encrypt secrets but only those with access to the private key can decrypt the secret file.
Since the encryption feature is safely available to all, the creation of secrets is now in the hands of developers, release managers and other personas. A simple command line operation will encrypt the file on disk, ready to be pushed to the repository:
sops --encrypt --in-place ./secrets/.env
SOPS will read the information from a .sops.yaml file in the repository to understand which system to use. An example of this file for Azure Key Vault:
creation_rules:
- azure_keyvault: https://my-kv.vault.azure.net/keys/sops/8773d4dd9cb1426fa9561726377188
This can be configured for other methods of encryption.
The person who creates the secret to begin with can encrypt it and make it available to the wider platform in a safe and secure manner. No more passing secrets around in plain text before they reach a key vault.
Typically, only your CI/CD process would have access to the private key in order to decrypt secrets just-in-time in automation. In our example, we are backing off to Azure Key Vault. Access control over the Key Vault is controlled by native Azure RBAC roles. Our CI/CD service principal has the Key Vault Secrets Officer role assigned and is therefore able to access the private key in order to decrypt secrets. Since we are using Azure roles to control access to the resources, it follows that is would be possible to use Azure Privileged Identity Management (PIM) to elevate permissions for troubleshooting any issues with secrets.