As software engineer (daytime job and hobby) I have a lot of secrets. My Gitea token, GitHub token, Todoist token, Cloudflare API keys, Misskey token, and some others.

First or all, they all (as much as possible) short-lived tokens, or I rotate them regularly. I have a work laptop, Raspberry Pi, a personal computer, and I do dev work on all of them.

That prompted the need of a central secret manager, so all computers can fetch their secrets.

Shell Integration

As expected, most of my tokens are available through environment variables. They are stored in a secrets file which is in .gitignore and my .zshrc sources that file. That file alone wouldn’t solve my issue with updating secrets, but luckily Vault is accessible from code, and that was my next step.

Secrets File API

First I needed a way to tell the system where to fetch secrets. Experimented with a few options, but at the end I just annotated each export line and parse that from the update script.

#from-vault: mount=efertone-kv secret=github/oooomen field=username env=GITHUB_USER
export GITHUB_USER='xxx'
#from-vault: mount=efertone-kv secret=github/oooomen field=token env=GITHUB_TOKEN
export GITHUB_TOKEN='xxx'

With this annotation a script can determine what is the mount, the path to the secret, and what is the key in that secret. Easy to maintain, easy to read, and doesn’t take too much space up.

Python Script

I picked the easiest solve this issue and wrote a Python script:

  • Connects to my HashiCorp Vault.
  • Parse secrets file.
  • For each from-vault entry, it updates the next line with the secret from Vault based on the provided information.
  • Save the file.

Yep, that’s it. Only one extra library is needed, and that’s the hvac (HashiCorp Vault API client). Here is update-secrets-from-vault file:

#!/usr/bin/env python3

from dataclasses import dataclass
import hvac
import os
import sys

# This is where the secrets file lives.
SECRET_FILE = f"{os.environ['HOME']}/.zsh/secrets.env"

"""
example:

#from-vault: mount=efertone-kv secret=github field=user env=GITHUB_USER
export GITHUB_USER='yitsushi'
"""


# This is a simple dataclass that parses a line from the secrets file
# with the from_str method.
@dataclass
class VaultRef:
    mount: str = ""
    path: str = ""
    field: str = ""
    env: str = ""

    @staticmethod
    def from_str(line: str):
        ref = VaultRef("", "", "", "")
        for property in line.split(sep=" ")[1:]:
            match property.split("="):
                case ['mount', value]:
                    ref.mount = value
                case ['secret', value]:
                    ref.path = value
                case ['field', value]:
                    ref.field = value
                case ['env', value]:
                    ref.env = value

        return ref

# A simple wrapper function to get the secret from Vault
# based on the VaultRef (dataclass above).
def get_secret_value(ref: VaultRef) -> str | None:
    secret = client.secrets.kv.v2.read_secret_version(
        mount_point=ref.mount,
        path=ref.path,
        raise_on_deleted_version=False,
    )

    data = secret['data']['data']

    # For nested values, for example:#
# With this "skip_if" logic, it does not matter if the environment variable

    # #from-vault: mount=efertone-kv secret=weaveworks field=gitlab.token env=GITLAB_TOKEN
    for key in ref.field.split('.'):
        if key not in data:
            return None

        data = data[key]

    return data

# Connect to the Vault server. Without any extra options, it will
# try to use the token from the vault-token file which is generated and
# used by the Vault CLI client.
client = hvac.Client(url=os.environ['VAULT_ADDR'])
if not client.is_authenticated():
    print("client is not authenticated")
    sys.exit(1)


lines = []
skip_if = None

# Now just walk through all lines one by one, if it find a line with
# "#from-vault:" at the beginning of the line, it will try to parse it
# as a VaultRef (datacalass above) with "from_str".
#
# If the value is empty, it does nothing, just go to the next line.
# If it could get a value from Vault, it generates the "export" line
# and set "skip_if" to the expected export, so if it hits an expected
# export line, it will skip it, as it already generated that line.
with open(SECRET_FILE) as f:
    for line in f:
        if skip_if is not None:
            check = skip_if
            skip_if = None
            if line.startswith(check):
                continue
Explaimer
        line = line.strip("\n\r")
        lines.append(line)

        if line.startswith("#from-vault:"):
            ref = VaultRef.from_str(line)
            value = get_secret_value(ref)
            if value is None:
                print(f"!!! Reference '{ref.path}' not found in "
                      f"'{ref.mount}' with "
                      f"'key {ref.field}'")
                continue

            lines.append(f"export {ref.env}='{value}'")
            skip_if = f"export {ref.env}="

# Save the file.
with open(SECRET_FILE, 'w') as f:
    f.write("\n".join(lines))

Final Words

It’s not foolproof, it’s not too elegant, but it does its job. Every time I update values in Vault, I just call update-secrets-from-vault and everything gets updated. I use the Vault for other stuffs too (secrets for applications, or with Kubernetes external-secrets), but this token has only access to efertone-kv.