Docker Swarm, Docker Compose, and Kubernetes each handle secrets differently - different injection paths, different storage backends, different failure modes. Most breakage produces no error: the app just reads an empty file or an old value. This page covers how each mechanism works, where each silently fails, and the exact commands to confirm your secrets are live in containers.
Each runtime has a different injection path, storage backend, and lifecycle. Understanding the mechanics tells you where to look when something is wrong.
/run/secrets/<name> inside the container.
Never written to disk on the worker.
--secret at deploy time or via docker service update.
Tasks must restart to see a new secret version - Swarm has no in-place rotation.
docker inspect <container> shows secret name and
target path in Spec.TaskTemplate.ContainerSpec.Secrets.
It does not show the value. The value is only accessible
inside the running container via the mounted file.
/run/secrets/<name>.
Not a tmpfs - it's a bind-mount from the host filesystem.
Compose v1 (docker-compose) silently ignores the
secrets: directive entirely.
docker inspect <container> shows the bind-mount source path.
If secrets are passed as environment variables instead of via the
secrets: directive, the value is fully visible
in Config.Env.
environment: entry with a secret value is visible to anyone who can run docker inspect on the host.
kubectl describe pod.
EncryptionConfiguration - not enabled by default on most clusters.
kubectl describe pod shows env var names and their Secret source.
kubectl get secret -o yaml shows base64-encoded values - decode
with base64 -d and you have the plaintext.
RBAC controls who can run these commands.
| Property | Docker Swarm | Docker Compose | Kubernetes |
|---|---|---|---|
| Encrypted at rest | Yes - Raft store, AES-256-GCM | No - plain host file | Only with EncryptionConfig |
| In-memory mount (tmpfs) | Yes - never written to disk | No - bind-mount from host | Yes - kubelet uses tmpfs |
| Value visible via inspect | No - name only | Yes if using environment: |
Yes via kubectl get secret -o yaml |
| Multi-service sharing | Native - attach to multiple services | Manual - each service reads same file | Native - any pod in namespace with RBAC |
| Rotation requires restart | Yes - must --force update |
No - bind-mount refreshes on file change | Env vars: yes. Volume mounts: no (60s sync) |
| Compose v1 support | N/A | Silently ignored | N/A |
Runnable commands per runtime. Each check has pass criteria and what to fix on failure. Click a card to mark it done.
echo "value" | docker secret create <name> - or from a file: docker secret create <name> ./secret.txtmy-secret -> /run/secrets/my-secretdocker service update --secret-add <name> <service> - then docker service update --force <service> to restart tasks.cat /run/secrets/<name> inside the container to confirm content.docker service update --force <service>. If still missing, verify secret is attached (check 2) and the worker node is healthy.ENV SECRET_VALUE=... or a compose file uses environment: SECRET=value, the value is baked into the image layer and visible here.DB_HOST=postgres). No plaintext secret values visible.DB_PASSWORD=hunter2). Remove ENV from Dockerfile. Read secrets from /run/secrets/<name> in application code instead. Rotate the exposed credential immediately.secrets: directive is silently ignored by docker-compose v1. This is the most common reason secrets appear to be configured but are not present in containers.docker compose version shows v2.x.x.apt install docker-compose-plugin (Debian/Ubuntu). Replace all docker-compose calls with docker compose (no hyphen) in scripts and CI.secrets: block with a file source. Environment variable substitution from .env files passes values as plaintext env vars.file: or external: true entries. The service definition references them under secrets:, not environment:.environment: MY_SECRET: ${MY_SECRET}. Move to secrets: directive: define the secret at top level with a file: source, mount it in the service under secrets:, and read from /run/secrets/<name> in code.secrets: directive with Compose v2 bind-mounts the secret file into /run/secrets/.cat /run/secrets/<name> returns expected value.secrets: block defined at top level and referenced in service definition, (c) secret source file exists on host at the path specified in file:. Recreate containers: docker compose up -d --force-recreate..env file is COPY'd into the image or bind-mounted into the container, its contents are accessible to any process in the container and to anyone who can exec into it..env file found in container. Config.Env shows only non-sensitive vars (PATH, hostname, framework settings)..env accessible in container, or plaintext secrets in Config.Env. Remove .env from Dockerfile COPY instructions. Add .env to .dockerignore. Replace environment: secret entries with secrets: directive. Rotate any exposed values.kubectl describe shows correct key names and non-zero byte counts.kubectl create secret generic <name> --from-literal=key=value -n <namespace> or from a file: kubectl create secret generic <name> --from-file=./secret.txt -n <namespace>kubectl describe pod and /proc/<pid>/environ.projected or secret volume. File visible inside container at the expected mountPath.valueFrom.secretKeyRef). Move to a volume mount in the pod spec. Update application to read from the mounted file path. Restart the pod after spec change.kubectl describe pod output with their source. This is better than hardcoded values but the value is accessible to any process in the container and shows in describe output for anyone with pod read access.EncryptionConfiguration resource - it is not enabled by default on most managed clusters (EKS, GKE, AKS have it optionally).aescbc, aesgcm, or KMS provider. Anonymous access denied for secrets.EncryptionConfiguration and restart the API server. Consider migrating to an external secrets operator (External Secrets Operator, Vault Agent) that stores values outside etcd entirely.Failures observed running Docker Swarm secrets for customers in production. Each one produces no error at deploy time. The app breaks later, or silently reads the wrong value.
docker service update without --forcedocker secret rm old-db-pass, docker secret create new-db-pass,
docker service update --secret-rm old-db-pass --secret-add new-db-pass <svc>.
Command exits 0. App continues authenticating with the old password - which has already been rotated
at the database. Auth failures start.
docker service inspect <svc> --format '{{range .Spec.TaskTemplate.ContainerSpec.Secrets}}{{.SecretName}}{{"\n"}}{{end}}'
shows the new secret name. But docker exec <container> cat /run/secrets/new-db-pass
still returns the old value. Tasks are still running old containers.
--secret-rm + --secret-add updates the service spec, but Swarm may determine
the running tasks are already compliant with the new spec and skip restarting them.
The mounted file at /run/secrets/ reflects what was injected at task start - it does not update in-place.
docker service update --force <svc> after any secret rotation.
--force increments the task spec version and triggers a rolling restart of all tasks,
causing each new container to mount the updated secret.
docker node update --availability drain <node>).
Tasks reschedule to other nodes. App throws "permission denied" or "no such file" reading
/run/secrets/<name>. Secret exists in docker secret ls.
docker service ps <svc> shows tasks in Failed or Starting
state on new nodes. Check the failure reason:
active: verify all nodes are Ready
before rescheduling tasks. Force a reroll to let the manager re-deliver secrets cleanly.
docker inspect shows secret value when ENV used instead of --secretdocker inspect <container> and finds database passwords, API keys,
or tokens in plain text in the Config.Env array. The values are in the
image layer history too (docker history --no-trunc).
ENV SECRET= instructions or ARGs used
with ENV. Check the compose file for environment: MYVAR: ${MYVAR}
entries sourcing values from .env.
ENV in a Dockerfile embeds values into image layers at build time. These layers
are stored in the image manifest and visible to anyone who can run docker inspect
or docker history on the host. Environment variables set via compose
environment: are also stored in container metadata and exposed by inspect.
ENV instructions and environment: blocks.
For runtime secrets: use the secrets: directive (Compose) or --secret
flag (Swarm). Application code reads from /run/secrets/<name>.
For build-time secrets (npm tokens, etc.): use Docker BuildKit's --secret build
flag - the value is never written to any image layer.
secrets: directive silently ignored by docker-compose v1secrets: block. Running
docker-compose up succeeds with no errors or warnings. But
/run/secrets/ inside the container is empty. App reads empty string
from secret file.
docker-compose) was written before
the Compose Specification added the secrets: key. It recognizes the key well enough
to not throw a parse error, but does not implement the injection behavior. The block is silently
ignored.
docker-compose with docker compose
everywhere - shell scripts, CI configs, Makefiles, cron jobs.
docker exec <container> cat /run/secrets/<name> returns empty output
or "Is a directory" error. The secret exists in Swarm (docker secret ls).
The service spec correctly references it. But the file is empty or a directory inside
the container.
/run,
/run/secrets, or any path that overlaps with the secret's target path, the
later mount overlays the earlier one. The secret was injected into /run/secrets/
but the bind-mount placed an empty directory there afterward, hiding the file.
/run/secrets or /run. Change conflicting
bind-mounts to use a different target path. If you need to share runtime files via a volume,
use a path like /data/runtime instead. Update the secret's target
path in the service spec if /run/secrets itself is reserved by something else.
vmfarms has run Docker Swarm for every customer since 2009. Secrets management was one of the first operational problems we had to solve at scale - before most of the tooling existed. Every failure in the catalog above is something we have seen in production, usually at 3am, usually because a deploy looked clean at the time.
The pattern we landed on: secrets are versioned objects in the Swarm store, services declare their secret dependencies explicitly, and every rotation is followed by a forced task reroll. We verify injection on every deploy using the same commands in the checklist above. The bind-mount conflict (F-05) still catches people - it is the hardest one to diagnose because nothing in the deploy output indicates a problem.
For teams hitting more complex scenarios - secret scoping across services, rotation without downtime, audit trails for who changed what - see the upcoming secret blast radius guide (coming soon, tracking issue #362).