Shell Lesson 26 of 42

Shell Secrets Handling: Env-Vars vs Files, Vault Integration, Ephemeral Credentials, ps/journal Leaks & no_log Discipline

Secrets — API keys, passwords, tokens, certificates — are uniquely hostile to shell. Unlike a high-level language where you can keep a secret string in memory and explicitly pass it to a function, every shell command is a process invocation, and secrets pass through:

This is the common cause of secret leaks in DevOps. A senior engineer adds aws s3 cp --secret-key $KEY ... to a script, the script runs in CI, the CI uploads logs to S3, and the secret is now searchable forever. CIs regularly publish this pattern to public artifact buckets.

This lesson covers:

By the end, your scripts will handle secrets without leaking them, even when something goes wrong.


1. The six leak channels

1.1 Command-line arguments — visible in ps

$ aws --secret-access-key AKIAIOSFODNN7EXAMPLE s3 ls

While that command runs, anyone on the system can see it:

$ ps -ef | grep aws
ubuntu  1234  ... aws --secret-access-key AKIAIOSFODNN7EXAMPLE s3 ls

ps reads /proc/$pid/cmdline, which is world-readable on Linux (mode 444 by default). Even unprivileged users on the same host can see it. On shared CI runners, multi-tenant containers, jump hosts — this is a leak.

Fix: never pass secrets as command-line arguments. Use environment variables, files, or stdin (next sections).

1.2 Environment variables — visible in /proc/$pid/environ

Environment variables are less leaky than argv but still readable:

$ AWS_SECRET_ACCESS_KEY=foo aws s3 ls

# In another terminal:
$ cat /proc/$pid/environ | tr '\0' '\n' | grep AWS_SECRET
AWS_SECRET_ACCESS_KEY=foo

By default, /proc/$pid/environ is mode 400only the process owner can read it. So same-user processes can read each other’s env, but cross-user reads require root.

This is “good enough” for most cases, but be aware:

For most production: env vars are the standard secret transport for CLI tools (AWS, GCP, Azure all use them).

1.3 Shell history — ~/.bash_history

$ MY_TOKEN=secret ./deploy.sh
$ history | grep TOKEN
1234  MY_TOKEN=secret ./deploy.sh

Bash records every command. Years later, a colleague greps your history, finds the token. Or a backup of ~/.bash_history ends up somewhere it shouldn’t.

Fixes:

1.4 Tracing — set -x

#!/usr/bin/env bash
set -Eeuxo pipefail   # ← `x` is the killer

KEY="$AWS_SECRET_ACCESS_KEY"
aws --secret-access-key "$KEY" s3 ls

set -x prints every command before execution, with all variables expanded:

+ KEY=AKIAIOSFODNN7EXAMPLE
+ aws --secret-access-key AKIAIOSFODNN7EXAMPLE s3 ls

If your CI captures stderr (which it always does), the secret is now in your build log. Permanent.

Fixes:

1.5 Logging and echo

echo "DEBUG: using token=$MY_TOKEN" >&2

That goes to stderr. In production, stderr goes to journald, syslog, or a log file. Now your secret is in /var/log/syslog and shipped to your central logging system. Indexed. Searchable.

Rule: never log secrets. Even in debug mode. Even temporarily. The “I’ll remove the debug line later” pattern fails 100% of the time.

If you must log that an operation happened, log it without the secret value:

echo "Authenticating with token (length=${#MY_TOKEN})" >&2

${#var} is the length — useful for debugging “is the token even set?” without revealing its value.

1.6 Core dumps

If your script forks a binary (e.g. python, node) that crashes, the kernel writes a core dump containing memory contents — including secrets that were in the address space. Core dumps land in /var/lib/systemd/coredump/ or wherever core_pattern points.

Fix:

For most scripts this is overkill. For privileged or secret-handling daemons, set it as a defensive baseline.


2. The four ways to pass secrets — pick one

2.1 Environment variable (most common)

# Caller:
export AWS_SECRET_ACCESS_KEY=$(get_secret aws/s3-key)

# Script:
[[ -n "${AWS_SECRET_ACCESS_KEY:-}" ]] || die "AWS_SECRET_ACCESS_KEY required"
aws s3 ls    # aws CLI reads from env automatically

Pros:

Cons:

2.2 File (best for keys, certs, multi-line secrets)

# Caller writes the secret to a file with mode 600:
get_secret aws/s3-key > /tmp/aws-key.tmp.$$
chmod 600 /tmp/aws-key.tmp.$$

# Script reads:
KEY=$(< /tmp/aws-key.tmp.$$)

# Always clean up:
trap 'rm -f /tmp/aws-key.tmp.$$' EXIT

Pros:

Cons:

For SSH keys, TLS certs, GCP service account JSON, always use file mode.

2.3 Stdin (best for one-shot operations)

# Pass secret via stdin to a tool that supports reading it:
echo "$PASSWORD" | sudo -S cmd
get_secret db/admin | psql --no-password -h "$DB_HOST" -U admin -d mydb -f schema.sql

Pros:

Cons:

2.4 Argument (the worst — avoid)

# DON'T:
mysql -u admin -psupersecret mydb

The password is in ps for the duration of the connection. Never do this. Most CLIs that accept -p PASSWORD also accept -p (no value, prompts) or have a --password-file=FILE form. Use those.

2.5 Decision matrix

Use case Best transport
Cloud CLI (aws, gcloud, az) Env var (their default)
TLS cert / private key File with 0600
Multi-line JSON service account File with 0600
Database connection string with password .pgpass file or env var
One-off admin command via sudo Stdin
Container orchestration (Docker/K8s) Mounted secret file (volume)
systemd-managed service EnvironmentFile= (dropped after read) or LoadCredential=

3. Vault, Secrets Manager, Key Vault — fetching secrets at runtime

The principle: secrets are not in code, not in env at deploy time. They’re fetched at runtime from a vault, used briefly, and discarded.

3.1 HashiCorp Vault

# Authenticate (assuming approle):
VAULT_TOKEN=$(vault write -field=token auth/approle/login \
  role_id="$ROLE_ID" secret_id="$SECRET_ID")
export VAULT_TOKEN

# Fetch a secret:
SECRET=$(vault kv get -field=password secret/myapp/db)

# Use it briefly:
PGPASSWORD="$SECRET" psql -h db.example.com -U myapp -c "SELECT 1"

# Clear it:
unset SECRET PGPASSWORD VAULT_TOKEN

vault reads VAULT_ADDR and VAULT_TOKEN from env. The -field=password flag makes it print just the value, not formatted output — easy to capture into a variable.

For service identity, you bootstrap with AppRole (role_id + secret_id, similar to OIDC client credentials) or with Kubernetes auth method (the pod’s service account token authenticates to Vault). The secret_id can be short-lived and machine-specific, dramatically limiting blast radius.

3.2 AWS Secrets Manager

# Fetch with awscli:
SECRET=$(aws secretsmanager get-secret-value \
  --secret-id myapp/db/password \
  --query SecretString \
  --output text)

# Or with jq if it's structured JSON:
RAW=$(aws secretsmanager get-secret-value --secret-id myapp/db --query SecretString --output text)
USER=$(echo "$RAW" | jq -r .username)
PASS=$(echo "$RAW" | jq -r .password)

For credentials to call AWS itself, use IAM roles attached to the EC2 instance / Lambda / ECS task — never embed AWS credentials in scripts. Secrets Manager is for secrets your app needs (database passwords, third-party API keys), not AWS credentials.

3.3 GCP Secret Manager

SECRET=$(gcloud secrets versions access latest \
  --secret=myapp-db-password \
  --project=my-project)

gcloud authenticates from the metadata server (when running on GCE/GKE/Cloud Run) or from ~/.config/gcloud (when on a developer machine). No secrets in scripts.

3.4 Azure Key Vault

SECRET=$(az keyvault secret show \
  --vault-name my-vault \
  --name myapp-db-password \
  --query value \
  --output tsv)

Same pattern: managed identity authenticates az, no secrets in scripts.

3.5 The reusable lib/secrets.sh

# lib/secrets.sh — drop-in secret helpers
# Source from any script. Usage: secret=$(get_secret aws|gcp|az|vault PATH)

get_secret_aws() {
  aws secretsmanager get-secret-value \
    --secret-id "$1" \
    --query SecretString --output text
}

get_secret_gcp() {
  gcloud secrets versions access latest --secret="$1"
}

get_secret_az() {
  local vault=${VAULT_NAME:?VAULT_NAME required}
  az keyvault secret show --vault-name "$vault" --name "$1" --query value --output tsv
}

get_secret_vault() {
  local field=${SECRET_FIELD:-value}
  vault kv get -field="$field" "$1"
}

# Generic dispatch — picks backend from SECRET_BACKEND env var:
get_secret() {
  local path=$1
  case ${SECRET_BACKEND:-aws} in
    aws)   get_secret_aws   "$path" ;;
    gcp)   get_secret_gcp   "$path" ;;
    az)    get_secret_az    "$path" ;;
    vault) get_secret_vault "$path" ;;
    *) echo "Unknown SECRET_BACKEND: ${SECRET_BACKEND}" >&2; return 1 ;;
  esac
}

Usage:

source /usr/local/lib/myapp/secrets.sh

DB_PASSWORD=$(get_secret myapp/db/password)
PGPASSWORD="$DB_PASSWORD" psql ...
unset DB_PASSWORD PGPASSWORD

Same script works on AWS, GCP, Azure, or Vault — pick by setting SECRET_BACKEND in the environment.


4. Ephemeral credentials — short-lived is safer than long-lived

The pattern: credentials live for minutes, not weeks. If they leak, blast radius is naturally limited by their TTL.

4.1 AWS STS — assume-role for short-lived credentials

# Get 15-minute credentials by assuming a role:
CREDS=$(aws sts assume-role \
  --role-arn arn:aws:iam::123456789012:role/MyAppRole \
  --role-session-name "deploy-$(date +%s)" \
  --duration-seconds 900)

export AWS_ACCESS_KEY_ID=$(echo "$CREDS" | jq -r .Credentials.AccessKeyId)
export AWS_SECRET_ACCESS_KEY=$(echo "$CREDS" | jq -r .Credentials.SecretAccessKey)
export AWS_SESSION_TOKEN=$(echo "$CREDS" | jq -r .Credentials.SessionToken)

# Use them. After 15 minutes they expire automatically.
aws s3 ls

If those credentials leak, they’re useless after 15 minutes. The TTL is the safety net.

4.2 OIDC federation — no static credentials anywhere

GitHub Actions and most modern CI systems support OIDC: the CI runner gets a short-lived JWT token from the IDP, exchanges it for cloud credentials with no static secrets stored anywhere.

# .github/workflows/deploy.yml
permissions:
  id-token: write          # Required for OIDC
  contents: read

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/GitHubActions
          aws-region: us-east-1
      # Now `aws` is configured with short-lived creds, no AWS_SECRET_ACCESS_KEY needed.
      - run: aws s3 ls

The OIDC trust policy on the IAM role specifies which GitHub repo and branch can assume it. No static credentials anywhere in the repo, in GitHub, or in CI. Compromise of the GitHub repo gives an attacker code-edit access but no cloud credentials.

This is the modern best practice. If you’re still using long-lived AWS access keys in GitHub secrets, migrate to OIDC.

4.3 Kubernetes service accounts

In Kubernetes, pods carry their service account’s token via a mounted file (/var/run/secrets/kubernetes.io/serviceaccount/token). With the IRSA pattern (AWS) or workload identity (GCP/Azure), these tokens federate to cloud credentials — same OIDC mechanism, automatic.

# In a pod with IRSA configured:
aws s3 ls         # Just works. AWS SDK reads the K8s token, exchanges for IAM creds.

No secrets in the script, no secrets in the pod spec. The CSI driver provides the token; AWS SDK does the federation.


5. The “no_log” discipline

Borrowed from Ansible: mark sensitive operations as no-log, suppress all output, and audit the script for accidental exposure.

5.1 The pattern

# A wrapper that ensures the inner command's output and trace are silenced:
no_log() {
  local saved_xtrace=""
  case $- in *x*) saved_xtrace=on; set +x ;; esac
  "$@"
  local rc=$?
  [[ $saved_xtrace == on ]] && set -x
  return $rc
}

# Use it for sensitive operations:
no_log mysql -u admin -p"$PASSWORD" -e "SELECT 1"

It saves whether set -x is currently on, disables it during the command, restores afterwards. The command’s stderr/stdout is unchanged; only the trace is suppressed.

5.2 Suppressing output entirely

For commands whose output might leak secrets:

# Run this command and discard all output:
no_log_quiet() {
  no_log "$@" >/dev/null 2>&1
}

no_log_quiet mysql -u admin -p"$PASSWORD" -e "DROP DATABASE temp_data"

Drops both the trace and the output. Only the exit code is observable.

5.3 Audit the script for leaks

# In CI:
grep -nE '(echo|printf).*\$.*(PASSWORD|TOKEN|SECRET|KEY)' bin/* lib/*.sh

Catches lines like echo "Using token: $TOKEN" that would leak. Add to your pre-commit hook or CI lint.

5.4 The set +x zone discipline

For long sections that touch secrets:

set +x                                  # Disable trace
{
  PASSWORD=$(get_secret db/admin)
  PGPASSWORD="$PASSWORD" psql ...
  unset PASSWORD PGPASSWORD
} >/dev/null 2>&1                       # And suppress stdout/stderr
set -x                                  # Re-enable

# Continue with normal operations.

Wrapping in braces creates a logical zone; the > /dev/null 2>&1 is for the output of the commands, not just the trace.


6. Container secrets — the right way

6.1 Docker build-time vs run-time

The biggest mistake: baking secrets into images.

# DON'T:
FROM ubuntu:22.04
ENV DB_PASSWORD=supersecret
RUN apt-get install -y mypackage

That DB_PASSWORD is now in the image forever, in a layer, recoverable by anyone who has the image. Every push to a public registry is a leak.

The fix: secrets are runtime-only, never image-time.

# OK — image is generic, accepts secrets at run time:
FROM ubuntu:22.04
RUN apt-get install -y mypackage
COPY entrypoint.sh /
ENTRYPOINT ["/entrypoint.sh"]
# Run with secret via env (one-time):
docker run --env-file <(get_secret myapp/env) myimage

6.2 Docker BuildKit secrets

If you genuinely need a secret during build (to download a private artifact), use BuildKit secret mounts:

# syntax=docker/dockerfile:1.4
FROM ubuntu:22.04
RUN --mount=type=secret,id=npmtoken \
    NPM_TOKEN=$(cat /run/secrets/npmtoken) && \
    npm install --registry=https://my-private-npm
DOCKER_BUILDKIT=1 docker build --secret id=npmtoken,src=$HOME/.npmrc -t myimage .

The secret is mounted as a tmpfs file during the RUN, never written to a layer. After the build, it’s gone.

6.3 Kubernetes secrets

Mount as files (preferred over env):

volumes:
  - name: db-credentials
    secret:
      secretName: db-credentials
containers:
  - name: app
    volumeMounts:
      - name: db-credentials
        mountPath: /var/run/secrets/db
        readOnly: true

Then in your container script:

DB_PASSWORD=$(< /var/run/secrets/db/password)

Files are visible only inside the pod, only to the container’s user, with proper mode. Env-var secrets in K8s are visible in pod spec (often readable), so prefer file mounts.

6.4 Sealed secrets / SOPS for git-stored secrets

If you must put encrypted secrets in git (gitops pattern), use SOPS:

# Encrypt:
sops -e --aws-kms arn:aws:kms:... secrets.yaml > secrets.enc.yaml
# Decrypt at runtime (in a script that already has KMS access):
sops -d secrets.enc.yaml > secrets.yaml

The encryption key (KMS, age, gpg) is the real secret; SOPS handles the encrypted content. With KMS, only your IAM principal can decrypt — even with the encrypted file, an attacker can’t read the secret without your IAM credentials.


7. Auditing existing scripts for leaked secrets

7.1 The grep-able patterns

For any codebase, these regexes catch the common leaks:

# AWS access key:
grep -rnE 'AKIA[0-9A-Z]{16}' .

# AWS secret key (40 chars base64-ish):
grep -rnE '[A-Za-z0-9/+=]{40}' . | head     # Lots of false positives; review.

# GitHub PAT:
grep -rnE '(ghp|gho|ghu|ghs|ghr)_[A-Za-z0-9]{36}' .

# Generic API tokens:
grep -rinE '(api[_-]?key|secret|password|token)\s*=\s*["'\'']?[A-Za-z0-9_-]{16,}' .

# Private keys:
grep -rln 'BEGIN .* PRIVATE KEY' .

# .env files:
find . -name '.env' -o -name '.env.*' | grep -v .gitignore

7.2 Tools

In CI:

- uses: gitleaks/gitleaks-action@v2
  with:
    config-path: .gitleaks.toml

This blocks any PR that introduces a secret.

7.3 If a secret leaked — what to do

  1. Rotate immediately. Generate a new credential, update consumers, revoke the old one. Speed matters.
  2. Audit usage logs for the leaked credential. AWS CloudTrail, GitHub audit log, etc. — when was it used, by whom, from where?
  3. Purge from history, but accept that anyone with prior git clone still has it. Rotation is the only real fix.
  4. Postmortem: how did it get committed? Was the pre-commit hook missing? Was a .env file not in .gitignore?

The git filter-branch / git filter-repo route is for “I want this gone from history” — but if the repo has been pushed and seen, it’s already cached, scraped by bots, in CI artifacts. Rotate, don’t try to redact.


8. The reusable patterns

8.1 The strict-mode preamble for secret-handling scripts

#!/usr/bin/env bash
set -Eeuo pipefail -f
IFS=$'\n\t'

# Pin environment:
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
export PATH
TZ=UTC LC_ALL=C
export TZ LC_ALL

# Disable core dumps for safety:
ulimit -c 0

# Disable shell history:
export HISTFILE=/dev/null
unset HISTSIZE
unset HISTFILESIZE

# Source the secret helpers:
source /usr/local/lib/myapp/secrets.sh

# At the END of the script (or via trap), make sure secrets are unset:
cleanup() {
  unset DB_PASSWORD AWS_SECRET_ACCESS_KEY API_TOKEN
}
trap cleanup EXIT

8.2 The “credential lifetime” pattern

Always scope secrets to the smallest block possible:

do_db_thing() (
  # Subshell: secret only exists in this subshell, no leak to caller.
  PGPASSWORD=$(get_secret db/admin)
  export PGPASSWORD
  psql -h db.example.com -U admin "$@"
)

# Caller doesn't see PGPASSWORD:
do_db_thing -c "SELECT 1"
do_db_thing -f /path/to/migration.sql

The subshell () makes the variable local. After the function returns, the secret is garbage-collected.

8.3 The “no static credentials” pattern

# Top of script — assert that we're using ephemeral creds:
require_ephemeral_credentials() {
  # AWS-specific:
  if [[ -n "${AWS_SESSION_TOKEN:-}" ]]; then
    return 0  # Has session token = ephemeral.
  fi
  if [[ -f ~/.aws/credentials ]]; then
    if grep -q 'aws_session_token' ~/.aws/credentials; then
      return 0
    fi
  fi
  if [[ -n "${AWS_WEB_IDENTITY_TOKEN_FILE:-}" ]]; then
    return 0  # OIDC.
  fi
  echo "ERROR: this script requires ephemeral AWS credentials." >&2
  echo "Use OIDC (gh-actions), assume-role (sts), or instance profile." >&2
  exit 1
}
require_ephemeral_credentials

This refuses to run with long-lived static credentials. Forces the operator to use the safer mode.

8.4 The “secret was used, prove it works” pattern

After fetching, verify before using:

DB_PASSWORD=$(get_secret myapp/db)

# Verify the password actually authenticates before doing anything destructive:
if ! PGPASSWORD="$DB_PASSWORD" psql -h "$DB_HOST" -U myapp -c '\q' >/dev/null 2>&1; then
  echo "Authentication failed. Aborting." >&2
  exit 1
fi

# Now safe to do the real work:
PGPASSWORD="$DB_PASSWORD" psql -h "$DB_HOST" -U myapp -f migration.sql

This catches “secret was rotated, my cache is stale” before you’ve already deleted half the database.


9. The full secrets-aware script template

#!/usr/bin/env bash
# myscript - description
# Handles secrets safely. See SECRETS.md for the discipline.

set -Eeuo pipefail -f
IFS=$'\n\t'

# Hardening:
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
export PATH
TZ=UTC LC_ALL=C
export TZ LC_ALL
ulimit -c 0
export HISTFILE=/dev/null

unset BASH_ENV ENV CDPATH GLOBIGNORE LD_PRELOAD LD_LIBRARY_PATH

# Helpers:
source /usr/local/lib/myapp/secrets.sh

# Cleanup on exit:
cleanup() {
  # Unset all secret variables:
  unset -v DB_PASSWORD API_TOKEN PGPASSWORD AWS_SECRET_ACCESS_KEY
  # Remove temp files:
  [[ -n "${TMPDIR:-}" ]] && rm -rf -- "$TMPDIR"
}
trap cleanup EXIT INT TERM

TMPDIR=$(mktemp -d -t myscript.XXXXXX)
chmod 700 "$TMPDIR"

# Main logic:
DB_PASSWORD=$(get_secret myapp/db/password)
[[ -n "$DB_PASSWORD" ]] || { echo "Failed to fetch DB password" >&2; exit 1; }

# Verify before destructive use:
if ! PGPASSWORD="$DB_PASSWORD" psql -h "$DB_HOST" -U myapp -c '\q' >/dev/null 2>&1; then
  echo "Authentication failed. Aborting." >&2
  exit 1
fi

# Use briefly, in a subshell so the secret doesn't escape:
(
  export PGPASSWORD="$DB_PASSWORD"
  psql -h "$DB_HOST" -U myapp -f /usr/share/myapp/migration.sql
)

# Cleanup runs automatically via trap.

This is the pattern for any script that touches secrets. Copy it as a starting point.


10. Quick reference card

The six leak channels

1. argv         — visible in ps. NEVER pass secrets as args.
2. environ      — visible in /proc/$pid/environ to same user.
3. history      — ~/.bash_history. Use HISTCONTROL=ignorespace.
4. xtrace       — set -x prints expanded vars. Wrap secrets in set +x ... set +x.
5. logging      — echo/printf to stderr. NEVER log a secret value.
6. core dumps   — ulimit -c 0. Or LimitCore=0 in systemd.

Pick the transport

Cloud CLI       → env var (their default)
TLS / SSH key   → file with 0600
Multi-line JSON → file with 0600
SQL connection  → .pgpass / env var
sudo password   → stdin (sudo -S)
Container       → mounted secret file
systemd service → LoadCredential= or EnvironmentFile=

Vault dispatch

SECRET=$(get_secret myapp/db/password)        # generic
# Set SECRET_BACKEND=aws|gcp|az|vault to choose.

Ephemeral creds

# AWS STS:
aws sts assume-role --role-arn ... --duration-seconds 900

# OIDC in CI: configure-aws-credentials@v4 with role-to-assume
# K8s: IRSA / workload identity — automatic

no_log wrapper

no_log() {
  case $- in *x*) set +x; "$@"; rc=$?; set -x; return $rc ;; esac
  "$@"
}
no_log psql -p "$PASSWORD" -e "..."

The 7 commandments of secrets

  1. Never in argv. Use env, file, or stdin.
  2. Never logged. Even in debug. Even temporarily.
  3. Never in source/git. Fetch at runtime from a vault.
  4. Ephemeral lifetime. STS assume-role, OIDC, ≤ 1 hour TTL.
  5. unset on exit. Trap-based cleanup.
  6. Permissions 0600 on secret files. chmod immediately after creation.
  7. Audit with gitleaks/trufflehog in CI on every push.

11. Wrap-up

Secrets are the highest-stakes data your script ever handles. A leaked password, key, or token can cost millions, take down a service, or end careers. The good news: the discipline is mechanical:

Layer on:

Get those right and the most common shell-secret leak class — accidental exposure — disappears. The rare hard cases (memory dumps, side-channel attacks) are real, but most leaks are these mundane ones, and most are avoidable with mechanical discipline.

Next: L27 — idempotency. We’ll cover state files, reconciliation loops, and dry-run flags — the patterns that turn “run once or break” scripts into “run any time, end up in the right state.”

shellbashsecretsvaultcredentialsiamsecurityleaks
Need this built for real?

Vinod is a Senior Cloud Architect (22+ yrs) — available for Azure / AWS / GCP architecture, landing zones, and migrations.

Work with me

Comments