Eliminating Secrets: Key Vault and Workload Identity Federation End to End

Every stored credential is a liability with a half-life: secrets expire at the worst moment, leak into logs and .env files, and outlive the engineer who created them. This guide walks the full path to a secret-free estate — Azure Key Vault as the system of record for the few secrets you cannot avoid, managed identities for anything running inside Azure, and workload identity federation (OIDC) to extend that passwordless model to GitHub Actions and AKS. The destination is an estate where the only thing you rotate is trust, not strings.

The reason this is hard is not the vault — creating a vault takes one command. The reason is the bootstrap: to read a secret you must first authenticate, and if that authentication is itself a stored secret you have moved the problem one hop upstream, not solved it. The entire discipline in this article is closing that last gap so that no stored credential anywhere grants access to your secrets. You will learn the exact trust assertions (issuer, subject, audience), the RBAC roles that gate the data plane, the federation subjects for each platform, and — because this is operational — the precise az and portal paths to confirm why a passwordless sign-in failed, since the failure modes are subtle and the error messages are deliberately vague.

By the end you will be able to stand up Key Vault with the right authorization and network posture, attach the right flavour of managed identity to each Azure workload, federate GitHub Actions and AKS service accounts to Entra ID with no stored secret, rotate secrets with zero downtime, and prove the whole thing is secret-free with Resource Graph and audit logs. Because you will return to this mid-incident, the federation subjects, the RBAC roles, the error codes, and the failure playbook are all laid out as scannable tables — read the prose once, then keep the tables open when a deploy fails at 02:00 with AADSTS70021.

What problem this solves

Secrets do not fail loudly. They fail at 02:00 on a Saturday when a certificate expires, or six months after an engineer leaves and their personal access token is finally revoked, or the day a .env file lands in a public repo. The pain in production terms is fourfold: expiry (a rotated database password that nobody propagated takes the app down), leakage (a secret in CI logs, an image layer, a Slack message), sprawl (the same credential copied into twelve app settings, none of which you can find when you must rotate), and attribution loss (a shared service-principal secret used by forty pipelines, so a breach implicates all of them and the audit trail names one principal for every action).

What breaks without this: teams hand-roll secret rotation and it desynchronises; they store an AZURE_CREDENTIALS JSON blob in every GitHub repo and can never rotate it without coordinating forty pipelines; they grant a runtime workload Key Vault Contributor (a control-plane role) and accidentally let it grant itself more access. The instinct — “we have a vault, we are secure” — is the trap. A vault you authenticate to with a stored secret is a vault with a key under the doormat.

Who hits this: every team running workloads that need credentials — which is every team. It bites hardest on CI/CD pipelines (the long-lived deploy credential is the single most over-privileged, most-copied secret in most estates), AKS workloads (pod-managed identity is deprecated and the migration is non-obvious), and multi-repo platforms (the 20-federated-credential ceiling arrives fast when you model identity per repository). The fix is almost never “add another secret to the vault” — it is “make the platform vouch for the workload so there is no secret to store.”

To frame the whole field before the deep dive, here is every identity-bootstrap mechanism this article covers, where the trust originates, and the one failure that defines it:

Learning objectives

By the end of this article you can:

• Explain the secret-zero problem and how platform-issued identity (managed identity and federation) eliminates the bootstrap credential entirely.
• Stand up Key Vault with the correct authorization model (RBAC over access policies), soft-delete, purge protection, and private networking — and justify each choice.
• Choose between system-assigned and user-assigned managed identities per workload, and assign least-privilege Key Vault data-plane roles at the right scope.
• Configure a federated identity credential (issuer / subject / audience) and federate GitHub Actions and AKS service accounts to Entra ID with no stored secret.
• Use Key Vault references and the Secrets Store CSI driver so application code never handles a secret string, and rotate without redeploying.
• Implement zero-downtime rotation with versionless references, CSI polling, and Event Grid notifications — treating rotation as a vault-side event consumers observe.
• Audit the estate for orphaned secrets with Resource Graph, route Key Vault AuditEvent logs to Log Analytics, and alert on anomalous access.
• Diagnose a failed passwordless sign-in to a specific cause — wrong subject, missing role, vault firewall, IMDS unreachable, 20-FIC ceiling — using exact commands.

Prerequisites & where this fits

You should already understand Entra ID basics: a tenant, an app registration (the identity of an application), a service principal (the local instance of that app in a tenant), and Azure RBAC (role assignments on a scope). You should be comfortable running az in Cloud Shell, reading JSON output, and reading a Bicep resource. Familiarity with OIDC at the level of “an issuer mints a signed token with claims, a relying party validates it” is assumed; you do not need to know JOSE internals.

This sits at the centre of the Identity & Platform Security track. Upstream of it is Azure Entra ID Fundamentals: Tenants, Users, Groups & RBAC, which defines the principals you assign roles to, and Entra Managed Identities Deep Dive: User-Assigned, FIC & RBAC, which goes deeper on the identity objects themselves. It pairs tightly with Azure Key Vault: Secrets, Keys & Certificates (the data-plane objects you are protecting) and Azure Key Vault Secret Rotation with Managed Identity. The federation half generalises across clouds — see GitHub Actions OIDC: Keyless Deploys to Multi-Cloud and Workload Identity Federation for Secretless CI/CD.

A quick map of who owns each layer, so you escalate to the right team when a passwordless flow breaks:

Core concepts

Five mental models make every later step obvious.

Secret-zero is the only hard part. To read a secret from Key Vault, a workload must authenticate to Entra ID. If that authentication relies on a stored client secret, you have only moved the problem one hop upstream. The answer is platform-issued identity: the platform a workload runs on (an Azure VM, an AKS pod, a GitHub runner) issues it a short-lived token, and Entra ID is configured to trust that platform. No secret is stored anywhere. Everything in this article is a variation on that single idea.

Managed identity is “Azure trusts itself”; federation is “Entra trusts a named external subject.” Inside Azure, the platform mints and rotates an identity bound to a resource and exposes it via IMDS (the Instance Metadata Service at 169.254.169.254). Outside Azure, an external OIDC issuer mints a token and Entra ID validates it against a configured federated identity credential (FIC). Both paths end in a normal short-lived Entra access token and zero stored secrets. The fork is purely “is the workload inside Azure’s IMDS reach?”

A FIC is a three-field trust assertion, matched exactly. A federated identity credential says: I will accept a token from this issuer, identifying this subject, for this audience. All three must match the incoming token exactly — subjects are case- and string-sensitive. Issuer is the OIDC issuer URL; subject is the sub claim (a repo+environment, or a Kubernetes service account); audience for Entra is always api://AzureADTokenExchange. Get one character wrong and Entra returns “no matching federated identity credential,” not “access denied” — a distinction that wastes hours if you do not know it.

Authorization has two planes, and confusing them is the classic mistake. The control plane (manage the vault: create it, set networking, assign roles) is governed by Azure RBAC roles like Key Vault Contributor. The data plane (read/write secret values) is governed either by legacy access policies or by Azure RBAC data roles like Key Vault Secrets User. A runtime workload needs data-plane read and nothing else; giving it Contributor lets it grant itself more — a privilege-escalation path that RBAC-for-data-plane closes.

Rotation is a vault-side event consumers observe, never a coordinated deploy. The discipline is: store each secret in exactly one place (the vault), reference it versionlessly everywhere, and let resolvers follow the current version. A versioned URI or a hardcoded value anywhere reintroduces a rotation outage. Done right, rotating a secret is one operation in the vault; every consumer picks it up on its own refresh cadence.

The vocabulary in one table

Before the deep sections, pin down every moving part. The glossary repeats these for lookup; this table is the model side by side:

The authorization & error reference

Before the per-step detail, here is the lookup table you scan first when a passwordless flow fails: every error you realistically see across Key Vault, managed identity, and federation, what it means, the likely cause, how to confirm it, and the fix. The non-obvious ones are the AADSTS codes (Entra token-exchange failures) and the difference between a control-plane 403 and a data-plane Forbidden.

Three reading notes that save the most time:

Step 1 — Key Vault foundations

Before federating anything, get the vault right. Two decisions dominate: the authorization model and data protection. Both are one-way doors in practice.

RBAC over access policies. Legacy access policies are a flat list on the vault; anyone with Microsoft.KeyVault/vaults/write (Contributor, Key Vault Contributor) can grant themselves data access — a privilege-escalation path. Azure RBAC uses the standard role-assignment plane, supports scoping down to an individual secret, and is the recommended model. As of recent Key Vault API versions, RBAC is the default for newly created vaults.

az keyvault create \
  --name kv-plat-prod-001 \
  --resource-group rg-platform-prod \
  --location australiaeast \
  --enable-rbac-authorization true \
  --enable-purge-protection true \
  --retention-days 90 \
  --public-network-access Disabled \
  --sku standard

resource kv 'Microsoft.KeyVault/vaults@2023-07-01' = {
  name: 'kv-plat-prod-001'
  location: location
  properties: {
    tenantId: subscription().tenantId
    sku: { family: 'A', name: 'standard' }
    enableRbacAuthorization: true       // RBAC data plane, not access policies
    enableSoftDelete: true              // always on; explicit for clarity
    softDeleteRetentionInDays: 90
    enablePurgeProtection: true         // irreversible — production default
    publicNetworkAccess: 'Disabled'
    networkAcls: { defaultAction: 'Deny', bypass: 'AzureServices' }
  }
}

The two authorization models, side by side — pick RBAC unless you have a specific legacy reason:

The data-plane RBAC roles you will actually use, and who gets each:

Assign least privilege at the secret scope where you can, and never hand a runtime workload more than Secrets User:

az role assignment create \
  --role "Key Vault Secrets User" \
  --assignee-object-id "$APP_PRINCIPAL_ID" \
  --assignee-principal-type ServicePrincipal \
  --scope "/subscriptions/$SUB/resourceGroups/rg-platform-prod/providers/Microsoft.KeyVault/vaults/kv-plat-prod-001/secrets/orders-db-conn"

Soft-delete and purge protection. Soft-delete (always on) recovers a deleted vault or secret within the retention window. Purge protection blocks even a privileged actor from hard-deleting before that window elapses, defeating a ransomware-style destroy. It is irreversible once enabled — that is the point. The data-protection knobs and their trade-offs:

Network isolation. --public-network-access Disabled plus a private endpoint keeps the data plane off the internet. Pair it with a Key Vault firewall that allows trusted Azure services so platform integrations still resolve. The network options, ordered by how locked-down they are:

Step 2 — Managed identities, decoded

Inside Azure, you almost never need federation — you need a managed identity. There are two flavours, and choosing wrong creates real operational pain.

• System-assigned: lifecycle tied 1:1 to a single resource — created and deleted with it. Good for a standalone service where the identity should never outlive the workload.
• User-assigned (UAMI): a standalone resource you create once and attach to many workloads. This is what you want at platform scale: assign Key Vault RBAC to the UAMI once, and every App Service, VM, or AKS pod that carries it inherits access. It also survives blue/green resource replacement.

# A UAMI shared across a workload family
az identity create \
  --name id-orders-api \
  --resource-group rg-platform-prod \
  --location australiaeast

APP_PRINCIPAL_ID=$(az identity show -n id-orders-api -g rg-platform-prod --query principalId -o tsv)
APP_CLIENT_ID=$(az identity show -n id-orders-api -g rg-platform-prod --query clientId -o tsv)

The two flavours, decided as a table — most platforms standardise on UAMI:

Where you can attach a managed identity, and how the token is delivered — this determines whether you even can use one:

For an App Service, attach the UAMI and point app settings at the vault using Key Vault references — the platform resolves them at startup using the identity, so your code never sees a secret string:

az webapp identity assign \
  --name app-orders-prod --resource-group rg-platform-prod \
  --identities "/subscriptions/$SUB/resourceGroups/rg-platform-prod/providers/Microsoft.ManagedIdentity/userAssignedIdentities/id-orders-api"

az webapp config appsettings set \
  --name app-orders-prod --resource-group rg-platform-prod \
  --settings "Db__ConnString=@Microsoft.KeyVault(SecretUri=https://kv-plat-prod-001.vault.azure.net/secrets/orders-db-conn/)"

resource site 'Microsoft.Web/sites@2023-12-01' = {
  name: 'app-orders-prod'
  location: location
  identity: {
    type: 'UserAssigned'
    userAssignedIdentities: { '${uami.id}': {} }
  }
  properties: {
    serverFarmId: plan.id
    keyVaultReferenceIdentity: uami.id   // which identity resolves KV references
    siteConfig: {
      appSettings: [
        {
          name: 'Db__ConnString'
          value: '@Microsoft.KeyVault(SecretUri=https://kv-plat-prod-001.vault.azure.net/secrets/orders-db-conn/)'
        }
      ]
    }
  }
}

The SecretUri without a version (trailing /) resolves the current version. That single decision is the foundation of zero-downtime rotation in Step 6. With multiple UAMIs attached, you must set keyVaultReferenceIdentity or the platform does not know which identity to use and the reference fails.

The Key Vault reference syntax has exactly two forms — know both and their reload behaviour:

Step 3 — Workload identity federation: how the trust works

Federation lets Entra ID accept an OIDC token from an external issuer in exchange for an Entra access token — no client secret involved. You configure a federated identity credential (FIC) on either an app registration or a user-assigned managed identity. A FIC is a trust assertion with three fields that must all match the incoming token:

• issuer — the OIDC issuer URL (e.g. https://token.actions.githubusercontent.com, or your AKS cluster’s OIDC issuer URL)
• subject — the exact sub claim identifying the workload (a repo+branch, a repo+environment, or a Kubernetes service account)
• audience — for Entra this is api://AzureADTokenExchange

At runtime the external platform issues a short-lived OIDC token, the workload presents it to Entra ID’s token endpoint, Entra validates issuer/subject/audience against a configured FIC, and returns a normal access token. The OIDC token lives minutes; nothing durable is stored.

The three fields, what each does, and the exact failure when it is wrong:

Where you can host a FIC, and the trade-off:

Limit: a single managed identity (or app) supports a maximum of 20 federated identity credentials. Plan subjects accordingly — one FIC per branch and per environment adds up fast. Flexible federated credentials (claims matching with wildcards) exist for GitHub/GitLab/Terraform Cloud on app objects if you outgrow exact-match.

The federation limits you will actually hit:

Step 4 — Federating GitHub Actions to Azure

This kills the AZURE_CREDENTIALS JSON secret that haunts so many pipelines. Create (or reuse) an app registration, then add a FIC whose subject pins the exact repo and ref.

APP_ID=$(az ad app create --display-name "gh-orders-deploy" --query appId -o tsv)
az ad sp create --id "$APP_ID"

The subject claim is where least privilege lives. Pin to a branch or a GitHub Environment — environment scoping is stronger because it lets you gate on approvals and environment protection rules:

# Environment-scoped: only the 'prod' environment of this repo can assume the identity
az ad app federated-credential create \
  --id "$APP_ID" \
  --parameters '{
    "name": "gh-orders-prod-env",
    "issuer": "https://token.actions.githubusercontent.com",
    "subject": "repo:contoso/orders-api:environment:prod",
    "audiences": ["api://AzureADTokenExchange"]
  }'

Common subject formats — copy the one that matches how the workflow is triggered:

Grant the app’s service principal only the roles that deployment needs — scoped to the target resource group, never the subscription. Then the workflow needs the id-token: write permission and the azure/login action with no secret:

name: deploy-orders
on:
  push:
    branches: [main]

permissions:
  id-token: write        # required to request the GitHub OIDC token
  contents: read

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: prod      # must match the FIC subject 'environment:prod'
    steps:
      - uses: actions/checkout@v4
      - uses: azure/login@v2
        with:
          client-id: ${{ vars.AZURE_CLIENT_ID }}
          tenant-id: ${{ vars.AZURE_TENANT_ID }}
          subscription-id: ${{ vars.AZURE_SUBSCRIPTION_ID }}
      - run: az webapp deploy --name app-orders-prod --resource-group rg-platform-prod --src-path ./app.zip --type zip

Note AZURE_CLIENT_ID and friends are repository variables, not secrets — they are identifiers, not credentials, and leaking them grants nothing without the matching OIDC trust. The two workflow permissions that gate this, and what breaks without them:

The GitHub-vs-secret comparison that justifies the migration:

Step 5 — AKS workload identity

Inside the cluster, pod-managed identity is deprecated; Microsoft Entra Workload ID is the model. The cluster runs an OIDC issuer, and a mutating webhook injects a projected service-account token plus the environment variables the Azure SDKs expect. Enable both:

az aks update \
  --name aks-plat-prod --resource-group rg-platform-prod \
  --enable-oidc-issuer \
  --enable-workload-identity

OIDC_ISSUER=$(az aks show -n aks-plat-prod -g rg-platform-prod \
  --query "oidcIssuerProfile.issuerUrl" -o tsv)

Federate a UAMI to a specific Kubernetes service account. The subject is system:serviceaccount:<namespace>:<name> and the issuer is the cluster’s OIDC URL:

az identity federated-credential create \
  --name fic-orders-sa \
  --identity-name id-orders-api \
  --resource-group rg-platform-prod \
  --issuer "$OIDC_ISSUER" \
  --subject "system:serviceaccount:orders:sa-orders" \
  --audiences "api://AzureADTokenExchange"

Annotate the service account with the UAMI client ID, and label pods to opt in. The annotation tells the webhook which identity to broker; the pod label flips the workload into the webhook’s injection path.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: sa-orders
  namespace: orders
  annotations:
    azure.workload.identity/client-id: "<APP_CLIENT_ID of id-orders-api>"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-api
  namespace: orders
spec:
  template:
    metadata:
      labels:
        azure.workload.identity/use: "true"   # opt this pod into the webhook
    spec:
      serviceAccountName: sa-orders
      containers:
        - name: orders-api
          image: acrplatprod.azurecr.io/orders-api:1.4.0

The four moving parts of AKS workload identity, and the failure when each is missing — this is the table to keep open when a pod can’t get a token:

The environment variables the webhook injects (your SDK reads these automatically):

With DefaultAzureCredential, the SDK inside the pod now authenticates with zero config. If you prefer secrets mounted as files, layer the Azure Key Vault provider for Secrets Store CSI Driver, which also works in workload-identity mode:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: spc-orders-kv
  namespace: orders
spec:
  provider: azure
  parameters:
    usePodIdentity: "false"
    clientID: "<APP_CLIENT_ID of id-orders-api>"   # workload identity mode
    keyvaultName: "kv-plat-prod-001"
    tenantId: "<TENANT_ID>"
    objects: |
      array:
        - |
          objectName: orders-db-conn
          objectType: secret

Enable the add-on with rotation when you create or update the cluster:

az aks enable-addons \
  --addons azure-keyvault-secrets-provider \
  --name aks-plat-prod --resource-group rg-platform-prod \
  --enable-secret-rotation \
  --rotation-poll-interval 2m

The two ways an AKS pod consumes a vault secret, side by side — DefaultAzureCredential vs CSI mount:

Step 6 — Rotation without downtime

Rotation breaks applications when code pins a version. The discipline is to reference secrets without a version and let the resolver follow the current one.

• App Service / Key Vault references: a versionless SecretUri (Step 2) re-resolves on app restart and on a periodic refresh, so rotating the secret in the vault propagates without a redeploy.
• CSI driver: with --enable-secret-rotation, the provider polls the vault every rotation-poll-interval (default 2 minutes) and updates both the mounted files and any synced Kubernetes Secret. Mounted file content updates in place; apps that read the file per request pick it up automatically. Apps that read once at startup still need a signal — watch the file or subscribe to the rotation.
• Event-driven: Key Vault emits Microsoft.KeyVault.SecretNewVersionCreated to Event Grid. Wire that to a Function or webhook to trigger graceful cache invalidation or a rolling restart the moment a new version lands, rather than waiting on a poll interval.

How each consumer picks up a rotated secret, and the latency you should expect:

The golden rule: store the secret in exactly one place (the vault), reference it versionlessly everywhere, and treat rotation as a vault-side operation that consumers observe — never a coordinated multi-system deploy.

The Event Grid event types Key Vault emits, and what to wire each to:

Step 7 — Auditing and detecting orphaned secrets

You cannot claim “secret-free” without proving it. Two fronts: find the secrets you missed, and watch the vault you kept.

Find orphaned secrets. Sweep app settings and pipeline definitions for plaintext that should be a Key Vault reference or a federated identity:

# App settings that look like inline secrets rather than KV references
az webapp config appsettings list -n app-orders-prod -g rg-platform-prod \
  --query "[?!contains(value, '@Microsoft.KeyVault')].name" -o tsv

Hunt the classic offenders across the estate with Resource Graph — for example, web apps inventory, then app registrations that still carry password credentials (a federation candidate):

az graph query -q "
  resources
  | where type == 'microsoft.web/sites'
  | extend kind = tostring(kind)
  | project name, resourceGroup, kind"

The estate-wide checks worth scripting into a weekly job:

Diagnostic logs. Route Key Vault AuditEvent logs to Log Analytics so every data-plane access is queryable and retained:

az monitor diagnostic-settings create \
  --name kv-audit \
  --resource "/subscriptions/$SUB/resourceGroups/rg-platform-prod/providers/Microsoft.KeyVault/vaults/kv-plat-prod-001" \
  --logs '[{"category":"AuditEvent","enabled":true}]' \
  --workspace "/subscriptions/$SUB/resourceGroups/rg-obs/providers/Microsoft.OperationalInsights/workspaces/law-platform"

The Key Vault log categories and what each is the source of truth for:

Alert on anomalies. A KQL alert for access from an unexpected identity or a spike in SecretGet denials catches both misconfiguration and intrusion:

AzureDiagnostics
| where ResourceType == "VAULTS" and OperationName == "SecretGet"
| where ResultType != "Success"
| summarize denials = count() by identity_claim_appid_g, bin(TimeGenerated, 15m)
| where denials > 10

The KQL you will reach for most — one query per question you ask during an incident or audit:

Architecture at a glance

The diagram traces the credential path exactly as a workload travels it, left to right, with each numbered badge marking the precise hop where a passwordless flow fails. Read it as the secret-zero journey: a workload (a GitHub runner, an AKS pod, or an App Service) starts with no stored secret. Off-Azure, the external OIDC issuer mints a short-lived token whose sub claim names the workload; on-Azure, IMDS plays the same role. That token is presented to Entra ID, where a federated identity credential (or the managed identity itself) is matched on issuer/subject/audience and exchanged for a normal Entra access token. Only then does the workload reach the Key Vault data plane, where an RBAC data role (Key Vault Secrets User) gates whether it can read the secret value — which finally resolves the versionless reference the app consumes. The private endpoint on the right keeps that last hop off the internet.

Notice the badges cluster where trust is actually established and where it most often breaks: badge 1 on the issuer/subject (the AADSTS70021 subject-drift trap), badge 2 on the Entra FIC (issuer mismatch and the 20-FIC ceiling), badge 3 on the data-plane role grant (the Forbidden that means “no Secrets User,” not “wrong subject”), badge 4 on the versionless reference (a pinned version that silently never rotates), and badge 5 on the network path (a private endpoint whose DNS resolves to a public IP). The legend narrates each as symptom · confirm · fix — that is the whole diagnostic method: localise the failure to one hop, read the confirm command, apply the fix.

Real-world scenario

Meridian Retail runs a forty-service microservice platform on Azure: AKS for the runtime, App Service for a handful of legacy APIs, and GitHub Actions for every deployment. The platform team is six engineers; the estate spans three subscriptions in australiaeast. Their mandate from a post-incident review was blunt: after a contractor’s leaked personal access token was found to still hold deploy rights two months after offboarding, no long-lived deploy secret may exist anywhere in the estate within one quarter.

They started where the risk was highest — CI/CD. Every one of the forty repos carried the same AZURE_CREDENTIALS JSON secret for a single shared service principal. They federated each repo’s prod environment to one shared gh-deploy app registration, one FIC per repo. Within two weeks they hit the wall: the 21st az ad app federated-credential create failed with The number of federated identity credentials on the application has reached the maximum allowed value of 20. The instinct was to mint more app registrations — but that scatters role assignments and audit identity across dozens of principals, exactly the sprawl they were trying to kill.

The fix was to stop modelling identity per repo. They created one user-assigned managed identity per deployment tier (id-deploy-prod, id-deploy-nonprod) and adopted GitHub’s repository_owner claim instead of pinning each repo. Crucially, a plain sub match cannot express “any repo in this org,” so they switched to a flexible federated credential on an app registration, using claimsMatchingExpression against assertion.repository_owner gated on the prod environment:

az ad app federated-credential create \
  --id "$APP_ID" \
  --parameters '{
    "name": "gh-org-prod",
    "issuer": "https://token.actions.githubusercontent.com",
    "audiences": ["api://AzureADTokenExchange"],
    "claimsMatchingExpression": {
      "value": "claims['"'"'repository_owner'"'"'] eq '"'"'meridian'"'"' and claims['"'"'environment'"'"'] eq '"'"'prod'"'"'",
      "languageVersion": 1
    }
  }'

One credential now covered every repo the org owned, gated on prod so approvals still applied. Forty FICs collapsed to two, role assignments lived on two identities, and sign-in logs attributed every deploy to one auditable principal.

The AKS side had its own trap. Three teams had copied a working SecretProviderClass but their pods kept failing with secrets that mounted empty. The platform on-call traced it to two distinct causes via the failure table: two teams had omitted the azure.workload.identity/use: "true" pod label (so the webhook never injected the token — kubectl exec … env | grep AZURE_ came back empty), and one team’s FIC subject read system:serviceaccount:orders:orders-sa while the deployment used serviceAccountName: sa-orders — a one-token mismatch that produced AADSTS70021, not a Key Vault error, which is why they had spent a day staring at vault RBAC.

By quarter end: every pipeline federated, the AZURE_CREDENTIALS secret deleted from all forty repos, AKS on workload identity with CSI rotation polling every two minutes, App Service on versionless references, and a Resource Graph job that fails the nightly build if any app registration still carries a password credential. The contractor-token class of incident became impossible — there was no longer a stored credential to leak. The lesson on the wall: “Federation subjects map to a trust boundary, not to a repository. Model the boundary first and the credential count takes care of itself.”

The migration as a timeline, because the order of moves is the lesson:

Advantages and disadvantages

The passwordless model removes the credential you most fear leaking, but it relocates the complexity into trust configuration — which has its own sharp edges. Weigh it honestly:

The model is right for any estate that runs workloads needing credentials — which is all of them — and especially for CI/CD and AKS where the long-lived deploy secret is the crown-jewel risk. It bites hardest on teams that model identity per repository (the FIC ceiling), teams new to the control/data-plane split (wrong-role grants), and anyone who flips network isolation before landing a private path. Every disadvantage is manageable — but only if you know it exists, which is the point of this article.

Hands-on lab

Stand up a vault with RBAC, attach a user-assigned identity, store a secret, grant least privilege, and read it back as that identity — all free-tier-friendly. Then reproduce the classic Forbidden failure and fix it. Run in Cloud Shell (Bash).

Step 1 — Variables and resource group.

RG=rg-kv-lab
LOC=australiaeast
KV=kv-lab-$RANDOM        # globally-unique vault name
UAMI=id-kv-lab
az group create -n $RG -l $LOC -o table
SUB=$(az account show --query id -o tsv)

Step 2 — Create a vault with RBAC authorization (no access policies).

az keyvault create -n $KV -g $RG -l $LOC \
  --enable-rbac-authorization true \
  --sku standard -o table

Expected: a vault row; properties.enableRbacAuthorization = true.

Step 3 — Create a user-assigned identity and capture its principal ID.

az identity create -n $UAMI -g $RG -l $LOC -o table
PID=$(az identity show -n $UAMI -g $RG --query principalId -o tsv)
CID=$(az identity show -n $UAMI -g $RG --query clientId -o tsv)
echo "principalId=$PID clientId=$CID"

Step 4 — Seed a secret (as yourself — you need Secrets Officer). First grant yourself, then write:

ME=$(az ad signed-in-user show --query id -o tsv)
az role assignment create --assignee-object-id $ME --assignee-principal-type User \
  --role "Key Vault Secrets Officer" \
  --scope "/subscriptions/$SUB/resourceGroups/$RG/providers/Microsoft.KeyVault/vaults/$KV"
# wait a few seconds for RBAC to propagate, then:
az keyvault secret set --vault-name $KV --name demo-conn --value "Server=db;Pwd=p@ss" -o table

Step 5 — Reproduce the Forbidden failure. The UAMI has no data role yet. Simulate its access check:

# This lists role assignments for the UAMI on the vault — expect EMPTY (the bug)
az role assignment list --assignee $PID \
  --scope "/subscriptions/$SUB/resourceGroups/$RG/providers/Microsoft.KeyVault/vaults/$KV" -o table

Empty output is the root cause: a workload carrying this UAMI would get Forbidden on SecretGet, which surfaces as an empty Key Vault reference and a crash loop — not an obvious “denied” in the app.

Step 6 — Grant least privilege and confirm.

az role assignment create --assignee-object-id $PID --assignee-principal-type ServicePrincipal \
  --role "Key Vault Secrets User" \
  --scope "/subscriptions/$SUB/resourceGroups/$RG/providers/Microsoft.KeyVault/vaults/$KV"

az role assignment list --assignee $PID \
  --scope "/subscriptions/$SUB/resourceGroups/$RG/providers/Microsoft.KeyVault/vaults/$KV" \
  --query "[].roleDefinitionName" -o tsv
# Expected: Key Vault Secrets User

Any workload (App Service, AKS pod) carrying this UAMI can now read demo-conn via a versionless reference — with zero stored secret.

Validation checklist. You created an RBAC vault, attached a reusable identity, hit the exact Forbidden/empty-reference failure from a missing data role, and fixed it with least privilege. No secret was stored to authenticate. The steps mapped to what each proves:

Cleanup (avoid lingering charges and a soft-deleted name).

az group delete -n $RG --yes --no-wait
# The vault soft-deletes; purge if you want the name back immediately (no purge protection here):
az keyvault purge --name $KV --no-wait 2>/dev/null || true

Cost note. A Standard vault has no hourly charge — you pay per 10,000 operations (fractions of a rupee for this lab). The UAMI is free. Deleting the resource group stops everything; the vault soft-deletes for 90 days unless purged.

Common mistakes & troubleshooting

This is the playbook — the part you bookmark. First as a scannable table you read mid-incident, then the entries that bite hardest in full.

The expanded form, with the full reasoning for the entries that waste the most time:

1. azure/login fails with AADSTS70021 “No matching federated identity record found.” Root cause: The OIDC token’s sub claim does not match any FIC subject. Most often the workflow lacks the environment: key (so the sub is ref:… not environment:prod), or a branch/tag/environment was renamed. Confirm: Print the FIC subjects with az ad app federated-credential list --id "$APP_ID" --query "[].subject" and compare to how the workflow is actually triggered. Add a debug step to dump the token claims if unsure. Fix: Make the subject string match the token exactly — including environment:prod when the job sets environment: prod. Subjects are case- and string-sensitive.

4. The 21st az ad app federated-credential create fails: “maximum allowed value of 20.” Root cause: The 20-FIC ceiling per app/UAMI, reached because identity was modelled per repo/branch. Confirm: az ad app federated-credential list --id "$APP_ID" | jq length returns 20. Fix: Stop pinning each repo. Use environment-scoped subjects, or a flexible federated credential matching repository_owner on an app registration, or one identity per trust boundary (deployment tier) rather than per repo. Minting more app registrations scatters audit identity — avoid it.

5. App boots but a Key Vault-backed app setting is empty and the app crash-loops. Root cause: The app’s identity has no Key Vault Secrets User role (or no identity is enabled, or the SecretUri is wrong), so the reference resolves to nothing. The app never sees “denied” — it sees an empty connection string. Confirm: Portal → Environment variables shows the reference with a red error; az webapp config appsettings list --query "[?contains(value,'KeyVault')]"; check az webapp identity show and az role assignment list --assignee <principalId>. Fix: Enable the identity; grant Key Vault Secrets User; set keyVaultReferenceIdentity if multiple UAMIs are attached; verify the secret exists/enabled and the URI (drop any pinned version).

7. The Key Vault reference resolves to a public IP or times out behind a private endpoint. Root cause: The vault is private but DNS resolves the public name, or the vault firewall blocks the caller. Confirm: nslookup kv-plat-prod-001.vault.azure.net returns a public IP instead of the private endpoint IP; the vault’s Networking blade shows “selected networks” without your path. Fix: Link the privatelink.vaultcore.azure.net private DNS zone to the VNet (group id vault); allow trusted Azure services on the firewall for App Service KV references; ensure the app’s outbound routes through the VNet.

8 & 9. AKS pod can’t authenticate. Two distinct failures that look identical from the app: 8 — no AZURE_* env vars at all: the pod is missing azure.workload.identity/use: "true", so the webhook never injected the token. Confirm with kubectl exec … env | grep AZURE_ (empty). Fix: add the label, restart. 9 — env vars present but AADSTS70021: the FIC subject does not match the pod’s service account. Confirm by comparing the FIC subject to the deployment’s serviceAccountName. Fix: align the subject to system:serviceaccount:<ns>:<name> exactly.

11. A rotated secret is ignored; the app keeps using the old value. Root cause: A pinned-version SecretUri (which never refreshes) or an app that reads the secret once at startup and caches it forever. Confirm: Grep the config/Bicep for a version segment after /secrets/<name>/; check whether the app re-reads on each use. Fix: Use a versionless SecretUri; for CSI mounts read the file per request; subscribe to SecretNewVersionCreated for an immediate signal, or restart on rotation.

12 & 13. Vault name conflicts and purge. 12 — “name already exists” on create: a soft-deleted vault still holds the name. az keyvault list-deleted to see it; recover with az keyvault recover, or az keyvault purge if it is not purge-protected. 13 — purge fails Conflict: purge protection is on and the retention window has not elapsed. This is by design — there is no override. Plan vault names so you do not need to recreate them.

Best practices

• RBAC for the data plane, always. Set enableRbacAuthorization: true and use Key Vault Secrets User/Officer/Administrator. Access policies let a control-plane role self-grant data access.
• Least privilege, scoped to the secret where you can. A runtime workload gets Secrets User and nothing more — never Administrator or Contributor. Scope to the individual secret if the role plane allows it.
• Federate, don’t store. Anything off-Azure (GitHub, GitLab, on-prem) gets a FIC, not a stored secret. Delete AZURE_CREDENTIALS and SP passwords once federation is proven.
• Model FIC subjects to a trust boundary, not a repository. One identity per deployment tier with environment-scoped or flexible subjects, so you never hit the 20-FIC ceiling or scatter audit identity.
• Prefer UAMI over system-assigned at scale. Grant Key Vault access to the UAMI once; every workload that carries it inherits access and survives blue/green.
• Reference secrets versionlessly. A versionless SecretUri is the foundation of zero-downtime rotation; a pinned version anywhere reintroduces an outage.
• Enable purge protection and soft-delete in production. They defeat accidental and malicious destruction; accept that purge protection is irreversible.
• Disable public network access and add a private endpoint — but land the private path (DNS zone, trusted services) before flipping the switch, or you lock out your own pipelines.
• Set keyVaultReferenceIdentity when multiple UAMIs are attached. Otherwise the platform cannot decide which identity resolves references, and they fail.
• Treat subjects as a contract. Renaming a GitHub Environment, branch, or Kubernetes service account silently breaks the FIC — change them deliberately and update the FIC in lockstep.
• Route AuditEvent to Log Analytics and alert on anomalies. Denial spikes and unexpected caller identities catch both misconfiguration and intrusion.
• Gate the build on a secret-free invariant. A nightly Resource Graph sweep that fails if any app registration still carries a password credential keeps the estate honest.

The settings worth standardising across every vault and identity, with the value you want:

Security notes

• Eliminate the credential, don’t just hide it. The strongest control is that no stored secret grants access at all — federation and managed identity achieve that. Where a secret is unavoidable, it lives in exactly one vault, read via a least-privilege data role.
• Separate control and data planes. Operators who manage the vault (Key Vault Contributor) should not automatically read secret values; data access is a separate Secrets User/Officer grant. RBAC enforces this split; access policies do not.
• Least privilege and scope. Grant Secrets User at the secret scope where possible, not the whole vault, and never grant Administrator to a runtime identity. Use PIM to make break-glass Administrator eligible, not standing.
• Network-isolate the data plane. Private endpoint plus publicNetworkAccess: Disabled keeps secret reads on the backbone. A firewall with defaultAction: Deny and trusted-services bypass admits only the platform integrations you need.
• Protect against destruction. Purge protection plus soft-delete defeats a ransomware-style destroy; diagnostic logs make every delete attributable.
• Pin and scan what pulls secrets. For AKS, pin image digests and scan images; the workload identity is only as trustworthy as the code it runs.
• Treat federation subjects as security boundaries. An over-broad subject (e.g. pull_request from any fork) is an access-grant; prefer environment-scoped subjects with approvals, and audit flexible-FIC claim expressions like any other policy.
• Watch the audit log. A spike in SecretGet, access from an unexpected appid, or reads from outside expected IPs is an early intrusion or misconfiguration signal — alert on it.

The security controls and what each defends against:

Cost & sizing

The bill for this whole pattern is dominated by operations, not capacity — which is why it is one of the cheapest security wins available.

• Key Vault is priced per operation, not per hour. A Standard vault charges per 10,000 secret operations (a few rupees per 10k); a Premium vault adds HSM-backed key operations at a higher per-op rate and a small per-key monthly charge for HSM-protected keys. Most secret-only workloads never leave Standard.
• Managed identities and FICs are free. UAMIs, system-assigned identities, and federated credentials cost nothing — the savings versus rotating and storing secrets are pure upside.
• The hidden cost is operation volume. Apps that read a secret on every request instead of caching it can generate millions of vault operations and a surprising bill; cache with a sane TTL (Key Vault references and the CSI driver do this for you).
• Private endpoints add a small hourly charge plus per-GB processing — modest, and the right trade for keeping the data plane private. Budget one private endpoint per vault per VNet that needs it.
• Log Analytics ingestion for AuditEvent is billed per GB — trivial for most vaults, but a high-traffic vault’s audit stream is worth a retention/sampling review.

A rough monthly picture and what drives each line:

The cache-vs-cost trade-off as a table — pick a TTL that fits the secret’s rotation cadence:

Interview & exam questions

1. What is the secret-zero problem and how do managed identity and federation solve it? Secret-zero is the bootstrap credential: to read a secret you must authenticate, and if that authentication is itself a stored secret you have only moved the problem upstream. Managed identity (inside Azure) and workload identity federation (outside) solve it by having the platform issue a short-lived token that Entra ID trusts, so no durable credential is stored anywhere.

2. Why prefer Azure RBAC over access policies for a Key Vault data plane? Access policies are a flat per-vault list, and anyone with vaults/write (Contributor) can self-grant data access — a privilege-escalation path. RBAC separates control-plane from data-plane permissions, supports scoping to a single secret, integrates with PIM, and is the default for new vaults. A runtime workload should get Key Vault Secrets User and nothing more.

3. A GitHub Actions deploy fails with AADSTS70021. What’s wrong and how do you confirm? The OIDC token’s sub claim does not match any federated identity credential’s subject — usually the workflow sets environment: prod but the FIC subject pins a branch (or vice versa), or something was renamed. Confirm by listing FIC subjects (az ad app federated-credential list) and comparing to how the job is triggered. Fix the subject to match exactly; subjects are string-sensitive.

4. What are the three fields of a federated identity credential, and what does each match? Issuer (the OIDC issuer URL, matched against the token’s iss), subject (the exact sub claim identifying the workload — a repo+environment or a Kubernetes service account), and audience (for Entra, always api://AzureADTokenExchange). All three must match the incoming token exactly or Entra returns “no matching federated identity,” not “access denied.”

5. When do you use a system-assigned versus a user-assigned managed identity? System-assigned when the identity should live and die with one resource (a standalone service). User-assigned (UAMI) when many workloads share access or the identity must survive blue/green replacement — you grant Key Vault RBAC to the UAMI once and every workload that carries it inherits access. AKS workload identity and external federation require a UAMI (or app), since system-assigned identities cannot hold FICs.

6. An app boots but a Key Vault-backed setting is empty and it crash-loops, with no exception. What do you check? A Key Vault reference resolved to nothing because the identity isn’t enabled, lacks Key Vault Secrets User, the vault firewall blocks it, or the SecretUri is wrong. Check the Environment variables blade for a red reference error, az webapp identity show, and az role assignment list --assignee <principalId>. With multiple UAMIs attached, also set keyVaultReferenceIdentity.

7. Why might an AKS pod fail to get a token even though workload identity is enabled? Two distinct causes: the pod is missing the azure.workload.identity/use: "true" label, so the webhook never injects the env vars/token (kubectl exec … env | grep AZURE_ is empty); or the FIC subject doesn’t match the pod’s service account (env vars present but AADSTS70021). Fix the label or align the subject to system:serviceaccount:<ns>:<name>.

8. You hit “maximum allowed value of 20” creating federated credentials. What now? You hit the 20-FIC ceiling because identity was modelled per repo/branch. Don’t mint more app registrations (that scatters audit identity). Consolidate with environment-scoped subjects, or a flexible federated credential matching repository_owner on an app registration, or one identity per deployment tier — model the trust boundary, not the repository.

9. How do you rotate a secret with zero downtime? Reference it versionlessly everywhere (a SecretUri ending in /), store it in exactly one vault, and let consumers follow the current version: App Service KV references re-resolve on restart/refresh, the CSI driver polls at rotation-poll-interval, and SecretNewVersionCreated via Event Grid can trigger immediate invalidation. Never pin a version or hardcode the value — that reintroduces a coordinated-deploy outage.

10. What does purge protection do, and what’s the catch? It blocks even a privileged actor from hard-deleting a vault or secret before the soft-delete retention window elapses, defeating a ransomware-style destroy. The catch: it is irreversible once enabled, and it blocks redeploys that try to recreate the same vault name within retention — so name vaults deliberately and don’t enable it in throwaway environments you recreate often.

11. How do you keep a Key Vault’s data plane off the internet without breaking App Service references? Set publicNetworkAccess: Disabled and add a private endpoint with the privatelink.vaultcore.azure.net DNS zone linked to the VNet, and enable the firewall’s trusted Azure services bypass so platform integrations (App Service KV references) still resolve. Land that private path before disabling public access, or you lock out your own pipelines.

12. What’s the difference between a control-plane 403 and a data-plane Forbidden on Key Vault? A control-plane 403 (e.g. on vaults/write) means the caller lacks an RBAC management role like Key Vault Contributor. A data-plane Forbidden (on secrets/getValue) means it lacks a data role/access policy like Key Vault Secrets User. They are governed by different planes — granting the wrong one is the classic mistake.

These map to AZ-500 (Security Engineer) — manage Key Vault, secrets, keys, certificates; configure managed identities; workload identity — and AZ-204 (Developer) — secure app configuration data, implement managed identities and Key Vault references. The federation and AKS angles touch AZ-400 and the Kubernetes specialty. A compact cert-mapping for revision:

Quick check

1. To read a secret from Key Vault a workload must authenticate to Entra ID. What is the name of the problem where that authentication itself needs a stored credential, and what mechanism removes it?
2. A GitHub Actions job sets environment: prod but azure/login fails AADSTS70021. Where is the mismatch, and what one command shows you the configured value to compare against?
3. True or false: granting a runtime web app Key Vault Contributor is the correct least-privilege way to let it read a secret.
4. Your AKS pod has none of the AZURE_* environment variables. What single piece of Kubernetes YAML is almost certainly missing?
5. You rotated a secret in the vault but the App Service still uses the old value. Name the most likely cause in the reference URI.

Answers

1. The secret-zero problem. It is removed by platform-issued identity — a managed identity (inside Azure, via IMDS) or workload identity federation (outside, via an OIDC token Entra trusts) — so no durable credential is stored.
2. The FIC subject does not match the token’s sub. The job’s sub is repo:ORG/REPO:environment:prod, so the FIC subject must be exactly that. Confirm the configured value with az ad app federated-credential list --id <appId> --query "[].subject".
3. False. Key Vault Contributor is a control-plane role (manage the vault) and lets the identity self-grant more access. The least-privilege data-plane role to read secret values is Key Vault Secrets User.
4. The pod (template) label azure.workload.identity/use: "true". Without it the mutating webhook does not inject the projected token or the AZURE_* env vars, so the SDK has nothing to exchange.
5. A pinned secret version in the SecretUri (a version segment after …/secrets/<name>/). A pinned version never refreshes; use a versionless URI ending in / so the current version is resolved.

Glossary

• Key Vault — Azure’s managed store for secrets, keys, and certificates, with a control plane (manage the vault) and a data plane (read/write values).
• Secret-zero — the bootstrap credential you would need to authenticate in order to read a secret; the problem managed identity and federation eliminate.
• Managed identity — a platform-minted, platform-rotated identity for an Azure resource, exposed via IMDS; needs no stored secret.
• System-assigned identity — a managed identity whose lifecycle is tied 1:1 to a single resource (created and deleted with it).
• User-assigned managed identity (UAMI) — a standalone, reusable managed-identity resource attached to many workloads; can hold federated credentials.
• IMDS — the Instance Metadata Service (169.254.169.254) that issues an in-Azure workload its managed-identity token.
• Workload identity federation — configuring Entra ID to trust an external OIDC issuer’s token in exchange for an Entra access token, with no stored secret.
• Federated identity credential (FIC) — the trust assertion (issuer, subject, audience) on an app or UAMI that Entra matches against an incoming OIDC token.
• Issuer / subject / audience — the three FIC fields: the OIDC issuer URL, the exact sub claim of the workload, and api://AzureADTokenExchange for Entra.
• Access policy — the legacy, escalation-prone, per-vault data-plane permission model; superseded by RBAC.
• RBAC data role — Key Vault Secrets User/Officer/Administrator (and crypto/cert equivalents) granting least-privilege data-plane access.
• Key Vault reference — an app setting of the form @Microsoft.KeyVault(SecretUri=…) resolved at boot by the app’s managed identity.
• keyVaultReferenceIdentity — the site setting that tells App Service which attached identity resolves Key Vault references when more than one is present.
• CSI Secrets Store — the Kubernetes provider that mounts Key Vault secrets as files in a pod, working in workload-identity mode.
• Versionless URI — a SecretUri ending in / (no version segment) that resolves the current version, enabling zero-downtime rotation.
• Purge protection — an irreversible vault setting blocking hard-delete before the soft-delete retention window elapses.
• Soft-delete — always-on recovery of a deleted vault/secret within a retention window (7–90 days).
• Flexible federated credential — a FIC on an app registration that matches token claims with expressions/wildcards (e.g. repository_owner), beyond exact-subject matching.

Next steps

You can now stand up a secret-free path end to end and diagnose where a passwordless flow breaks. Build outward:

• Next: Azure Key Vault: Secrets, Keys & Certificates — go deep on the data-plane objects you are protecting and their lifecycle.
• Related: Azure Key Vault Secret Rotation with Managed Identity — automate the rotation half with managed identity and Event Grid.
• Related: Entra Managed Identities Deep Dive: User-Assigned, FIC & RBAC — the identity objects and federated credentials in full.
• Related: AKS Secrets Store CSI: Key Vault Sync & Rotation — the file-mount path for cluster workloads, with rotation.
• Related: GitHub Actions OIDC: Keyless Deploys to Multi-Cloud — generalise federation across Azure, AWS, and GCP from one workflow.
• Related: Azure Private Endpoints & Private DNS at Scale — get the vault’s private network path right so references never resolve to a public IP.