Most CI systems are a black box bolted to the side of your cluster: a YAML dialect on someone else’s controller, scaling logic you cannot see, provenance that is at best a log line. Tekton inverts that. Every build step is a pod, every pipeline is a custom resource, and the same RBAC, admission, and observability you run for workloads applies to your CI. Once builds are Kubernetes objects, emitting tamper-evident SLSA provenance stops being a pipeline stage and becomes a controller that watches PipelineRun completions. This article builds reusable pipelines from the CRDs up, then wires Tekton Chains so every artifact is signed and every build produces an in-toto attestation without a single extra task.
1. The Tekton CRD model: Task, Pipeline, PipelineRun, results
Tekton is four core nouns. A Task is an ordered list of steps, each a container. A Pipeline arranges Tasks into a DAG with explicit ordering and data flow. A PipelineRun (or TaskRun) is the execution — it binds parameters, workspaces, and a service account, and it is the object Chains later signs. Task and Pipeline are reusable templates with zero runtime state; the Run objects hold all instance data.
Install the core component and confirm the API is serving:
kubectl apply -f https://storage.googleapis.com/tekton-releases/pipeline/latest/release.yaml
# Wait for the controller and webhook to be Ready
kubectl wait --for=condition=Ready pods --all -n tekton-pipelines --timeout=180s
kubectl api-resources --api-group=tekton.dev
You should see tasks, pipelines, pipelineruns, and taskruns under tekton.dev/v1 — the stable API you should author against. v1beta1 still resolves via conversion but is deprecated.
The two data primitives that make Tasks composable are results and workspaces. A result is a small string a step writes to $(results.<name>.path) that downstream Tasks consume as $(tasks.<task>.results.<name>). Results are for facts — a digest, a tag, a commit SHA — and are capped at roughly 4 KB total per TaskRun when stored in the termination message. A workspace is a shared filesystem (a PVC, emptyDir, Secret, or ConfigMap) mounted across steps and Tasks for bulk data like source trees and caches.
A minimal but real Task that clones a repo and emits the resolved commit as a result:
apiVersion: tekton.dev/v1
kind: Task
metadata:
name: git-clone-min
spec:
params:
- name: url
type: string
- name: revision
type: string
default: "main"
workspaces:
- name: source
description: Where the repo is checked out
results:
- name: commit
description: The resolved commit SHA
steps:
- name: clone
image: cgr.dev/chainguard/git:latest
script: |
#!/usr/bin/env sh
set -eu
cd "$(workspaces.source.path)"
git clone "$(params.url)" .
git checkout "$(params.revision)"
git rev-parse HEAD | tr -d '\n' > "$(results.commit.path)"
2. Reusable Tasks and pulling shared ones from Tekton Hub
You should not hand-write a git-clone Task in production. The community maintains hardened, parameterized Tasks on Tekton Hub. Install the tkn CLI and pull one in:
# Install the official git-clone Task (cluster-scoped, namespaced install)
tkn hub install task git-clone --version 0.9
tkn task list
For supply-chain hygiene, pin a specific catalog version rather than tracking latest, and review the Task source — a Task is arbitrary container execution with whatever service account you bind. The catalog git-clone Task exposes results you rely on downstream: commit, url, and committer-date.
The better pattern for fleet-wide reuse is Tekton Resolvers, which lets a Pipeline reference a Task by remote ref instead of vendoring YAML. The hub and git resolvers are both built in:
# Inside a Pipeline spec, reference a remote Task without copying it locally
- name: fetch-source
taskRef:
resolver: hub
params:
- name: kind
value: task
- name: name
value: git-clone
- name: version
value: "0.9"
workspaces:
- name: output
workspace: shared-data
params:
- name: url
value: $(params.repo-url)
This keeps a single source of truth: bump the version in one Pipeline, not fifty copied manifests. Use the git resolver for your own internal Tasks — point it at a tag or commit in your platform repo so the resolved Task is pinned and auditable.
3. Sharing data with workspaces, volumes, and result passing
A Pipeline ties Tasks together along two axes: ordering (runAfter or implicit results dependencies) and data (workspaces and results). The pattern below clones into a shared workspace, builds an image with Kaniko, and passes the digest forward as a result so later Tasks — and Chains — can reference the exact artifact.
apiVersion: tekton.dev/v1
kind: Pipeline
metadata:
name: build-and-push
spec:
params:
- name: repo-url
type: string
- name: image-ref
type: string
workspaces:
- name: shared-data
- name: docker-credentials
tasks:
- name: fetch-source
taskRef:
resolver: hub
params:
- name: kind
value: task
- name: name
value: git-clone
- name: version
value: "0.9"
workspaces:
- name: output
workspace: shared-data
params:
- name: url
value: $(params.repo-url)
- name: build-push
runAfter: ["fetch-source"]
taskRef:
resolver: hub
params:
- name: kind
value: task
- name: name
value: kaniko
- name: version
value: "0.6"
workspaces:
- name: source
workspace: shared-data
- name: dockerconfig
workspace: docker-credentials
params:
- name: IMAGE
value: $(params.image-ref)
results:
- name: image-digest
value: $(tasks.build-push.results.IMAGE_DIGEST)
The Pipeline-level results block re-exports a Task result so it appears on the PipelineRun status. This matters for Chains, which keys off Pipeline parameters and results to know what was built. The catalog kaniko Task already writes IMAGE_DIGEST and IMAGE_URL — that is not an accident, it is the convention Chains expects.
For workspace backing, use a volumeClaimTemplate so each PipelineRun gets its own ephemeral PVC garbage-collected with the run, rather than a shared PVC that serializes concurrent builds:
apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
generateName: build-and-push-
spec:
pipelineRef:
name: build-and-push
taskRunTemplate:
serviceAccountName: tekton-builder
params:
- name: repo-url
value: https://github.com/acme/widget-api
- name: image-ref
value: registry.acme.io/widget-api:$(context.pipelineRun.uid)
workspaces:
- name: shared-data
volumeClaimTemplate:
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 1Gi
- name: docker-credentials
secret:
secretName: registry-creds
4. Triggering pipelines from webhooks with EventListeners
A PipelineRun you apply by hand is a demo; production CI fires on git push. Tekton Triggers turns an inbound webhook into a PipelineRun via three nouns: EventListener (an HTTP sink backed by a pod and Service), TriggerBinding (extracts fields from the payload), and TriggerTemplate (the parameterized object to create). Interceptors sit in front to validate, filter, and enrich.
kubectl apply -f https://storage.googleapis.com/tekton-releases/triggers/latest/release.yaml
kubectl apply -f https://storage.googleapis.com/tekton-releases/triggers/latest/interceptors.yaml
kubectl wait --for=condition=Ready pods --all -n tekton-pipelines --timeout=180s
The wiring below validates the GitHub HMAC signature, fires only on pushes to main, binds the repo URL and commit, and stamps out a PipelineRun:
apiVersion: triggers.tekton.dev/v1beta1
kind: TriggerBinding
metadata:
name: github-push-binding
spec:
params:
- name: repo-url
value: $(body.repository.clone_url)
- name: revision
value: $(body.after)
---
apiVersion: triggers.tekton.dev/v1beta1
kind: TriggerTemplate
metadata:
name: build-trigger-template
spec:
params:
- name: repo-url
- name: revision
resourcetemplates:
- apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
generateName: build-and-push-
spec:
pipelineRef:
name: build-and-push
taskRunTemplate:
serviceAccountName: tekton-builder
params:
- name: repo-url
value: $(tt.params.repo-url)
- name: image-ref
value: registry.acme.io/widget-api:$(tt.params.revision)
workspaces:
- name: shared-data
volumeClaimTemplate:
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 1Gi
- name: docker-credentials
secret:
secretName: registry-creds
---
apiVersion: triggers.tekton.dev/v1beta1
kind: EventListener
metadata:
name: github-listener
spec:
serviceAccountName: tekton-triggers-sa
triggers:
- name: github-push
interceptors:
- ref:
name: "github"
params:
- name: secretRef
value:
secretName: github-webhook-secret
secretKey: secretToken
- name: eventTypes
value: ["push"]
- ref:
name: "cel"
params:
- name: filter
value: "body.ref == 'refs/heads/main'"
bindings:
- ref: github-push-binding
template:
ref: build-trigger-template
The github interceptor performs the HMAC check — this is your authentication boundary, so the secret must be a real random token configured identically in the GitHub webhook. The cel interceptor is where branch and path filtering belongs; pushing that logic upstream means you never spin up a pod for an irrelevant event. Expose the EventListener Service (Ingress, or a Route on OpenShift) and register that URL as the webhook target.
5. Generating SLSA provenance automatically with Tekton Chains
Here is the payoff. Tekton Chains is a controller that watches TaskRun and PipelineRun objects; on completion it observes their inputs (parameters), outputs (results), and produced image references, then generates a signed in-toto attestation describing exactly what was built from what. You do not add a task; you install a controller and configure it.
kubectl apply -f https://storage.googleapis.com/tekton-releases/chains/latest/release.yaml
kubectl wait --for=condition=Ready pods --all -n tekton-chains --timeout=180s
Chains is configured entirely through the chains-config ConfigMap in tekton-chains. The defaults are conservative; you want the SLSA provenance format and storage in the OCI registry alongside the image:
kubectl patch configmap chains-config -n tekton-chains -p '{
"data": {
"artifacts.taskrun.format": "slsa/v2alpha4",
"artifacts.taskrun.storage": "oci",
"artifacts.pipelinerun.format": "slsa/v2alpha4",
"artifacts.pipelinerun.storage": "oci",
"artifacts.oci.storage": "oci",
"artifacts.oci.format": "simplesigning",
"transparency.enabled": "true"
}
}'
# Restart the controller to pick up config changes
kubectl rollout restart deployment tekton-chains-controller -n tekton-chains
What each key does:
| Key | Purpose |
|---|---|
artifacts.pipelinerun.format |
Attestation format; slsa/v2alpha4 emits SLSA v1.0 provenance for the whole PipelineRun |
artifacts.pipelinerun.storage |
Where the attestation goes — oci co-locates it with the image; tekton stores it as an annotation on the run |
artifacts.oci.format |
simplesigning produces a cosign-compatible signature over the built image |
transparency.enabled |
Records the signature in a Rekor transparency log for tamper-evidence |
For Chains to recognize what a TaskRun built, the run must expose results it can map to artifacts. With the catalog kaniko Task that means IMAGE_URL and IMAGE_DIGEST; Chains reads those, fetches the image, and signs it. This is why the result naming in step 3 was load-bearing — Chains is convention-driven, and getting the names right is the difference between a signed artifact and a silent no-op.
6. Signing artifacts and attestations with cosign and KMS
Chains needs a key to sign with. For a lab, generate a cosign keypair stored as a Kubernetes Secret in the Chains namespace — Chains looks for a Secret named signing-secrets:
# cosign writes directly to the secret Chains expects
cosign generate-key-pair k8s://tekton-chains/signing-secrets
That populates cosign.key, cosign.pub, and cosign.password inside signing-secrets. A static key on the cluster proves the flow, but in production you do not want a long-lived private key in etcd. Point Chains at a cloud KMS instead, so the private key never leaves the HSM:
kubectl patch configmap chains-config -n tekton-chains -p '{
"data": {
"signers.kms.kmsref": "gcpkms://projects/acme-prod/locations/us-central1/keyRings/tekton/cryptoKeys/chains-signer/versions/1"
}
}'
kubectl rollout restart deployment tekton-chains-controller -n tekton-chains
Chains supports the cosign KMS reference scheme across providers — gcpkms://, awskms://, azurekms://, and hashivault://. The controller’s workload identity (IRSA on EKS, a workload-identity binding on GKE) needs sign and get-public-key permission on that key and nothing more. The fully keyless path also works: set signers.x509.fulcio.enabled and Chains requests a short-lived certificate from Fulcio bound to the controller’s OIDC identity — no key material at all, at the cost of a hard dependency on a Fulcio instance.
7. Securing PipelineRuns: service accounts, pod security, limits
CI is the highest-value target in your cluster — it can push to your registry and often holds cloud credentials. Treat it that way.
Least-privilege service accounts. The builder SA needs registry push and nothing else; the triggers SA needs to create PipelineRuns and read its triggers resources. Keep them split:
apiVersion: v1
kind: ServiceAccount
metadata:
name: tekton-builder
namespace: ci
secrets:
- name: registry-creds
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: tekton-triggers-createonly
namespace: ci
rules:
- apiGroups: ["tekton.dev"]
resources: ["pipelineruns", "taskruns"]
verbs: ["create"]
- apiGroups: ["triggers.tekton.dev"]
resources: ["eventlisteners", "triggerbindings", "triggertemplates", "triggers", "clusterinterceptors", "interceptors"]
verbs: ["get", "list", "watch"]
Pod Security and step hardening. Label the CI namespace for the restricted Pod Security Standard. Kaniko historically needed a relaxed profile because it manipulates the filesystem; prefer rootless build tooling (Kaniko rootless mode, or Buildah running unprivileged) so the namespace can stay restricted:
apiVersion: v1
kind: Namespace
metadata:
name: ci
labels:
pod-security.kubernetes.io/enforce: restricted
pod-security.kubernetes.io/enforce-version: latest
Resource limits. Every step is a container; without limits one runaway build starves the node. Set computeResources so the scheduler can bin-pack and the kubelet can evict fairly:
steps:
- name: build
image: gcr.io/kaniko-project/executor:latest
computeResources:
requests:
cpu: "500m"
memory: 1Gi
limits:
cpu: "2"
memory: 4Gi
A final non-obvious control: enable enforce-nonfalsifiable: true in Chains config. It hashes the TaskRun spec into the provenance, so a run mutated mid-flight (for example by a compromised admission webhook) cannot produce a clean attestation.
8. Sharding executions, pruning runs, and dashboard observability
At fleet scale, three operational facts bite. First, completed PipelineRun and TaskRun objects accumulate in etcd and never leave on their own — you must prune them:
# Imperative cleanup: keep the 50 most recent runs in the ci namespace,
# delete the rest (wrap in a CronJob for unattended pruning)
tkn pipelinerun delete --keep 50 --all -n ci
For declarative pruning under the Tekton Operator, the TektonConfig CR exposes a pruner block (schedule plus keep/keep-since and which resources to prune) — the supported way to bound history without a hand-rolled CronJob.
Second, a single controller pod is a throughput ceiling. Tekton supports HA via leader election with sharded buckets — increase buckets in config-leader-election and scale the controller Deployment so reconciliation is partitioned across replicas:
kubectl patch configmap config-leader-election -n tekton-pipelines \
-p '{"data":{"buckets":"3"}}'
kubectl scale deployment tekton-pipelines-controller -n tekton-pipelines --replicas=3
Third, observability. Install the Tekton Dashboard for a read-only view of runs, logs, and the DAG, and scrape the controller’s Prometheus metrics for tekton_pipelines_controller_pipelinerun_duration_seconds and reconcile latency:
kubectl apply -f https://storage.googleapis.com/tekton-releases/dashboard/latest/release.yaml
kubectl port-forward -n tekton-pipelines svc/tekton-dashboard 9097:9097
Verify
Run an end-to-end build and prove the artifact is signed and carries provenance.
# 1. Fire a build
kubectl create -f pipelinerun.yaml -n ci
tkn pipelinerun logs --last -f -n ci
# 2. Confirm Chains signed it — the run gets annotated when signing succeeds
kubectl get pipelinerun --sort-by=.metadata.creationTimestamp -n ci -o jsonpath \
'{range .items[-1:]}{.metadata.annotations.chains\.tekton\.dev/signed}{"\n"}{end}'
# Expect: true
# 3. Verify the image signature with the public key
cosign verify --key k8s://tekton-chains/signing-secrets \
registry.acme.io/widget-api:<tag>
# 4. Pull and inspect the SLSA provenance attestation
cosign verify-attestation --key k8s://tekton-chains/signing-secrets \
--type slsaprovenance \
registry.acme.io/widget-api:<tag> | jq -r '.payload' | base64 -d | jq .
Step 2 returning true is the controller’s receipt that signing completed. Step 4 should print an in-toto statement whose predicate lists the builder ID, the materials (your git URL and resolved commit), and the invocation parameters. With transparency.enabled on, cosign verify-attestation also confirms the Rekor entry exists.
If the
signedannotation never appears, the usual cause is missing or misnamed results: Chains only signs artifacts it can identify, so a Task that does not emitIMAGE_URL/IMAGE_DIGESTproduces an unsigned run with no error. Check the controller logs intekton-chainsforno signable targets found.
Enterprise scenario
A platform team on a regulated payments product needed every production container to carry SLSA Build L2 provenance, but auditors raised a hard objection to the first design: cosign signing keys lived as Kubernetes Secrets, so any cluster-admin (and the etcd backup process) could exfiltrate the private key and forge provenance for a malicious image. The control was theater if the key was reachable.
They re-architected signing to be fully keyless: stood up Fulcio and Rekor instances, bound the Chains controller to a dedicated workload identity with no other permissions, and switched Chains to request short-lived Fulcio certificates instead of a static key — so there was no private key to steal, and every signature anchored to the controller’s OIDC identity plus a transparency-log entry.
kubectl patch configmap chains-config -n tekton-chains -p '{
"data": {
"signers.x509.fulcio.enabled": "true",
"signers.x509.fulcio.address": "https://fulcio.internal.acme.io",
"signers.x509.fulcio.issuer": "https://oidc.acme.io",
"signers.x509.fulcio.provider": "spiffe",
"transparency.enabled": "true",
"transparency.url": "https://rekor.internal.acme.io"
}
}'
The audit win was decisive: with no key material anywhere, “who signed this build” was answerable from the certificate identity and “was it tampered with after signing” from Rekor — without trusting a single long-lived secret. The trade-off was operating Fulcio and Rekor as tier-1 services, since a Fulcio outage now blocked production builds; they mitigated that with a regional active-passive deployment and an alert on certificate-issuance latency.