Configure Confluent Cloud Cluster Linking and Tiered Storage for Multi-Region Kafka

A payments company runs its order and settlement events through Kafka on Confluent Cloud in us-east-1. The mandate from the new platform lead is blunt: a regional outage cannot stop European settlement, and auditors want seven years of every settlement.completed event re-readable on demand — not in a cold archive nobody can replay, but as a live Kafka topic a consumer group can rewind into. Today the setup keeps seven days of retention on broker disk and replicates nothing across regions. This guide builds the fix: Cluster Linking to mirror topics from a us-east-1 source into a eu-west-1 destination with byte-for-byte offset fidelity and synced consumer offsets, and Tiered Storage so both clusters offload old segments to object storage and keep months-to-years of history without paying for hot broker disk.

Two features carry the design, and they solve orthogonal problems. Cluster Linking is geo-replication: a managed, destination-initiated link that pulls a configured set of topics into read-only mirror topics preserving partition count, keys, and exact offsets — the foundation for active-passive DR, regional read locality, and zero-downtime migration. Tiered Storage is the retention economics: every Dedicated cluster offloads closed segments older than a small hotset to managed cloud object storage (S3/GCS/Azure Blob), so a topic holds years while brokers hold only hours. Together they let you say “every settlement event, in two regions, for seven years, re-readable as a live topic” — and mean it, at a bill you can defend.

This is the implementation playbook, written for someone who will run it. You will stand up two clusters, wire a link, create mirror topics, tune retention and hotset for cost, fail over to the second region with consumer offsets intact, and bolt on the operating model — Terraform, Vault, Okta/Entra SSO, the Metrics API into Datadog/Dynatrace, and ServiceNow. Every operation comes with both the real confluent CLI command and the Terraform that makes it declarative, and the cluster types, link settings, ACL grants, failure modes, and cost levers are laid out as scannable tables. Read the prose once, then keep the tables open while you build.

What problem this solves

Single-region Kafka is a single point of failure with a short memory. The two pains it creates are independent, and most teams hit them in this order.

The first pain is resilience. A regional outage — control-plane degradation, a cascading zone failure, or a deliberate evacuation — takes your entire event backbone offline: producers can’t write, consumers can’t read, downstream systems stall. If your settlement pipeline lives only in us-east-1, a us-east-1 incident is a settlement outage, full stop. The classic answer was MirrorMaker 2 on a Connect cluster you operate — a second piece of infrastructure that fails on its own schedule and mangles offsets unless you run the offset-sync connectors perfectly. Cluster Linking removes that burden: the link is managed, offsets are preserved natively, and there is no Connect cluster to run.

The second pain is retention economics. Kafka keeps data on broker local disk by default, and broker disk on a Dedicated cluster is expensive, finite, and tied to your CKU (Confluent Unit for Kafka) sizing. Seven days is routine; seven years on broker disk would mean enormous, mostly-idle storage at hot-disk prices every hour. So teams compromise: short retention plus a fragile sink connector dumping old events into a lake nobody can replay as Kafka — and when an audit needs the history, the export turns out to have dropped fields or lost ordering. Tiered Storage fixes this directly: closed segments move to object storage (cents per GB), the topic still presents a continuous offset space, and a consumer reading an old offset transparently fetches from the cold tier — slightly slower, but it is Kafka, with the original bytes and ordering.

Who hits this: any team running Kafka for events that matter across regions or over long horizons — payments and settlement (the canonical case), order/inventory streams feeding multiple geographies, IoT and clickstream platforms retaining raw events for ML backfills, and regulated workloads (finance, healthcare, telco) with multi-year audit-replay needs. Without these features a regional incident becomes a business outage and an audit request becomes an archaeology project. The whole field, framed before the deep dive:

Learning objectives

By the end of this article you can:

• Choose the right cluster type (Basic / Standard / Enterprise / Dedicated) for a multi-region, long-retention workload, and explain which features each tier gates.
• Stand up source and destination Dedicated clusters and configure a destination-initiated Cluster Link with the right link.mode, connection.mode, security, and offset-sync settings — in CLI and Terraform.
• Create and manage mirror topics: how they preserve partitions/keys/offsets, why they are read-only, and how to pause/resume/promote/failover them.
• Tune Tiered Storage so long retention stays cheap: set retention.ms and confluent.tier.local.hotset.ms, reason about hot vs cold reads, and predict the cost split.
• Execute a clean regional failover that preserves committed offsets via consumer.offset.sync.enable, and run a controlled migration cutover the same way.
• Secure it: scoped Vault-backed keys per principal, least-privilege ACLs/RBAC, SSO for humans, and Schema Linking so the destination can deserialize.
• Monitor replication with the Metrics API — cluster_link_mirror_topic_offset_lag, link state, tiered-fetch — and alert before lag becomes data loss.
• Reason about the cost of multi-region Kafka (CKUs, mirrored bytes, object-storage GB) and replicate only what earns its keep.

Prerequisites & where this fits

You should already understand Kafka fundamentals: topics and partitions, per-partition ordering, what an offset is (a record’s monotonic position within a partition), what a consumer group is and how it commits offsets, and the producer/consumer model. If those are fuzzy, read Event Hubs Fundamentals: Partitions, Consumer Groups, Checkpoints and Offsets for Beginners first — the partition/offset model is identical across Kafka-compatible systems, and the whole DR story rests on why offset preservation is the hard part.

You should also be comfortable with the operating tools: Terraform ≥ 1.6 with the confluentinc/confluent provider, the confluent CLI v3.x+, and the idea that secrets live in HashiCorp Vault, not in terraform.tfvars. Disaster-recovery vocabulary helps — RTO (how fast you recover) and RPO (how much data you can lose) frame every choice here; High Availability vs Disaster Recovery: RTO and RPO Explained is the primer, and BCDR Foundations on Azure: Making Sense of RTO, RPO, and the Resilience Spectrum maps the spectrum.

This sits in the streaming-platform track. Cluster Linking is the cross-region transport; the broader question of which replication topology and consistency model to pick is Multi-Region Data: Choosing Replication and Consistency Without Losing Writes. Once data flows in two regions you feed it into analytics — Real-Time Analytics with Databricks and Confluent Kafka on AWS — or use Kafka as the CDC backbone, as in Real-Time CDC with Debezium and Kafka. A map of who owns what during this build:

Core concepts

Six mental models make every later step obvious. Read these once; the deep sections assume them.

Cluster Linking is destination-initiated and managed. You create the link object on the destination cluster, point it at the source’s bootstrap, and the link pulls topics from source to destination as mirror topics. There is no Connect cluster or MirrorMaker process to scale or patch — it is a first-class managed component of the destination. “Destination-initiated, source-pulled” explains every CLI flag: you run link create and mirror create on the destination, never the source.

A mirror topic is a read-only, offset-faithful copy. Unlike MirrorMaker 2 (which republishes records with new offsets, then runs separate offset-sync to translate), a mirror preserves the exact offsets, partition count, keys, and topic settings — offset 4,418,002 on partition 7 of the source is offset 4,418,002 on partition 7 of the mirror. That fidelity is why consumer failover is clean: a group’s committed offset means the same thing on both sides. A mirror is read-only until you promote or failover it, which severs the link and makes it a normal writeable topic.

Offset sync makes failover correct, not just possible. Mirroring records is necessary but not sufficient. If a group in us-east-1 is at offset 1,000,000 and the region dies, the failed-over group must resume at 1,000,001, not restart from the beginning (re-processing millions of events) or from latest (silently skipping them). consumer.offset.sync.enable=true copies committed group offsets source→destination continuously, so the destination knows where each group was. It is set at link creation and awkward to retrofit — decide it up front.

Tiered Storage decouples retention from disk. On a Dedicated cluster, every topic’s closed segments offload to object storage. Two per-topic levers control it: retention.ms (how long data is kept anywhere) and confluent.tier.local.hotset.ms (how much recent data stays on broker disk). Set retention to seven years and the hotset to six hours, and you keep seven years total while brokers hold six — the object store holds the rest at a fraction of the cost. A read older than the hotset transparently fetches from object storage with modestly higher latency. This is what makes “years of replayable history” economically real.

Hot reads and cold reads differ. Reads from the hotset (broker disk, page cache) are normal low-latency Kafka. Reads of older offsets fetch from object storage — higher first-byte latency, and a large backfill drives tiered fetch throughput and can momentarily pressure the cluster. That suits replay and audit (bursty, latency-tolerant), but a heavy backfill is an operation to plan, not a casual --from-beginning against a multi-terabyte topic in business hours.

Cluster type gates the features. Cluster Linking and per-topic Tiered Storage tuning are not on every tier. Basic and Standard are multi-tenant, cheaper, limited; Dedicated gives isolated CKU capacity, Cluster Linking, full Tiered Storage control, private networking, and the scale this design needs. Choosing the tier is the first real decision, and getting it wrong means re-provisioning. The vocabulary, side by side:

Cluster types: Basic vs Standard vs Enterprise vs Dedicated

The first decision is which cluster type to provision, because Cluster Linking and full Tiered Storage control are Dedicated-class capabilities — attempting them on a Basic or Standard cluster fails at create time, and choosing wrong means re-provisioning and migrating. The comparison that matters for this workload (multi-region, long retention, production scale):

As a decision table — match your requirement to the smallest type that satisfies it:

For the payments design — two regions, seven-year retention tuned per topic, private networking, and a hard SLA — the answer is Dedicated on both ends. A CKU bundles throughput, partition, and connection capacity; the minimum multi-zone Dedicated cluster is 2 CKUs (single-zone can start at 1, but you do not run production payments single-zone). Sizing is covered in Cost & sizing below.

Authenticate, pin the environment, and stage credentials in Vault

Log in (SSO-backed, so the browser hands you off to Okta/Entra for MFA), then pin the environment so every later command is unambiguous. Acting on the wrong environment is a classic, expensive mistake when you have prod and non-prod side by side.

confluent login --save                          # SSO -> Okta/Entra, MFA, token cached
confluent environment list
confluent environment use env-payments-prod      # pin so we never act on the wrong env
confluent kafka cluster list                      # see what already exists in this env

Create a bootstrap Cloud API key and store it in Vault immediately — never leave it in shell history or a CI log. It is an org-level credential used only to provision; runtime principals get their own scoped keys. CI reads it from Vault with a short-lived token, so the static secret is fetched at job time, not baked into a pipeline.

confluent api-key create --resource cloud --description "tf-bootstrap-payments"
# -> prints API Key + Secret ONCE. Pipe straight into Vault; do not echo to a file or commit it.
vault kv put secret/confluent/payments-bootstrap \
  api_key="<KEY>" api_secret="<SECRET>"

In Terraform, the provider reads those values from Vault rather than from variables on disk:

terraform {
  required_providers {
    confluent = { source = "confluentinc/confluent", version = "~> 2.0" }
    vault     = { source = "hashicorp/vault" }
  }
  required_version = ">= 1.6"
}

data "vault_kv_secret_v2" "cc_bootstrap" {
  mount = "secret"
  name  = "confluent/payments-bootstrap"
}

provider "confluent" {
  cloud_api_key    = data.vault_kv_secret_v2.cc_bootstrap.data["api_key"]
  cloud_api_secret = data.vault_kv_secret_v2.cc_bootstrap.data["api_secret"]
}

The credential model for the whole build. Each layer gets a different identity, and each secret lands in Vault under its own path so a leak is a scoped, rotatable blast radius:

Declare both Dedicated clusters in Terraform

Cluster Linking needs Dedicated clusters. Declare both regions and the environment in one config so the topology is reviewable, reproducible, and drift-controlled — the artifact CI runs and Argo CD reconciles. MULTI_ZONE availability is non-negotiable for production: it spreads replicas across zones so a single-zone failure doesn’t take the cluster down — the in-region HA that complements the cross-region DR the link provides.

resource "confluent_environment" "payments" {
  display_name = "payments-prod"
  stream_governance { package = "ADVANCED" }   # Schema Registry for both regions
}

resource "confluent_kafka_cluster" "source_us" {
  display_name = "payments-source-use1"
  availability = "MULTI_ZONE"
  cloud        = "AWS"
  region       = "us-east-1"
  dedicated { cku = 2 }
  environment { id = confluent_environment.payments.id }
}

resource "confluent_kafka_cluster" "dest_eu" {
  display_name = "payments-dest-euw1"
  availability = "MULTI_ZONE"
  cloud        = "AWS"
  region       = "eu-west-1"
  dedicated { cku = 2 }
  environment { id = confluent_environment.payments.id }
}

Apply through CI, not a laptop, so the state file is authoritative and the change is reviewed:

terraform init && terraform plan -out tfplan
terraform apply tfplan          # provisioning Dedicated CKUs takes several minutes

The cluster resource arguments, what they control, and the gotcha for each:

A note on networking, because the link must reach the source’s bootstrap. With public clusters the destination dials the source over the public internet on :9092 (SASL_SSL). For production you usually want PrivateLink, VPC/VNet peering, or Transit Gateway so replication never traverses the public internet — that adds a confluent_network and access-point resources, and the link’s bootstrap.servers points at the private endpoint. Decide the network model before the link, because it changes the bootstrap address the link config carries.

Create service accounts and scoped, Vault-backed credentials

Give producers, consumers, and the link their own identities — never reuse the bootstrap key for runtime. Each service account gets a cluster-scoped API key whose secret lands in Vault under a per-service path, so a leaked consumer key rotates without touching producers.

# identities
confluent iam service-account create sa-payments-producer --description "EAST producers"
confluent iam service-account create sa-payments-consumer --description "EU consumers"
confluent iam service-account create sa-payments-link     --description "Cluster Link principal"

# scoped key for the producer, against the SOURCE cluster
confluent api-key create --service-account sa-payments-producer \
  --resource lkc-source-use1
vault kv put secret/confluent/payments-producer api_key="<KEY>" api_secret="<SECRET>"

# scoped key for the link principal, also against the SOURCE (it reads the source)
confluent api-key create --service-account sa-payments-link \
  --resource lkc-source-use1
vault kv put secret/confluent/payments-link api_key="<KEY>" api_secret="<SECRET>"

Then grant least-privilege ACLs (full matrix below): producers WRITE/DESCRIBE the source topic prefix, the link principal READ/DESCRIBE the source topics plus DESCRIBE the cluster, consumers READ the destination mirrors and their group.

# SOURCE: producers may only WRITE the payments topics
confluent kafka acl create --cluster lkc-source-use1 \
  --allow --service-account sa-payments-producer \
  --operations WRITE,DESCRIBE --topic settlement --prefix

# SOURCE: the link principal may READ the topics it mirrors + DESCRIBE the cluster
confluent kafka acl create --cluster lkc-source-use1 \
  --allow --service-account sa-payments-link \
  --operations READ,DESCRIBE --topic settlement --prefix
confluent kafka acl create --cluster lkc-source-use1 \
  --allow --service-account sa-payments-link \
  --operations DESCRIBE --cluster-scope

# DESTINATION: EU consumers may READ the mirror topics + their group
confluent kafka acl create --cluster lkc-dest-euw1 \
  --allow --service-account sa-payments-consumer \
  --operations READ,DESCRIBE --topic settlement --prefix
confluent kafka acl create --cluster lkc-dest-euw1 \
  --allow --service-account sa-payments-consumer \
  --operations READ --consumer-group eu-settlement --prefix

The complete least-privilege grant matrix. A distinct link principal lets the audit trail show what the link copied versus what an app wrote:

In Terraform the same grants are declarative via confluent_kafka_acl (or confluent_role_binding for RBAC), keyed to the service-account resources — a reviewer sees who can do what, and a removed block revokes access on the next apply.

Turn on Tiered Storage retention on the source topics

Dedicated clusters already tier closed segments to object storage; what you control per topic is how long data is retained (retention.ms) and the hotset that stays on broker disk (confluent.tier.local.hotset.ms). Create the topics with long retention and a modest hotset so brokers stay lean while the topic holds years.

# 7-year retention, but only ~last 6h of segments kept on broker disk (rest in object store)
confluent kafka topic create settlement.completed \
  --cluster lkc-source-use1 \
  --partitions 12 \
  --config retention.ms=220752000000 \
  --config confluent.tier.local.hotset.ms=21600000 \
  --config cleanup.policy=delete

confluent kafka topic create order.events \
  --cluster lkc-source-use1 --partitions 12 \
  --config retention.ms=220752000000 \
  --config confluent.tier.local.hotset.ms=21600000

The same in Terraform, declarative and drift-controlled:

resource "confluent_kafka_topic" "settlement" {
  kafka_cluster { id = confluent_kafka_cluster.source_us.id }
  topic_name       = "settlement.completed"
  partitions_count = 12
  config = {
    "retention.ms"                   = "220752000000" # ~7 years
    "confluent.tier.local.hotset.ms" = "21600000"     # ~6h hot on disk
    "cleanup.policy"                 = "delete"
  }
  rest_endpoint = confluent_kafka_cluster.source_us.rest_endpoint
  credentials {
    key    = data.vault_kv_secret_v2.cc_admin.data["api_key"]
    secret = data.vault_kv_secret_v2.cc_admin.data["api_secret"]
  }
}

confluent.tier.local.hotset.ms is the lever that makes long retention cheap: it caps disk-resident data regardless of retention.ms, while reads of older offsets transparently fetch from object storage — the right trade for replay and audit. The Tiered Storage config reference you will return to:

The crucial distinction between the two storage tiers — confusing them is how people mis-budget or panic about latency:

Create the Cluster Link from the destination

Cluster Linking is destination-initiated: you create the link on the eu-west-1 cluster and point it at us-east-1, and it runs as managed infrastructure. First write a config file giving the link the source’s bootstrap and a source-scoped key (from Vault):

# link-config.properties
link.mode=DESTINATION
connection.mode=OUTBOUND
bootstrap.servers=<SOURCE_BOOTSTRAP>:9092
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule \
  required username="<LINK_KEY>" password="<LINK_SECRET>";
consumer.offset.sync.enable=true
consumer.offset.sync.ms=30000
acl.sync.enable=true
auto.create.mirror.topics.enable=false

Then create the link, naming source and destination cluster IDs:

confluent kafka link create payments-east-to-west \
  --cluster lkc-dest-euw1 \
  --source-cluster lkc-source-use1 \
  --source-bootstrap-server <SOURCE_BOOTSTRAP>:9092 \
  --config-file link-config.properties

confluent kafka link list --cluster lkc-dest-euw1     # expect the link in 'ACTIVE'
confluent kafka link describe payments-east-to-west --cluster lkc-dest-euw1

The same link in Terraform, source-scoped credential from Vault:

resource "confluent_cluster_link" "east_to_west" {
  link_name = "payments-east-to-west"
  link_mode = "DESTINATION"
  connection_mode = "OUTBOUND"

  source_kafka_cluster {
    id            = confluent_kafka_cluster.source_us.id
    bootstrap_endpoint = confluent_kafka_cluster.source_us.bootstrap_endpoint
    credentials {
      key    = data.vault_kv_secret_v2.cc_link.data["api_key"]
      secret = data.vault_kv_secret_v2.cc_link.data["api_secret"]
    }
  }
  destination_kafka_cluster {
    id            = confluent_kafka_cluster.dest_eu.id
    rest_endpoint = confluent_kafka_cluster.dest_eu.rest_endpoint
    credentials {
      key    = data.vault_kv_secret_v2.cc_admin.data["api_key"]
      secret = data.vault_kv_secret_v2.cc_admin.data["api_secret"]
    }
  }
  config = {
    "consumer.offset.sync.enable"      = "true"
    "consumer.offset.sync.ms"          = "30000"
    "acl.sync.enable"                  = "true"
    "auto.create.mirror.topics.enable" = "false"
  }
}

Every link setting that matters, what it does, and why this design sets it — these levers determine whether your failover is clean:

Again: consumer.offset.sync.enable=true is what makes failover clean, it is set at link creation, and it cannot be retrofitted cleanly — which is why the link decisions belong at the start of the design.

Create mirror topics on the destination

Mirror topics are read-only copies that inherit the source’s partition count and exact offsets. Create one per topic you want in Europe, over the link. They do not exist until you create them — auto.create.mirror.topics.enable=false means nothing is replicated implicitly, exactly the control you want over cross-region cost.

confluent kafka mirror create settlement.completed \
  --cluster lkc-dest-euw1 --link payments-east-to-west

confluent kafka mirror create order.events \
  --cluster lkc-dest-euw1 --link payments-east-to-west

confluent kafka mirror list --cluster lkc-dest-euw1 --link payments-east-to-west
confluent kafka mirror describe settlement.completed \
  --cluster lkc-dest-euw1 --link payments-east-to-west

In Terraform, each mirror is a resource on the link:

resource "confluent_kafka_mirror_topic" "settlement" {
  source_kafka_topic { topic_name = confluent_kafka_topic.settlement.topic_name }
  cluster_link { link_name = confluent_cluster_link.east_to_west.link_name }
  kafka_cluster {
    id            = confluent_kafka_cluster.dest_eu.id
    rest_endpoint = confluent_kafka_cluster.dest_eu.rest_endpoint
    credentials {
      key    = data.vault_kv_secret_v2.cc_admin.data["api_key"]
      secret = data.vault_kv_secret_v2.cc_admin.data["api_secret"]
    }
  }
}

Set the destination topics’ own Tiered Storage retention so Europe keeps the same seven-year horizon — mirror topics tier independently, so the destination needs its own retention/hotset config or it quietly falls back to a ~week default:

confluent kafka topic update settlement.completed --cluster lkc-dest-euw1 \
  --config retention.ms=220752000000 \
  --config confluent.tier.local.hotset.ms=21600000

A mirror topic moves through a small lifecycle of states, and the state dictates which operation is legal and what failover will do:

Now eu-west-1 consumers read settlement.completed locally while writes still happen only in us-east-1 — read locality and a warm DR copy from one managed pipe.

Schema Linking: making the destination able to deserialize

A subtle production trap: the mirror carries the records, but a consumer in eu-west-1 using Avro/Protobuf/JSON-Schema needs the schemas they reference. Each record embeds a schema ID the consumer resolves against a Schema Registry; if the destination’s registry lacks those schemas, EU consumers fail to deserialize — the data is there, but unreadable. Decide the schema strategy up front, alongside the link.

The options:

With Stream Governance ADVANCED (declared on the environment earlier), set up a schema exporter so subjects flow to the destination SR and stay in step with the source:

# Create a schema exporter on the source SR that pushes subjects to the destination SR
confluent schema-registry exporter create payments-schema-link \
  --subjects ":*:" \
  --context-type AUTO \
  --config-file schema-exporter.properties

confluent schema-registry exporter status payments-schema-link    # expect RUNNING

Destination consumers then resolve schema IDs locally. Keep compatibility mode (BACKWARD is the common default) consistent across both registries — a mismatch turns a routine producer schema bump into a destination-side break.

Consumer failover and the migration cutover

This is the payoff. Two operations look similar but mean different things, and choosing wrong loses data or causes split-brain.

Promote is a graceful cutover used when the source is still reachable — a planned migration or DR drill. It stops mirroring only after catching the destination fully up to the source, then makes the mirror writeable, so a promote is lossless.

Failover is a forced cutover used when the source is gone — a real outage where you cannot reach us-east-1. It severs the link immediately and makes the mirror writeable using whatever it had already replicated; anything not yet mirrored (the replication lag at the moment of failure) is your RPO — the data you may lose. This is why you alert on lag: lag is your potential data loss.

# PLANNED migration / drill — source reachable, lossless:
confluent kafka mirror promote settlement.completed \
  --cluster lkc-dest-euw1 --link payments-east-to-west

# REAL outage — source unreachable, accept lag-as-RPO:
confluent kafka mirror failover settlement.completed \
  --cluster lkc-dest-euw1 --link payments-east-to-west

Promote vs failover, decided:

After either cutover, producers fail over — repointed to the now-writeable eu-west-1 topic — and because offsets were preserved and synced, consumers resume at the right place. The full failover runbook:

For a migration (decommissioning the old cluster, not an outage) the flow is the same shape with promote instead of failover, in a maintenance window, producers cut over only after the promote confirms the destination is caught up. That is the zero-downtime cluster move: link, mirror, catch up, promote, repoint — no dual-write window, no offset translation.

Monitoring replication and tiered storage with the Metrics API

Replication you don’t watch is replication you don’t trust. Confluent exposes a Prometheus-compatible Metrics API you scrape (or pull into Datadog/Dynatrace via their Confluent integrations). The single most important metric here is cluster_link_mirror_topic_offset_lag — the per-mirror gap between source and destination — because on a forced failover that lag is your data loss. Wire it before you need it.

Pull the metrics over HTTP with a Metrics-API key (read-only telemetry scope, stored in Vault like every credential):

# Query mirror-topic offset lag over the last hour for the destination cluster
curl -s -u "$METRICS_KEY:$METRICS_SECRET" \
  "https://api.telemetry.confluent.cloud/v2/metrics/cloud/query" \
  -H 'Content-Type: application/json' -d '{
    "aggregations":[{"metric":"io.confluent.kafka.server/cluster_link_mirror_topic_offset_lag"}],
    "filter":{"field":"resource.kafka.id","op":"EQ","value":"lkc-dest-euw1"},
    "granularity":"PT1M","intervals":["PT1H/now"]
  }'

The metrics that tell you whether this system is healthy, and what a bad value indicates:

Turn those into alerts — on the leading indicators (lag, link state) so you act before consumers or auditors notice:

A sustained lag spike or a link leaving ACTIVE should page on-call and auto-open a ServiceNow incident, so a replication problem is a ticket with an owner, not a graph nobody watches.

Architecture at a glance

The shape is two Dedicated clusters and one managed link, wrapped in an operating model. Read the diagram left to right. The source in us-east-1 owns the writeable topics — producers only write here, scoped to WRITE on the settlement* prefix. A Cluster Link payments-east-to-west runs on the destination in eu-west-1 and continuously pulls the configured topics into read-only mirror topics that preserve partition count, keys, and exact offsets, while consumer.offset.sync.enable=true keeps committed group offsets in step so failover resumes cleanly. Each cluster independently runs Tiered Storage: the recent hotset stays on broker disk for low-latency tail reads while everything older offloads to object storage, so a topic holds seven years while brokers hold six hours. EU consumers read the mirror; if us-east-1 is lost, you failover the mirror to writeable and repoint producers — and because offsets matched, consumer groups resume where they were.

Around the data plane sit the operating tools, each mapping to a layer above: Terraform declares the environment, clusters, topics, link, mirrors, and ACLs; Vault issues every scoped credential as a short-lived lease; Okta/Entra SSO gates operators with MFA and RBAC bound to IdP groups; Jenkins/GitHub Actions plus Argo CD apply and reconcile the declared state; Datadog/Dynatrace ingest the Metrics API and alert on cluster_link_mirror_topic_offset_lag; ServiceNow records the failover change; Wiz audits the account and the Tiered Storage buckets. Follow producer → source topic → link → mirror → EU consumer, cold tier below each cluster, and the system explains itself.

Real-world scenario

Veltrix Pay runs the settlement backbone for a card-issuing fintech: order, authorization, and settlement.completed events on a single Dedicated 4-CKU cluster in AWS us-east-1 — 18 topics, ~22,000 partitions, sustained ingress ~140 MB/s peaking to 380 MB/s at end-of-day settlement. Retention was 7 days on broker disk; nothing was replicated. Six platform engineers; monthly Confluent spend ~₹14,50,000 (~USD 17.5k). Two pressures landed the same quarter: a new EU banking entity needed local settlement that could survive a us-east-1 incident, and a regulator mandated seven-year retrievability of every settlement event as a replayable stream.

The first attempt was the obvious wrong one: MirrorMaker 2 on a small self-managed Connect cluster in eu-west-1 for the four settlement-critical topics. It demoed fine and fell apart in week two — the Connect cluster needed its own capacity planning, the offset-sync connector lagged under end-of-day load, and in a deliberate DR test the failed-over consumer groups restarted from the beginning of a 9-day topic, re-processing ~600 million events and double-firing settlement reconciliation. The post-mortem line: “we replicated the data but not the position, and added a system that fails on its own.”

They rebuilt on Cluster Linking and Tiered Storage. A second Dedicated 4-CKU cluster went up in eu-west-1 via Terraform with PrivateLink between the regions. One destination-initiated link, pay-east-to-west, replicated the four settlement topics — not all 18; clickstream and internal-ops topics stayed single-region to avoid cross-region egress Europe did not need. They set consumer.offset.sync.enable=true and consumer.offset.sync.ms=15000, and Schema Linking carried the Avro subjects so EU consumers could deserialize. On both clusters the four settlement topics got retention.ms = 7 years and confluent.tier.local.hotset.ms = 12 hours.

The numbers told the story. Moving the heavy settlement topics from 7-day broker retention to a 12-hour hotset shrank the hot-disk footprint enough to drop one CKU per cluster, and seven years of history accrued in object storage at roughly ₹2.20/GB-month. Cross-region mirroring billed on the ~140 MB/s of settlement bytes, not the full 380 MB/s — a deliberate egress line. The redone DR drill was the proof: they promoted the EU mirror in a maintenance window, repointed a canary producer, and the consumer groups resumed at their exact committed offsets — zero re-processing, zero double-settlement. Total spend landed at ₹16,80,000/month across two regions with seven-year retention — only ~16% more than the single-region, 7-day starting point.

The payoff came four months later when AWS us-east-1 had a multi-hour control-plane degradation. Veltrix failover-ed the four mirror topics, repointed producers to eu-west-1, and kept settling. Measured RPO was 1.8 seconds (the offset lag at the moment of failure, which they had alerted on and kept low); RTO was 11 minutes, almost all of it human decision-and-approval, not the technical cutover. The wall-mounted lesson: “Cluster Linking replicates the position, not just the bytes — that one difference is the entire value of doing DR right.”

The two designs, compared on what actually mattered:

Advantages and disadvantages

The managed-link, tiered-storage model both enables multi-region long-retention Kafka and imposes real trade-offs. Weigh it honestly:

The model is right when you genuinely need a second region (DR, locality, or migration) and long retention on a Dedicated footprint you can justify. It is overkill for a single-region dev workload, and it is not a substitute for in-region HA — MULTI_ZONE handles a zone failure, the link handles a region failure. The disadvantages are all manageable, but only if you make the up-front decisions (Dedicated, offset sync, schema strategy, what-to-replicate) deliberately rather than discovering them in an incident.

Hands-on lab

Stand up the whole pattern at small scale, prove the three things that matter — data flows, offsets match, old data comes from the cold tier — execute a clean cutover, then tear it all down. This uses two Dedicated clusters (linking requires Dedicated), so it is not free-tier; budget a small spend and delete everything at the end (teardown is Step 11). Run in a shell with the confluent CLI logged in.

Step 1 — Pin the environment and set variables.

confluent login --save
confluent environment use env-payments-prod
SRC=lkc-source-use1     # replace with your real source cluster id after Step 2
DST=lkc-dest-euw1       # replace with your real destination cluster id after Step 2
LINK=payments-east-to-west

Step 2 — Provision two Dedicated clusters (or reuse existing). Smallest multi-zone is 2 CKUs; for a short lab a single-zone 1-CKU pair is cheaper if your environment allows it.

confluent kafka cluster create lab-source --cloud aws --region us-east-1 \
  --type dedicated --cku 1 --availability single-zone
confluent kafka cluster create lab-dest --cloud aws --region eu-west-1 \
  --type dedicated --cku 1 --availability single-zone
confluent kafka cluster list   # note the two lkc-... ids; set SRC and DST to them

Expected: two clusters provisioning; list shows PROVISIONING then UP after a few minutes. Wait for both to be UP before proceeding.

Step 3 — Create the source topic with Tiered Storage retention. Long retention, tiny hotset, so old reads hit the cold tier.

confluent kafka topic create settlement.completed --cluster $SRC \
  --partitions 6 \
  --config retention.ms=220752000000 \
  --config confluent.tier.local.hotset.ms=60000 \
  --config cleanup.policy=delete
confluent kafka topic describe settlement.completed --cluster $SRC

Expected: describe lists the configs back, including retention.ms=220752000000 and confluent.tier.local.hotset.ms=60000 — a 1-minute hotset, deliberately tiny so the lab can force a cold read fast.

Step 4 — Create a service account and a source-scoped key for the link.

confluent iam service-account create sa-lab-link --description "lab link principal"
confluent api-key create --service-account sa-lab-link --resource $SRC
# capture the printed key/secret into LINK_KEY / LINK_SECRET shell vars (do NOT commit)
confluent kafka acl create --cluster $SRC --allow --service-account sa-lab-link \
  --operations READ,DESCRIBE --topic settlement --prefix
confluent kafka acl create --cluster $SRC --allow --service-account sa-lab-link \
  --operations DESCRIBE --cluster-scope

Expected: an sa-... id, an API key/secret printed once, and two ACL rows created.

Step 5 — Create the Cluster Link on the destination, pointing at the source. Get the source bootstrap first.

SRC_BOOT=$(confluent kafka cluster describe $SRC -o json | jq -r '.endpoint')
cat > /tmp/link.properties <<EOF
link.mode=DESTINATION
connection.mode=OUTBOUND
bootstrap.servers=${SRC_BOOT}
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="${LINK_KEY}" password="${LINK_SECRET}";
consumer.offset.sync.enable=true
auto.create.mirror.topics.enable=false
EOF

confluent kafka link create $LINK --cluster $DST \
  --source-cluster $SRC --source-bootstrap-server ${SRC_BOOT} \
  --config-file /tmp/link.properties
confluent kafka link list --cluster $DST

Expected: the link appears and reaches ACTIVE within a minute or two. If it stays in a non-active state, jump to the troubleshooting table (most often a bad bootstrap or key).

Step 6 — Create the mirror topic on the destination.

confluent kafka mirror create settlement.completed --cluster $DST --link $LINK
confluent kafka mirror list --cluster $DST --link $LINK
confluent kafka topic update settlement.completed --cluster $DST \
  --config retention.ms=220752000000 \
  --config confluent.tier.local.hotset.ms=60000

Expected: mirror list shows settlement.completed in ACTIVE (or MIRROR) state with 6 partitions matching the source.

Step 7 — Produce on the source, consume the mirror on the destination. Prove data flows across the link.

# Produce 5 records on the SOURCE
for i in 1 2 3 4 5; do
  echo "{\"id\":\"s-100$i\",\"amt\":$((RANDOM%500))}"
done | confluent kafka topic produce settlement.completed --cluster $SRC

# Consume the MIRROR on the DESTINATION (should appear within seconds)
confluent kafka topic consume settlement.completed --cluster $DST \
  --from-beginning --print-key --timeout 10000

Expected: the five records appear on the destination within a few seconds — written in us-east-1, read in eu-west-1, over the managed link.

Step 8 — Prove offset fidelity. The mirror’s high-water marks should track the source partition-for-partition:

confluent kafka mirror describe settlement.completed --cluster $DST --link $LINK
# Look at per-partition lag (should be ~0) and that mirror offsets match the source

Expected: mirror describe shows per-partition lag at or near zero and offsets tracking the source — the offset-fidelity guarantee, visible.

Step 9 — Force a cold-tier read. With a 1-minute hotset, records older than a minute are served from object storage. Wait, then read from the beginning.

sleep 90                          # let the first records age past the 60s hotset
confluent kafka topic consume settlement.completed --cluster $SRC \
  --from-beginning --timeout 10000
# The earliest records now come from the COLD tier (object storage), transparently.

Expected: all records still return, including the earliest — but those are now fetched from the cold tier (in production, confirm via the Metrics API showing non-zero tiered-bytes-fetched).

Step 10 — Execute a graceful cutover (promote). Simulate a planned migration — the source is reachable, so use promote for a lossless cutover that makes the mirror writeable.

confluent kafka mirror promote settlement.completed --cluster $DST --link $LINK
confluent kafka mirror list --cluster $DST --link $LINK     # state -> PROMOTED
# The destination topic is now writeable — producers can target eu-west-1:
echo '{"id":"s-2001","amt":99}' | \
  confluent kafka topic produce settlement.completed --cluster $DST

Expected: the mirror moves to PROMOTED, and a produce against the destination now succeeds (it was rejected while the topic was a read-only mirror). That transition — read-only mirror to writeable topic — is the failover/migration mechanism.

Step 11 — Teardown (do this; Dedicated clusters bill continuously). Tear down in reverse dependency order.

confluent kafka mirror delete settlement.completed --cluster $DST --link $LINK
confluent kafka link delete $LINK --cluster $DST
confluent kafka cluster delete $DST
confluent kafka cluster delete $SRC
confluent api-key list --service-account sa-lab-link   # then delete the key
confluent iam service-account delete sa-lab-link

Expected: each resource deletes cleanly. Verify both clusters are gone with confluent kafka cluster list — a forgotten Dedicated cluster is the most expensive lab mistake you can make.

The lab steps mapped to what each proves, and its production analogue:

Common mistakes & troubleshooting

The failure modes that bite, as a symptom → cause → confirm → fix playbook. Read the table first; expanded notes follow for the worst offenders.

The expanded reasoning for the entries that cost the most time:

3. Failed-over consumer group restarts from the beginning. The expensive DR mistake — a failover that re-processes the whole topic and double-fires downstream side effects. The cause is almost always that consumer.offset.sync.enable was not enabled at link creation; without it the destination never knew where groups were, so a group resets to its auto.offset.reset (often earliest). Confirm with confluent kafka link describe <link>. You cannot cleanly retrofit it — re-create the link with offset sync on. This is why the decision belongs at design time, not after an incident.

8. Mirror lag growing steadily. On a forced failover, the source-to-mirror lag is your data loss. Steady growth means the link can’t keep up with source ingress (under-sized CKUs or too many replicated topics) or is degrading. Confirm via cluster_link_mirror_topic_offset_lag trending up; fix by adding CKUs, replicating only the topics that need a second region, and checking link health — never let lag silently grow. (Items 4 and 5 are covered in the Schema Linking and Tiered Storage sections above; the table rows are the quick reference.)

Best practices

• Use Dedicated on both ends for production multi-region. Basic/Standard limits will bite; Dedicated gives Cluster Linking, full Tiered Storage control, private networking, BYOK, and the SLA. Saving on the cluster type now means migrating later.
• Set consumer.offset.sync.enable=true at link creation, every time. It is the difference between a clean failover and re-processing the whole topic. It cannot be cleanly retrofitted, so make it a non-negotiable in your link template.
• Replicate only the topics that need a second region. Cross-region mirroring bills on bytes. Keep auto.create.mirror.topics.enable=false and create mirrors explicitly so cross-region cost is a deliberate, reviewed choice — not an accident.
• Set retention and hotset on the destination topics too. Mirror topics tier independently; if you forget, the destination quietly keeps only the default ~7 days while you think you have seven years in both regions.
• Keep the hotset small to keep long retention cheap. Hours of hotset, years of retention. The hotset caps expensive broker disk; everything else lives in cheap object storage. This single lever is the whole cost story.
• Decide the schema strategy up front. Schema Linking or a shared Stream Governance package, with aligned compatibility modes, so the destination can always deserialize. A mirror without schemas is unreadable.
• Manage the entire topology as code. Terraform for environment, clusters, topics, link, mirrors, ACLs; Argo CD to reconcile so console drift is reverted. A topology you can plan is one you can review, audit, and rebuild.
• Scope every credential and store it in Vault. Separate service accounts for producer, consumer, and link, each with least-privilege ACLs and a Vault-leased key, so a leak is small and rotation is a key-roll, not a redeploy.
• Alert on lag before it becomes data loss. cluster_link_mirror_topic_offset_lag and link state into Datadog/Dynatrace; a sustained lag spike or a link leaving ACTIVE should page and open a ticket. Lag is your RPO.
• Use promote for planned cutovers, failover only for real outages. Promote is lossless (catches up first); failover accepts lag-as-RPO. Rehearse both in DR drills so the runbook is muscle memory, not improvisation.
• Plan large backfills off-peak. A --from-beginning over a multi-terabyte tiered topic drives cold-tier fetch and can pressure the cluster. Schedule it, bound it, size CKUs to absorb it.
• Run MULTI_ZONE for in-region HA. The link handles a region failure; multi-zone handles a zone failure. You want both; they are complementary, not redundant.

Security notes

Every credential and path in this build is least-privilege and short-lived, because a payments backbone is a high-value target and a replicated one doubles the surface.

• Scoped, Vault-leased credentials. The bootstrap Cloud key provisions only and lives in Vault; runtime producers, consumers, and the link each get a cluster-scoped key (also from Vault), so a leaked consumer key cannot touch producers or the source’s write path, and rotation is a key-roll, not a redeploy.
• Least-privilege ACLs/RBAC. Producers get WRITE only on their prefix; the link principal READ/DESCRIBE only; consumers READ on the mirror and their group; nobody ever gets WRITE on a mirror. RBAC bindings map to IdP groups so access follows employment.
• Humans authenticate via SSO with MFA. The Console and CLI are reachable only through Okta/Entra ID with MFA; IdP de-provisioning instantly removes access. No shared logins, ever.
• Encryption everywhere. All traffic is SASL_SSL/TLS; tiered data at rest is encrypted in the object store, and Dedicated supports BYOK. Wiz audits the account and the tiered-storage buckets for drift (public exposure, weakened encryption).
• Private networking between regions. Prefer PrivateLink, peering, or Transit Gateway so replication never crosses the public internet — a security and reliability win, and a common audit requirement.
• Audit the position, not just access. The distinct link principal makes the audit trail show exactly what the link copied versus what an app wrote; a link leaving ACTIVE or an unexpected ACL change opens a ServiceNow incident.

The security controls, what each defends, and what else it buys:

Cost & sizing

Three levers dominate the bill, and they pull in different directions, so the goal is to right-size each rather than minimize any one.

CKU sizing drives each Dedicated cluster’s base cost, and two regions means two footprints. Size to sustained throughput with headroom for failover surge and cold-read backfills, not peak, and scale CKUs through Terraform when utilization climbs — over-provisioning “to be safe” is the most common waste. Tiered Storage is the cost win that makes seven-year retention viable: object storage is far cheaper per GB than broker disk, so a small confluent.tier.local.hotset.ms keeps the hot tier tiny while history accumulates cheaply — and shrinking the hot footprint can drop a CKU. Cluster Linking bills on bytes mirrored across regions, so replicate only what genuinely needs a second region; cross-region egress is a deliberate line item, not a side effect of auto.create.mirror.topics.enable=true.

The cost drivers, what each buys, the watch-out:

Right-size in order of leverage: shrink the hotset to hours (biggest saving, may drop a CKU); replicate only DR-critical topics (cuts cross-region bytes and destination storage); size CKUs to sustained and scale on metrics; and budget the cold tier’s GB growth explicitly. Pipe the Confluent billing and Metrics API into your cost dashboard so mirrored-GB and tiered-GB trends are visible. There is no free tier for Dedicated — the floor is two small Dedicated clusters, which is why the lab insists on teardown.

Interview & exam questions

1. Why is a Cluster Linking mirror topic better for DR than MirrorMaker 2? A mirror topic preserves the source’s exact offsets, partition count, and keys, and (with consumer.offset.sync.enable) syncs committed group offsets — so on failover a group resumes at the right position. MirrorMaker 2 republishes records with new offsets and needs separate offset-translation, which is fragile and the common cause of groups restarting from the beginning. Cluster Linking is also fully managed.

2. Cluster Linking is “destination-initiated.” What does that mean operationally? You create the link object and the mirror topics on the destination cluster, and the link pulls topics from the source. There is nothing to deploy or scale on the source side, and link create/mirror create always run against the destination. The destination dials out to the source’s bootstrap (connection.mode=OUTBOUND).

3. What is the single most important link setting for a clean failover, and why can’t you add it later? consumer.offset.sync.enable=true. It continuously copies committed consumer-group offsets from source to destination, so a failed-over group knows where it was. It must be set at link creation because it governs how offsets are tracked from the start; retrofitting it does not retroactively reconstruct the sync, so groups may still reset. Decide it at design time.

4. Difference between promote and failover on a mirror topic? Both convert a read-only mirror into a writeable topic. Promote is graceful — it first catches the destination fully up to a reachable source, so it is lossless; use it for planned migrations and drills. Failover is forced — it severs the link immediately when the source is unreachable, accepting the current replication lag as your RPO; use it in a real outage.

5. How does Tiered Storage make seven-year retention economically viable? It offloads closed segments older than the hotset to cheap object storage while keeping only the hotset on expensive broker disk. Set retention.ms to seven years and confluent.tier.local.hotset.ms to a few hours: brokers hold hours, object storage holds years, old offsets fetch transparently from the cold tier, and the topic stays a continuous, replayable Kafka topic.

6. What’s the relationship between retention.ms and confluent.tier.local.hotset.ms? retention.ms is how long data is kept anywhere before deletion; hotset.ms is how much recent data stays on local broker disk. The hotset caps hot-disk cost regardless of total retention — that decoupling is the whole point, and long retention with a small hotset is the cheap-and-long sweet spot.

7. Which cluster types support Cluster Linking, and which type does a production multi-region payments design need? Standard supports it with limits; Dedicated supports it fully (any direction) with isolated CKU capacity, full Tiered Storage control, private networking, BYOK, and the highest SLA. A mission-critical multi-region payments backbone needs Dedicated on both ends. Basic does not support production linking.

8. A consumer in the destination region throws “Schema not found.” What happened and how do you fix it? The mirror replicated the records, but each record’s embedded schema ID must resolve against the destination’s Schema Registry, which doesn’t have the schema. Fix by setting up Schema Linking to replicate subjects to the destination SR, or by sharing one Stream Governance package across both clusters, keeping compatibility modes aligned.

9. You enabled seven-year retention and your CKU cost jumped. What’s the most likely cause? You left the hotset large (or default) while setting a huge retention.ms, so years of data sit on expensive broker disk. Set confluent.tier.local.hotset.ms small (hours); it caps disk-resident data regardless of retention, pushing the rest into cheap object storage — often enough to drop a CKU.

10. Why must you set retention on the destination mirror topic, not just the source? Mirror topics tier independently of the source. If you only configure the source, the destination falls back to defaults (≈7 days) and you silently lose your long retention in the second region. Always set retention.ms and hotset.ms on both sides.

11. How do you keep cross-region replication cost under control? Keep auto.create.mirror.topics.enable=false and create mirror topics explicitly for only the topics that genuinely need a second region, so cross-region mirroring bytes are a deliberate choice. Monitor mirrored-GB via the Metrics API and challenge any topic whose replication isn’t earning its keep.

12. What does mirror lag tell you in DR terms, and how do you treat it? On a forced failover, the lag between source and mirror is exactly the data you may lose (your RPO). Treat it as a first-class SLO: alert on cluster_link_mirror_topic_offset_lag so a sustained spike pages on-call, and keep it low by sizing CKUs and replicating only what’s needed.

These map to the Confluent Certified Administrator for Apache Kafka (CCAAK) and Confluent Certified Developer for Apache Kafka (CCDAK) for the Kafka, replication, and retention concepts, and to general cloud-architecture DR objectives (RTO/RPO, multi-region) that appear across AWS / Azure / GCP professional architect exams. A compact mapping for revision:

Quick check

1. You create a Cluster Link but forget consumer.offset.sync.enable=true. A month later you do a real failover. What goes wrong, and why can’t you just fix it now?
2. Your source topic has retention.ms set to seven years but your CKU bill jumped after you applied it. What single config did you most likely get wrong?
3. True or false: a mirror topic is writeable as soon as it’s created, so EU producers can write to it directly for active-active.
4. EU consumers throw “Schema not found” even though the mirror is healthy and records are arriving. What’s missing?
5. When do you use promote versus failover on a mirror topic, and what’s the data-loss difference?

Answers

1. The failed-over consumer groups restart from the beginning (or latest), re-processing or skipping data, because the destination never had their offsets synced. You can’t cleanly retrofit offset sync, so you must re-create the link with consumer.offset.sync.enable=true — which is why the decision belongs at design time.
2. You left confluent.tier.local.hotset.ms large or default while setting a huge retention.ms, so years of data sit on expensive broker disk instead of cheap object storage. Set the hotset small (hours); it caps hot-disk cost regardless of total retention.
3. False. Mirror topics are read-only until you explicitly promote or failover them. Producers stay on the source until a deliberate cutover; writing to a mirror returns an error. (Active-active is a different, bidirectional topology.)
4. The destination Schema Registry doesn’t have the schemas the records’ embedded schema IDs reference. Set up Schema Linking (or share a Stream Governance package) so subjects replicate to the destination, with aligned compatibility modes.
5. Use promote for a planned cutover when the source is reachable — it catches the destination fully up first, so it’s lossless. Use failover when the source is gone — it severs the link immediately and accepts the current replication lag as your RPO (potential data loss).

Glossary

• Cluster Linking — Confluent’s managed, destination-initiated replication that pulls topics from a source cluster into read-only mirror topics on a destination cluster, preserving offsets.
• Mirror topic — a read-only copy of a source topic created over a Cluster Link, inheriting the source’s partition count, keys, and exact offsets; becomes writeable only on promote/failover.
• CKU (Confluent Unit for Kafka) — the capacity unit of a Dedicated cluster, bundling ingress/egress throughput, a partition ceiling, and connection limits; the primary cost driver.
• Tiered Storage — automatic offload of closed log segments from broker local disk to cloud object storage, decoupling retention from disk and making long retention cheap.
• Hotset (confluent.tier.local.hotset.ms / .bytes) — the amount of recent data kept on broker local disk for low-latency reads; everything older lives in the cold tier.
• retention.ms — how long data is retained anywhere before deletion (the total horizon); independent of the hotset.
• Offset — the monotonic position of a record within a partition; offset preservation across regions is what makes consumer failover correct.
• Offset sync (consumer.offset.sync.enable) — link setting that copies committed consumer-group offsets from source to destination so a failed-over group resumes at the right place.
• Promote — a graceful cutover that catches the destination fully up to a reachable source, then makes the mirror writeable (lossless); used for planned migration/drills.
• Failover — a forced cutover that severs the link immediately when the source is unreachable and makes the mirror writeable, accepting the replication lag as RPO; used in a real outage.
• Schema Linking / Stream Governance — replication of Schema Registry subjects from one cluster to another (so the destination can deserialize), built on Confluent’s governance package (ESSENTIALS / ADVANCED); ADVANCED enables Schema Linking.
• Service account — a non-human identity that holds scoped API keys (producer, consumer, link), each with least-privilege ACLs.
• ACL / RBAC — authorization controls on topics, consumer groups, and the cluster; the basis of least-privilege access.
• RTO / RPO — Recovery Time Objective (how fast you recover) and Recovery Point Objective (how much data you can lose); the link’s offset lag is your RPO on a forced failover.
• Dedicated cluster — a single-tenant Confluent Cloud cluster with provisioned CKUs, full Cluster Linking and Tiered Storage control, private networking, and BYOK; required for this design.

Next steps

You can now stand up multi-region Kafka with offset-faithful replication and cheap long retention, and fail over cleanly. Build outward:

• Next: Multi-Region Data: Choosing Replication and Consistency Without Losing Writes — the broader decision of which replication topology and consistency model to pick before you wire links.
• Related: Event Hubs Fundamentals: Partitions, Consumer Groups, Checkpoints and Offsets for Beginners — the partition/offset mental model the whole failover story rests on.
• Related: Real-Time Analytics with Databricks and Confluent Kafka on AWS — what to do with the replicated, long-retained stream once it’s flowing in two regions.
• Related: Real-Time CDC with Debezium and Kafka — using Kafka as the change-data-capture backbone that these clusters often carry.
• Related: HashiCorp Vault as Central Secrets Broker for Multi-Cloud Workloads — the secrets-brokering pattern behind every scoped credential in this build.
• Related: High Availability vs Disaster Recovery: RTO and RPO Explained — the resilience vocabulary that frames every promote-vs-failover choice here.