Azure Cache for Redis Enterprise: Clustering, Active Geo-Replication, and Resilient Failover Patterns

Most “Redis is down” pages I have been dragged into were not Redis failing. They were a client library that opened a single connection to a single node, hardcoded a regional hostname, and treated MOVED as a fatal error instead of a routing hint. Azure Cache for Redis Enterprise – the tier built on the commercial Redis Enterprise runtime rather than OSS Redis – gives you clustering, multi-region active-active replication, durable persistence, and the Redis modules (Search, JSON, TimeSeries, Bloom). But every one of those features changes the contract your client must honor. Cross-slot multi-key commands are no longer free. A node can move under you mid-request. Two regions can both accept a write to the same key and you have to decide who wins. This guide wires up the Enterprise tier correctly and, just as importantly, builds the client-side behavior that survives the day the topology shifts.

Everything here targets the Enterprise and Enterprise Flash tiers, with notes on where Premium diverges. The provider resource is Microsoft.Cache/redisEnterprise, a different ARM resource type from the Microsoft.Cache/redis you use for Basic/Standard/Premium. That distinction trips up Terraform and Bicep modules constantly, and it is the first thing to get right because a module written for one resource type silently does nothing useful against the other.

By the end you will stop treating “the cache is down” as a single event. You will know whether you are looking at a CROSSSLOT error from a keyspace that ignored hash tags, a MOVED that escaped to application code because cluster mode is off, an OOM because NoEviction met an undersized cluster, a connection storm because the client pools per-request instead of multiplexing, or a genuine regional outage that an active-active CRDB should have made a non-event. Knowing which in ninety seconds is what separates a five-minute blip from a two-hour incident.

What problem this solves

A cache that holds session state, idempotency keys, feature flags, rate-limit counters, or a hot read-through layer is on the critical path of every request. When it stalls, the application stalls behind it – or worse, falls back to the database and runs at a fraction of normal throughput until the database itself buckles. The naive answer (“just add a replica”) solves node loss but not the three failure classes that actually page senior engineers: a client that cannot follow a topology change, a single-region cache that cannot accept writes during a regional incident, and a cache used as an LRU eviction store in a geo-replicated topology where evictions silently diverge the regions.

What breaks without the Enterprise patterns: a passive geo-replica that is read-only at exactly the moment you need to write to it (during the primary region’s outage), turning an eleven-minute regional blip into eleven minutes of either blocked writes or duplicate processing. A MULTI/EXEC transaction that worked in dev against a single node and throws CROSSSLOT the first time it runs against a real cluster. A ConnectionMultiplexer created with the default AbortOnConnectFail = true that throws permanently the first time a maintenance window blips the connection and never recovers. A cache sized for the average working set that OOMs the moment a campaign doubles the keyspace, returning OOM command not allowed on every write because the policy is NoEviction.

Who hits this: any team running Redis on the critical path at scale. It bites hardest on multi-region applications (where passive replication is mistaken for an availability tool), on stateful workloads that model everything as opaque strings (and so eat last-write-wins data loss), on teams that adopt clustering without auditing their client library’s cluster support, and on anyone who treats Redis persistence as a backup. The fix is rarely “buy a bigger SKU” – it is “model the data as the right CRDT, pick the clustering policy the client can actually drive, size for the full working set under NoEviction, and make the client survive a node move.”

To frame the field before the deep dive, here is every failure class this article covers, the question it forces, and where to look first:

Learning objectives

By the end of this article you can:

• Choose between Standard, Premium, Enterprise, and Enterprise Flash from durability and topology requirements rather than raw memory size, and explain why Enterprise is a separate runtime, not the next rung on a ladder.
• Pick the right clustering policy (OSS vs Enterprise/proxy) at creation time, knowing it is permanent, and design a keyspace with hash tags so transactions, Lua, and MGET/MSET stay single-slot.
• Stand up active geo-replication as a conflict-free replicated database (CRDB) across regions, and model write data as the correct CRDT (counter, set, hash) where last-write-wins on strings is unacceptable.
• Configure RDB and AOF persistence with the right durability trade-off, and explain why persistence is not a backup and how it relates to (but differs from) replication.
• Lock the cache into a VNet with a Private Endpoint, Private DNS, TLS-only on port 10000, and Microsoft Entra ID token auth instead of a shared access key.
• Build a resilient client: a singleton multiplexer with AbortOnConnectFail = false, periodic health checks, and a jittered operation-level retry that survives scaling, patching, and node moves.
• Run online scale-up and scale-out operations, validate them with a load test through a live reshard, and alert on the leading indicators (used-memory %, eviction rate, server load, client-side p99) before they become outages.

Prerequisites & where this fits

You should already understand Redis at the data-structure level: strings, hashes, sets, sorted sets, TTL/EXPIRE, and the basic command set (SET, GET, INCR, MGET, MULTI/EXEC). You should know how to run az in Cloud Shell, read JSON output, and reason about a VNet, subnet, NSG, and Private DNS zone. Familiarity with at least one Redis client library (StackExchange.Redis, Lettuce/Jedis, redis-py, go-redis) helps, because half of the resilience story lives in client configuration.

This sits in the Data & Caching track of the Zero-to-Hero program. It assumes the networking fundamentals from Azure Virtual Network Basics: Subnets, NSGs, and Peering and the private-connectivity patterns from Azure Private Endpoints and Private DNS at Scale. The identity and secret-handling side leans on Azure Key Vault: Secret Rotation with Managed Identity. For multi-region thinking beyond the cache, it pairs with Cosmos DB Multi-Region Writes and Conflict Resolution (a useful contrast: Cosmos lets you plug in conflict logic; Redis CRDTs resolve automatically per type) and Azure Multi-Region Active-Active Disaster Recovery. When the cache fronts a relational store, Azure SQL Database: Hyperscale, Elastic Pools, Ledger is the backing tier the cache protects.

A quick map of who owns what during a cache incident, so you call the right person fast:

Core concepts

Six mental models make every later decision obvious.

Enterprise is a different runtime, not a bigger Premium. Basic/Standard/Premium run OSS Redis under Microsoft.Cache/redis. Enterprise and Enterprise Flash run the commercial Redis Enterprise software under Microsoft.Cache/redisEnterprise, with a parent cluster resource and a child database resource. The database is the thing your client connects to. This split is why a Bicep module that creates a Microsoft.Cache/redis resource cannot produce an Enterprise cache, and why the endpoint, port, and capabilities differ.

The clustering policy is a permanent client contract. You choose OSS or Enterprise clustering policy at database creation and you cannot change it without recreating the database. OSS exposes the native Redis Cluster API: the client discovers every shard, computes the CRC16 hash slot (16384 slots) for each key, and connects directly to the owning node – lowest latency, but it requires a cluster-aware client and the client sees every node address. Enterprise policy puts a proxy in front of the shards so the client talks to a single endpoint like a standalone Redis – simplest for clients and networking, at the cost of a proxy hop.

Multi-key commands need one hash slot. In any clustered Redis, a command touching multiple keys requires all of them in the same hash slot. MSET user:1001 a user:1002 b hashes the two keys to different slots and fails with CROSSSLOT under OSS policy. Hash tags – the substring inside the first {} – force co-location: MSET user:{t42}:1001 a user:{t42}:1002 b hashes both on t42. Design the keyspace so everything you co-access (a tenant, an order, a session) shares a hash tag; otherwise transactions, Lua scripts, and MGET/MSET break.

Active-active means no primary and automatic conflict resolution. Enterprise active geo-replication builds an Active-Active CRDB (conflict-free replicated database). Every region accepts reads and writes; changes replicate full-mesh to all peers. A region outage means you keep serving from the survivors with no failover step. Concurrent writes converge deterministically because each data type is reimplemented as a CRDT (conflict-free replicated data type): strings are last-write-wins by timestamp, counters are additive (both increments apply), and sets/hashes/sorted-sets merge per element. You do not write conflict logic; you choose the data type that gives the convergence you need.

Persistence survives restart; replication survives node loss; neither is a backup. RDB snapshots the dataset on an interval; you lose everything since the last snapshot on a hard failure. AOF logs every write, and with fsync every second the worst-case loss is ~1 second. Both protect against a full cluster restart. Replication protects against losing a node or a region. Neither protects against a bad FLUSHALL or a logic bug – that is what an export to a storage account is for.

The client is where outages are made or prevented. A correctly provisioned, multi-region, persistent cluster behind a broken client is still an outage. The client must multiplex (one long-lived connection, not pool-per-request), keep retrying instead of throwing on first-connect failure, follow MOVED/ASK redirects (OSS policy), health-check idle connections, and retry the operation with jittered backoff through a node move. Every one of those is a configuration choice, and the defaults are frequently wrong for Azure.

The vocabulary in one table

Before the deep sections, pin down every moving part. The glossary at the end repeats these for lookup; this table is the mental model side by side:

Tier selection: Standard, Premium, Enterprise, Enterprise Flash

Pick the tier from your durability and topology requirements, not from raw memory size. The tiers are not a linear ladder – Enterprise is a separate runtime, and Flash trades RAM for NVMe to cut cost on large skewed datasets.

The deciding factors, as a decision table:

A note on each tier’s sweet spot, because the table compresses real trade-offs:

# Enterprise tier uses a distinct resource: redisenterprise, with a child database.
# --capacity must be EVEN for Enterprise SKUs (nodes deploy in HA pairs).
az redisenterprise create \
  --name kv-redis-prod \
  --resource-group rg-data-prod \
  --location eastus2 \
  --sku Enterprise_E10 \
  --capacity 2 \
  --zones 1 2 3

# The database (the actual Redis endpoint) is a child resource.
az redisenterprise database create \
  --cluster-name kv-redis-prod \
  --resource-group rg-data-prod \
  --client-protocol Encrypted \
  --clustering-policy EnterpriseCluster \
  --eviction-policy NoEviction \
  --persistence aof-enabled=true aof-frequency=1s

The SKU name encodes both the engine (Enterprise_E10, EnterpriseFlash_F300) and a capacity unit. --capacity must be an even number for Enterprise SKUs because nodes deploy in pairs for HA. Always pass --zones 1 2 3 at create time; you cannot add zone redundancy to an existing cluster in place.

The SKU families and what the suffix encodes:

Clustering policies and key distribution

This is the single most consequential decision and it is permanent for the database’s lifetime: you choose it at creation and it dictates how your client connects and how multi-key operations behave.

OSS clustering policy exposes the native Redis Cluster API. The client discovers all shards, computes the CRC16 hash slot for each key, and connects directly to the owning node. Lowest latency and highest throughput because there is no proxy hop – but it requires a cluster-aware client, and the client sees every node’s address, which complicates private networking (every node IP must be routable).

Enterprise clustering policy puts a proxy in front of the shards. The client connects to a single endpoint as if it were a standalone Redis; the proxy routes commands to the correct shard. Far simpler for clients (any standard client works, no cluster mode) and for networking (one endpoint), at the cost of a proxy hop.

OSS vs Enterprise policy, side by side

The multi-key contract

The behavior that surprises people is multi-key commands. A command touching multiple keys requires all those keys to live in the same hash slot:

# This fails across slots -- the keys hash to different slots
MSET user:1001 alice user:1002 bob   # CROSSSLOT error under OSS policy

# Hash tags force keys into the same slot using the {...} substring
MSET user:{tenant42}:1001 alice user:{tenant42}:1002 bob   # both hash on "tenant42"

Only the substring inside the first {} is hashed. Design your keyspace with hash tags around the entity you co-access so transactions and MGET/MSET stay single-slot. Under the Enterprise policy, the proxy makes some cross-slot multi-key commands appear to work by fanning out, but MULTI/EXEC transactions and Lua scripts still require single-slot keys – so the hash-tag discipline is non-negotiable either way.

Which operations are slot-sensitive, and what each policy does:

Hash-tag design patterns that keep co-accessed keys together:

Choose OSS policy when you control the client and want maximum performance, and you are comfortable with cluster-aware libraries (StackExchange.Redis, Lettuce, redis-py with cluster mode, go-redis ClusterClient). Choose Enterprise policy when you need a single endpoint for private-networking simplicity, or your client cannot do cluster mode. You cannot change it later without recreating the database – so this decision deserves a design review, not a default.

Active geo-replication topologies and conflict handling

Enterprise active geo-replication builds an active-active database (an Active-Active CRDB). Every participating cluster accepts both reads and writes, and changes replicate to all peers. There is no primary. A region outage means you keep serving from the survivors with no failover step.

The mechanism that makes concurrent writes safe is CRDTs. Redis Enterprise reimplements each data type as a CRDT so concurrent writes in different regions converge deterministically. The key insight: you do not get to plug in custom conflict logic the way Cosmos DB does. You pick the data type whose built-in convergence matches your correctness requirement.

CRDT semantics per data type

Choosing the data type for the convergence you need

# Create an active geo-replication group spanning two regions.
# Each region is its own redisenterprise cluster + database; you link them
# via a shared group nickname and mutual linkedDatabase references.

az redisenterprise database create \
  --cluster-name kv-redis-eastus2 \
  --resource-group rg-data-eastus2 \
  --client-protocol Encrypted \
  --clustering-policy EnterpriseCluster \
  --group-nickname global-sessions \
  --linked-databases id="/subscriptions/<sub>/resourceGroups/rg-data-eastus2/providers/Microsoft.Cache/redisEnterprise/kv-redis-eastus2/databases/default" \
  --linked-databases id="/subscriptions/<sub>/resourceGroups/rg-data-westeurope/providers/Microsoft.Cache/redisEnterprise/kv-redis-westeurope/databases/default"

The --linked-databases list must include this database plus every peer, and the same group nickname must be used on every member. Designing the topology:

• Keep the geo group to regions you can tolerate replicating all writes to – replication is full mesh, so N regions means each write fans out to N-1 peers. Bandwidth and cross-region egress cost scale with the mesh.
• Active-active forces NoEviction semantics conceptually: do not run an active-active cache as an LRU eviction cache, because evictions are local and create divergence. Use it for data you intend to keep (sessions, counters, feature flags), and size for the full working set.
• Conflict resolution is per data type and automatic. If LWW on strings is unacceptable for a key, model it as a counter, set, or hash instead.

Topology trade-offs as you add regions:

Active geo-replication vs the Premium passive geo-replica:

Data persistence: RDB, AOF, and durability trade-offs

Enterprise supports both persistence mechanisms, and they answer different questions. Persistence is about surviving a full cluster restart; it is orthogonal to replication, which is about surviving node loss, and to backup, which is about surviving a bad command or logic bug.

RDB (snapshot) writes a point-in-time dump on an interval (e.g., every 1h/6h/12h). Cheap, low overhead, but you lose everything since the last snapshot on a hard failure.

AOF (append-only file) logs every write. With fsync every second (aof-frequency=1s), worst-case data loss is ~1 second. The cost is write amplification and larger files. This is the right default for anything you cannot regenerate.

RDB vs AOF, with the numbers

AOF fsync frequency trade-off

# AOF with per-second fsync -- the resilient default for stateful caches
az redisenterprise database update \
  --cluster-name kv-redis-prod \
  --resource-group rg-data-prod \
  --persistence aof-enabled=true aof-frequency=1s

# RDB hourly -- acceptable only for regenerable caches where restart speed matters
az redisenterprise database update \
  --cluster-name kv-redis-prod \
  --resource-group rg-data-prod \
  --persistence rdb-enabled=true rdb-frequency=1h

The RDB snapshot intervals you can choose, and what each implies:

Persistence vs replication vs backup – three different jobs:

Two correctness notes. First, in an active-active geo group you generally rely on the peer regions for recovery and persistence is a secondary safety net – a surviving region rehydrates a recovered one. Second, persistence is not a backup: it protects against process restart, not against a bad FLUSHALL or a logic bug that corrupts data; a corrupting command replicates to every peer in the mesh. Enterprise persists to the cluster’s local managed disks, not to your storage account, so treat exports separately if you need point-in-time backups.

Private endpoint, VNet injection, and TLS hardening

Never expose a production cache to the public internet. The Enterprise tier supports Private Link, which projects the cache into your VNet via a private endpoint and a private IP – the public FQDN resolves to a private address through Private DNS.

resource cache 'Microsoft.Cache/redisEnterprise@2024-09-01-preview' = {
  name: 'kv-redis-prod'
  location: 'eastus2'
  sku: { name: 'Enterprise_E10', capacity: 2 }
  zones: ['1', '2', '3']
}

resource db 'Microsoft.Cache/redisEnterprise/databases@2024-09-01-preview' = {
  parent: cache
  name: 'default'
  properties: {
    clientProtocol: 'Encrypted'        // TLS-only; rejects plaintext
    clusteringPolicy: 'EnterpriseCluster'
    evictionPolicy: 'NoEviction'
    port: 10000
    persistence: { aofEnabled: true, aofFrequency: '1s' }
  }
}

resource pe 'Microsoft.Network/privateEndpoints@2024-05-01' = {
  name: 'pe-kv-redis-prod'
  location: 'eastus2'
  properties: {
    subnet: { id: dataSubnetId }
    privateLinkServiceConnections: [
      {
        name: 'redis'
        properties: {
          privateLinkServiceId: cache.id
          groupIds: ['redisEnterprise']
        }
      }
    ]
  }
}

Hardening checklist that actually matters:

• clientProtocol: 'Encrypted' forces TLS. The Enterprise tier listens on port 10000 (not 6380 like Premium) – a frequent connection-string bug when migrating. Set your client’s TLS port accordingly.
• Wire a Private DNS zone (privatelink.redisenterprise.cache.azure.net) linked to the VNet so the public FQDN resolves privately. Without the zone link, in-VNet clients still resolve the public IP and the private endpoint does nothing for them.
• Prefer Microsoft Entra ID (token) authentication over the access key where your client supports it; it removes the long-lived shared secret. The access key still exists as a fallback – rotate it and store it in Key Vault, never in app config.

Networking and TLS settings reference

The port-10000 trap and other connection-string mistakes

Identity and secret-handling options, ranked:

Client resilience: multiplexing, retries, and reconnect

This is where most outages are actually caused or prevented. A correctly provisioned cluster behind a broken client is still an outage.

Multiplex one connection, do not pool-per-request. Redis clients like StackExchange.Redis are built around a single long-lived multiplexer that pipelines all commands over a few connections. Opening a connection per operation exhausts ports and ignores the library’s pipelining. Create the multiplexer once as a singleton:

// Singleton ConnectionMultiplexer -- created once, shared process-wide.
var config = new ConfigurationOptions
{
    EndPoints = { "kv-redis-prod.eastus2.redisenterprise.cache.azure.net:10000" },
    Ssl = true,
    AbortOnConnectFail = false,          // keep retrying instead of throwing at startup
    ConnectRetry = 5,
    ConnectTimeout = 15000,
    KeepAlive = 30,
    ReconnectRetryPolicy = new ExponentialRetry(5000)
};
// Token auth (Entra ID) instead of an access key:
await config.ConfigureForAzureWithTokenCredentialAsync(new DefaultAzureCredential());

var muxer = await ConnectionMultiplexer.ConnectAsync(config);

The non-obvious settings that matter on Azure, enumerated:

The non-obvious behaviors:

• AbortOnConnectFail = false is mandatory. The default true throws permanently if the first connect fails (e.g., during a maintenance window), and the multiplexer never recovers. With false, it reconnects in the background.
• During scaling and patching, Azure issues a brief connection blip per node. Your code must retry the operation, not just rely on the multiplexer reconnecting. Wrap commands in a bounded retry (Polly) that handles RedisConnectionException and RedisTimeoutException with jittered backoff.
• Under OSS clustering policy, the client must follow MOVED/ASK redirects automatically – every mainstream cluster client does, but only if you enabled cluster mode. A MOVED reaching your application code means the client is misconfigured.

# redis-py against the Enterprise (proxy) policy -- a single endpoint, TLS, retry on timeout
from redis import Redis
from redis.retry import Retry
from redis.backoff import ExponentialBackoff
from redis.exceptions import ConnectionError, TimeoutError

r = Redis(
    host="kv-redis-prod.eastus2.redisenterprise.cache.azure.net",
    port=10000, ssl=True,
    socket_timeout=5, socket_connect_timeout=5,
    retry=Retry(ExponentialBackoff(cap=2, base=0.1), retries=3),
    retry_on_error=[ConnectionError, TimeoutError],
    health_check_interval=30,
)

health_check_interval sends a periodic PING so idle connections that were silently dropped (by a node move or an Azure load-balancer idle timeout) are detected and rebuilt before a real request hits the dead socket. Without it, the first request after an idle period eats the failure.

Client library cluster support matrix

Retry policy design

Scaling, reshard operations, and zero-downtime maintenance

Enterprise scales two ways: scale up (a bigger SKU – E10 to E20) and scale out (more capacity units, which add shards and rebalance slots). Both are online operations, but “online” assumes a resilient client (previous section).

# Scale up the SKU (more memory/throughput per node)
az redisenterprise update --name kv-redis-prod --resource-group rg-data-prod \
  --sku Enterprise_E20

# Scale out capacity (adds nodes/shards; triggers a reshard/rebalance)
az redisenterprise update --name kv-redis-prod --resource-group rg-data-prod \
  --capacity 4

Scale-up vs scale-out

What happens during a reshard, and how to survive it:

• Hash slots migrate between shards. Under OSS policy, in-flight keys briefly answer ASK/MOVED and the client re-routes – transparent only if the client handles redirects. Under Enterprise policy, the proxy absorbs this and clients see at most brief latency.
• A small number of connections drop as nodes are added. This is exactly the blip your retry policy exists for. Validate by running a scale operation in a load test and confirming zero application errors, only a latency bump.
• Maintenance windows. Enterprise patches the OS and Redis software with rolling, one-node-at-a-time updates so the database stays available. Configure a maintenance window aligned to your low-traffic hours, and never assume “no failover during maintenance” – assume a connection reset per node and make the client idempotent.

Idempotency under retried writes

Caches are naturally idempotent for reads; for write paths, a retried operation after a successful-but-unacknowledged write can corrupt data. Map the operation to a safe pattern:

Events that cause a connection blip

Monitoring memory pressure, evictions, and latency percentiles

Redis fails loudly on CPU and silently on memory. Watch both, and alert on the leading indicators rather than the outage.

The metrics that predict incidents (all available in Azure Monitor for the Enterprise resource):

The metrics to alert on, with the action each alert should trigger:

// Memory pressure trend + eviction correlation over the last 24h
AzureMetrics
| where ResourceProvider == "MICROSOFT.CACHE"
| where ResourceId contains "kv-redis-prod"
| where MetricName in ("usedmemorypercentage", "evictedkeys", "serverLoad")
| summarize avg(Average), max(Maximum) by MetricName, bin(TimeGenerated, 5m)
| order by TimeGenerated desc

For latency, do not trust server-side averages – measure client-side percentiles, because an average of 1ms hides a p99 of 200ms caused by a single hot shard or a GC pause in your own process. Track p50/p99 per operation from the application, and correlate p99 spikes against serverLoad and reshard events. A latency cliff that lines up with a scaling operation is your retry policy working; one that does not is a hot key or a cross-slot fan-out.

What a latency spike is telling you, by what it correlates with:

Architecture at a glance

The diagram traces the data and control path of a two-region active-active deployment, left to right, and maps each failure class to the exact hop where it bites. Read it as four zones. On the left, clients (App Service / AKS pods) hold a singleton multiplexer and reach the cache over TLS on port 10000 through a Private Endpoint – never the public internet. The endpoint resolves via a Private DNS zone linked to the VNet, which is the hop where a missing zone link silently sends you to the public IP. In the middle, the East US 2 Enterprise cluster terminates the connection: under Enterprise policy a proxy fronts the shards; under OSS policy the client talks to shards directly and must follow MOVED/ASK. The shards enforce the eviction policy (NoEviction here) and write AOF at one-second fsync to local managed disk.

On the right, the West Europe Enterprise cluster is the active-active peer: a full-mesh CRDB link replicates every write both directions, so both regions accept reads and writes with no primary and no failover step. The numbered badges mark the five places this design fails and how you confirm each: a CROSSSLOT from a keyspace that ignored hash tags, an OOM where NoEviction met an undersized cluster, a connection storm from a pool-per-request client, replica divergence from running eviction on a geo database, and a stale-read window from replication lag. The legend narrates each as symptom, the metric or command that confirms it, and the fix. The whole method: localize the symptom to a hop, read the cause, run the named check, apply the fix.

Real-world scenario

Paywell, a fictional payments platform, ran a global idempotency cache on Premium with a passive geo-replica: EU writes went to West Europe, a one-way replica fed East US for reads, and failover was a manual DNS swap. The cache held one thing that mattered above all – the answer to “have I already processed this request id?” – and the platform’s entire duplicate-protection guarantee rested on it. Average load was 3,000 ops/sec, the monthly cache spend about ₹95,000, and the team was four engineers.

During a West Europe zone incident the replica was read-only, so for eleven minutes every in-flight payment in the US that needed to check idempotency either blocked on the manual failover or fell back to the database and ran at a fraction of normal throughput. Worse, after failover a handful of duplicate captures slipped through because the idempotency keys written in the US during the gap had not replicated back – the one-way link only flowed EU→US, so US writes during the outage were invisible to West Europe when it recovered. Two customers were double-charged. The post-incident review put it bluntly: passive replication is a DR tool, not an availability tool.

The first instinct was to “add another replica” – which would have solved nothing, because the problem was not node loss, it was that the secondary could not accept writes. The breakthrough was reframing the requirement: a system that must never lose a write across regions has to be active-active and conflict-free by construction. They moved to Enterprise active-active geo-replication across West Europe and East US, modeling idempotency state two ways. The “have I seen this request” check became a CRDT set (SADD seen <id>) so adds in either region converge with observed-remove semantics, and the per-request lock used SET payment:idem:<id> processing NX EX 86400 – string LWW plus NX gives “first writer in either region wins,” which is exactly the duplicate-protection semantic they needed.

They kept AOF at 1s as a restart safety net and sized for NoEviction: idempotency keys carry a 24-hour TTL via SET ... EX, never LRU eviction, so divergence is impossible. They moved both databases behind Private Endpoints with a linked Private DNS zone, switched clients to Entra token auth, and rebuilt the .NET client as a singleton multiplexer with AbortOnConnectFail = false and a Polly retry. The one subtlety they hit in testing: an early version used INCR for a per-merchant attempt counter and double-counted on a retried-after-timeout write; they switched the critical counter to an idempotency-keyed SET of a computed value.

# Idempotency check, region-local, on an active-active CRDB.
# SET NX EX is the primitive: succeeds only if the key is new, with a TTL.
# Converges across regions because string LWW + NX gives "first writer in either region wins".
SET payment:idem:7f3c-9a21 processing NX EX 86400
# -> OK    (first time, in either region: proceed)
# -> nil   (already seen anywhere in the mesh: this is a duplicate, reject)

The measurable result: the next regional zone failure was a non-event – no manual step, p99 unchanged at 1.1 ms, zero duplicate captures. Monthly spend rose to about ₹1,40,000 for the two Enterprise clusters, which the team judged trivial against a single double-capture chargeback plus reputational cost. The lesson on the wall: the duplicate-capture class of bug was designed out by construction, not monitored for.

The incident as a timeline, because the order of moves is the lesson:

Advantages and disadvantages

The Enterprise active-active model both removes the regional-write failure class and adds operational discipline you must respect. Weigh it honestly:

The model is right when you genuinely need multi-region writes, conflict-free convergence, or Redis modules, and you can size for the full working set under NoEviction. It is the wrong tool for a cheap, regenerable, single-region read cache – that is what Standard or Premium are for. The disadvantages are all manageable, but only if you respect them: model the data type deliberately, size for the working set, lock down the network, and make the client resilient.

Hands-on lab

Stand up an Enterprise cache, prove the clustering and persistence behavior, and watch active-active counter convergence – then tear it down. Enterprise is not free-tier, so this lab uses the smallest Enterprise SKU and deletes everything at the end; budget a small hourly charge while it runs. Run in Cloud Shell (Bash).

Step 1 — Variables and resource group.

RG=rg-redis-lab
LOC=eastus2
CLUSTER=kv-redis-lab-$RANDOM   # cluster name must be unique
az group create -n $RG -l $LOC -o table

Step 2 — Create the smallest Enterprise cluster (zone-redundant).

az redisenterprise create \
  -n $CLUSTER -g $RG -l $LOC \
  --sku Enterprise_E5 --capacity 2 --zones 1 2 3 -o table

Expected: a cluster row with provisioningState: Succeeded (this takes several minutes).

Step 3 — Create the database with Enterprise (proxy) policy, NoEviction, AOF.

az redisenterprise database create \
  --cluster-name $CLUSTER -g $RG \
  --client-protocol Encrypted \
  --clustering-policy EnterpriseCluster \
  --eviction-policy NoEviction \
  --persistence aof-enabled=true aof-frequency=1s -o table

Step 4 — Get the host and access key, then connect with TLS on port 10000.

HOST=$(az redisenterprise show -n $CLUSTER -g $RG --query hostName -o tsv)
KEY=$(az redisenterprise database list-keys --cluster-name $CLUSTER -g $RG \
  --query primaryKey -o tsv)

redis-cli -h $HOST -p 10000 --tls -a "$KEY" PING
# -> PONG

Step 5 — Prove persistence is on and eviction is off.

redis-cli -h $HOST -p 10000 --tls -a "$KEY" CONFIG GET appendonly   # -> appendonly yes
redis-cli -h $HOST -p 10000 --tls -a "$KEY" CONFIG GET maxmemory-policy  # -> noeviction

Step 6 — Prove the slot contract (under Enterprise policy the proxy fans out simple MSET, but transactions still need one slot). Demonstrate hash-tag co-location:

# Co-located keys via hash tag -- guaranteed one slot, safe for MULTI/EXEC and Lua
redis-cli -h $HOST -p 10000 --tls -a "$KEY" MSET 'u:{t1}:a' 1 'u:{t1}:b' 2
redis-cli -h $HOST -p 10000 --tls -a "$KEY" MGET 'u:{t1}:a' 'u:{t1}:b'   # -> 1, 2

Step 7 — Counter additive behavior (single region here; in active-active this is what converges).

redis-cli -h $HOST -p 10000 --tls -a "$KEY" INCR global:signups
redis-cli -h $HOST -p 10000 --tls -a "$KEY" INCR global:signups
redis-cli -h $HOST -p 10000 --tls -a "$KEY" GET  global:signups   # -> 2

Step 8 — Teardown (stop the meter).

az group delete -n $RG --yes --no-wait

What each lab step proves, mapped to a section above:

To make this a real active-active test, repeat steps 2–4 in a second region with a shared --group-nickname and mutual --linked-databases, then INCR global:signups once in each region and GET from either – the value converges to the sum, not a lost update. That is the closest thing to a real cross-region failover you can run on demand.

Common mistakes & troubleshooting

Eleven real failure modes, each as symptom → root cause → how to confirm → fix. This is the playbook to keep open at 02:14.

Decision table: which failure am I looking at?

The error/limit reference

Best practices

Production-grade rules, learned the hard way:

Security notes

The cache often holds the most sensitive transient data you have – session tokens, idempotency keys, PII in flight. Lock it down on every axis:

Least-privilege auth options, ranked from most to least secure:

Cost & sizing

Enterprise costs more than Standard/Premium because you are paying for a commercial runtime, and active-active doubles the footprint because both regions run full clusters. The bill is driven by SKU (RAM/CPU per node), capacity (number of nodes/shards), the number of regions in the mesh, and cross-region egress for replication.

What drives the bill, and how to control each lever:

Right-sizing approach:

Rough figures (list-price ballpark, varies by region and commitment – always confirm with the pricing calculator):

There is no free tier for Enterprise. For dev and learning, use a Standard C0/C1 (which is cheap) to practice client patterns, and reserve Enterprise spend for the features that require it (modules, active-active). Commit to a reservation once steady-state size is known to cut the hourly rate.

Interview & exam questions

Maps to the AZ-204 (developing solutions), AZ-305 (designing infrastructure), and Redis-specific knowledge expected of senior Azure roles.

1. Why is Azure Cache for Redis Enterprise a different ARM resource type from Premium, and why does it matter? Enterprise/Enterprise Flash use Microsoft.Cache/redisEnterprise (a parent cluster + child database), running the commercial Redis Enterprise runtime, while Basic/Standard/Premium use Microsoft.Cache/redis running OSS Redis. It matters because IaC modules, endpoints, ports (10000 vs 6380), and capabilities (modules, active-active) differ; a Bicep/Terraform module for one type does nothing for the other.

2. Contrast OSS and Enterprise clustering policies. OSS exposes the native Redis Cluster API – the client discovers shards, computes CRC16 hash slots, and connects directly (lowest latency, needs a cluster-aware client and routable node IPs). Enterprise puts a proxy in front so the client uses one endpoint like a standalone Redis (simplest networking, any client, one extra hop). The policy is chosen at creation and is permanent.

3. What is a CROSSSLOT error and how do you prevent it? A multi-key command (or MULTI/EXEC/Lua) whose keys hash to different slots. Prevent it by co-locating keys with a hash tag – the {...} substring is what’s hashed – so all co-accessed keys (e.g. t:{tenant}:...) land in one slot. Even the Enterprise proxy requires single-slot keys for transactions and Lua.

4. Difference between Premium passive geo-replica and Enterprise active geo-replication? Passive is a one-way link to a read-mostly secondary with manual failover – a DR tool. Active-active is a full-mesh CRDB where every region accepts reads and writes with automatic CRDT conflict resolution and no failover step – an availability tool. If both regions must accept writes, you need Enterprise active-active.

5. How do CRDTs resolve concurrent writes, and why can’t you use a string counter in active-active? Each data type is reimplemented as a CRDT: strings are last-write-wins, counters are additive (both increments apply), sets/hashes merge per element/field. A string SET n for a count is LWW, so concurrent increments in two regions lose updates; use INCR (additive) instead.

6. Why model an idempotency check as a CRDT set, and how does SET NX EX behave across regions? A set’s observed-remove semantics make concurrent adds of the same id converge cleanly. SET key val NX EX succeeds only if the key is new; with string LWW across regions, it gives “first writer in either region wins,” which is exactly the duplicate-protection guarantee for idempotency.

7. RDB vs AOF – when do you choose each, and what’s the data-loss window? RDB snapshots periodically (loss up to the interval, e.g. 1h) – cheap, fast restart, good for regenerable caches. AOF logs every write; at 1s fsync, worst-case loss is ~1 second – the default for data you can’t recreate. Both protect against restart, not against bad commands.

8. Why is NoEviction mandatory for an active-active geo cache? Evictions are local to each region, so an LRU eviction in one region but not another silently diverges the dataset – a correctness bug, not just a hit-rate dip. Run geo databases as NoEviction and size for the full working set.

9. Name three client configurations that prevent Redis outages on Azure. A singleton multiplexer (not pool-per-request) to avoid connection storms; AbortOnConnectFail = false so the client recovers after a maintenance blip instead of throwing permanently; and an operation-level retry with jittered backoff so reshard/patch blips don’t surface as errors. Add periodic health checks to detect dead idle sockets.

10. What happens during a scale-out reshard, and how do you make it invisible? Hash slots migrate between shards; under OSS policy keys briefly answer ASK/MOVED and a cluster-aware client re-routes, while the Enterprise proxy absorbs it. Make it invisible with a resilient client (redirect-following, AbortOnConnectFail=false, op retries) and validate with a load test through a live scale-out expecting zero errors and only a p99 bump.

11. Why measure client-side latency percentiles instead of server averages? A server-side average of 1 ms hides a p99 of 200 ms from a hot shard, a cross-slot fan-out, or a client GC pause. Track p50/p99 per operation from the app and correlate spikes with serverLoad and reshard events to tell “retry working” from “real hot key.”

12. How do you secure an Enterprise cache end to end? TLS-only (clientProtocol: Encrypted) on port 10000; Private Endpoint with public access disabled and a linked Private DNS zone; Entra token auth via managed identity (key in Key Vault, rotated, as fallback); short TTLs on sensitive keys; diagnostic logs to Log Analytics; and compliance-aware region choice since writes replicate to every mesh peer.

Quick check

1. Which ARM resource type backs the Enterprise tier, and what is the default TLS port?
2. You need both East US and West Europe to accept writes to the same key with no failover step. Which tier and replication mode?
3. A MULTI/EXEC transaction throws CROSSSLOT. What single keyspace change fixes it?
4. Your active-active counter is losing increments across regions. What’s the likely modeling mistake?
5. Your app throws permanently the first time a maintenance window blips the connection. Which one client setting fixes it?

Answers

1. Microsoft.Cache/redisEnterprise (a parent cluster + child database), and the default TLS port is 10000 (Premium uses 6380).
2. Enterprise (or Enterprise Flash) with active-active geo-replication (a CRDB) – both regions accept reads and writes with automatic CRDT conflict resolution and no failover.
3. Add a hash tag so every key in the transaction shares the {...} substring (e.g. k:{order123}:...), forcing them into one hash slot.
4. The counter is modeled as a string SET (last-write-wins), so concurrent writes lose updates. Use INCR (additive CRDT) instead.
5. Set AbortOnConnectFail = false so the multiplexer reconnects in the background instead of throwing permanently after the first failed connect.

Glossary

Next steps

• Azure Private Endpoints and Private DNS at Scale — lock the cache into your VNet correctly so in-VNet clients resolve the private IP.
• Azure Key Vault: Secret Rotation with Managed Identity — store and rotate the access key, or move to managed-identity token auth.
• Cosmos DB Multi-Region Writes and Conflict Resolution — contrast Redis automatic CRDT merge with Cosmos’s pluggable conflict policies.
• Azure Multi-Region Active-Active Disaster Recovery — fit the active-active cache into a full multi-region architecture.
• Azure Monitor Deep Dive: Every Option — build the alerts on used-memory %, eviction, server load, and client-side p99.