Cruxly › System design › Databases by Name: Internals and Terms

Redis in depth: cluster slots, eviction, persistence and the lock debate

an in-memory structure server with a durability story that is mostly about which seconds you can lose

Redis is easy to start and hard to run well, because every default is a decision about what you are willing to lose: a write, a second of writes, a key, or the whole dataset. This lesson covers the mechanisms behind those decisions: hash slots and failover in Redis Cluster, the eviction policy that defines what "full" means, RDB and AOF persistence, the replication backlog, and why a Redis lock is fine for efficiency and risky for correctness.

Hash slots: how Cluster shards keys

Redis Cluster does not hash keys onto nodes. It hashes keys onto 16384 slots, and nodes own slots. The Redis cluster specification gives the formula as HASH_SLOT = CRC16(key) mod 16384. Every master serves a subset of the slots, so adding or removing capacity means moving slots between nodes, never rehashing the whole key space. That is the same idea as the consistent-hashing ring in Scaling Postgres and MySQL before you leave them, with a fixed number of buckets instead of a continuous ring.

Why 16384. The spec states the consequence rather than the reason: the key space is split into 16384 slots, which sets a hard upper limit of 16384 masters, though the suggested maximum cluster size is on the order of 1,000 nodes. It also says every heartbeat packet carries a bitmap of the slots served by the sender (or, for a replica, by its master). A bitmap of 16384 slots is 16384 / 8 = 2,048 bytes. That is our arithmetic, not a number from the spec, but it shows why the slot count matters: it is paid on every ping and pong.

A key you can check by hand. With the CRC16 that the Cluster spec ships as its reference implementation, the key foo maps to slot 12182 and user:1000 maps to slot 1649 (worked arithmetic: CRC16 of the key, modulo 16384).

Hash tags. If a key contains {...}, only the text between the first { and the first } after it is hashed, provided that text is not empty. The spec's own cases make the rules concrete:

Multi-key commands, transactions and Lua scripts only work when every key is in one slot, so hash tags are how you keep related keys together. The cost is the usual one for a partition key: if every key of a busy tenant carries the same tag, that tenant lives in one slot on one master. Slots are the unit of both placement and load, so a hot tag is a hot node that adding nodes cannot fix.

MOVED versus ASK. Both are redirections a client must understand:

• MOVED says the slot is now served by another node. The client should update its cached slot map and retry there.
• ASK says the slot is being migrated and this specific key may already be on the target. The client sends ASKING followed by the command to the target, once, and does not update its map. The spec describes ASKING as setting a one-time flag on the connection.

The cluster bus. Nodes gossip over a second TCP port, the data port plus 10000 by default (or whatever cluster-port sets). A firewall that allows 6379 but blocks 16379 breaks failure detection and config propagation while ordinary reads and writes keep working, which is a nasty half-failure.

What experts say in the room. "Cluster is sharding with a fixed bucket count; the question is whether our access pattern can live with one slot per transaction." The number is 16384; the failure is a single hash tag turning one master into the bottleneck.

What a senior checks. That all 16384 slots are covered, that the cluster-bus port is open between every pair of nodes, that the client library handles both MOVED and ASK, and the rate of MOVED errors during normal operation (a steady stream means a stale or broken slot map). They also look for any key prefix with a hash tag that concentrates traffic.

:::key Verified October 2026 Numbers above are from the Redis cluster specification (docs "latest", read 2026-10-29): 16384 slots, suggested maximum around 1,000 nodes, bus port equals data port plus 10000. :::

Failure detection, failover and the write you can lose

Redis Cluster is available and partition-tolerant on the majority side, and it is not a CP system. The cluster specification states the design goal in one sentence: it tries, best effort, to keep writes from clients connected to the majority of masters, and it says plainly that there are small windows where acknowledged writes can be lost, with larger windows when the client sits on a minority side of a partition.

How a node is declared dead. Each node pings others over the bus. When a node has not answered within NODE_TIMEOUT (the cluster-node-timeout setting), the asking node marks it PFAIL, possible failure, a purely local opinion. PFAIL becomes FAIL when a majority of masters have reported the node as PFAIL or FAIL within a window of two times NODE_TIMEOUT (the spec says the validity factor is 2 in the current implementation). FAIL is the trigger that lets a replica ask for promotion.

How a replica wins. After FAIL, a replica waits a short delay so the state can spread, then asks every master for a vote. It needs acks from a majority of masters. The spec says it waits up to two times NODE_TIMEOUT for replies, and never less than 2 seconds, and a failed election is retried after four times NODE_TIMEOUT (never less than 4 seconds). The majority side of a partition becomes available again after NODE_TIMEOUT plus a few more seconds for election and promotion. Each win gets a new, higher configEpoch, which is how the cluster decides which version of a slot's ownership is the latest.

Why acknowledged writes disappear. Replication between a master and its replicas is asynchronous, and the merge rule is, in the spec's words, last failover wins: the dataset of the last elected master eventually replaces all others. Two failure modes follow from the spec:

1. The master acknowledges a client, then dies before the write reaches any replica. A replica is promoted. The write is gone. The spec notes this is hard to observe on a sudden crash because the master tries to answer the client and feed replicas at about the same moment, but it is possible.
2. A master is cut off, failed over, and a client with an outdated routing table writes to it before it learns it was demoted. The spec calls this unlikely, because a master that cannot reach a majority stops accepting writes after a while.

The partition case is much bigger. The spec says a master has to be unreachable by the majority for at least NODE_TIMEOUT before it is failed over. If the partition heals before then, no writes are lost. If it lasts longer, every write the minority-side masters accepted up to that point may be lost, because the majority side has promoted replicas. The minority side starts refusing writes after NODE_TIMEOUT, which bounds the damage but does not remove it.

WAIT narrows the window, it does not close it. The replication docs say WAIT blocks until a number of replicas have acknowledged, but it does not turn Redis into a CP system with strong consistency: acknowledged writes can still be lost in a failover, depending on persistence settings. It reduces the probability of loss to hard-to-trigger failure modes.

The cost of small timeouts. Every node pings any node it has not heard from for half of NODE_TIMEOUT. The spec's example: 100 nodes with a 60 second timeout means each node sends 99 pings every 30 seconds, 3.3 pings per second per node, 330 per second cluster-wide. A shorter timeout detects failure sooner and multiplies that traffic, and it also raises false positives during GC pauses, snapshots and network blips.

What experts say in the room. "Cluster gives us partition-tolerant sharding with asynchronous replication, so the contract is: we can lose the tail of writes at a failover, and a longer partition widens it. If we cannot live with that, Redis is a cache or a derived store here, not the system of record." The numbers are NODE_TIMEOUT and the 330-pings-per-second example; the failure is an order or ledger entry that was acknowledged and then not there. This is the Redis as database versus cache argument in its sharpest form, and it connects to the consistency discussion in Consistency models, for the person choosing.

What a senior checks. The cluster state and slot coverage on every node, replica count per master (at least one reachable replica per master is the survival condition in the spec), the configured NODE_TIMEOUT relative to the longest expected pause on a node, and whether any workload on this cluster assumes an acknowledged write is durable.

:::warn Do not call WAIT a durability guarantee The Redis replication docs explicitly say WAIT does not make Redis strongly consistent. If you need to claim a write survives failover, say "bounded loss", not "no loss". :::

maxmemory and eviction: the policy decides what 'full' means

An unconfigured Redis grows until the operating system kills it. In the Redis key-eviction docs, maxmemory defaults to 0 on 64-bit systems, meaning no limit (32-bit systems have an implicit 3 GB limit). With no limit, your eviction policy is the kernel's out-of-memory killer, which chooses a victim you did not pick.

What happens at the limit. Whenever a client command adds data, Redis checks memory. If usage is over maxmemory, it evicts keys under maxmemory-policy until it is back under, or refuses the command. The docs warn that a single command that adds a lot of data, such as a large set intersection stored into a new key, can overshoot the limit by a large amount.

Replication and AOF buffers are not counted. The docs say output buffers to replicas and the AOF buffer are excluded from the number compared with maxmemory, because evictions generate updates for those buffers and counting them would cause a feedback loop of needless evictions. INFO memory shows the amount as mem_not_counted_for_evict; leave RAM free for it. Illustration, ours: maxmemory 8gb on a 10 GB host leaves 2 GB for buffers, fork copy-on-write and the OS.

The policies (docs, latest):

The classic outage. A team picks volatile-lru because it sounds safe, but never sets TTLs on cache keys. The docs state that volatile policies behave like noeviction when no key has an expiry. When memory fills, every write that adds data starts failing while reads still work, so the symptom is "the site works but nothing new saves". The docs add that volatile-* policies are mainly for one instance that mixes cache and persistent keys, and they suggest two separate instances where possible. Also, a TTL costs memory per key, so allkeys-lru is more memory efficient.

LFU and its two knobs. LFU uses a Morris counter, a probabilistic counter that fits in a few bits, with decay so old popularity fades. The docs give the defaults: lfu-log-factor 10 and lfu-decay-time 1 (minutes; 0 means never decay). By default the counter saturates at around one million requests. The docs' table of the 0-255 counter shows the effect of the factor:

A low factor separates keys with few hits well and saturates early; a high factor keeps distinguishing very hot keys. LFU helps when a scan touches many keys once, because frequency, unlike recency, does not let a one-time scan wipe the hot set.

LRU is approximate, and the sample size is a knob. Redis does not keep an exact LRU list (it costs memory). It samples a few random keys and evicts the one idle longest, helped since Redis 3.0 by a pool of good candidates. maxmemory-samples defaults to 5; the docs say raising it to 10 approximates true LRU closely at the price of some CPU, and you can try it live with CONFIG SET maxmemory-samples 10. The same setting governs LRM, which since Redis 8.6 stamps a key only on writes, so reads do not keep a key alive: useful when the data being read is not the data being refreshed.

Reading the hit ratio. Compute keyspace_hits / (keyspace_hits + keyspace_misses) from INFO stats. The docs add a trap: EXISTS on an absent key counts as a miss. If the ratio is lower than the working set predicts, a high evicted_keys rate suggests the policy is evicting the wrong keys; a low evicted_keys with a high expired_keys suggests TTLs that are too short.

What experts say in the room. "The eviction policy is a product decision: what may the system forget first?" The numbers are samples 5 versus 10 and log-factor 10. The failure is volatile-lru with no TTLs. This ties to the cache rung in Scaling Postgres and MySQL before you leave them: a cache with noeviction is not a cache.

What a senior checks. used_memory against maxmemory and mem_not_counted_for_evict, the evicted_keys rate, the hit ratio against what the working set predicts, current_eviction_exceeded_time, and rejected commands in commandstats when the policy is noeviction or volatile-*. They also confirm replicas have more RAM than the master's maxmemory (see the replication section).

:::key Verified October 2026 Defaults above come from the Redis key-eviction docs (latest, which reference redis.conf 7.4). allkeys-lrm needs Redis 8.6 or later. Check CONFIG GET maxmemory* on your own version. :::

Persistence: RDB, AOF and the fork

Redis persistence is a choice of how many seconds of writes you can afford to lose. The persistence docs describe four modes: RDB snapshots, an append-only file (AOF), both, or none. Compare this with the commit path in Storage engines: B-trees, LSM-trees, row vs column: Redis applies the write in memory first, and the log is a copy.

RDB: snapshot by fork. Redis forks a child process, the child writes the dataset to a temporary file, and on completion the file replaces the old dump.rdb atomically. The parent keeps serving because the fork uses copy-on-write memory. A save point such as save 60 1000 means "every 60 seconds if at least 1,000 keys changed". Strengths from the docs: one compact file that is good for backups and disaster recovery, and faster restarts on large datasets than replaying an AOF. The weakness is loss: snapshots usually happen every five minutes or more, so a crash without a clean shutdown loses the latest minutes. The other cost is the fork itself. The docs say fork() can take time on a big dataset and may stop Redis serving clients for some milliseconds, or even a second when the dataset is large and the CPU is weak. Watch latest_fork_usec in INFO.

AOF: log every write. With appendonly yes, every write command is appended and replayed on restart. The appendfsync setting is the durability dial:

Even always is a group commit: one write and one fsync cover a batch, and that is the same trick described for the WAL in Storage engines: B-trees, LSM-trees, row vs column.

AOF rewrite. The log grows: incrementing a counter 100 times leaves one live value and 100 log entries. A rewrite builds the shortest command sequence that recreates the dataset, in a forked child. Before Redis 7.0 the parent buffered new writes in memory during the rewrite, wrote them twice, and could freeze while appending the buffer at the end. Since 7.0 the AOF is multi-part: at most one base file plus incremental files in a directory tracked by a manifest. The parent opens a new incremental file when the rewrite starts, so no big in-memory buffer is needed, and the switch is an atomic manifest swap. If a rewrite fails and retries, the retries slow down so repeated failures do not create many incremental files.

What the docs recommend. Use both RDB and AOF if you want data safety comparable to what PostgreSQL can provide, RDB alone if a few minutes of loss in a disaster is acceptable, and the docs discourage AOF alone because an RDB snapshot is good for backups, faster restarts, and as protection against bugs in the AOF engine. When both are enabled, a restart uses the AOF because it is the most complete.

Do not run replicated Redis with persistence off and auto-restart. The replication docs describe the failure: node A is master with persistence off, B and C replicate from it. A crashes and an init system restarts it with an empty dataset. B and C then sync from the empty master and erase their own copies. The docs say that with Sentinel the master can even restart fast enough that no failure is detected. If persistence has to be off, disable automatic restart.

What experts say in the room. "Pick the seconds you can afford to lose, then prove restore works." The numbers: a 1 second everysec window and a fork stall that the docs bound at up to about a second on very large datasets with weak CPUs. The failure: the empty-master wipe.

What a senior checks. latest_fork_usec, aof_last_bgrewrite_status, free disk for a rewrite alongside the live files, whether persistence is on for replicas as well as the master, a tested restore from an RDB copy off the host, and that no supervisor restarts a persistence-off master.

:::key Verified October 2026 Defaults and behaviour from the Redis persistence and replication docs (latest). Multi-part AOF requires Redis 7.0 or later; Redis 8.10 adds a BACKUP command family (START, SEAL, LIST, CLEANUP) for online backups built on the multi-part AOF format; nothing else here depends on it. :::

Replication, the backlog and the resync storm

Redis replication is asynchronous, offset-based and driven by a backlog buffer. The replication docs call it leader-follower: the master streams the effects of writes, expiries and evictions to replicas, and replicas acknowledge how much they have processed. It is non-blocking on the master side.

Replication ID and offset. Each master has a replication ID, a long random string that names one history of the dataset, and an offset that grows with every byte of replication stream produced, even if no replica is connected. The pair identifies an exact version of the data. A replica reconnects with PSYNC, sending the ID and offset it last processed. If the master still holds that stretch of the stream in its backlog buffer, it sends only the missing part (a partial resync). If the replica is too far behind, or refers to an unknown history, the master does a full resync: it forks, produces an RDB, buffers new writes meanwhile, transfers the file, and then streams the buffer. Several replicas asking at once share a single background save.

The resync storm. The backlog size is a setting (the docs refer to the example redis.conf for the sizes). If the backlog is smaller than the write volume during a network blip, every replica that was disconnected for that long needs a full resync. Each full resync costs the master a fork and a snapshot transfer, and the replica must discard its old dataset and load the new one in its main thread, blocking incoming connections for what the docs call a brief window that can be many seconds for very large datasets. Illustration, ours: at 5 MB/s of replication stream, a 60 second disconnect needs 5 x 60 = 300 MB of backlog to resume partially. Measure your own stream rate rather than copy this figure. Diskless replication (repl-diskless-sync, with repl-diskless-sync-delay to wait for more replicas to join) sends the RDB straight over the socket, which helps when the disk is slow.

Replica behaviour that surprises people.

• Replicas do not expire keys. The master expires or evicts a key and sends a DEL. Because the DEL can arrive late, the replica uses its own clock to hide logically expired keys from reads. That is why key counts can differ between nodes, and why Lua scripts see time frozen while they run.
• Replicas ignore maxmemory by default (replica-ignore-maxmemory yes): eviction happens on the master and arrives as DELs. The docs warn that a replica may use more memory than maxmemory because some buffers and structures are larger there, so give replicas more RAM than the master's limit and monitor them.
• Writable replicas exist for historical reasons; the docs recommend against them because propagated commands may behave differently on a diverged replica.

Bounding loss: min-replicas-to-write. The master can refuse writes unless at least N replicas are connected with lag no more than M seconds (min-replicas-to-write, min-replicas-max-lag). Replicas ping the master every second, so it knows each lag. The docs describe this as best effort: it does not guarantee a given write reached a replica, but it limits the time window of loss. They say bounded loss is better than unbounded loss.

What experts say in the room. "Backlog sized by write rate times the longest disconnect we expect to ride out, otherwise every network blip becomes a fork on the master." The failure: a full-resync storm on a large master, which is also when memory spikes from copy-on-write.

What a senior checks. each replica's offset against the master's (ROLE and INFO replication show them), the number of full resyncs, backlog size against peak write rate, replica RAM headroom above the master's maxmemory, and whether failover tooling keeps replicas able to resync partially after a promotion.

:::warn Fork plus full resync equals a memory spike Copy-on-write means pages the parent modifies during the snapshot are duplicated. A busy master that forks for a resync needs headroom for that, and that headroom is exactly what maxmemory does not count. Plan RAM for the fork, not just for the dataset. :::

Redis as a queue under memory pressure: Slack's job queue

A Redis queue has a failure mode a disk queue does not: it can become impossible to drain because draining needs memory. Slack Engineering's post "Scaling Slack's Job Queue" (2017-12-06) describes this exactly.

The setup. Slack's job queue ran on Redis. The web app hashed a job's identifier and logical queue to pick a Redis host, workers polled Redis and moved each job from a pending queue to an in-flight list before running it, failed jobs went to a retry queue and then a permanently-failed list that engineers inspected by hand. The post gives the scale as more than 1.4 billion jobs on the busiest days, at up to 33,000 jobs per second at peak. For scale, our arithmetic: 1.4 billion over 86,400 seconds averages about 16,200 per second, so the peak is about twice that average.

The incident. The post dates it only as about a year before the article. A database contention problem slowed job execution. Jobs piled up in Redis until it reached its configured memory limit, so new jobs could not be enqueued and Slack features that depended on them failed. Dequeuing needs a little free memory too, because a job is moved to the in-flight list before it runs. So even after the database recovered, the queue stayed locked and needed extensive manual recovery.

Why this is structural. The post lists the underlying problems: little memory headroom, so a sustained enqueue rate above the dequeue rate exhausts memory; every client connecting to every Redis instance; workers that could not scale independently of Redis, since each added worker added polling load; dequeue work growing with queue length; and unclear delivery and deduplication semantics.

What Slack changed. It kept Redis as the job store and put Kafka in front of it as a durable buffer, a minimum change that preserved the existing interfaces. Two stateless Go services did the work:

• Kafkagate takes an HTTP POST with a topic, partition and content and writes to Kafka synchronously. It waits only for the partition leader's acknowledgement, which the post says favours availability and latency over consistency, at the risk of losing a job if a broker dies before replicating.
• JQRelay reads a topic and writes to the matching Redis cluster, advancing the Kafka offset only after a successful Redis write. If Redis is down it retries indefinitely, and job-specific errors are re-enqueued to Kafka so they do not block the queue. Rate limits are applied when writing to Redis. A Consul lock gives each topic exactly one relay.

Rollout and result. The Kafka cluster in the post (Kafka 0.10.1.2) had 16 brokers, 32 partitions per topic, replication factor 3 and 2-day retention. The rollout double-wrote to Redis and Kafka with JQRelay in shadow mode, compared job counts at each hop, and sent canaries to every partition each minute across 50 Redis clusters and 1,600 Kafka partitions (50 topics x 32, which matches). Kafka now absorbs buildup, so the web app keeps enqueuing while the relay's rate limit matches what Redis can dequeue.

The architect's point. Slack did not conclude "never use Redis for queues". It concluded that a bounded-memory store needs a buffer in front of it, so backpressure turns into queue depth on disk instead of a failed enqueue. In interview terms: a queue's capacity limit is a design parameter, and "what happens when it is full" should have a better answer than "nothing can move".

What experts say in the room. "If the consumer needs memory to make progress, memory pressure is a deadlock, not a slowdown." The numbers: 1.4 billion jobs a day, 33,000 a second peak. The failure: a full Redis that cannot dequeue.

What a senior checks. Headroom to maxmemory at the peak enqueue-minus-dequeue rate, how long the queue can grow before that is hit (headroom divided by the net inflow rate), what happens at the limit (noeviction fails enqueues, any evicting policy silently drops jobs), a durable buffer in front for work you cannot lose, and a drain runbook rehearsed before it is needed.

:::key Verified October 2026 Figures are from Slack Engineering's 2017 post as read on 2026-10-29. They describe Slack's system in 2017 and say nothing about its current architecture. :::

Redis as a lock service: the Redlock debate

A lock with an expiry is a lease, and a lease is only safe if the holder knows when it ended. The design question behind the long Redlock argument is whether Redis can provide mutual exclusion that the protected resource can rely on. Martin Kleppmann's 2016 post "How to do distributed locking" is the standard statement of the sceptical side. Below we describe his arguments as his. Antirez published a reply defending Redlock; read both.

The single-instance lock. SET key token NX PX ttl sets the key only if absent and with an expiry; the token is random, and release deletes the key only if the token matches. This is fine when the lock merely prevents duplicate work.

Redlock. The algorithm tries to take the lock on a majority of N independent Redis nodes within a validity time (Kleppmann's critique discusses a five-node setup). Its aim is to survive the loss of a minority of nodes.

Two reasons to take a lock. Kleppmann separates them:

• Efficiency: avoid doing work twice. A failure costs a little money or an extra email. A single Redis instance is enough, documented as approximate, and Redlock's five nodes are overkill.
• Correctness: two holders corrupt data. This needs a stronger guarantee.

Why a lease is not enough for correctness. His example: a client takes the lock, then pauses for longer than the lease, for example in a stop-the-world GC pause, a page fault or CPU starvation. The lease expires, a second client takes the lock and writes. The first client wakes up, still believing it holds the lock, and writes stale data. Network delay produces the same result when a request sits in the network past the expiry. No lock service can prevent this on its own, because the lock holder cannot know in the instant of writing that its lease is still valid.

Fencing tokens. The fix he proposes is a number that increases every time the lock is granted, passed with every write. The storage service remembers the highest token seen and rejects older ones. In his diagram, client 1 holds token 33, pauses and expires; client 2 gets 34 and writes; client 1 resumes and writes with 33, and the storage rejects it. He argues Redlock has no way to generate such tokens, and doubts a simple fix exists, because a counter across several nodes would need a consensus algorithm.

Timing assumptions. His second critique: Redis expires keys with wall-clock time rather than a monotonic clock, so a forward clock jump can expire a lock early, letting two clients each hold a majority. More generally, Redlock is safe only in a synchronous system model with bounded network delay, bounded pauses and bounded clock error. Real data centres meet those assumptions most of the time, he argues, which is not good enough for correctness. Consensus systems such as Raft, Paxos and ZooKeeper's Zab do not use timing for safety.

His conclusion (Kleppmann's position, not a consensus of all experts): for best-effort locks, use a single Redis node; for correctness, use a consensus-based system such as ZooKeeper, or a database with solid transactions, together with fencing tokens on every access to the protected resource. In practice, the second half often means putting the check in the resource itself: a conditional write such as "update only if version is still N" is a fencing token you already have, as in Isolation levels, in depth.

What experts say in the room. "Is this lock for efficiency or for correctness? If correctness, does the resource check a token?" Three failure sources recur in his argument: process pauses, delayed packets, and clock jumps. The failure: two workers both believe they own the lock and both write.

What a senior checks. What the lock protects and what the cost of a double hold is; whether the protected store can reject stale holders (a version column, a monotonic token, a conditional write); the lease length against the worst observed GC or pause; and whether renewal exists and fails safe.

:::warn Contested ground This debate is a published disagreement between two practitioners. Present it as such in an interview: state the efficiency versus correctness distinction, give Kleppmann's argument, and say you would check the resource side before choosing. :::

How to say it in the interview

"Redis Cluster splits the key space into 16384 hash slots and replicates each master asynchronously, so it is available and partitioned but not strongly consistent: an acknowledged write can vanish after a failover, and WAIT only narrows that window. The default AOF policy is everysec, so I accept losing about one second, and maxmemory defaults to 0, so I always set it and pick the eviction policy on purpose. For locks I would use Redis to avoid duplicate work, and use a consensus system plus fencing tokens when correctness depends on the lock."

Checklist

• Stated that Redis Cluster uses 16384 hash slots and asynchronous replication, and that acknowledged writes can be lost at failover
• Confirmed all slots are covered, the bus port (data port + 10000) is open between all nodes, and the client handles MOVED and ASK
• Checked for hash tags that concentrate traffic into one slot
• Set maxmemory on purpose and left headroom for replica and AOF buffers (mem_not_counted_for_evict) and fork copy-on-write
• Matched the eviction policy to the role: allkeys-lru or lfu for a cache, noeviction plus alerting for a store
• Chose RDB, AOF or both from the seconds of loss I can accept, and watched latest_fork_usec and AOF rewrite status
• Verified persistence is on for replicas, or that a persistence-off master cannot auto-restart
• Sized the replication backlog to write rate times the longest disconnect, and tracked full-resync counts
• Gave replicas more RAM than the master's maxmemory and monitored them
• Decided whether each lock is for efficiency or correctness, and required a fencing token or conditional write when it is for correctness

Related lessons

• PostgreSQL internals: MVCC cost, VACUUM, wraparound and WAL
• MySQL and InnoDB: locks, buffer pool, replication and Vitess
• Cassandra and ScyllaDB: the ring, repair, compaction and Discord's migration
• DynamoDB under the hood: partitions, adaptive capacity, quotas and the 2015 outage
• MongoDB internals: oplog, WiredTiger, shard keys, the balancer and Stripe's DocDB
• Kafka as a log store: ISR, acks, compaction, transactions and LinkedIn scale