Cruxly › System design › Scaling Building Blocks

Scaling Up, Scaling Out and Splitting

buy time with a bigger box, buy capacity with more boxes, and split along the line that actually carries the load

Every design hits the moment where one machine is not enough, and the interviewer wants to hear which move you make first and why. There are only three: make the machine bigger, run more copies of it, or cut the system into pieces that grow independently. This lesson gives you the conditions each one needs, the arithmetic for how many instances you need, why autoscaling is slower than it looks, and how to find the bottleneck before you spend money on the wrong tier.

It builds on Estimating the Load, State Gravity, Caching, Load Balancing, CDNs, and the API Gateway and Scaling Postgres and MySQL before you leave them, and does not repeat them. Further reading: The Art of Scalability by Martin L. Abbott and Michael T. Fisher.

Vertical scaling: the cheapest move and its ceiling

Scaling up is the right first move more often than interviewers expect, because it changes nothing about the design. A bigger instance has the same code, the same deployment, the same failure story and no new coordination. You pay in money and in a ceiling, not in complexity.

What it buys and what it does not:

• It buys headroom immediately. More cores, more RAM (a bigger cache or buffer pool), faster disks and network, in the time it takes to resize. Scaling Postgres and MySQL before you leave them treats this as rung 2 of the database ladder and shows how Figma used it to buy about a year.
• It does not buy availability. One big machine is still one machine. When it dies, everything on it dies, and the larger it is, the larger the share of your traffic that failure takes down.
• It does not scale linearly. Doubling the cores of a process that contends on one lock, one connection pool or one disk does not double throughput. The shared resource caps you, so measure before you buy.
• Price is not linear either. The largest sizes carry a premium per core, and a resize can mean a restart, so schedule it.
• It has a hard ceiling. There is a biggest machine.

:::key Verified November 2026 The AWS High Memory instances page lists U-1 sizes up to 6 TiB (6,144 GiB) of memory and 448 logical processors (hyperthreads on 224 cores). Its notice says the larger U-1 sizes (9, 12, 18 and 24 TB) and further U-1 sales ended on 6/20/25, with U7i recommended for new customers; the page does not give U7i sizes. So the single-instance ceiling is a few terabytes of RAM and a few hundred logical processors, and it is perishable: it moves every hardware generation. The interview point is that the ceiling exists and that you can name roughly where it is. :::

When scale-up is the answer. The workload has one hot shared structure that is hard to split (a single-writer database primary, an in-memory index), the load is within a few times of today's, and the downtime of a resize is acceptable. Estimating the Load gave the workhorse number: roughly 10 to 20 thousand small durable writes per second is where one primary starts to hurt. Below that, a bigger primary beats a distributed rewrite.

When it is a trap. When the real reason is availability (you need two machines for failover anyway), when the load grows by orders of magnitude, or when the cost per unit of capacity rises faster than the traffic.

What the senior engineer says in the room: "Scale up first, because it is reversible and adds no new failure modes. I would still run at least two of whatever we scale up, because a single big box is a single point of failure." What a senior checks: the saturating resource (CPU, memory, IOPS or locks) before the purchase, and whether the next size up actually relieves that resource.

Horizontal scaling: what must be true first

Scaling out means any instance can serve any request, so that adding an instance adds capacity. That sentence is a condition, not a fact. It holds only when the instance keeps nothing that the next request needs. State Gravity is the principle behind this, and Load Balancing, CDNs, and the API Gateway covers the balancer itself. This section lists what has to be true before you draw "N copies" on the board.

What must move out of the instance, and where it goes:

The shared tier you moved state into is now the thing that must scale. Pushing sessions to a shared store does not delete the state, it concentrates it. That is the right trade (see State Gravity), but you have to size the new tier: a session store taking one read per request sees the whole fleet's request rate.

Cloning has a coordination bill even when stateless. Each instance opens its own connections downstream, so 50 instances with a pool of 20 means 1,000 connections at the database (worked arithmetic). Deploys now roll across many instances, so two versions run at once and must be compatible. Health checks and graceful shutdown (drain, then stop) become mandatory, otherwise scale-in kills in-flight requests.

What the senior engineer says: "Before I add instances I list what each one holds. If the list is not empty, that is the first piece of work." What a senior checks: kill one instance in staging and see whether any user notices; count total downstream connections at N instances.

Three ways to split: clone, split by function, split by data

There are three directions to grow in, and naming them tells the room which problem you are solving. AKF Partners' scale cube page describes them as three axes, which this lesson restates in its own words and cites by name.

AKF's page also notes that Z-style pods can follow local privacy rules such as GDPR, since a customer's data can be kept in a region. It gives no performance numbers, and you should not invent any.

How the axes combine. They are independent and you can use all three: a service split by function (Y), each service cloned behind its own balancer (X), and the data-heavy ones sharded by tenant (Z). The skill is picking the axis that matches the saturated resource, as the next sections show.

• Clone (X) when the bottleneck is stateless compute or read traffic. It is the cheapest and the most reversible.
• Split by function (Y) when one function has a different scaling shape from the rest (search is read-heavy and memory-hungry, checkout is write-heavy and must not fail), or when teams block each other. Load Balancing, CDNs, and the API Gateway covers routing between the resulting services.
• Split by data (Z) when one dataset or one write stream outgrows a single node. Scaling Postgres and MySQL before you leave them places this at the top of the ladder (rungs 8 and 9), and Partitioning and Sharding covers key choice.

:::warn Y is a cost you pay for a benefit, not a goal Splitting by function turns local calls into network calls and takes away joins and cross-service transactions. Split when a function's load shape or ownership differs, not because microservices sound modern. :::

What the senior engineer says: "X first for compute, Y when functions have different shapes or owners, Z when data or writes outgrow a node. They compose, and I would say which one each tier uses." What a senior checks: whether the proposed split lines up with a real difference in load shape, failure tolerance or ownership.

Headroom and capacity planning: from peak load to instance count

Capacity is a number you calculate from three inputs: peak load, per-instance capacity at a safe utilisation, and how many failures you must survive. Estimating the Load gives you the peak; this section turns it into an instance count.

Measure capacity in resources, not requests per second. The Google SRE book (chapter 21) argues that queries per second is a poor capacity measure, because the cost of a request varies by client, time of day and software version, so the ratio keeps moving. It recommends measuring in available resources such as CPU, and says that using CPU consumption as the provisioning signal works well in most cases. The practical form: find, by load test, the request rate at which one instance's latency leaves its target, then back off from that knee.

Why not run at the knee: queueing. Using the simplest queueing model (one server, random arrivals, mean latency = service time / (1 - utilisation)), the arithmetic for a 20 ms service time is:

This is worked arithmetic from a textbook model, not a measurement of any system. The shape is the point: latency does not rise gently, it explodes near the top, and the tail rises earlier than the mean. That is why a safe target is well below 100%.

Worked example (assumed numbers). Peak is 20,000 requests per second. A load test shows one instance holds its latency target up to 500 requests per second; you plan to run at 60% of that, so each instance counts as 300.

1. Instances for peak: 20,000 / 300 = 66.7, so 67.
2. Survive the loss of one of three availability zones: the remaining two zones must carry the whole peak. The two survivors must hold 66.7 instances' worth, so each zone needs at least 34 (2 x 34 x 300 = 20,400). Total 3 x 34 = 102.
3. The headroom is therefore not 10%; surviving one zone costs about 50% more machines than the bare minimum (102 versus 67).

Add the cost of a deploy: if you roll 10% of instances at a time, 10% of capacity is out of service during the roll. The zone-loss headroom does not also cover that: if a zone fails mid-deploy, the survivors run hotter than planned, so pause deploys during a zone failure or add margin.

Reading Google's own example. The SRE chapter shows a per-datacenter ceiling as tasks times per-task rate (100 tasks at 500 requests per second each gives 50,000 requests per second for that datacenter). Treat that as the model: a ceiling per failure domain, then ask what happens at the ceiling.

What the senior engineer says: "I size for the peak with one failure domain gone, at about 60% utilisation, and I say what the 60% is based on: a load test." What a senior checks: the load-test knee, headroom after losing a zone, and the growth rate against the time it takes to order capacity.

Autoscaling: signals, lag and cold-start cost

Autoscaling is a feedback loop with delay, and a loop with delay oscillates or reacts too late unless you design for the delay. Its job is to track load, not to rescue you from a spike that is faster than the loop. The two public documents read for this lesson, the Amazon EC2 Auto Scaling user guide and the Kubernetes horizontal pod autoscaler page, describe the same shape.

Pick a signal that scales with the fleet. The AWS guide says a target-tracking metric must describe how busy an instance is, and its value must rise or fall in proportion to the number of instances. CPU utilisation of the group works if load is spread across the instances; the guide lists the load balancer's total request count and its latency as metrics that do not work, because the total does not change with the group's utilisation. A queue's depth does not work directly either, but queue depth per instance can.

The loop and the formula. Kubernetes computes desiredReplicas = ceil[currentReplicas x (currentMetric / desiredMetric)]. Worked arithmetic: 10 replicas at 90% average CPU against a 60% target gives ceil(10 x 1.5) = 15. A tolerance of 0.1 means no change when the ratio sits within 10% of 1.0, so 63% against a 60% target (ratio 1.05) does nothing. AWS describes the same idea as a thermostat: you give a target (say 50% average CPU) and the service manages the alarms.

:::key Verified November 2026 Kubernetes HPA defaults (kubernetes.io): the controller loop runs every 15 seconds; tolerance 0.1; scale-down stabilisation window 300 seconds, taking the highest recommendation inside the window; default scale-up allows the larger of 100% of current replicas or 4 pods per 15 seconds, and scale-down allows 100% per 15 seconds; CPU samples are ignored for 5 minutes after pod start (cpu-initialization-period) unless the pod is Ready, and the initial readiness delay is 30 seconds. Pods that are not yet ready are assumed to use 0% of the target when scaling up, which damps the reaction. AWS: EC2 metrics are published at five-minute intervals by default, one minute with detailed monitoring, and AWS recommends one-minute or finer metrics for target tracking. The default instance warmup is not enabled or configured by default; AWS strongly recommends enabling it and, if unsure, suggests starting with 300 seconds and tuning. :::

Where the lag comes from. Add the stages in your own environment (assumed numbers for the arithmetic): metric period 60 seconds, alarm evaluation 60, instance boot and image pull 90, application warm-up 120. That is about 5.5 minutes between the spike and capacity that serves it. Traffic that doubles in two minutes outruns that.

• Warm-up protects the loop. AWS keeps a newly launched instance out of the aggregated metrics until its warmup expires, because a freshly started instance often burns CPU while it loads. While instances warm up, the group scales out again only if the instances that are not warming exceed the target, and warming instances still count as capacity for the next scale-out, so repeated breaches add one batch, not many. Scale-in is blocked until they finish.
• Scale in conservatively. AWS notes a target-tracking policy scales in more gradually than out, to protect availability, and rounds up or down conservatively; Kubernetes uses the 300-second window for the same reason. Scaling in too fast turns each lull into the next outage.
• Small groups look far from target. With 2 instances, adding one changes utilisation by a large step, which AWS says explains the gap between target and actual.
• Cold starts add cost beyond time. A new instance has an empty cache, cold connection pools and code that has not yet been optimised by the runtime; the first requests are slower, and a thousand new instances all missing a cache at once hit the database together (Caching covers the herd).

Autoscaling is not a substitute for headroom. Keep a floor (minimum capacity) that survives the failure you chose in the previous section, scale on a signal that leads (queue age or request concurrency) where you can, and for known events scale on a schedule. If demand outruns the loop, the answer is to shed or queue load (the SRE chapter's criticality levels and client-side throttling exist for this), not to autoscale harder.

What the senior engineer says: "Autoscaling handles the slope, headroom handles the step. I put warm-up and stabilisation numbers in the design because the lag is the design." What a senior checks: time from alarm to serving traffic, minimum capacity against the failure case, and what the scale-in rule does after a spike.

Finding the bottleneck: utilisation, queueing, the saturated resource

Scaling the wrong tier is the most common waste, so find what saturates first. A request passes through a chain: balancer, app CPU, connection pool, cache, database CPU, database disk, a downstream service. The slowest link sets throughput, and only relieving that link changes anything.

A method that works in the room. For each resource on the path, ask three questions: how busy is it (utilisation), is work waiting for it (a queue, a run queue, a pool wait), and is it failing (errors, rejects, timeouts). A resource that is busy, with a queue forming, is your bottleneck. A busy resource without a queue has room. A queue in front of a resource with spare capacity usually points to a limit in the software (a pool, a lock, a thread count), not the hardware.

Fan-out turns rare slowness into common slowness. Dean and Barroso (The Tail at Scale, 2013) work this out: if a server answers slowly (over one second) for one request in a hundred, and a user request must collect answers from 100 such servers in parallel, then 63% of user requests take more than one second. Check: 1 - 0.99^100 = 63.4%. Their second example: with only one slow request in 10,000 per server, a service of 2,000 such servers sees almost one in five user requests over a second (1 - 0.9999^2000 = 18.1%). Scaling out a fan-out tier, therefore, can make the tail worse, since each user request now touches more servers.

The paper's remedies are the practical vocabulary for this:

• Hedged requests: send the request to a second replica after a short delay and use the first answer, cancelling the rest. Their Bigtable benchmark (1,000 keys across 100 servers) cut the 99.9th percentile of the whole read from 1,800 ms to 74 ms with a hedge after 10 ms, at 2% more requests.
• Micro-partitions: more partitions than machines (they cite 20 to 1,000 tablets per machine for Bigtable), so load balances in small pieces and recovery after a failure spreads across many machines.
• Canary requests: for very high fan-out, send a request to one or two servers first, so a request that crashes servers does not do it to thousands at once.

Hot spots are a different kind of bottleneck. Skew sends one key, tenant or shard far more load than its neighbours. Adding nodes spreads average load but the hot partition stays hot, which is why the question is "is it saturated everywhere or in one place?"

What the senior engineer says: "I do not add capacity until I can name the saturated resource, and for fan-out I check p99, not the mean, because the tail grows with the number of servers touched." What a senior checks: per-tier utilisation plus queue depth, per-partition load against the average, and percentiles for fan-out calls.

Scaling the data tier last, and what each rung costs

Compute scales by copying, data scales by dividing, and dividing is where the pain is. Every stateless tier you cloned still talks to something stateful. So the order is: make the request tier horizontally scalable, protect the data tier with caches and replicas, and divide the data only when a measured limit forces it.

Scaling Postgres and MySQL before you leave them is the full ladder with signals and costs. Here is how its rungs line up with the scale cube, and what each one costs the application.

Reads and writes scale differently. Copies of data take reads; every write must go to every copy of that item or to its one owner. Replicas multiply read capacity but not write capacity; only splitting the data (Y or Z) or reducing writes does. That is why the write rate decides when you reach the top of the ladder. The Estimating the Load anchor of roughly 10 to 20 thousand small durable writes per second per primary is the line.

Read amplification from your own scale-out. Cloning the app tier multiplies the connections and the cache misses reaching the data tier. If the app fleet grows 10x and each instance holds the same pool, the database sees 10x the connections (worked arithmetic: 100 instances times 20 = 2,000). Pool or proxy before you scale the app fleet, not after the database complains.

:::warn Do not shard to fix a read problem If the signal is read load, caches and replicas are cheaper, reversible and keep joins. Sharding is chosen for write throughput or data size, and it is the hardest rung to undo. :::

What the senior engineer says: "I make the stateless tier elastic, I make the data tier boring for as long as I can, and I say which write rate or size forces the split." What a senior checks: write rate and data size per node against their limits, replica lag, and whether the next rung is reversible.

Interview script: taking a design from 1x to 100x

Interviewers ask "what if traffic grows 100x?" to hear whether you scale in the right order and can say what breaks at each step. Use a ladder with numbers, and attach each step to a signal. The figures below are assumed for the example, not drawn from a company.

Start state, 1x: 1,000 requests per second at peak, 10% writes. One app instance and one database. Writes: 100 per second. Say plainly that one box is fine.

10x (10,000 requests per second, 1,000 writes per second).

1. Move state out of the app (sessions, uploads), put a load balancer in front, run at least 3 instances across zones. At 300 per instance, two surviving zones must carry 10,000 / 300 = 33.3 instances, so 17 per zone (2 x 17 x 300 = 10,200) and 51 in total (worked arithmetic). Say the formula, then the number.
2. Add a cache for hot reads and replicas if the primary's reads stay high. Writes at 1,000 per second are still comfortably one primary.
3. Pool connections.

100x (100,000 requests per second, 10,000 writes per second, 90,000 reads).

1. App tier: 100,000 / 300 = 334 instances needed with all zones up; with zone-loss headroom 167 per zone, 501 in total (worked arithmetic). Autoscaling with a floor at the zone-loss number, plus schedules for known peaks.
2. Reads: with a 90% cache hit rate (assumed), 9,000 reads per second reach the database instead of 90,000 (worked arithmetic).
3. Writes: 10,000 per second is now at the low end of one primary's range, so you are choosing between a bigger box and a split. Split by function if one table family dominates writes, otherwise split by data. State which key, and what you lose (cross-shard queries).
4. Fan-out: if one request touches many backends, plan for p99 (hedging, micro-partitions) because the tail grows with the number of servers.
5. Isolation: stop one function's spike from taking the rest (Y), and cap the blast radius of a bad deploy or a bad tenant (Z pods).

What to say at each step: the signal that triggers it, what it costs, and what you would measure to confirm. End with the failure you chose to survive and the number that shows you do.

:::key Wrong statements to avoid "Just add more servers" without naming the saturated tier. "Autoscaling handles spikes" without the lag numbers. "Shard the database" for a read problem. "The load balancer fixes availability" when the data tier is one box. :::

Further reading. The Art of Scalability by Martin L. Abbott and Michael T. Fisher is the source of the AKF scale cube and covers capacity planning (its chapter "Determining Headroom for Applications") and the cube (its chapter "Introduction to the AKF Scale Cube"). Its companion, Scalability Rules, is a shorter catalogue. For tail latency, read Dean and Barroso's paper. For overload, read the Google SRE book chapter "Handling Overload".

How to say it in the interview

"First I find which resource saturates, because adding boxes to the wrong tier is the classic waste. If it is the request tier and it is stateless, I clone it behind a load balancer and size it so that losing one availability zone still leaves the peak under about 60% utilisation. That is arithmetic, not a feeling. If one function dominates, I split it out as its own service. If one table or one tenant dominates, I split by data. The database goes last, because it is the one tier I cannot just clone, and I would say which rung of the ladder the signal points to."

Checklist

• Name the saturated resource (busy and queueing) before adding capacity to any tier
• List what each instance holds (sessions, files, caches, timers, counters) and move it out before cloning
• Run at least two of anything you scale up, and say a single large box is still a single point of failure
• Compute instance count from peak, per-instance capacity measured by load test, and a failure domain you survive
• Plan to run at a utilisation well below the knee (about 60% in this lesson's example) and check p99, not the mean
• Write down the autoscaling signal, warm-up, stabilisation and scale-in rule, and the end-to-end lag in seconds
• Keep a minimum capacity that covers the failure case, and scale on a schedule for known events
• Count downstream connections and cache misses at N instances before growing the fleet
• Pick the scale-cube axis that matches the saturated resource: clone for compute, split by function for different shapes or owners, split by data for writes and size
• Keep the data tier on the cheapest reversible rung, and name the write rate or size that forces a split

Related lessons

• Load Balancing, CDNs, and the API Gateway
• Caching
• Cache Eviction and Topology
• Distributed Cache Cluster
• Content Delivery Network
• Rate Limiting