Cruxly

Cruxly › Cloud and DevOps › Cloud & DevOps: Cloud Cost Engineering

The Bill You Don't See Coming: Egress, Cross-AZ Charges, and Real Cost Attribution

traffic that costs money without ever looking slow

Open the interactive lesson

Prompt: a service's multi-AZ architecture (load balancers, application servers, and a database spread across three availability zones for redundancy) is functioning correctly — good latency, good error rate — and the monthly cloud bill has grown noticeably without any corresponding growth in request volume. Find and fix the actual source, and design cost attribution so a similar surprise doesn't take this long to trace next time.

30 minutes first. The naive investigation starts with performance metrics, because "the bill went up" feels like it should correlate with load or latency. It often doesn't — the cost driver can be invisible to every dashboard that only measures speed and correctness.

Data transfer is billed separately from compute, and none of it looks like a performance problem

Cross-AZ transfer (within one region), cross-region transfer, and internet egress are each billed at their own real per-GB rate — on AWS, $0.01/GB for cross-AZ transfer in each direction (so every GB moved between zones costs $0.02 in total) and roughly $0.09/GB for the first tier of internet egress, rates that are individually small but multiply into a real, visible line item at meaningful traffic volume. None of this transfer makes a request slower or less correct — a request routed cross-AZ instead of staying within one AZ can have perfectly good latency and succeed every time, while still costing measurably more than the same request would have cost routed AZ-locally.

A multi-AZ architecture, specifically, is a common source of surprise cross-AZ billing precisely because it's working as designed. Spreading load balancers, application servers, and a database across availability zones for redundancy is the correct reliability decision — but without AZ-affinity-aware routing (preferring to route a request to an application server and database replica in the same AZ when a same-AZ option exists and is healthy), a meaningful fraction of otherwise identical, correctly-functioning traffic crosses AZ boundaries and gets billed for it, purely as a side effect of load-balancing logic that never considered AZ locality as a routing input.

NAT Gateway processing fees are a frequently-missed compounding cost on top of the base transfer rate — traffic routed through a NAT Gateway is billed the underlying data-transfer rate plus $0.045/GB NAT processing (AWS, us-east-1), so 10 TB a month through NAT adds about $450 before any transfer charge. That second charge is easy to miss specifically because it's a distinct line item most teams don't think to look for when investigating "why did egress cost more than expected."

Attribution has to be tagged before the bill arrives, not reconstructed after

A shared cloud bill answers "how much did we spend" and nothing about "who's responsible for which part" unless resources are tagged with team/service/environment context at the point they're provisioned — reconstructing that attribution after the fact, from resource names or informal conventions, is unreliable and gets categorically harder the longer untagged spend is allowed to accumulate, for exactly the same reason retrofitting cost attribution onto untagged LLM API calls is unreliable months later: the context that would answer the question was never captured at the only point it was cheap to capture.

Enforce tagging at the infrastructure-provisioning layer (a policy that rejects untagged resource creation, not a wiki page asking people to remember) rather than relying on convention. A norm that isn't enforced by the platform creating the resources decays the same way an unenforced code-review requirement does — correctly followed by disciplined teams under no time pressure, and the first thing skipped by anyone moving fast during an incident or a deadline, which is precisely when a new, unattributed resource is most likely to be created.

Tags on resources aren't enough on their own:

Review the "unattributed" bucket on a schedule, treating its growth as a leading indicator of a tagging gap, not just a bookkeeping nuisance. A shrinking or stable unattributed bucket means tagging enforcement is actually working; a growing one means new untagged spend is being created faster than anyone is finding and fixing the gap that allows it — a signal worth acting on before the bucket is large enough that reconstructing it becomes its own multi-week project.

The cheap fixes: endpoints, NAT, public IPs, and in-zone Kubernetes traffic

Several of the biggest network line items have simple fixes. AWS prices (us-east-1) are used below; other providers have the same kinds of charges.

Know which cross-AZ paths are free. Inter-AZ transfer within a region is charged $0.01/GB in each direction. Some paths are exempt:

Traffic between your own pods, from a database client to a replica in another zone, and Kafka replication on self-managed brokers are all charged.

In Kubernetes, keep Service traffic in-zone where you can. A ClusterIP Service spreads requests across endpoints in every zone by default. Setting trafficDistribution: PreferClose on the Service (GA in Kubernetes 1.33) sends traffic to endpoints in the client's zone when there are some, and falls back to other zones when there aren't. Check two things when you use it:

How to say it in the interview

"Cross-AZ and egress charges don't make anything slower - they make it more expensive, which is exactly why a latency and error-rate dashboard has no reason to ever flag them. A multi-AZ architecture that works perfectly and costs a lot more than expected is not a contradiction; it's traffic doing exactly what it was designed to do, billed at a per-GB rate nobody was watching."

Checklist

Practice this interactively
Share this lessonWhatsAppLinkedInX

Related lessons