Cruxly › System design › Microservices and Events

Service-to-Service Communication: Discovery, Mesh, Identity and Compatibility

every call between services is a promise that two teams can break independently, so choose the style, find the peer, prove who you are, and change the contract with that in mind

Once a system is split into services, most of what used to be a function call becomes a network call between things deployed by different teams on different days. This lesson is the communication layer: synchronous versus asynchronous and what each does to availability, how callers find callees, what a service mesh gives and costs, how services prove identity to each other, where the gateway ends and a backend-for-frontend begins, how a request is followed across hops, how contracts change without a flag day, and a table for choosing per interaction. It builds on Designing the API (contracts, versioning, idempotency keys), Overload, Load Shedding and Cascading Failure (retry budgets, deadlines, breakers, bulkheads), API Gateway and Load Balancing, CDNs, and the API Gateway, Boundaries Are Where Guarantees Die, Queues and Async Work, Distributed Message Queue, SLAs, Availability Math and Error Budgets as Design Inputs and Distributed Tracing System, and does not repeat them.

Synchronous or asynchronous: what each does to coupling and availability

Claim: the choice is not about speed or taste. It decides whether the callee's outage is your outage.

microservices.io describes the two basic styles. With remote procedure invocation the client sends a request and waits for a reply; its listed benefits are that it is simple and familiar and that there is no intermediate broker, and its main drawback is reduced availability, because "the client and the service must be available for the duration of the interaction". With messaging the sender and consumer are coupled less at runtime, and availability improves because the broker holds the message until the consumer can handle it; the cost is that the broker is a new component that must itself be highly available, and request/reply over messages is harder to build.

Two kinds of coupling are in play, and people mix them up.

• Runtime coupling: must the other side be up right now? Sync says yes, a queue says no.
• Design coupling: must the other side know my message format, and does a change to it break me? Both styles have this one. A queue does not remove it; it moves it from an API schema to an event schema (see the last-but-one section).

Worked arithmetic (the lesson's own, using the series-composition rule from SLAs, Availability Math and Error Budgets as Design Inputs). A checkout handler hard-depends on five services, each 99.9% available and failing independently. The handler can succeed at most 0.999^5 = 0.99501 of the time. A 30-day month is 43,200 minutes, so the unavailable share of 0.499% is about 215.6 minutes, roughly 3.6 hours, before the handler adds its own failures. Ten such calls give 0.999^10 = 0.99004, about 7.2 hours. Now move one step, the "send a confirmation email" call, behind a queue. The handler's success then depends on four services plus the broker. If the broker is 99.99% (an assumption chosen for the illustration, not a vendor figure), the product is 0.999^4 x 0.9999 = 0.99591, about 177 minutes. The email service can be down for an hour and the order still succeeds; the email is late, not lost. Independence is an assumption: a shared database, a shared deploy or a shared network zone breaks it, and then the real number is lower.

What async costs. You give up the immediate answer, so the user interface has to say "accepted", not "done". You take on at-least-once delivery, duplicates and ordering questions, all covered in Queues and Async Work and Distributed Message Queue; and you lose the stack trace, which is why the tracing section matters. Async also hides overload: a queue that grows quietly is a latency bug nobody sees, so alert on queue age.

What sync costs, beyond availability. Latency adds along the chain, and a chain of calls has the retry and timeout multiplication in Overload, Load Shedding and Cascading Failure. A staff engineer's rule: a synchronous call is only free when the work cannot continue without the answer.

Three styles, not two. Request/reply (ask and wait), notification (tell and move on; the sender does not care who listens), and request/async-reply (ask now, receive the answer later on a callback or reply queue). The third is the right answer when the callee is slow and the caller does not want to hold a thread: pay with a correlation id and a place for the reply to land.

What a senior checks: for every synchronous edge, whether the caller has a fallback if it fails (a default, a cached value, a degraded page), because an edge with no fallback belongs in the availability product and an edge with one does not.

Service discovery: client-side, server-side, DNS and registries

Claim: discovery is a data-freshness problem. Every scheme answers "where is the healthy instance" from data that is some seconds old, and the interesting question is how old and what happens to calls aimed at a dead address.

Instances come and go with deploys and autoscaling, so addresses cannot live in config files. microservices.io names two patterns.

Load Balancing, CDNs, and the API Gateway covered the balancing algorithms; here the point is where the lookup lives.

Registries and how instances get into them. microservices.io's self-registration page: the instance registers at startup, renews its registration periodically and unregisters on shutdown. Its stated advantage is that the instance knows its own state (starting, available) and its stated drawbacks are coupling to the registry, per-language code, and that "a service instance that is running but cannot handle requests" often cannot unregister itself. The alternative is third-party registration, where a platform component watches instances and registers them for you, which is what an orchestrator does.

:::key Verified November 2026 Netflix's Eureka wiki ("Eureka at a glance") states: instances renew their lease with a heartbeat every 30 seconds; if renewals fail several times the instance is removed after about 90 seconds; clients refresh their registry view about every 30 seconds and keep a local copy, so they keep working even when every Eureka server is down. Each region runs its own Eureka cluster, with at least one server per zone. :::

Worked arithmetic on those numbers: an instance that dies just after a heartbeat is still listed for up to about 90 seconds, and a client can hold its previous copy of the registry for up to 30 seconds more, so the window in which a client may still send to a dead address is roughly 90 + 30 = 120 seconds (the sum of the two stated figures; the wiki does not give this total, and real servers add their own eviction sweep and response caching, so treat 120 seconds as a floor for the worst case, not a ceiling). At 500 requests per second to that instance's share, a window that long is tens of thousands of requests aimed at an address that is gone. That is why client-side discovery always comes with client-side timeouts, retries to a different instance, and passive ejection of an instance that has just failed; the registry cannot be fast enough on its own.

DNS and Kubernetes. The Kubernetes DNS documentation: an ordinary Service gets a record my-svc.my-namespace.svc.cluster-domain.example that resolves to the Service's cluster IP; a headless Service (no cluster IP) resolves to the set of pod IPs, and clients are "expected to consume the set or else use standard round-robin selection". So the first gives you server-side discovery (a virtual IP in front of the pods) and the second gives you client-side discovery over DNS.

Why does Kubernetes put a virtual IP in front instead of handing out pod addresses by DNS? The Virtual IPs page gives three reasons: DNS implementations have a long history of not respecting record TTLs and caching lookups too long; some applications resolve once and cache forever; and even with correct re-resolution, low or zero TTLs would load DNS heavily. This is the practical lesson: a headless Service is only as good as the client's resolver behaviour. A JVM or client library that caches the first answer will keep calling a stale pod after a rolling deploy.

On the virtual IP itself: kube-proxy programs forwarding rules, and in iptables mode it installs rules per endpoint that pick a backend pod at random. The same page notes that clusters with tens of thousands of pods and services mean tens of thousands of iptables rules and a long time to update them when endpoints change, with minSyncPeriod and syncPeriod available to tune. The page also says the default mode is changing in a future version, and recommends setting the mode explicitly.

Also from the DNS docs: a pod's resolv.conf lists search domains (<namespace>.svc.cluster.local svc.cluster.local cluster.local) and the example shows ndots:5, so a short name such as data is expanded through those domains before being tried as given. The consequence worth knowing is that a name with fewer than five dots can cause several failed lookups before the right one; write the fully qualified name, with its trailing dot where the resolver supports it, in hot paths.

What goes wrong, in the order it usually happens. Stale entries after a crash (the 120-second window); a registry outage (clients with a cached copy survive, clients without it do not); thundering re-registration after a registry restart; clients that cache DNS forever; and a partitioned registry that deletes healthy instances. Eureka's wiki says only that its servers are built to be resilient to peer failures and partitions, so find out how your registry prefers to be wrong: showing dead instances or hiding live ones. Showing too many is the safer error, since a client can retry another address, whereas an empty list is an outage.

What a senior checks: the worst-case window between an instance dying and callers stopping, measured in a game day; that the caller's HTTP client re-resolves on a schedule; and that the platform's default discovery is used before a second one is added.

The service mesh: what moves out of application code, and what it costs

Claim: a mesh moves cross-cutting network behaviour (mTLS, retries, load balancing, telemetry) out of every service and into a proxy layer you operate. You pay in resources, in hops and in a new thing that can break every call.

Istio's architecture page describes the model. The data plane is a set of Envoy proxies deployed as sidecars next to each workload, mediating all network traffic between services; the page lists load balancing, TLS termination, circuit breakers, health checks and fault injection, plus telemetry, all without application changes. The control plane, istiod, handles service discovery, configuration and certificates: it translates high-level routing rules into proxy configuration and pushes it, and it acts as the certificate authority for mTLS. Linkerd has the same split with its own proxy, which its architecture docs call an ultralight micro-proxy written in Rust, providing automatic mTLS, latency-aware layer-7 load balancing, retries, timeouts, circuit breaking and Prometheus metrics, with an identity service acting as a certificate authority.

What this removes from the code: the client library that every language needed for retries, TLS and metrics. A polyglot company with 12 languages writes that logic once. That uniformity is the strongest reason to adopt a mesh, more than any single feature. What it does not remove: deadlines (the app must still propagate them, per Overload, Load Shedding and Cascading Failure), idempotency (a proxy that retries a non-idempotent POST makes duplicates, the exact trap in Boundaries Are Where Guarantees Die), and trace propagation.

What it costs. Three bills.

1. Resources. The Istio performance page (for the Istio 1.24 test; the page itself is in the 1.31 docs) reports, for 1,000 HTTP requests per second, 1 KB payloads, 2 proxy worker threads and mTLS: a sidecar uses about 0.20 vCPU and 60 MB, a waypoint about 0.25 vCPU and 60 MB, and a ztunnel about 0.06 vCPU and 12 MB. The load test described a mesh of 1,000 services, 2,000 pods and 70,000 mesh-wide requests per second (the page does not state that test's hardware; its separate latency test used a small bare-metal cluster), and the page says different hardware gives different values and that memory depends on the configuration the proxy holds. Worked arithmetic, not Istio's figure: if each of 2,000 pods ran a sidecar at that load, that is 2,000 x 0.20 = 400 vCPU and 2,000 x 60 MB = 120,000 MB, about 117 GB. A per-node ztunnel on, say, 100 nodes (an assumed size) is 100 x 0.06 = 6 vCPU and 1.2 GB. The comparison depends on the pods-per-node ratio, and a sidecar sized for peak is reserved even when idle, which is the over-provisioning complaint Istio's own 2022 ambient post makes.
2. Latency and hops. Every call crosses two proxies in sidecar mode (the client's and the server's), so a chain of five calls crosses ten. Istio's page reports latency only as charts against connection counts, comparing no mesh, ambient L4, ambient with waypoints and sidecars, so no number is quoted here; measure on your own hardware, at your p99, with mTLS on.
3. Operations. The control plane is now on the critical path for configuration and certificates, proxies need upgrades, and a bad config pushed to every proxy is a fleet-wide outage. The 2022 ambient post lists sidecar problems in Istio's own words: installing or upgrading sidecars requires restarting application pods, traffic capture and HTTP processing are expensive and can break applications with non-conformant HTTP, and there is a fixed cost per workload even if all you wanted was transport security.

Sidecar versus ambient. Ambient mode splits the proxy into a per-node ztunnel (written in Rust; it does layer-3/4 work: mTLS, authentication, layer-4 authorisation and telemetry, and does not parse workload HTTP) and an optional per-namespace waypoint (Envoy, run outside the application pods, scaled and upgraded independently) for layer-7 policy and routing. Sidecar and ambient workloads can coexist in one mesh, and adoption can go from no mesh to a layer-4 overlay to layer 7 per namespace. The transport between ztunnels is HBONE, an HTTP CONNECT-based tunnel.

:::key Verified November 2026 Istio announced general availability of ambient mode in the 1.24.0 release on 7 November 2024, saying ztunnel, waypoints and the APIs were marked Stable by the TOC; the current docs read were version 1.31. The introducing-ambient post is dated 7 September 2022. Some waypoint extensions (WebAssembly plugins, Lua scripts) were still labelled Alpha in the navigation. Resource figures are from the Istio performance page for one test setup, not a guarantee. :::

When not to adopt one. Few services, one language with a good shared library, no regulatory need for mTLS everywhere, or no platform team to own it. A mesh is an operational product; if nobody is on call for it, it is a liability. What a staff engineer says: "What exact problem are we paying a proxy per pod for? If the answer is just mTLS, can we get it from the platform without layer-7 processing?"

What a senior checks: the proxy's CPU and memory requests against the sum for the cluster, the p99 of a representative call with and without the mesh, and what happens to traffic if the control plane is down (proxies keep the last config they were given; confirm it for your mesh).

mTLS and service identity: proving who is calling

Claim: network location is not identity. In a cluster where pods are rescheduled every day, an IP address names nothing, so services need a short-lived cryptographic identity tied to the workload and a policy that says which identity may call which.

Mutual TLS gives both ends a certificate. The server proves it is the service you meant to reach, the client proves what it is, and the channel is encrypted. It answers "who is on the other end" but not "may they do this"; authorisation is a separate policy on top.

SPIFFE is the open standard for the identity. Its concepts page defines a SPIFFE ID as a URI of the form spiffe://trust-domain/workload-identifier (its example is spiffe://acme.com/billing/payments), carried in an SVID, which is either an X.509 certificate or a JWT. The page advises X.509 SVIDs when possible because tokens can be replayed, with JWTs sometimes needed when a layer-7 proxy or load balancer sits between workloads. A trust domain is the trust root and can stand for an organisation, environment or department; the page recommends separate trust domains for, say, production and staging. The Workload API hands a workload its identity, short-lived credentials and trust bundles without the workload having to present any credential of its own, and keys are rotated automatically.

How a mesh does it. Istio's security page: on Kubernetes the service identity is the service account. An agent beside each Envoy proxy creates a private key and a certificate signing request, istiod validates and signs it, the proxy receives the certificate over SDS, and the agent watches expiry and repeats the process. The result is that nothing in the application has to hold or renew a certificate. Linkerd's identity service does the equivalent, issuing certificates to proxies at start-up that they use between each other.

Enforcement has two knobs. In Istio, PeerAuthentication sets the mTLS mode: PERMISSIVE accepts both mTLS and plaintext (recommended during migration), STRICT accepts only mTLS, and DISABLE turns it off. AuthorizationPolicy selects workloads and applies ALLOW, DENY or CUSTOM rules; evaluation order is CUSTOM, then DENY, then ALLOW. Two defaults to remember: a workload with no authorisation policy allows all requests, and once a workload has at least one ALLOW policy, anything matching no ALLOW rule is denied.

:::warn The migration trap PERMISSIVE is the right state to start in and the wrong state to stay in. A mesh with mTLS "enabled" in permissive mode still accepts plaintext from anything that can reach the pod, so the control you believe you have is absent. Track the share of traffic that is actually mTLS per service, and move services to STRICT one at a time once that share is 100%. A mesh with no authorisation policies is encrypted but open: any identity may call anything. :::

Worked arithmetic on rotation: if workload certificates live for 24 hours (an assumed lifetime, not Istio's default, which varies by version and configuration), a leaked key is useful to an attacker for at most 24 hours and a revocation list is rarely needed. The price is that the CA must issue one certificate per workload per day: with 5,000 workloads that is 5,000 signings a day, about one every 17 seconds on average (86,400 / 5,000 = 17.3), which is trivial, and the real risk is a CA outage that outlasts the certificate lifetime. Size the lifetime to be several times longer than your longest realistic CA outage.

Application-level identity still matters. Service identity says "the billing service is calling". It does not say "on behalf of user 42". The end-user token is a separate credential that must be forwarded and checked; a service that trusts any caller with a valid mesh identity to pass a user id invites the confused-deputy problem. Verify the user's token where the data is, or at least validate a signed assertion forwarded by the gateway.

What a senior checks: the fraction of in-mesh traffic that is mTLS (not "is it enabled"), that every service has an authorisation policy listing its allowed callers, certificate lifetime against CA recovery time, and that production and staging are different trust domains.

API gateway versus backend for frontend

Claim: a gateway is about the edge, one door for everyone. A BFF is about the client, one door per kind of client. They are different layers, and the usual mistake is putting client-specific logic into a shared gateway until it becomes the new monolith.

API Gateway and Load Balancing, CDNs, and the API Gateway covered what belongs in a gateway: authentication, rate limits, routing and protocol translation. microservices.io's API Gateway page lists the forces that justify it: fine-grained services mean a client may need several calls for one screen (its example is a product page needing product info, pricing, orders, inventory and reviews); desktop and mobile need different data; mobile networks allow only a few requests; locations change; and some services speak protocols that are not web-friendly. The page names the benefits (clients insulated from partitioning, fewer round trips, a protocol bridge) and the drawbacks (another component to develop and run, and an extra hop, which it argues is usually insignificant).

The same page describes Backends for Frontends as a variant with a separate gateway for each kind of client; its example has one each for a web app, a mobile app and an external third-party app. The Azure Architecture Center page (dated 19 March 2025) states the problem in team terms: when one backend serves several interfaces, competing demands cause frequent updates, conflicting changes and a bottleneck, and a separate team owning the backend creates delays because one frontend's requested change has to be validated with the others. Its solution is a layer per interface owned by the frontend team, which then controls language, release cadence and priorities.

The arrangement Azure shows puts both in series: a gateway (API Management in its example) handles the cross-cutting work, token validation, monitoring, caching and routing, and each client has its own BFF behind it, which calls the microservices. The page warns that a BFF should hold only client-specific logic and cross-cutting features should be abstracted out, that code duplication is a probable outcome, that each extra service adds an operational and latency cost, and that BFFs may add nothing when interfaces make the same requests or when only one interface exists.

The tests for whether you need one. Do two clients need different shapes of the same data, or different request counts? Do their release cadences clash? Is one gateway team the bottleneck for every client change? If not, one gateway is enough. Azure's page also notes that GraphQL can remove the need for a separate BFF layer because clients ask for the fields they need, and that BFFs may add little if you already use GraphQL with frontend-specific resolvers. Designing the API covers the GraphQL trade-offs, including query cost.

Worked arithmetic on why aggregation matters on mobile: a screen that needs 5 services, called one by one over a mobile network with a 100 ms round trip, spends at least 5 x 100 = 500 ms on the network sequentially, or 100 ms if the five calls are parallel, against 100 ms plus the BFF's data-centre calls (which are around a millisecond each inside a region) when one request goes to a BFF that fans out. The saving comes from moving the fan-out from the slow network to the fast one. The 100 ms figure is an assumption for illustration.

What a senior checks: that no BFF holds business rules (those belong in the domain services, or two BFFs will disagree), that the gateway has no per-client conditionals, and that each BFF has its own deadline and retry policy so one client's behaviour does not consume another's capacity.

Following one request across hops

Claim: the moment a request crosses a service boundary, the evidence of what happened to it splits into as many log files as there were hops. Tracing is the discipline of carrying one identifier through every one.

Distributed Tracing System builds the tracing system itself: collection, storage and sampling. This section is only about what the communication layer must do to make a trace possible.

Context has to travel. Each service must accept a trace identifier on the way in, put it in every log line, and attach it to every call it makes on the way out, including calls it makes from a background thread, a queue consumer or a scheduled job. One service that drops the header splits the trace into two unrelated halves, which is the failure the tracing lesson walks through.

A proxy can see hops but not causes. A sidecar or ztunnel observes each connection and can report request counts, latency and errors per service pair, and Istio's architecture page says its proxies collect telemetry on mesh traffic. By its position, the proxy sees an inbound request and, separately, an outbound one. Nothing on the wire says that the outbound call happened because of that inbound request unless the application copied the context from one to the other. So the mesh gives you the per-edge golden signals for free and a full trace only if the application propagates context. Teams that adopt a mesh "for tracing" learn this late.

Async edges need explicit work. A message on a queue is a hop with no connection to observe. Put the trace context in the message headers on publish, and in the consumer start a new span linked to that context. A common design is to link and not nest: the consumer's span links back to the producer's, since the consumer may run minutes later and a parent span that ended long before is misleading. Fan-out creates one trace with many branches; batch consumers that process 100 messages in one call carry up to 100 parent contexts, so link rather than pick one.

Sampling and correlation. Sampling decides what you can ever see (head versus tail sampling is covered in Distributed Tracing System), and a trace id in every log line, error response and message envelope turns "something is slow" into a lookup. Designing for Observability is the lesson for the signals themselves.

What a senior checks: the percentage of requests that arrive at the deepest service with a context (the broken-link rate), that queue consumers link to the producer span, and that the gateway creates the trace id when the client does not send one.

Versioning and compatibility across independently deployed services

Claim: in a system with independent deploys, "the new version" does not exist as a moment. For hours or weeks, old and new callers and callees run together, and every contract change must be safe in all four pairings.

Designing the API covered how to version a public API: URL, header, media type, and the cost of each. This section is about the internal case, where you control both sides but not their timing.

Four pairings matter during a rollout: old client with old server, old client with new server, new client with old server, and new client with new server. A change is safe only if all four work. Two names help: backward compatible (new readers accept old data) and forward compatible (old readers accept new data). Rolling deployments need both, because during the rollout either side can be the newer one.

The Protocol Buffers rules are a good model, because they are written down, and Boundaries Are Where Guarantees Die already teaches them (adding fields is safe, never reuse a field number, reserve what you delete, unknown fields survive only if you do not rebuild the message). The proto3 guide adds two rollout details worth knowing here: you must reserve a deleted field's number and should reserve its name too, so JSON and text encodings keep parsing, and moving a field into an existing oneof is not safe. Some wire-compatible changes are lossy, such as widening int32 to int64, so the guide says to roll them out carefully and to avoid them on schemas published outside your organisation.

The same logic applies to JSON, with a rule of thumb sometimes called Postel's law: be strict in what you send and tolerant in what you accept. Tolerant means ignoring unknown fields instead of failing on them. A strict JSON schema validator on the consumer side turns every producer addition into an outage.

Expand and contract. For a breaking change, split it into steps that are each safe: (1) expand: add the new field or endpoint alongside the old, and deploy readers that understand both; (2) migrate: move writers and callers to the new form; (3) contract: remove the old form only when telemetry shows nothing uses it. Worked arithmetic on timing: if the slowest consumer deploys every 4 weeks, step 3 cannot start until at least 4 weeks after step 2 completes for all consumers, so a rename costs about 8 weeks of calendar time. That is the real price of a breaking change, and it argues for adding instead of renaming.

Consumer-driven contracts invert the usual test. Each consumer publishes the parts of the producer's response it relies on; the producer's build runs all of them, so it fails before deploy and not after. This is cheaper than an end-to-end environment shared by everyone and answers the question "can I delete this field?" with evidence. The limit: it covers shape, not behaviour such as ordering or timing.

Events are contracts too. A published event is read by consumers you may not know and may replay for years. Adding optional fields is safe; changing the meaning of an existing field is a breaking change even if the type stays the same. Version the event type name when meaning changes, and keep the old event flowing while consumers migrate. Distributed Message Queue covers retention and replay, which is why event schemas outlive the code that wrote them.

:::warn The dual-write window During migration the producer often writes both the old and the new field. If a consumer reads one and another reads the other, a bug that sets them differently shows up only in the second consumer. Assert in the producer that the two agree, and keep the dual-write window short. :::

What a senior checks: that removed fields are reserved, that consumers ignore unknown fields, that a usage metric exists for every deprecated field, and that CI runs the consumers' contract tests against the producer's change.

Choosing the style per interaction

Claim: a system uses several styles at once, chosen per interaction by three questions: does the caller need the answer to continue, can the callee be down when the caller runs, and who owns the data being asked about.

Walk each interaction through the table below. The right column names the failure to design for, which is where the choice really pays off.

Rules of thumb a staff engineer will defend.

1. Default to synchronous for queries, asynchronous for commands that fan out. A query wants an answer; a command's side effects rarely need to finish before you reply.
2. Count sync edges on the critical path. More than three or four hard-dependent hops on a user request is a design smell, by the availability product in the first section. Collapse services, cache, or move an edge behind a queue.
3. Never share a database between services as the communication channel. It is the tightest coupling of all and hides the contract.
4. One retry layer, one deadline, whatever the style. The mesh can retry, the client library can retry, the application can retry; choose one owner per call path.
5. Pick the platform default for discovery and identity, and add machinery only when a named requirement forces it.

A worked example. Order placement in a shop. The checkout service calls Payments synchronously (needs the result; deadline 2 s; idempotency key from the client). It reserves stock with a synchronous call to Inventory, with a fallback of "backorder". It publishes OrderPlaced to a broker; Email, Analytics and Recommendations consume it independently, and the order succeeds even if all three are down. The mobile BFF builds the order-confirmation screen by reading from an order read model, not from three services. With payments and inventory both at 99.9% and the broker at an assumed 99.99%, independent failures give 0.999 x 0.999 x 0.9999 = 0.99790, roughly 90.7 minutes of unavailability a month, against the 3.6 hours of a five-call chain (the lesson's own arithmetic).

What to say in the room. "I would not choose one style. The request path stays synchronous and short, side effects go through events, and I cap hard-dependent hops so the availability product still meets the SLO. Discovery and identity come from the platform. I would add a mesh if we need uniform mTLS and policy across many languages and have the team to run it."

Further reading: Building Microservices, second edition, by Sam Newman, chapters 4 (Microservice Communication Styles) and 5 (Implementing Microservice Communication); the microservices.io pattern catalogue by Chris Richardson; and the Istio, Linkerd and SPIFFE documentation for the mesh and identity details.

How to say it in the interview

"For each interaction I ask first whether the caller needs the answer to finish its own work. If it does, it is a synchronous call and the callee's availability multiplies into mine, so five hard dependencies at 99.9% each give me about 99.5%. If it does not, I put a queue between us and my availability depends on the broker, not on the consumer. Discovery I leave to the platform, a Kubernetes Service or a registry with a known staleness window, and I know that number. I only add a mesh when I need uniform mTLS, identity-based authorisation and telemetry across many languages, and I know it costs a proxy per pod or per node. Identity is a short-lived certificate tied to a workload, not an IP address. Contracts change by adding, never reusing, and by running old and new side by side."

Checklist

• Listed every synchronous call on the critical path and computed the availability product, with a fallback or a queue for any edge that need not block
• Stated the worst-case window between an instance dying and callers stopping, and set client timeouts and retry-to-another-instance to cover it
• Confirmed the client resolver re-resolves names on a schedule, and used fully qualified names in hot paths
• Justified a mesh with a named requirement (uniform mTLS, identity policy, telemetry across languages) and sized proxy CPU and memory for the whole cluster
• Measured p99 with and without the proxy path, with mTLS on, on our own hardware
• Tracked the share of traffic that is actually mTLS and moved services from PERMISSIVE to STRICT one by one
• Wrote an authorisation policy per service listing allowed caller identities, and kept production and staging in separate trust domains
• Kept client-specific shaping in a BFF owned by that client's team and shared concerns in the gateway, with no business rules in either
• Propagated the trace context through every outbound call, including queue messages, and monitored the share of requests that reach the deepest service with a context
• Changed contracts by expand, migrate, contract; reserved removed field numbers; made consumers ignore unknown fields; measured use of each deprecated field

Related lessons

• Decomposing a System into Services
• API Gateway
• Queues and Async Work
• Event-Driven Architecture: Choreography, Orchestration and Sagas