Cruxly › System design › Advanced Trade-offs

Workflow Orchestration Engine

resuming from exactly where it crashed, days later, without redoing what already happened

Prompt: design a workflow orchestration engine (like Temporal or AWS Step Functions) — long-running, multi-step business processes (e.g. "process an order: charge card, reserve inventory, ship, send confirmation email") that must survive worker crashes, retries, and steps that take minutes to days, resuming exactly where they left off.

40 minutes first. This is a different problem from the job scheduler exercise elsewhere in this track — that one runs one job once; this one runs a sequence of steps and must never repeat a step that already had a side effect.

Applying the four moves: Scope — long-running multi-step processes that survive crashes and resume exactly where they left off; the business logic inside each step out. Estimate — number of in-flight workflows and how long they live (minutes to days), which means state is mostly at rest, not hot in memory. Sketch — a durable event history per workflow, workers that execute steps, and a replay mechanism that rebuilds a workflow's state from its history. Deep dive — why this isn't the job scheduler again: resuming exactly where it left off requires deterministic workflow code replayed against recorded results, and each side-effecting step must be idempotent or compensable (a saga) when a later step fails.

Why this isn't just 'the job scheduler exercise, but longer'

Scope. In scope: defining a multi-step workflow, durable execution across worker crashes, retries with backoff per step, human-readable workflow state/history. Out of scope: the workflow definition language's full expressiveness (branching, loops — real systems support these, but the interesting mechanism doesn't depend on the language's richness), the business logic inside each step itself.

The job scheduler exercise (elsewhere in this track) solves 'run this one job exactly once.' This exercise is a level up: run a sequence of steps, where each step is its own 'exactly once,' and the sequence itself must be resumable from an arbitrary point of failure — not just retried from the start. A workflow that charges a card (step 1), reserves inventory (step 2), and ships (step 3) cannot simply be restarted from step 1 after a crash during step 3, because step 1's side effect (the charge) already, correctly, happened — restarting from the top would charge the customer again. This single fact — a crash mid-workflow must resume from the next unexecuted step, not the first step — is what makes this a genuinely different problem from single-job scheduling, and stating that distinction explicitly is the strongest opening move.

Event sourcing: replaying history instead of storing current state

The event history is the source of truth for what already happened; a resuming worker derives "where was I" by replaying it, rather than trusting any single stored "current step" field that could itself be stale or lost.

Every workflow's execution is recorded as an append-only event log — ChargeStarted, ChargeCompleted, ReserveStarted, ReserveCompleted, and so on — rather than the engine simply overwriting a "current state" field. When a worker needs to resume a workflow (because the previous worker crashed, or because the workflow slept for a scheduled delay and is now waking up), it replays this event history from the beginning, deterministically re-executing the workflow definition's control flow (which step comes next, given what already completed) without re-executing the side-effecting work itself — a completed step's actual external action (the card charge, the API call) is never repeated; only the bookkeeping of "which step are we on" is recomputed by replay.

This is the same event-sourcing idea used by any audit-log-as-source- of-truth system, applied specifically to workflow control flow — the event history isn't just a debugging aid, it is the durable state; there is no separate "current workflow state" record that could itself become inconsistent with what actually happened.

Idempotent steps and the exactly-once-effect discipline

Each individual step still needs its own idempotency key, exactly as in this track's payment and job-scheduler exercises — because a step can be started multiple times (a worker crashes right after starting a step but before recording its completion, and the resuming worker, seeing a ChargeStarted with no matching ChargeCompleted, must re-attempt it) even though the workflow engine guarantees the step's position in the sequence is only reached once. "Exactly once in the workflow's control flow" and "exactly once in side effects" are two different guarantees, and only the first comes for free from event sourcing — the second still requires the step itself to be written idempotently (the charge call includes a stable idempotency key so a repeated charge attempt for the same ChargeStarted event is a no-op against the payment provider, not a second real charge).

Timers and long delays (a workflow step that says "wait 3 days, then check if the customer responded") must be durable, not an in-memory sleep — an in-memory timer is lost the instant the worker process restarts. The engine persists the timer as its own durable event (TimerScheduled with a wake time), and any worker, not necessarily the one that scheduled it, can pick up the workflow when the timer fires, because the timer's existence lives in the durable event history, not in any one process's memory.

Failure modes and the honest edges

Replaying the full event history from the beginning becomes slow for a very long-running or very step-heavy workflow — a workflow that has accumulated thousands of events over months of execution shouldn't need a full replay from event zero every time it wakes up. Periodic snapshotting of derived state (checkpoint "as of event 400, here's the reconstructed state" so a resume only replays events after the snapshot) is the same snapshot-plus-replay-the-tail pattern used in this track's collaborative-document exercise, applied to workflow state instead of document text.

Non-determinism in the workflow definition itself breaks replay correctness — if a workflow's control-flow logic calls something non-deterministic directly (the current wall-clock time, a random number, an external API call made from within the "decide what happens next" logic rather than as a recorded step), replaying the same event history twice could compute a different next step each time, silently corrupting resumption. Real engines solve this by making all non-deterministic operations (getting the current time, generating a UUID, calling an external service) themselves go through the durable event log — recorded once, replayed as the recorded value forever after, never recomputed live during replay. Naming this constraint explicitly is what separates "I understand event sourcing" from "I understand exactly why the workflow definition must be written to be replay-safe."

A step that fails permanently (not a transient failure, but a genuine business failure — the customer's card is declined) needs an explicit compensation or rollback path, not just "retry forever" — if inventory was already reserved before the charge failed, the workflow needs a defined "release the reservation" compensating step, the same saga-pattern compensating-transaction idea that a distributed, multi-step business process without a single database transaction always eventually needs.

How to say it in the interview

"The hard constraint is that step three of a five-step workflow might have already charged a customer's card, and if the worker executing the workflow crashes right after, restarting from step one would charge them twice — so the entire design is built around never re-executing a step whose side effect already happened."

Checklist

Explained why this is a different problem from single-job scheduling, not a longer version of it
Used an append-only event history as the durable source of truth, not an overwritten current-state field
Distinguished replaying control flow from re-executing side effects during resumption
Required each individual step to carry its own idempotency key, on top of the engine's event-sourcing guarantee
Persisted timers and long delays as durable events rather than in-memory sleeps
Added periodic state snapshotting so replay cost doesn't grow unbounded with workflow age
Required all non-deterministic operations to go through the durable event log, not be computed live during replay
Added an explicit compensating-transaction path for permanent business failures, not just retry-forever

Practice this interactively