TL;DR — A generic "job started / job finished" progress stream can't tell a user which phase of a multi-step job is running, can't distinguish an ordinary phase transition from a pause that's actually waiting on a human, and loses everything a client missed while briefly disconnected. The fix is three design choices working together: name every event after the specific phase it belongs to (not a generic "start"/"done"), give a paused-for-human-input state its own distinct event type rather than overloading the normal phase events, and persist every event to a durable, replayable log before pushing it live, so a reconnecting client can ask "what did I miss since event N" instead of losing state. In the harness below, a reconnecting client recovers all 5 events it missed — including the one signaling a human decision was needed — by replaying from the durable log rather than only listening live. This design stops paying for itself for jobs short enough that a simple spinner and a final result are all a user actually needs.
1. Simple explanation
A long-running job — grading a batch of submissions, processing a large document, running a multi-step pipeline — needs to show a user more than "loading." A bare progress stream that only emits generic start and done events tells a client that something began and something finished, but not what, and if the job pauses partway through to wait for a human decision, a generic stream has no way to say so — from the client's perspective, an ordinary phase taking a long time and a pause waiting indefinitely for a person look identical.
Analogy — a delivery tracker versus a blinking "in progress" light. A blinking light on a package that just says "in progress" tells you nothing about where the package actually is or whether anything has gone wrong — it looks the same whether the package is five minutes from arriving or stuck at customs waiting for a human to release it. A real delivery tracker names each stage — "left warehouse," "arrived at facility," "held for customs review" — and that last one is not just another stage with a different label; it needs its own distinct status, because it means something categorically different (waiting on an external decision, not just time passing) and the tracker should tell you that plainly rather than showing the same generic "in progress" dot it showed for ordinary transit.
2. Diagram
LAYER 1 ONLY: generic events LAYER 2: named events + interrupt
{"type": "start"} {"type": "ingest_start"}
{"type": "done"} {"type": "ingest_done"}
{"type": "start"} {"type": "grade_start"}
{"type": "done"} {"type": "interrupt",
{"type": "start"} "gate": "low_confidence_review",
{"type": "done"} "message": "awaiting human..."}
{"type": "grade_resumed",
Client CANNOT tell which "decision": "override_approved"}
phase #2's start/done {"type": "grade_done"}
belongs to without {"type": "persist_start"}
separately counting calls. {"type": "persist_done"}
Client can build a REAL timeline,
and can tell "paused for a human"
apart from "just running long."
DURABLE LOG + RECONNECTION
every event --append--> [durable log, indexed 0, 1, 2, ...] --push--> live client
|
| client disconnects after index 2
v
client reconnects: "give me everything since index 2"
|
v
log.since(2) -> replays events 3..7, including
the interrupt event the client would otherwise
have silently missed
MEASURED (harness in §5): client missed 5 of 8 total events while
disconnected; all 5, including the interrupt, were recovered by replay.
3. How it works
3.1 Name the event after the phase, not after the transition type
A generic event scheme has exactly two shapes, start and done, and relies on the client already knowing, from context, which phase each one belongs to. That's fragile the moment a job has more than one phase running in any order other than perfectly predictable strict sequence, and it makes the client's rendering logic depend on carefully counting events rather than reading them. Naming each event after its specific phase (ingest_start, grade_start, persist_done) means the event itself carries everything a client needs to render a real timeline — no external bookkeeping required, no assumptions about ordering baked into the client.
3.2 A pause waiting on a human is not just a slow phase — give it its own event type
The single most important distinction a progress stream can make is between "still working, just taking a while" and "stopped, waiting on someone to make a decision." These look identical from a bare timer's perspective but require completely different UI treatment — a spinner is the wrong thing to show for an indefinite wait on a human, and a banner asking someone to act is the wrong thing to show for ordinary in-progress work. Giving the paused state its own distinct event type (interrupt, carrying which gate triggered it and a human-readable reason) lets the client render the right UI for the right situation without inferring it from how long a phase has been "running."
3.3 Persist before you push, so reconnection means replay, not loss
A live-only stream — one where events are pushed to connected clients and then discarded — means anything that happens while a client is disconnected (a dropped connection, a tab backgrounded and throttled, a network blip) is simply gone from that client's point of view once it reconnects. Writing every event to a durable, indexed, append-only log before pushing it live means a reconnecting client can ask a well-defined question — "give me everything after the last event index I actually received" — and get a complete, ordered replay of what it missed, rather than either silently skipping ahead with gaps or replaying the entire job from the start.
Where this stops working: the value of named events, interrupt distinction, and durable replay all scale with how long the job runs and how much a user needs to understand its internal state. For a job that completes in under a second or two, none of this machinery earns its cost — a simple spinner followed by a final result is simpler to build, simpler to reason about, and gives the user nothing meaningfully worse, since there's no meaningful intermediate state worth surfacing in the first place.
4. The math
There's no formula to derive here — the measurable claim is about information recoverable after a gap, not a numeric relationship. With a durable log indexed from 0, a client that last confirmed receiving event at index k and reconnects can recover exactly len(log) - (k + 1) events via log.since(k), with no approximation and no loss, provided every event was durably appended before being pushed live. In the harness in §5, a client disconnected after index 2 in an 8-event log recovers exactly 8 - 3 = 5 events on reconnection — a number that matches precisely, because replay from a durable log is exact recovery, not a best-effort approximation.
5. Real code
class DurableEventLog:
"""Every event is appended here BEFORE being pushed to any live
listener, so a client that reconnects can replay everything it missed
instead of losing progress that happened while it was disconnected."""
def __init__(self):
self.events = []
def append(self, event):
self.events.append(event)
return len(self.events) - 1 # this event's index in the log
def since(self, last_seen_index):
"""What a reconnecting client asks for: everything after the last
event index it actually received."""
return self.events[last_seen_index + 1:]
def run_pipeline_generic(log):
"""Layer 1 only: generic start/done events with no semantic meaning.
A client seeing these cannot tell WHICH phase is running without
separately tracking call order itself."""
for _ in ["ingest", "grade", "persist"]:
log.append({"type": "start"})
log.append({"type": "done"})
def run_pipeline_named_with_interrupt(log):
"""Layer 2: named phase events PLUS a distinct interrupt event type.
A client can render a real progress UI directly from the event
stream, and can tell an ordinary phase transition apart from a
"waiting on a human" pause without any side-channel state."""
log.append({"type": "ingest_start"})
log.append({"type": "ingest_done"})
log.append({"type": "grade_start"})
log.append({
"type": "interrupt",
"gate": "low_confidence_review",
"message": "Confidence below threshold; awaiting human decision.",
})
log.append({"type": "grade_resumed", "decision": "override_approved"})
log.append({"type": "grade_done"})
log.append({"type": "persist_start"})
log.append({"type": "persist_done"})
def render_phase_timeline(events):
"""What a client can reconstruct from named events alone: a real
per-phase timeline, including which phase paused for a human."""
timeline = []
for e in events:
t = e["type"]
if t.endswith("_start"):
timeline.append(f"{t.removesuffix('_start')}: started")
elif t.endswith("_done"):
timeline.append(f"{t.removesuffix('_done')}: finished")
elif t == "interrupt":
timeline.append(f"PAUSED -- {e['gate']}: {e['message']}")
elif t.endswith("_resumed"):
timeline.append(f"resumed: {e['decision']}")
return timeline
generic_log = DurableEventLog()
run_pipeline_generic(generic_log)
print("=== Layer 1 only: generic events ===")
for e in generic_log.events:
print(" ", e)
print("(a client cannot tell which phase 'start'/'done' #2 refers to "
"without separately counting calls itself)")
named_log = DurableEventLog()
run_pipeline_named_with_interrupt(named_log)
print("\n=== Layer 2: named events + interrupt ===")
for line in render_phase_timeline(named_log.events):
print(" ", line)
print("\n=== Reconnection: a client disconnects after event index 2 ===")
client_last_seen = 2
print(f"client's last confirmed event index: {client_last_seen}")
missed = named_log.since(client_last_seen)
print(f"events the client missed while disconnected: {len(missed)}")
for e in missed:
print(" ", e)
assert len(missed) == len(named_log.events) - (client_last_seen + 1)
replayed_timeline = render_phase_timeline(missed)
assert any("PAUSED" in line for line in replayed_timeline)
print("\nassert passed: replay from the durable log recovers the interrupt "
"event the client would otherwise have silently missed while "
"disconnected -- including the fact a human decision was needed")
# Output:
# === Layer 1 only: generic events ===
# {'type': 'start'}
# {'type': 'done'}
# {'type': 'start'}
# {'type': 'done'}
# {'type': 'start'}
# {'type': 'done'}
# (a client cannot tell which phase 'start'/'done' #2 refers to without separately counting calls itself)
#
# === Layer 2: named events + interrupt ===
# ingest: started
# ingest: finished
# grade: started
# PAUSED -- low_confidence_review: Confidence below threshold; awaiting human decision.
# resumed: override_approved
# grade: finished
# persist: started
# persist: finished
#
# === Reconnection: a client disconnects after event index 2 ===
# client's last confirmed event index: 2
# events the client missed while disconnected: 5
# {'type': 'interrupt', 'gate': 'low_confidence_review', 'message': 'Confidence below threshold; awaiting human decision.'}
# {'type': 'grade_resumed', 'decision': 'override_approved'}
# {'type': 'grade_done'}
# {'type': 'persist_start'}
# {'type': 'persist_done'}
#
# assert passed: replay from the durable log recovers the interrupt event the client would otherwise have silently missed while disconnected -- including the fact a human decision was needed
Both asserts passed on the run that produced this output: the count of missed events matches the log-length arithmetic exactly, and the replayed events are confirmed to include the interrupt — the one event a client absolutely cannot afford to silently miss, since it represents a job waiting indefinitely on a person who doesn't yet know they need to act.
6. Real-world example
A team's first version of a progress UI for a multi-step job subscribed to a live event stream and rendered a simple spinner that turned into a checkmark on the final "done" event. It worked well in testing, where connections were stable and jobs ran on a fast local network. In production, mobile clients on flaky connections would occasionally drop the stream partway through a job and reconnect a few seconds later — and because events were only ever pushed live, never persisted, a reconnecting client had no way to know what had happened while it was disconnected. It would simply resume showing a spinner, with no information about whether the job was still running, had finished, or was paused waiting on a decision.
The worst version of this failure happened when a job paused for human review exactly during a client's brief disconnection. The user's client reconnected to a job that was, from the server's perspective, sitting idle waiting for a decision — but the client had no record that an interrupt had ever occurred, since that event was pushed while nobody was listening. The user saw an indefinitely spinning progress indicator with no explanation, assumed the job was broken, and in one recorded case, closed the tab and re-submitted the same job entirely, creating a duplicate that later needed manual cleanup.
The fix was exactly the durable-log-plus-replay design in §5: every event, including interrupts, gets written to a persistent, indexed log before being pushed to any live listener, and a reconnecting client's first action is to ask for everything since its last confirmed event index. After the change, the exact disconnect-during-interrupt scenario that caused the duplicate submission was retested deliberately and confirmed to recover correctly — the reconnecting client received the interrupt event on reconnection and rendered the correct "waiting for your review" state instead of a bare spinner.
7. Interview questions companies actually ask
Q1. Why isn't a generic "job started, job finished" event pair sufficient for a multi-phase long-running job? Because it gives a client no way to know which phase is currently running, how far through the job it is, or whether an unusually long wait is normal progress or something that needs the user's attention — all a client can show is an undifferentiated "in progress" state for the entire duration of a job that might have many meaningfully different stages.
Q2. Why does a human-review pause need its own distinct event type instead of just being a longer "phase in progress" period? Because the correct client behavior is completely different for the two cases: an ordinary phase taking a while should show progress or a spinner, while a pause waiting on a human decision should show an actionable prompt telling someone what's needed. Without a distinct event type, the client has no reliable signal for which UI to show, and defaults to treating an indefinite human wait the same as ordinary processing time.
Q3. What specifically breaks if events are only ever pushed to live listeners and never persisted? Any client that disconnects, even briefly, permanently loses every event that occurred during the disconnection — there's no way to ask "what did I miss," because nothing was recorded to answer that question. On reconnection, the client either has to assume nothing happened (wrong, and dangerous specifically when an interrupt occurred during the gap), skip ahead with an unexplained jump, or restart tracking from scratch.
Q4. How would you design the reconnection protocol between client and server for this kind of stream? The client should track the index (or a similar durable identifier) of the last event it actually received, and on reconnection, request everything after that index specifically, rather than either a full replay from the beginning or a live-only resubscription. This requires the server to keep a durable, indexed log of every event, appended to before being pushed live, so "everything since index N" is a well-defined, exactly-answerable query.
Q5. In the real-world example, a client saw an indefinitely spinning indicator during a missed interrupt and the user re-submitted the same job, creating a duplicate. Whose fault was that, architecturally? It's the streaming design's fault, not the user's — the user had no information available to them that would have suggested waiting was correct, since the one event that would have told them a decision was needed (the interrupt) was pushed while they were disconnected and never recorded anywhere they could later retrieve it. A user acting reasonably on the information available to them, given a design that fails to preserve that information, isn't a user error.
Q6. When would this whole design — named events, interrupt distinction, durable log, reconnection replay — be overkill? For a job that completes quickly enough that a simple spinner-then-result experience serves the user just as well as a detailed phase-by-phase timeline — if there's no meaningful intermediate state to show and no realistic chance of a client disconnecting mid-job in a way that matters, the added durability and event-naming machinery costs more to build and maintain than it returns in user experience.
8. When to use / tradeoffs
Reach for named events, interrupt distinction, and durable replay when:
- a job has multiple distinct phases a user would benefit from seeing named, not just a single undifferentiated "in progress" state
- the job can pause and wait on a human decision, and that state needs to be visually distinguishable from ordinary processing
- clients are expected to disconnect and reconnect during a job's lifetime (mobile networks, long-running jobs spanning minutes or hours)
| Situation | Why it breaks | Use instead |
|---|---|---|
| The job completes in a second or two with no meaningful intermediate state | The event-naming, interrupt-typing, and durable-log machinery costs more to build than it returns | A simple spinner and a final result |
| Events are pushed live only, never persisted | Any client disconnection permanently loses whatever happened during the gap, including interrupts | A durable, indexed log appended to before any live push |
| The job never pauses for human input at any point | There's no case that needs the interrupt/ordinary-phase distinction | Named phase events alone, without a separate interrupt type |
| Clients are always on stable, uninterrupted connections for the job's full duration | Reconnection-replay logic is complexity solving a problem that doesn't occur in this environment | A live-only stream is sufficient |
Honest limits. Named events and interrupt distinction only help if the client actually renders them differently — building a rich event vocabulary that a UI then displays as one generic "loading" indicator regardless of event type captures none of the design's actual benefit. Durable logging adds real storage and write cost per event, proportional to how granular and long-running the job is, and for extremely high-frequency events, logging every single one durably may itself become the bottleneck, requiring batching or sampling that has to be designed deliberately rather than assumed free. And reconnection-replay assumes the client can reliably track and report its own last-seen event index; a client that can't do this correctly gets no benefit from the server-side durability no matter how well the log itself is designed.
9. Summary + related articles
- A bare "start/done" progress stream can't tell a client which phase is running or distinguish an ordinary wait from a pause on a human decision.
- Naming events after their specific phase, and giving human-review pauses their own distinct event type, lets a client build a real, differentiated status display directly from the event stream.
- Persisting every event to a durable, indexed log before pushing it live turns reconnection into an exact replay ("everything since index N") instead of a silent loss of whatever happened during a disconnection.
- Measured: a client disconnected after event 2 of 8 recovered exactly the 5 missed events on reconnection, including the interrupt event signaling a human decision was needed.
- Boundary: none of this is worth building for jobs short enough, or simple enough, that a spinner and a final result already serve the user just as well.
Related:
- Effective Access: Combining Two Levels of Role-Based Permissions — a different system-design pattern from the same underlying platform this progress-streaming design was built for
- Consistency Memory: Making Repeated AI Judgments Agree With Each Other — another case where persisting state (a stored judgment, here a durable event log) rather than relying on live-only, in-memory behavior is what makes a system reliable across repeats or gaps
