Async Programming

PythonAsyncBeginner22 min

By: Anacodic Team

TL;DR — asyncio gives you concurrency on one thread by letting a coroutine hand control back to an event loop whenever it would otherwise wait. It is a large win for I/O-bound work — thousands of open connections on one thread — and does nothing for CPU-bound work. The failure that matters in production is a synchronous call inside async def: it never yields, so it holds the loop for its whole duration and every other request serialises behind it. In the runnable example below, four 20 ms requests take 420–481 ms because one 400 ms blocking call is in front of them; asyncio.to_thread fixes it and the same work finishes in 406 ms. The reason this survives code review is that it is invisible in single-request testing — p50 looks perfect and only breaks under concurrency, which is exactly when you least want to be debugging it.

1. Simple explanation

A normal function runs from start to finish, and if it has to wait — for a network reply, a disk read, a database row — it just sits there holding the thread.

A coroutine can pause. When it reaches something it must wait for, it says "I'm waiting, use the CPU for something else," and the event loop — a scheduler that keeps a list of paused coroutines and what each is waiting for — runs another one. When the reply arrives, the original coroutine resumes.

Nothing runs in parallel. There is one thread and one thing executing at a time. The win is that waiting stops costing anything, so a program that spends 95% of its life waiting on the network can handle hundreds of connections on a single thread.

The critical word is cooperative. A coroutine only yields at an await. If you call something that blocks without awaiting, the loop cannot take control back — it is stuck until that call returns, and every other pending task waits.

Analogy — one waiter versus one chef. A single good waiter serves twenty tables because the work is mostly waiting: take an order, hand it to the kitchen, and while that dish cooks go and take another order. That is asyncio, and it scales beautifully. But if the waiter stops to personally chop vegetables for ten minutes, all twenty tables wait — not because the restaurant is busy, but because the one person who moves between tables is stuck doing work that doesn't yield. That is a blocking call in an async handler, and adding more tables makes it worse rather than better.

2. Diagram

SYNCHRONOUS                     ASYNCIO (cooperative, one thread)

 req A |==work==|                req A |=|.......waiting.......|=|
 req B          |==work==|       req B   |=|....waiting....|=|
 req C                   |==|    req C     |=|..waiting..|=|
       └── total = sum ──┘             └─ total ≈ the SLOWEST one ─┘
                                       (|=| = actually using the CPU)


THE BUG:  blocking call inside `async def`

  async def handler():
      time.sleep(0.4)        # or requests.get(...), or a sync DB driver
      ^^^^^^^^^^^^^^^ never yields -> the loop is FROZEN for 400 ms

  loop: |========= 400 ms, one task, nothing else can run =========|
                                                        ▲
        4 cheap 20 ms requests queued here ─────────────┘
        they finish at 420, 440, 461, 481 ms


THE FIX:  await asyncio.to_thread(blocking_fn)

  loop:  |=| free, servicing everyone else ...
  thread:    |======== 400 ms blocking work ========|
             ▲
             cheap requests finish at ~26 ms


CHOOSING A CONCURRENCY MODEL

  waiting on network/disk?   -> asyncio            (thousands of tasks)
  CPU-bound maths?           -> multiprocessing    (asyncio does NOT help)
  blocking library, no async
    version available?       -> asyncio.to_thread  (bridge to a thread)

3. How it works

3.1 Coroutines, the event loop, and `await`

async def defines a coroutine function. Calling it does not run it — it returns a coroutine object, which does nothing until something schedules it. That mismatch is the most common beginner error, and it usually shows up as a RuntimeWarning: coroutine was never awaited.

await does two things: it suspends the current coroutine, and it registers interest in something that will complete later. Control returns to the event loop, which runs other ready tasks. asyncio.run(main()) creates a loop, runs one coroutine to completion, and shuts down.

The mental model that avoids most confusion: await is a yield point, and yield points are the only places anything else can run. Code between two awaits runs uninterrupted.

3.2 Concurrency versus parallelism

Two different things that share a colloquial meaning:

Concurrency — several tasks in progress, interleaved on one thread. This is asyncio.
Parallelism — several tasks executing at literally the same instant, on several cores.

asyncio gives concurrency, not parallelism. For CPU-bound work — parsing a large file, numerical loops — there is no waiting to exploit, so asyncio adds overhead and no benefit. That work needs multiprocessing, or a library that releases the GIL internally.

3.3 Running things at once, and the difference between `gather` and `create_task`

Sequential awaits do not overlap. This is the second common error:

await fetch(a)      # waits for a to finish
await fetch(b)      # THEN starts b       -> total = a + b

To overlap them:

asyncio.gather(*coros) — start all of them, wait for all, get results in order. Use it when you need every result before continuing. By default one exception cancels the rest and propagates; return_exceptions=True collects them instead.
asyncio.create_task(coro) — schedule immediately and carry on. Use it for work you want running in the background while you do something else. You must eventually await or cancel the task; a task nobody holds a reference to can be garbage collected mid-flight, and an exception in a task nobody awaits is easy to lose.
asyncio.TaskGroup (3.11+) — the modern structured-concurrency form. Tasks are scoped to a async with block, and a failure cancels siblings deterministically. Prefer it over bare gather in new code.

3.4 Timeouts, cancellation, and limiting concurrency

Every network call in production needs a bound. asyncio.timeout() (3.11+) or asyncio.wait_for() cancels the operation when it overruns. Cancellation works by raising CancelledError inside the coroutine at its next yield point — so a coroutine that never awaits also cannot be cancelled, which is the same underlying property as §3.5.

Unbounded concurrency is its own failure. gather over ten thousand URLs opens ten thousand connections at once and takes down either you or the other end. asyncio.Semaphore caps it:

sem = asyncio.Semaphore(20)
async def limited(u):
    async with sem:
        return await fetch(u)

The other primitives — Lock, Event, Condition, Queue — mirror their threading equivalents but are not thread-safe; they coordinate coroutines on one loop. asyncio.Queue is the idiomatic producer/consumer channel.

3.5 The one that causes outages: sync inside async

A function declared async def that calls blocking code holds the loop for the full duration. Nothing else runs — not other requests, not health checks, not timeouts.

Three common sources, all easy to miss in review:

looks fine	actually blocks	use instead
`requests.get(url)`	yes	`httpx.AsyncClient`, `aiohttp`
a synchronous DB driver	yes	the async driver, or `to_thread`
`time.sleep(n)`	yes	`await asyncio.sleep(n)`
`open(f).read()` on a big file	yes	`to_thread`, or `aiofiles`

When no async version exists, bridge to a thread: await asyncio.to_thread(fn, *args) (3.9+), or loop.run_in_executor(None, fn) on older versions. Web frameworks offer the same bridge — FastAPI and Starlette provide run_in_threadpool, and notably a def endpoint is already run in a threadpool while an async def endpoint is not. Declaring a handler async def and then blocking inside it is strictly worse than leaving it synchronous.

3.6 Why this bug survives testing, and where async stops helping

Single-request testing cannot see it. One request through a blocking async handler takes exactly as long as it should; the latency only appears when a second request has to wait. So it passes local testing, passes review, and appears in production as a p95 that collapses under load while p50 stays healthy — a signature worth memorising, because it points at contention rather than at any single slow component.

Where asyncio stops being the answer: CPU-bound work (use processes), code you do not control that blocks (bridge to threads and accept the cost), and codebases where only part of the stack is async — mixing models is where most real complexity lives, and "async all the way down" is a real constraint rather than a style preference.

4. The math

4.1 Why concurrency wins on I/O

For n tasks each spending w waiting and c on CPU:

  sequential  T = n * (w + c)
  concurrent  T ≈ max(w) + n * c        (one thread, CPU work still serialises)

  speedup ≈ (w + c) / c   when w >> c   -> large for I/O, ~1 for CPU-bound

4.2 What a blocking call does to the budget

  correct    T ≈ max(t_i)           tasks overlap
  blocking   T ≈ sum(t_i)           tasks serialise -- the loop is held

  and the latency of the i-th queued request becomes
      L_i = sum(t_1 .. t_i)         i.e. it inherits everyone ahead of it

4.3 Worked example

Five concurrent requests: one 400 ms model call and four 20 ms lookups. Ideally everything finishes inside 400 ms, because the cheap ones fit in the slow one's waiting time.

Blocking version — sum behaviour, and the cheap requests inherit the queue:

    model call   wanted   400 ms, took    400 ms
    cheap #1     wanted    20 ms, took    420 ms  <- QUEUED behind the slow one
    cheap #2     wanted    20 ms, took    440 ms  <- QUEUED
    cheap #3     wanted    20 ms, took    461 ms  <- QUEUED
    cheap #4     wanted    20 ms, took    481 ms  <- QUEUED
    WALL CLOCK                  481 ms

Each cheap request is 21x slower than the work it actually did, purely from waiting. With asyncio.to_thread:

    model call   wanted   400 ms, took    406 ms
    cheap #1     wanted    20 ms, took     26 ms
    cheap #2     wanted    20 ms, took     27 ms
    cheap #3     wanted    20 ms, took     27 ms
    cheap #4     wanted    20 ms, took     27 ms
    WALL CLOCK                  406 ms

481 ms → 406 ms wall clock, which is the small win. The large win is per-request: 481 ms → 27 ms for the cheap requests, a 17x improvement for the users who were only doing 20 ms of work. Note that with only five tasks the wall-clock difference looks unimpressive — scale to fifty concurrent requests and the blocking version's total grows linearly while the fixed version stays near 400 ms.

5. Real code

"""One blocking call inside `async def` stalls every other request on the loop."""
import asyncio
import time


def blocking_work(seconds: float) -> None:
    """Stands in for a synchronous DB driver, `requests.get`, or a sync SDK call."""
    time.sleep(seconds)


async def handler_wrong(t0: float, seconds: float) -> float:
    """Declared async, but calls blocking code directly -- freezes the event loop."""
    blocking_work(seconds)                      # <- the bug, in one line
    return time.perf_counter() - t0             # measured from SUBMISSION


async def handler_right(t0: float, seconds: float) -> float:
    """Hands the blocking call to a worker thread so the loop stays free."""
    await asyncio.to_thread(blocking_work, seconds)
    return time.perf_counter() - t0


async def serve(handler, label: str) -> None:
    """Five concurrent requests: one slow model call plus four cheap lookups.

    Latency is measured from the moment all five were submitted, which is what
    the caller experiences -- not from when the coroutine happened to get the CPU.
    """
    work = [("model call", 0.40)] + [(f"cheap #{i}", 0.02) for i in range(1, 5)]
    wall_start = time.perf_counter()
    waits = await asyncio.gather(*(handler(wall_start, s) for _n, s in work))
    wall = time.perf_counter() - wall_start

    print(f"{label}")
    for (name, want), got in zip(work, waits):
        queued = got - want
        flag = "  <- QUEUED behind the slow one" if queued > 0.05 else ""
        print(f"    {name:<12} wanted {want*1000:>5.0f} ms, "
              f"took {got*1000:>6.0f} ms{flag}")
    print(f"    {'WALL CLOCK':<12} {wall*1000:>18.0f} ms\n")
    return wall


async def main() -> None:
    wrong = await serve(handler_wrong, "BLOCKING inside async def  (the bug)")
    right = await serve(handler_right, "asyncio.to_thread          (the fix)")

    ideal = 0.40   # everything should finish inside the slowest single task
    print(f"ideal wall clock (all concurrent) : {ideal*1000:.0f} ms")
    print(f"blocking version                  : {wrong*1000:.0f} ms  "
          f"({wrong/ideal:.1f}x)")
    print(f"to_thread version                 : {right*1000:.0f} ms  "
          f"({right/ideal:.1f}x)")

    # The blocking version serialises: total is the SUM of every task.
    assert wrong > 0.45, wrong
    # The fixed version overlaps: total is close to the SLOWEST task.
    assert right < 0.45, right
    assert wrong > right
    print("\nNote the shape: p50 looks fine in BOTH runs if you only measure the")
    print("cheap requests alone. The bug only appears under concurrency.")
    print("all assertions passed")


asyncio.run(main())

# Output:
#   BLOCKING inside async def  (the bug)
#       model call   wanted   400 ms, took    400 ms
#       cheap #1     wanted    20 ms, took    420 ms  <- QUEUED behind the slow one
#       cheap #2     wanted    20 ms, took    440 ms  <- QUEUED behind the slow one
#       cheap #3     wanted    20 ms, took    461 ms  <- QUEUED behind the slow one
#       cheap #4     wanted    20 ms, took    481 ms  <- QUEUED behind the slow one
#       WALL CLOCK                  481 ms
#
#   asyncio.to_thread          (the fix)
#       model call   wanted   400 ms, took    406 ms
#       cheap #1     wanted    20 ms, took     26 ms
#       cheap #2     wanted    20 ms, took     27 ms
#       cheap #3     wanted    20 ms, took     27 ms
#       cheap #4     wanted    20 ms, took     27 ms
#       WALL CLOCK                  406 ms
#
#   ideal wall clock (all concurrent) : 400 ms
#   blocking version                  : 481 ms  (1.2x)
#   to_thread version                 : 406 ms  (1.0x)
#
#   Note the shape: p50 looks fine in BOTH runs if you only measure the
#   cheap requests alone. The bug only appears under concurrency.
#   all assertions passed

Millisecond values vary a little between runs — the ordering and the ratios are the stable part. Raise the task count and the gap widens: the blocking total grows linearly, the fixed one does not.

6. Real-world example

A team added a feature to an existing web service: one endpoint that called a slow external API taking two to four seconds. They declared the handler async def, used the vendor's Python SDK, and it worked in testing.

The service had been comfortably handling a few hundred requests per second on other endpoints. Within a day of the new endpoint going live, unrelated endpoints started timing out. Not the new one — the cheap existing ones, which had been reliably answering in under 30 ms.

The vendor SDK was synchronous. Each call to the new endpoint occupied the event loop for its full two to four seconds, during which nothing else on that worker ran: no other requests, no health checks. The health checks were the loudest symptom, because the orchestrator marked workers unhealthy and restarted them, which shifted load to the remaining workers and made it worse.

What made diagnosis slow was that every dashboard pointed at the wrong place. The new endpoint's own latency was normal — two to four seconds was expected. The alerts were on the old endpoints, which had not changed. And p50 across the service stayed acceptable, because most requests still landed on a worker that happened not to be blocked; only p95 and p99 exploded. That gap between a healthy median and a collapsed tail is the signature.

The immediate fix was one line — wrap the SDK call in asyncio.to_thread — plus sizing the threadpool for the expected concurrency. The durable fix was a lint rule banning known-blocking calls inside async def, because the same mistake had already been made twice with a different library.

7. Interview questions companies actually ask

Q1. What is the difference between async and await? async def declares a coroutine function; calling it returns a coroutine object and runs no code until something schedules it. await suspends the current coroutine at that point and hands control to the event loop, which runs other ready tasks until the awaited thing completes. The useful framing is that await marks the only places where anything else can run — code between two awaits is uninterruptible.

Q2. How does asyncio differ from threading? asyncio is cooperative and single-threaded: tasks yield at explicit await points, so you know precisely where a switch can happen and you rarely need locks for your own state. Threads are preemptive: the OS can switch at almost any instruction, so shared mutable state needs synchronisation. asyncio scales much further for I/O because a suspended coroutine is far cheaper than a blocked thread, but a single non-yielding call stalls everything — where a blocked thread only stalls itself.

Q3. When would you use gather versus create_task? gather when you need all the results before continuing — it starts everything, waits for everything, and returns results in argument order. create_task when you want work running in the background while you carry on, in which case you must keep a reference and eventually await or cancel it, or the task can be garbage collected mid-flight and its exception lost. In modern code prefer TaskGroup, which scopes tasks to a block and cancels siblings deterministically on failure.

Q4. What happens if you call a blocking function inside an async handler? It holds the event loop for its entire duration, so every other task on that loop waits, including health checks and timeout enforcement. Latency for queued requests becomes the cumulative sum of everything ahead of them. The fix is an async-native library, or asyncio.to_thread to bridge to a worker thread. Worth knowing that in FastAPI or Starlette a plain def endpoint already runs in a threadpool, so declaring a handler async def and then blocking is worse than leaving it synchronous.

Q5. Why does this bug pass testing? Because it needs concurrency to manifest. A single request through a blocking async handler takes exactly the time it should, so functional tests and manual checks pass. The symptom only appears when a second request has to wait, which is why it shows up in production as a healthy p50 with a collapsed p95 — a signature that indicates contention rather than a single slow dependency.

Q6. Does asyncio help with CPU-bound work? No. Concurrency exploits waiting, and CPU-bound work is not waiting — so there is nothing to interleave, and you pay event-loop overhead for no benefit. A tight numerical loop in a coroutine blocks the loop exactly like time.sleep does. Use multiprocessing, a process pool, or a library that releases the GIL internally.

Q7. How do you stop a gather over ten thousand URLs from destroying something? Bound the concurrency with asyncio.Semaphore, acquired inside each task, so only N are in flight at once. Add a per-request timeout with asyncio.timeout or wait_for, and decide explicitly whether one failure should cancel the rest — gather propagates and cancels siblings by default, return_exceptions=True collects them instead. Unbounded concurrency is a denial-of-service against yourself or the other end.

8. When to use / tradeoffs

Reach for asyncio when:

The work is I/O-bound — network calls, database queries, file and socket work
You need many simultaneous operations, especially many idle connections
You are writing a service that fans out to several backends per request
The libraries in your path have async-native versions

Reach for something else when:

The work is CPU-bound — use multiprocessing or a process pool
A key library is blocking with no async version — bridge with to_thread, or stay synchronous
The program is a simple script where sequential code is clearer and fast enough

Situation	Why it breaks	Use instead
`requests` / sync SDK in `async def`	Holds the loop for the whole call	`httpx.AsyncClient`, or `to_thread`
`time.sleep` in a coroutine	Blocks; does not yield	`await asyncio.sleep`
CPU-bound loop in a coroutine	Nothing to interleave; loop is held	`multiprocessing`
`gather` over a huge list	Unbounded concurrency	`Semaphore` to cap in-flight work
Sequential `await`s expecting overlap	Each waits for the previous	`gather` or `TaskGroup`
`async def` endpoint that blocks	Worse than a plain `def` endpoint	Remove `async`, or `run_in_threadpool`
Fire-and-forget `create_task`	Task can be collected; exception lost	Keep a reference; await or cancel

Honest limits. The measurements in §5 use time.sleep as a stand-in for real I/O, which is honest about the loop-blocking property but omits everything else: real network calls have variable latency, connection setup, DNS, and retries, so absolute numbers will not transfer. Millisecond values also shift between runs — the ratios are the stable finding. The five-task example understates the problem: with so few tasks the wall-clock difference is only about 1.2x, and the effect grows with concurrency, so do not read the small number as a small problem. to_thread is a bridge, not a cure — it costs a thread per concurrent call, so a threadpool sized for ten will queue at eleven, and it does not make a blocking library good. Finally, this article assumes one event loop in one process; production async services usually run several workers, which changes how contention presents and can make a blocked loop look like an intermittent problem rather than a consistent one.

asyncio gives concurrency, not parallelism: one thread, tasks interleaved at await points. It makes waiting free.
Calling an async def function does not run it. await is the only place another task can run.
Sequential awaits do not overlap — use gather, create_task, or preferably TaskGroup.
The production bug is a blocking call inside async def. It holds the loop, so queued requests inherit everything ahead of them: four 20 ms requests took 420–481 ms behind one 400 ms blocking call.
asyncio.to_thread fixes it: the same work finished in 406 ms, and the cheap requests went from 481 ms to 27 ms — a 17x improvement for those users.
It survives testing because it needs concurrency to appear. The signature is healthy p50, collapsed p95.
In FastAPI/Starlette a plain def endpoint already runs in a threadpool — async def plus blocking is worse than not being async at all.
Bound concurrency with Semaphore and always set timeouts. A coroutine that never awaits also cannot be cancelled.
asyncio does nothing for CPU-bound work, and to_thread is a bridge with a thread cost, not a cure.

Cost-Latency Tradeoffs — where async fits among cost and latency levers, and why it changes who waits
LLM Observability — the p50/p95 split that reveals this bug in production
Voice Agent Architectures: Cascaded vs Speech-to-Speech — a real-time system where a blocked loop is immediately audible
Production Agents — scaling, error handling, and retries around async calls
ML Inference Systems — serving concurrency and capacity
API Best Practices — timeouts and retries for the calls you are awaiting

Resources

Python asyncio documentation — the authoritative reference; the "Developing with asyncio" page covers debug mode, which reports coroutines that block the loop: https://docs.python.org/3/library/asyncio-dev.html
asyncio.to_thread — the bridge for blocking calls, added in 3.9: https://docs.python.org/3/library/asyncio-task.html#asyncio.to_thread
asyncio.TaskGroup — structured concurrency, added in 3.11, preferable to bare gather in new code: https://docs.python.org/3/library/asyncio-task.html#task-groups
Hettinger, R. — Keynote: Concurrency from the Ground Up and the wider PyCon concurrency talks; good grounding in why cooperative scheduling behaves as it does.
Ramalho, L. — Fluent Python (2nd ed.), Ch. 19–21 on concurrency models, asyncio, and when each applies.
Starlette concurrency documentation — run_in_threadpool, and the distinction between def and async def endpoints: https://www.starlette.io/threadpool/
Real Python — Async IO in Python: A Complete Walkthrough — a gentler introduction than the standard library docs: https://realpython.com/async-io-python/

Next: Python Fundamentals →