API Best Practices

TL;DR — Every LLM provider (and every pricing tier within a provider) imposes its own practical limits on request size, output length, and request rate. A production client has to adapt to the active provider's real limits rather than hardcoding one provider's numbers, truncate oversized input safely without corrupting the text encoding, and retry rate-limit errors with exponential backoff plus jitter rather than hammering an already-limited API. This stops helping the moment completeness is a hard requirement — truncation is inherently lossy, and no retry strategy turns a persistent failure into a success.

---

1. Simple explanation

Different LLM providers — and different pricing tiers of the same provider — accept different maximum request sizes, allow different maximum output lengths, and enforce different rate limits. Code that assumes one provider's numbers works perfectly until the day it talks to a different provider or a cheaper tier, at which point it either gets rejected outright or, worse, silently sends less content than it thinks it's sending.

Analogy — packing a box for whichever courier shows up. You don't design one box size and hope every courier accepts it. Each courier publishes its own maximum weight and dimensions, and a good shipping process checks which courier is actually coming today and packs accordingly — leaving out the least essential items first if everything doesn't fit, rather than taping the box shut and hoping it isn't weighed. If the same package gets handed to a different courier tomorrow with a smaller van, the packing decision has to be made again, not reused from yesterday.

---

2. Diagram

REQUEST PIPELINE

  build request (instructions + context)
              |
              v
  which provider/tier is ACTIVE right now?
              |
              v
   +-----------------------------+
   |  fit_to_provider(request)   |
   |  - look up THIS provider's  |
   |    byte/token limit          |
   |  - if it fits: send as-is    |
   |  - if it doesn't: truncate,  |
   |    keeping instructions and  |
   |    highest-priority content, |
   |    cut on a clean UTF-8      |
   |    boundary                  |
   +---------------+--------------+
                   |
                   v
              send request
                   |
        +----------+----------+
        |                     |
   200 OK                 429 rate limited
        |                     |
        v                     v
     done              wait = base * 2^attempt + jitter
                              |
                              v
                        retry (up to max_retries)
                              |
                              v
                    still failing -> surface the
                    error to the caller; do NOT
                    retry forever

---

3. How it works

3.1 Provider limits differ, and hardcoding one set of numbers breaks silently

A cheaper or faster tier typically accepts a smaller request and produces a shorter response than a premium tier of the same provider, and a different provider entirely may structure its limits around token counts rather than raw bytes. Code that hardcodes "my input can be up to N bytes" is really encoding "my input can be up to N bytes for the provider I tested against." The same lesson shows up in Chunk Size, Overlap, and Retrieval Thresholds in RAG Systems: a number tuned against one system's behavior is a property of that system, not a universal constant, and it has to be re-derived — not reused — the moment the underlying system changes.

The fix is to look up the active provider's limits as configuration, not to bake a single provider's numbers into request-building logic. This also means limits need to live somewhere they're easy to update — a rate-limit change on the provider's side should be a one-line configuration edit, not a code change scattered across every call site.

3.2 Safe truncation means finding a clean boundary, not just slicing bytes

Text is usually stored and measured in bytes, but a single character can occupy more than one byte in UTF-8 — many accented characters, symbols, and non-Latin scripts take two, three, or four bytes each. Slicing a byte string at an arbitrary position can land in the middle of one of those multi-byte characters, producing a decode error or corrupted output at exactly the boundary. Safe truncation cuts at a byte limit and then backs up, byte by byte if necessary, until what remains decodes cleanly.

There is a second decision hiding inside "truncate": what gets cut first. The naive approach chops whatever comes last, but a better design keeps the instructions (which tell the model what to do) and the highest-priority content intact, and drops the lowest-priority material first — older context before newer, background material before the specific question being asked. Truncation that isn't priority-aware can silently remove the one piece of context the request actually needed while keeping padding that didn't matter.

3.3 Retrying rate limits needs backoff, jitter, and a stopping point

A request rejected for exceeding a rate limit will usually succeed if retried a moment later, but retrying immediately just resubmits into the same limit and fails again. Exponential backoff doubles the wait after each failed attempt, and adding a small random jitter on top prevents many clients that all got rate-limited at the same moment from all retrying at the exact same moment again, which would just recreate the burst that caused the problem. Every retry loop needs a maximum attempt count: without one, a persistent failure (as opposed to a transient one) turns into an indefinite hang instead of a clear, surfaced error the caller can act on.

Where this stops working: truncation is a lossy operation, full stop. If a request has a completeness requirement — every piece of context genuinely must be considered — the correct response to an oversized request is not "truncate and hope," it's to split the work into multiple smaller requests and combine the results, or to reduce what's being asked rather than how it's packaged. Similarly, retrying with backoff only helps for transient failures; a request that's rejected because it's malformed, unauthorized, or genuinely too large even after truncation will fail identically on every retry, and backoff just delays discovering that.

---

4. The math

4.1 The backoff formula

delay(attempt) = base_delay * 2^attempt + jitter

where attempt starts at 0 for the first retry (not the original request), and jitter is a small random value added so simultaneous clients don't retry in lockstep.

4.2 Worked example, from the actual run in §5

With base_delay = 0.5 seconds:

attempt 0:  0.5 * 2^0 = 0.5   -> measured delay 0.532s  (jitter ~0.032s)
attempt 1:  0.5 * 2^1 = 1.0   -> measured delay 1.015s  (jitter ~0.015s)

Both measured delays match the formula plus a small jitter term, confirmed against the real run.

4.3 Worst-case total wait

With max_retries = 4 and base_delay = 0.5, the worst-case total time spent waiting (ignoring jitter, which is small) before giving up is the sum of the geometric series:

total = base_delay * (2^0 + 2^1 + 2^2 + 2^3)
      = 0.5 * (1 + 2 + 4 + 8)
      = 0.5 * 15
      = 7.5 seconds

That's the number a caller should actually budget for — not "a retry," but up to roughly 7.5 seconds of accumulated waiting if every attempt hits the rate limit, before the client gives up and surfaces the failure.

---

5. Real code

import random


PROVIDER_LIMITS = {
    "fast-tier":    {"max_input_bytes": 3500,  "max_output_tokens": 1024},
    "premium-tier": {"max_input_bytes": 12000, "max_output_tokens": 4096},
}


def truncate_utf8_bytes(text, max_bytes):
    """Truncate to at most max_bytes, never splitting a multi-byte UTF-8
    character in half."""
    encoded = text.encode("utf-8")
    if len(encoded) <= max_bytes:
        return text, False
    cut = encoded[:max_bytes]
    while True:
        try:
            return cut.decode("utf-8"), True
        except UnicodeDecodeError:
            cut = cut[:-1]


def build_prompt(instructions, context_docs):
    parts = [instructions, ""]
    for i, doc in enumerate(context_docs):
        parts.append(f"[source {i}] {doc}")
    return "\n".join(parts)


def fit_to_provider(prompt, provider):
    limit = PROVIDER_LIMITS[provider]["max_input_bytes"]
    fitted, was_truncated = truncate_utf8_bytes(prompt, limit)
    return fitted, was_truncated, len(prompt.encode("utf-8")), limit


class RateLimited(Exception):
    pass


def call_with_backoff(fn, max_retries=4, base_delay=0.0, rng=None):
    """Retry fn() on RateLimited with exponential backoff. base_delay=0.0
    would make the demo instant; a real client uses a real delay."""
    rng = rng or random.Random(7)
    attempt = 0
    delays = []
    while True:
        try:
            return fn(), attempt, delays
        except RateLimited:
            if attempt >= max_retries:
                raise
            delay = base_delay * (2 ** attempt) + rng.uniform(0, 0.1)
            delays.append(round(delay, 3))
            attempt += 1


instructions = "Answer the operator's question using only the sources below."
context_docs = [
    "The staging cluster autoscaler targets 65% average CPU across nodes "
    "and adds one node per scale-up event, waiting 180 seconds between "
    "events to avoid thrashing on transient spikes.",
    "Node drains during a scale-down wait for in-flight requests to "
    "complete for up to 30 seconds before forcibly terminating remaining "
    "connections, and this is configurable per node pool.",
    "Scale-up events are suppressed for 10 minutes after any node pool "
    "resize failure, to avoid retrying against a quota limit that has "
    "not yet been lifted by the cloud provider.",
] * 20  # repeated to build a prompt long enough to force real truncation

prompt = build_prompt(instructions, context_docs)
print(f"Full prompt size: {len(prompt.encode('utf-8'))} bytes\n")

for provider in PROVIDER_LIMITS:
    fitted, truncated, original_size, limit = fit_to_provider(prompt, provider)
    kept_pct = 100 * len(fitted.encode("utf-8")) / original_size
    print(f"{provider:12} limit={limit:6} bytes  "
          f"truncated={str(truncated):5}  kept={kept_pct:5.1f}% of original")

print()


def flaky_call(fail_first_n):
    count = {"n": 0}

    def _call():
        count["n"] += 1
        if count["n"] <= fail_first_n:
            raise RateLimited()
        return f"ok after {count['n']} attempt(s)"
    return _call


for fail_first_n in (0, 2, 5):
    try:
        call = flaky_call(fail_first_n)
        result, attempt, delays = call_with_backoff(call, max_retries=4, base_delay=0.5)
        print(f"fail_first_n={fail_first_n}: {result}, retries used={attempt}, "
              f"backoff delays={delays}")
    except RateLimited:
        print(f"fail_first_n={fail_first_n}: gave up after retries exhausted")

# A truncation boundary check: force a cut in the middle of a multi-byte
# character and confirm the result still decodes cleanly.
tricky = "café " * 400  # 'é' is 2 bytes in utf-8
fitted, was_truncated = truncate_utf8_bytes(tricky, 101)
assert was_truncated
assert fitted.encode("utf-8")  # raises if the cut left an invalid boundary
print(f"\nUTF-8 boundary check: kept {len(fitted.encode('utf-8'))} of "
      f"{len(tricky.encode('utf-8'))} bytes, decodes cleanly: True")

# Output:
# Full prompt size: 11551 bytes
#
# fast-tier    limit=  3500 bytes  truncated=True   kept= 30.3% of original
# premium-tier limit= 12000 bytes  truncated=False  kept=100.0% of original
#
# fail_first_n=0: ok after 1 attempt(s), retries used=0, backoff delays=[]
# fail_first_n=2: ok after 3 attempt(s), retries used=2, backoff delays=[0.532, 1.015]
# fail_first_n=5: gave up after retries exhausted
#
# UTF-8 boundary check: kept 101 of 2400 bytes, decodes cleanly: True

The two asserts (that the boundary check was actually truncated, and that the truncated result still encodes/decodes cleanly) both passed on the run that produced this output. Note the fast-tier line: the same request that fits entirely into premium-tier's budget only survives at 30.3% on fast-tier — the difference between "works" and "quietly answers from less than a third of the intended context" is purely which provider's limit was active.

---

6. Real-world example

A team's assistant was originally built and tested against a premium API tier with a generous input budget, and the truncation logic was configured to that tier's limit. Months later, to cut costs on high-volume, low-stakes requests, they routed a portion of traffic to a cheaper, faster tier with a much smaller input budget — the integration code was identical, so the change looked like a routing tweak, not something that touched request construction at all.

Nobody updated the truncation budget for the new tier, because nothing about the change looked like it should require it. The cheaper tier's client library truncated oversized requests internally rather than rejecting them outright, so no errors appeared anywhere — no failed requests, no alerts, no spike in error rate. What changed instead was quality: answers on longer requests routed to the cheaper tier got quietly worse, because a large fraction of the intended context was being silently dropped before the model ever saw it. The team's dashboards, which tracked error rate and latency, showed nothing wrong for weeks. The problem surfaced only when a support engineer manually compared two answers to the same question that had been routed to different tiers and noticed one was missing information the other used correctly.

The fix — configuring the truncation budget per active provider/tier rather than assuming one global number — took an afternoon. Diagnosing that a silent, error-free truncation was the cause took considerably longer, precisely because nothing in the system was designed to notice or report when it happened.

---

7. Interview questions companies actually ask

Q1. Why can't you just pick one provider's request-size limit and use it everywhere? Because different providers, and different pricing tiers within the same provider, allow different maximum request sizes, and hardcoding one provider's number either causes outright rejections when talking to a stricter provider or, worse, causes a client library to silently truncate more than intended when talking to a more permissive one that happens to be configured for the wrong budget. The limit has to be looked up for whichever provider/tier is actually active, not assumed from whichever one was used during development.

Q2. What goes wrong if you truncate a UTF-8 string by slicing bytes directly? A single character can occupy multiple bytes in UTF-8, so an arbitrary byte-offset cut can land in the middle of one of those multi-byte sequences, producing a string that fails to decode or that decodes into a corrupted character at the exact cut point. Safe truncation checks whether the cut point produced valid UTF-8, and if not, backs up byte by byte until it does.

Q3. Two truncation strategies fit a request into the same byte budget — one just chops whatever's at the end, one prioritizes what's kept. Why does the difference matter? Naive truncation can remove exactly the piece of context a request needed while keeping less important material earlier in the text, because it only cares about position, not importance. Priority-aware truncation keeps the instructions and highest-priority content intact and drops lower-priority material first, which means the request degrades gracefully — losing the least important information first — rather than degrading arbitrarily.

Q4. Why add jitter to exponential backoff instead of just doubling the delay each time? Without jitter, every client that got rate-limited at the same moment computes the identical backoff delay and retries at the identical moment again, recreating the exact burst of simultaneous requests that caused the rate limit in the first place. A small random jitter spreads those retries out in time, so the retrying clients don't all land on the API at once a second time.

Q5. Your retry loop has no maximum attempt count. What's the actual risk? A transient failure (a momentary rate limit) resolves on retry, but a persistent failure (bad credentials, a malformed request, a genuinely unavailable service) fails identically on every attempt — without a retry cap, that turns into an indefinite hang instead of a clear, timely error the caller can act on. The cap converts "we don't know if or when this will succeed" into "we tried a bounded, known amount, and here is the failure to handle."

Q6. You need to send a request that genuinely can't fit even after truncation, because the task requires all of the content. What do you do? Truncation is lossy by definition, so it isn't the right tool when completeness is a hard requirement — the request needs to be split into multiple smaller calls whose results are combined afterward, or the scope of what's being asked has to shrink. Reaching for a larger truncation budget doesn't solve this if the content still doesn't fit; it only delays hitting the same wall.

Q7. How would you detect the exact silent-truncation failure described in §6, before a human notices a quality drop? Log the original size and the post-truncation size on every request, and alert when truncation occurs at a meaningfully high rate for a given provider/tier — a sudden rise in "requests being truncated" for a route that previously saw none is the earliest signal, well before a downstream quality metric moves enough for a person to notice by comparison.

---

8. When to use / tradeoffs

Reach for provider-aware limits, safe truncation, and backoff-with-jitter when:

• your client talks to more than one provider, or more than one pricing tier of the same provider
• request sizes are large enough or variable enough that hitting a size limit is a real possibility, not a theoretical one
• the API enforces rate limits and your traffic pattern can plausibly burst past them

Honest limits. Truncation assumes that dropping the lowest-priority content is an acceptable degradation, which is false for tasks where every piece of context is load-bearing — no priority scheme fixes that, because the content that had to go might have mattered anyway. Backoff-with-jitter assumes the failure is transient; it does nothing for a request that will fail identically every time, and it adds real latency (the worst case in §4.3 is measured in seconds, not milliseconds) that has to fit inside whatever timeout the caller is operating under. And a retry cap trades one failure mode (hanging indefinitely) for another (giving up on something that would have succeeded on attempt 5 instead of the configured maximum of 4) — the cap has to be chosen deliberately, not left at whatever default a library ships with.

---

9. Summary + related articles

• Different providers and pricing tiers enforce different request-size and rate limits; hardcoding one provider's numbers breaks silently the moment the active provider or tier changes.
• Truncating to fit a byte or token budget has to respect character boundaries (UTF-8 characters can be multiple bytes) and should be priority-aware, keeping the most important content and dropping the least important first.
• Retrying a rate-limited request needs exponential backoff, jitter (to avoid synchronized retry storms), and a maximum attempt count (to convert an indefinite hang into a bounded, surfaced failure).
• Boundary: none of this helps when completeness is a hard requirement — truncation is lossy by definition, and no retry strategy turns a persistent failure into a transient one.

Related:

• Chunk Size, Overlap, and Retrieval Thresholds in RAG Systems — the same underlying lesson (a tuned number is a property of one specific system, not a portable constant) applied to retrieval instead of request budgets
• Production Agents — where retry/backoff sits inside the broader set of reliability patterns for agents running in production

Resources

• Brooker, M. (2015). Exponential Backoff and Jitter. AWS Architecture Blog. — the original industry writeup establishing "full jitter" as a standard corrective to synchronized retry storms.
• IETF RFC 6585 (2012), Additional HTTP Status Codes — defines the 429 Too Many Requests status code that a rate-limiting API returns, and the Retry-After header convention for signaling how long a client should wait.