Structured Extraction from Vision-Language Models

Structured OutputVision-LanguageIntermediate13 min

By: Anacodic Team

TL;DR — A vision-language model (VLM) turns an image into text, but a pipeline needs structured data — a JSON object with known fields and types. The reliable pattern: give the VLM a schema, get JSON back, then validate it against that schema and reject/repair anything that doesn't conform — never trust the raw string. Prefer real enforcement (OpenAI Structured Outputs' json_schema, Gemini response_schema, or constrained decoders like Outlines / XGrammar) so the model can't emit invalid JSON; where you can't, parse-then- validate and fail closed on invalid. It breaks when the field you want isn't actually visible in the image (the VLM will hallucinate a plausible value), so the schema and downstream logic must treat "not present" as a first-class answer.

1. Simple explanation

Models that read images (VLMs) are great at describing what they see, but a program can't act on a paragraph of prose — it needs fields: {"shapes": 3, "has_circle": true, "colors": ["red","blue"]}. The job of structured extraction is to get exactly that: a machine-readable object with the fields you asked for, the right types, every time.

The naive approach — "please return JSON" in the prompt, then json.loads the reply — fails constantly: the model wraps JSON in markdown fences or prose, emits a string where you wanted a number, omits a field, or trails a comma. So the real pattern has two guarantees layered on the request: (1) tell the model the exact schema, and ideally force it (schema-constrained decoding) so invalid output is impossible; and (2) validate whatever comes back against that schema and refuse anything that doesn't fit. The model proposes; your validator disposes.

Analogy — a form instead of a blank page. If you ask someone to "describe your trip," you get free text you must re-read and re-key. If you hand them a form with labeled boxes (dates as dates, cost as a number, checkboxes), you get back exactly the fields you need — and you can bounce the form if a box is blank or the wrong type. The schema is the form; validation is checking the boxes are filled correctly before you file it.

2. Diagram

  IMAGE ──▶ [ VLM: visual encoder + language decoder ] ──▶ TEXT that should be JSON
                              │  (guided by a SCHEMA)
                              ▼
        ┌───────────────────────────────────────────────┐
        │ ENFORCE / VALIDATE against the schema           │
        │   best:  schema-constrained decoding            │
        │          (model CAN'T emit invalid JSON)        │
        │   min:   parse -> check fields+types            │
        │            valid   -> use it                    │
        │            invalid -> reject (fail closed) / repair│
        └───────────────────────────────────────────────┘
                              ▼
                    typed object the pipeline can trust

  raw prose  {"shapes":3,...}      -> valid   -> {shapes:3, has_circle:True, ...}
  {"shapes":"three"} (wrong type)  -> invalid -> rejected (do NOT pass downstream)

3. How it works

3.1 What a VLM actually returns

A VLM pairs a visual encoder (turns the image into embeddings) with a language decoder (generates text conditioned on those embeddings). Its native output is a token stream — free text — not a data structure. Everything about "getting JSON" is about constraining and checking that text stream so it lands as valid structured data.

3.2 Three levels of "get me JSON", strongest first

Schema-constrained decoding (strongest). The decoder is restricted at generation time to only produce tokens allowed by your JSON schema/grammar, so the output is guaranteed well-formed and type-correct. This is what OpenAI's Structured Outputs (response_format with json_schema, 2024), Gemini's response_schema, and libraries like Outlines / XGrammar do.
Function/tool calling (strong). You declare a function signature; the model emits arguments matching it. Similar guarantee, framed as a tool call.
Prompt-and-parse (weakest). You ask for JSON in the prompt and parse the reply. No guarantee — you must validate and handle failure. Use only when the provider can't enforce a schema.

3.3 Always parse-then-validate (even with enforcement)

Extract the JSON (models still sometimes wrap it in prose/markdown, so pull the {...} block), then validate against the schema: required fields present, correct types, values in range/enum. Treat validation as a gate: a malformed or off-schema object must not flow downstream. When enforcement is available it makes invalid output nearly impossible; validation is the seatbelt for the cases it doesn't (older models, streaming glitches, prose leakage).

3.4 Fail closed on invalid; make "absent" explicit

Two failure modes need explicit handling. Invalid output (won't parse / wrong types) → reject it (optionally one repair retry), don't pass a half-parsed object on. The information isn't in the image → this is the dangerous one: asked for a field it can't see, a VLM will often invent a plausible value. So your schema should allow null / "unknown" and your prompt should say "use null if not visible" — turning a hallucination into an honest "not present."

3.5 Repair, but bounded

When output fails validation you have two moves: reject it, or attempt one repair — feed the malformed text plus the schema and the validation error back to the model and ask it to fix only the JSON. A single repair pass cheaply catches transient formatting slips (a stray fence, a trailing comma, a quote where a number belongs). But cap it at one retry: looping "fix it" burns tokens and, on a genuine misperception, just yields a differently-wrong object. And never let repair change the values — it should only re-format, or you've let the model launder a bad answer into a valid-looking one. If the single repair fails, reject and surface it for review.

Boundary condition. Structured extraction guarantees the shape of the output, not its truth. A perfectly schema-valid object can still be wrong — the VLM misread the image. Schema enforcement stops malformed JSON; it does nothing about misperception (Section 8), which is why safety-critical fields still need verification beyond "it parsed."

4. The math (the validation contract)

Little arithmetic; the core is a conformance predicate. Given a schema Σ (fields, types, constraints) and a candidate object o:

  valid_Σ(o) =  all required fields of Σ present in o
             AND every field's value has the declared type
             AND every value satisfies its constraints (range / enum / length)

  accept(text) =  let o = parse_json(extract_braces(text))
                  o        if valid_Σ(o)
                  REJECT   otherwise            # fail closed
                  (parse error also -> REJECT)

The guarantee you want is: everything that reaches the pipeline satisfies valid_Σ. Schema-constrained decoding makes valid_Σ(o) true by construction; prompt-and-parse makes it true by rejection. Either way, no object that fails valid_Σ is ever used.

4.x Worked example

Schema Σ: shapes = int ≥ 0, has_circle = bool, colors = list of strings.

A "good" VLM reply wraps clean JSON in surrounding prose (often a sentence, or a markdown code fence): e.g. Analysis: the drawing shows {"shapes": 3, "has_circle": true, "colors": ["red","blue"]} in total. Extract the {...} block, parse → {shapes:3, has_circle:True, colors:[...]}, check types → valid, use it.
A "bad" reply: {"shapes": "three", "has_circle": true}. It parses, but shapes is a string (schema wants int) and colors is missing. So valid_Σ is false → reject (fail closed), don't hand a broken object downstream. The code below returns the good object and None for the bad one.

5. Real code

import json
# A VLM returns TEXT; we must extract a valid, schema-conforming JSON object.
# Schema: {"shapes": int>=0, "has_circle": bool, "colors": list[str]}
raw_good = 'Analysis: the drawing shows {"shapes": 3, "has_circle": true, "colors": ["red","blue"]} in total.'
raw_bad  = '{"shapes": "three", "has_circle": true}'      # wrong type + missing field

def extract_json(text):                       # models wrap JSON in prose/markdown
    s, e = text.find("{"), text.rfind("}")
    return json.loads(text[s:e+1])

def valid(o):
    return (isinstance(o.get("shapes"), int) and o["shapes"] >= 0
            and isinstance(o.get("has_circle"), bool)
            and isinstance(o.get("colors"), list))

def safe_extract(text):                        # fail-closed: accept only if schema-valid
    try:
        o = extract_json(text)
        return o if valid(o) else None
    except Exception:
        return None

print("good ->", safe_extract(raw_good))
print("bad  ->", safe_extract(raw_bad))
assert safe_extract(raw_good) is not None      # clean JSON pulled from prose, valid
assert safe_extract(raw_bad) is None           # wrong types / missing field -> rejected
print("OK: schema validation accepts the good extraction, rejects the malformed one")

# Output:
#   good -> {'shapes': 3, 'has_circle': True, 'colors': ['red', 'blue']}
#   bad  -> None
#   OK: schema validation accepts the good extraction, rejects the malformed one

In production you'd let the provider enforce the schema (so raw_bad can't happen), but you keep safe_extract as the seatbelt — the exact same accept/reject logic guards the pipeline regardless of which model produced the text.

6. Real-world example

A team extracted fields from scanned forms with a VLM using "return JSON" in the prompt and a bare json.loads. In the demo it worked; in production ~6% of calls threw — the model had wrapped JSON in ```json fences, emitted "N/A" where a number was expected, or dropped a field on long documents — and each exception dropped the whole record. Worse, on fields that were simply missing from the scan, the VLM confidently filled in plausible-looking values, so some records were "complete" but fabricated.

Two changes fixed it. First, switch to schema-enforced output (json_schema with types and an explicit null-allowed for optional fields), which eliminated the parse failures outright. Second, make the prompt+schema say "use null if the value is not visible," turning silent fabrication into an explicit "not present" the pipeline could route to human review. The recurring lesson: "please return JSON" is not a contract — enforce the schema, validate on the way in, and design for "the answer isn't in the image" as a real outcome, not an error.

7. Interview questions companies actually ask

Q1. Why not just prompt "return JSON" and json.loads the reply? Because that's not a contract — models wrap JSON in prose/markdown, emit wrong types, drop fields, or add trailing commas, so parsing throws or yields a malformed object. You need schema enforcement and/or validation-then-reject, not raw trust.

Q2. What are the levels of getting structured output, strongest first? Schema-constrained decoding (the decoder can only emit tokens allowed by the schema, so output is valid by construction), function/tool calling (arguments match a declared signature), and prompt-and-parse (ask and hope, then validate). Prefer enforcement; keep validation regardless.

Q3. If the provider enforces the schema, why still validate? Defense in depth: older models, streaming, or prose leakage can still slip through, and you may swap providers. Validation is a cheap seatbelt that guarantees everything reaching the pipeline conforms, independent of who generated it.

Q4. A VLM must return a field that isn't visible in the image — what happens and how do you handle it? It tends to hallucinate a plausible value. Handle it by allowing null/"unknown" in the schema and instructing "use null if not visible," so a missing value becomes an explicit, routable "not present" instead of silent fabrication.

Q5. Does a schema-valid object mean the extraction is correct? No — schema guarantees shape and type, not truth. The VLM can misread the image and still produce valid JSON. Correctness needs separate verification (confidence checks, cross-fields consistency, human review for critical fields).

Q6. How would you make extraction robust in a pipeline? Enforce the schema at generation, extract the JSON block defensively, validate types/ranges, fail closed on invalid (with at most one repair retry), allow explicit nulls for absent data, and log rejects so you can see failure rates rather than silently dropping records.

Q7. How do you monitor structured extraction in production? Track the reject rate (schema-invalid outputs), the null / "not visible" rate, and per-field validation failures; alert when any of them spikes — a model or prompt change usually shows up first as a jump in the reject rate — and sample rejects for review. Silently dropping malformed records hides a degrading extractor; surfacing the reject rate makes the degradation visible before it corrupts downstream data.

8. When to use / tradeoffs

Reach for structured VLM extraction when:

You need typed fields from images/documents (counts, flags, tables, entities) to drive downstream code.
A provider or library can enforce a JSON schema / grammar.
"Absent / not visible" is a meaningful, handleable outcome you can model.

Do NOT rely on it (as-is) when:

Situation	Why it breaks	Use instead
The field is safety/decision-critical	valid JSON can still be misperceived	verification + human review of that field
Info often isn't in the image	VLM fabricates plausible values	schema with null + "not visible" instruction
Provider can't enforce a schema	prompt-and-parse leaks invalid output	a constrained decoder (Outlines/XGrammar) around it
Exact layout/coordinates needed	free VLM text loses geometry	a detection/OCR model with bounding boxes
High volume, tight latency/cost	schema decoding + retries add overhead	smaller model + cache + batch, measure first

Honest limits. Structured extraction guarantees the shape of the output, never its truth — a schema-valid object can be a confident misreading, so critical fields still need verification. Enforcement narrows but doesn't fully eliminate "absent → hallucinated" unless you explicitly model null and prompt for it. Constrained decoding and repair retries add latency and cost, and an overly rigid schema can force the model to coerce a genuinely ambiguous image into a wrong but valid value. Validate on the way in, model "not present," and treat schema-validity as necessary, not sufficient.

A VLM emits text; a pipeline needs typed JSON — bridge the gap with a schema plus validation.
Prefer schema-constrained decoding (OpenAI Structured Outputs, Gemini response_schema, Outlines/XGrammar) so invalid output is impossible; keep parse-then-validate as the seatbelt.
Fail closed on invalid output, and model "not visible" as null so the VLM reports absence instead of hallucinating it.
Boundary: schema guarantees shape and type, not correctness — a valid object can still be a misread image, so critical fields need separate verification.

LLM-as-a-Judge: Using a Model to Grade Model Output — a judge must return a structured verdict; the same schema-validate-fail-closed discipline applies.
Embeddings and Cosine Similarity — the other half of turning perception into machine-usable vectors.
Prompt Management Architecture: Prompts as Files, Not Strings — versioning the prompts/schemas that drive extraction.
Guardrails & Output Validation — validating and refusing untrusted model output, the same discipline applied to safety.

Resources

OpenAI, "Structured Outputs" (2024) — response_format with json_schema for guaranteed schema-conforming output. (Vendor docs; feature released Aug 2024.)
Google, Gemini API "Structured output / response_schema" (2024) — schema-enforced JSON from Gemini. (Vendor docs.)
"Generating Structured Outputs from Language Models: Benchmark and Studies" / JSONSchemaBench — evaluation of constrained-decoding frameworks (Guidance, Outlines, XGrammar, …). https://arxiv.org/abs/2501.10868 (verify arXiv id before citing).
Hugging Face, "Vision Language Models (2025)" — survey of open VLMs and their structured-output/function-calling support. (Blog; practical overview.)

← Previous: Prompt Management Architecture: Prompts as Files, Not Strings Next: Prompt Fundamentals →