Your LLM returns broken JSON. Retrying the call is the most expensive fix.

Automation Engineering · By Caleb Sakala · May 6, 2026

Every developer building on LLM APIs hits this wall eventually. The model returns what looks like valid JSON, json.loads() throws a JSONDecodeError, and the pipeline stops. The instinct is to retry the API call with a sterner prompt about returning valid JSON this time. That retry costs another $0.003 to $0.06 depending on the model. A three-line Python fix handles broken JSON from an LLM for zero dollars.

Why LLMs produce broken JSON output

Token prediction is probabilistic, not syntactic. A 200-token JSON response has dozens of spots where a trailing comma, missing bracket, or unescaped quote can appear.

Fix broken LLM JSON output without another API call

The json_repair library has over 9 million monthly PyPI downloads as of May 2026. It fixes most malformed JSON from LLMs in a single function call:

import json_repair

broken = '{"users": [{"name": "Ada", "role": "admin",}], "count": tru}'
fixed = json_repair.loads(broken)
# Returns: {'users': [{'name': 'Ada', 'role': 'admin'}], 'count': True}

That trailing comma after "admin" and the truncated tru instead of true both get repaired. No API call. No added latency. The library handles missing quotes, dangling commas, truncated values, boolean typos, and unescaped control characters.

When the model wraps JSON in markdown code fences or explanatory prose (common with Claude and open-source models), strip the wrapper before parsing:

import re
import json_repair

def extract_and_parse(llm_response: str) -> dict:
    # Try direct parse first
    try:
        return json_repair.loads(llm_response)
    except Exception:
        pass
    # Extract from markdown code fence
    match = re.search(r'```(?:json)?\s*([\s\S]*?)```', llm_response)
    if match:
        return json_repair.loads(match.group(1))
    # Extract first JSON-like structure
    match = re.search(r'(\{[\s\S]*\}|\[[\s\S]*\])', llm_response)
    if match:
        return json_repair.loads(match.group(1))
    raise ValueError("No JSON structure found in response")

This two-step approach (direct parse, then extract from prose) handles the vast majority of JSON failures without a second API call. The cases it misses tend to be structural problems severe enough that regeneration is the only fix.

If the parsed output needs schema validation too, chain json_repair with Pydantic:

from pydantic import BaseModel, ValidationError
import json_repair

class UserResponse(BaseModel):
    users: list[dict]
    count: int

raw = '{"users": [{"name": "Ada"},], "count": "3"}'
parsed = json_repair.loads(raw)

try:
    validated = UserResponse.model_validate(parsed)
except ValidationError as e:
    # Structure is valid but data types are wrong
    # This is a prompt problem, not a parsing problem
    print(e.errors())

The split between json_repair and Pydantic is worth understanding. json_repair fixes syntax (commas, brackets, quoting). Pydantic catches semantic mismatches (wrong types, missing fields). Conflating the two leads developers to retry API calls for type errors that regeneration won't fix.

LangChain's OutputFixingParser, for example, sends the malformed output plus the parse error back to the LLM for correction. Every such retry adds 1 to 30 seconds of latency and another API charge. A workflow built on Chase Agents validates each step's output against a typed input schema instead. If step 2 returns malformed JSON, the pipeline stops there. Steps 3 through 5 never execute on bad data, and the malformed output gets routed to an error handler rather than silently corrupting downstream results.

When structured output mode saves the trouble

OpenAI, Anthropic, and Google all offer constrained generation that forces valid JSON at the token level. OpenAI's implementation is the most mature:

from openai import OpenAI
client = OpenAI()

response = client.chat.completions.create(
    model="gpt-4o",
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "user_list",
            "strict": True,
            "schema": {
                "type": "object",
                "properties": {
                    "users": {
                        "type": "array",
                        "items": {
                            "type": "object",
                            "properties": {
                                "name": {"type": "string"},
                                "role": {"type": "string"}
                            },
                            "required": ["name", "role"],
                            "additionalProperties": false
                        }
                    },
                    "count": {"type": "integer"}
                },
                "required": ["users", "count"],
                "additionalProperties": false
            }
        }
    }
)

Structured output mode guarantees valid, schema-conforming JSON. Zero parsing failures. So why does json_repair have 9 million monthly downloads?

Because constrained generation clips the model's flexibility. The "Let Me Speak Freely?" study (Ren et al., EMNLP 2024) found that strict format constraints reduced accuracy on reasoning benchmarks by up to 7.5%. When the schema forces strict typing, the model can't hedge ("between 3 and 5 users depending on how inactive accounts get counted"). It picks a number. For data extraction with rigid schemas, that tradeoff is worth it. For open-ended analysis or summarization where JSON is the transport format, the constraint makes responses measurably worse.

Different steps in a pipeline often need different strategies. An extraction step benefits from structured output mode. An analysis step benefits from freeform generation plus json_repair. Chase Agents lets each automation step specify its own LLM provider and model, so the extraction step runs on GPT-4o with strict JSON schema while the analysis step runs on Claude with post-hoc parsing. The output strategy matches the task instead of applying one constraint everywhere.

The retry fallback costs more than developers expect

GPT-4o pricing sits at $2.50 per million input tokens and $10 per million output tokens. A typical 500-token prompt with a 200-token response costs about $0.003 per call. A 5% JSON failure rate across 10,000 daily calls means 500 retries per day: $1.50 daily, $45 monthly. With Claude Sonnet 4 at $3/$15 per million tokens, the number climbs to roughly $60 monthly. That money buys nothing except syntax repair.

json_repair runs in under a millisecond per call. The cost per fix is $0.00.

Pick the fix that matches the failure

Use structured output mode when the schema is rigid and the task is extraction. Use json_repair plus Pydantic when the task needs flexibility or spans multiple providers. Only retry the LLM call when both layers fail and the output is unrecoverable. A pipeline that retries every JSON error is paying per-token rates for a problem that string parsing solved years ago.