OpenAI function calling: why schema-valid calls still fail

Automation Engineering · By Caleb Sakala · June 7, 2026

OpenAI function calling earned its reputation in the demo. Ten minutes of wiring gets one weather function returning perfect JSON arguments on every run. Since strict mode arrived, the arguments match your schema 100% of the time, and teams reasonably conclude the reliability problem is solved. Then the integration ships, a customer types "cancel my last order," and the model calls cancel_order with an order ID it made up on the spot. The JSON was flawless. The value was fiction.

What strict mode guarantees

When OpenAI shipped structured outputs in August 2024, gpt-4o-2024-08-06 scored 100% on the company's eval of complex JSON schema following. The previous approach, prompting gpt-4-0613 to behave, scored under 40%. Behind the jump sits constrained decoding: at each generation step the API masks every token that would violate your schema, so the model cannot produce a missing field or a string where you declared an integer. Set strict: true in a function definition, as the function calling guide documents, and malformed tool calls disappear.

What constrained decoding cannot do is check your database. A schema knows that order_id must be a string in your declared format. It has no idea whether "ord_84412" exists or belongs to the customer in this conversation, and it certainly cannot tell that the model invented the value because the user said "my last order" and a required field demanded something. Every one of those fabrications validates perfectly.

Strict mode also removed the failure that used to protect you. Broken JSON crashed loudly in development, and the crash forced teams to build a validation layer before launch. A fabricated value that parses cleanly sails through the same code path as a real one. The visible failure was doing safety work, and now it is gone.

Where OpenAI function calling breaks in production

Fabricated arguments are the first failure class. The second shows up as your tool count grows: wrong-tool selection. With one weather function the model has no choices to get wrong. With twenty tools whose descriptions overlap, cancel_order sitting next to cancel_subscription, selection accuracy degrades in ways no schema can catch, because calling the wrong tool with valid arguments is still schema-valid. The third class is silent error continuation: a tool returns {"error": "not found"}, the model receives it as ordinary output, and some models cheerfully report the task complete instead of stopping.

Benchmarks show how much room these classes leave. The Berkeley Function-Calling Leaderboard V4 evaluates exactly this territory: multi-turn tool use with parallel and nested calls in agentic settings. On the llm-stats tracker of BFCL V4 results, the top score as of June 2026 is 0.750 (Qwen3.7 Max), with an average of 0.601 across the 12 models tracked there. Schema conformance is effectively solved, yet the best models still miss roughly a quarter of these agentic tasks. The misses live in selection, sequencing, and values.

Practitioners noticed before the benchmarks did. A widely shared r/OpenAI thread from May 2026 says it plainly in its title: function calling works great in demos, and production is a different story.

Grounding OpenAI function calling arguments before they execute

The fix fits in one sentence: treat every model-supplied identifier as a query, never as a fact.

A resolution layer sits between the tool call and the execution. Before any side effect runs, each identifier the model supplied gets looked up against a source of truth. Found exactly one match, proceed. Found zero or several, return a structured error and let the model ask the user.

def execute_refund(args: dict, ctx: Context):
    # The model supplied args["order_id"]. Resolve it; don't trust it.
    order = db.orders.find(id=args["order_id"], customer=ctx.customer_id)
    if order is None:
        recent = db.orders.recent(customer=ctx.customer_id, limit=3)
        return {
            "status": "unresolved",
            "message": f"No order {args['order_id']} for this customer.",
            "did_you_mean": [o.id for o in recent],
        }
    return refunds.create(order)  # side effect only after resolution

The did_you_mean field carries more weight than it looks. Returning candidates turns a dead end into a recovery path: the model can show the user real options and stop guessing.

Walk the refund scenario through this layer. The user writes "refund my last order." The model, needing a value for a required field, invents ord_84412. Without resolution, that string hits the payments API and the problem surfaces in a support ticket. With resolution, the lookup returns zero matches plus three real candidates, the model asks "did you mean the order from June 2?", and the user corrects course in one turn. Same hallucination, much smaller blast radius.

Workflows with irreversible side effects need one more boundary, placed where the model cannot reach it. In a procurement automation built on Chase Agents, a research step returns purchase_recommendation objects that route to an approval step, and only the orchestrator holds the tool access that initiates a purchase call. A fabricated vendor ID surfaces at the approval step as a recommendation nobody recognizes. It never becomes a paid invoice. The platform calls this action-type routing, and it does at the workflow level what resolution does at the argument level: it keeps a confident wrong value away from the irreversible thing.

The number worth alerting on

A dashboard that tracks schema failure rate, the number strict mode pinned near zero, glows green while refunds go to the wrong orders. The metric that predicts incidents is resolution failure rate: out of every model-supplied identifier, the share that failed lookup against a source of truth. The absolute level matters less than the trend. A sudden climb means a prompt change, a renamed field, or a new user behavior the tool descriptions never anticipated, and it appears in the logs days before it appears in support tickets.

Validation deserves the same treatment on the way in. An invoice-intake automation that declares "every extracted total must match the sum of its line items" as a success condition fails the run at validation instead of posting a wrong number to the ledger. Chase Agents builds this in as input schemas and success criteria on each automation, which also makes swapping the underlying model cheap to test, since tool-selection accuracy differs across providers and a validated workflow exposes the difference immediately.

A prediction for the next BFCL update: schema conformance stays a solved row on every model card, overall scores inch up a few points, and resolution failures in real systems do not move at all, because no amount of constrained decoding can know what sits inside your database. Before trusting the next green dashboard, pull the last hundred tool calls from the logs and count how many model-supplied identifiers were verified against anything before they executed. Usually that count is zero, and zero is the most informative number in the whole system.