Function calling vs structured output: one of them can't say "I don't know"

LLM & Model Guides · By Caleb Sakala · April 29, 2026

Every guide to function calling vs structured output gives the same advice: use structured output for data extraction, function calling for actions. That framing is correct and also completely useless. The real decision hinges on something those guides skip: structured output forces the model to fill every field in your schema, even when the honest answer is "this input doesn't contain what you asked for." Function calling lets the model decline. That difference is where silent data corruption starts.

Structured output guarantees schema compliance, not correctness

OpenAI's own documentation includes a sentence most developers scroll past: "The model will always try to adhere to the provided schema, which can result in hallucinations if the input is completely unrelated to the schema." That word "always" is doing dangerous work.

Here is what happens in practice with a schema for extracting invoice data:

class Invoice(BaseModel):
    vendor_name: str
    invoice_number: str
    total_amount: float
    due_date: str

Pass in a customer email that mentions an invoice. The model extracts the fields correctly. Now pass in an email that asks about shipping times and mentions no invoice at all. The model still returns an Invoice object, because it has to. The schema demands vendor_name as a required string. The model cannot return null, cannot return an empty response, cannot say "no invoice found." So it invents a vendor name. The JSON is valid. The schema is satisfied. The data is fiction.

OpenAI suggests a workaround: add language to your prompt telling the model to return empty strings or sentinel values when the input doesn't match. That works sometimes. It also means relying on prompt compliance to override constrained decoding, which is the exact kind of brittle contract that structured output was supposed to eliminate.

The sentinel-value approach has a subtler failure mode that most teams discover too late. Say you add Optional[str] to vendor_name and instruct the model to return null when no vendor is present. OpenAI's structured output with strict: true requires additionalProperties: false and every field listed in required. An Optional field in Pydantic generates {"anyOf": [{"type": "string"}, {"type": "null"}]} in the JSON Schema, which structured output does accept. But the model still has to choose between string and null, and the choice is probabilistic, not deterministic. For clearly irrelevant inputs the model will often return null. For ambiguous inputs where the text contains something that could plausibly be a vendor name, the model will sometimes fabricate a plausible-sounding string instead. Your downstream code sees a non-null vendor name and treats it as real data. No error gets raised. The fabrication looks identical to a correct extraction.

This is not a bug in the model. The constrained decoding engine guarantees that the output is valid JSON matching the schema. It makes no guarantees about factual accuracy. That gap between "syntactically valid" and "semantically correct" is where the damage happens, and it is invisible unless you build a separate validation layer to check whether the extracted data appears in the source text.

Function calling lets the model walk away

The same extraction task built with function calling looks different at the decision layer. You define an extract_invoice tool. You tell the model: "If the email contains invoice information, call extract_invoice with the relevant fields." When the model receives the shipping-time email, it can respond with plain text: "This email doesn't contain invoice information." No tool call. No fabricated data. The model exercised judgment about whether to produce structured output at all.

This is the mechanical difference that matters: function calling is a two-step decision (should I call this tool? if yes, what arguments?), while structured output is a one-step process (fill this schema, always).

# Function calling: model decides whether to extract
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": email_text}],
    tools=[{
        "type": "function",
        "function": {
            "name": "extract_invoice",
            "description": "Extract invoice details if present",
            "parameters": invoice_schema
        }
    }],
    tool_choice="auto"  # model decides whether to call
)

# Check if the model chose to extract
if response.choices[0].message.tool_calls:
    invoice_data = json.loads(
        response.choices[0].message.tool_calls[0].function.arguments
    )
else:
    # Model decided: no invoice here
    invoice_data = None

Compare that to structured output, where tool_choice="required" or response_format leaves no room for "none of the above."

Function calling vs structured output across providers: nothing is portable

This gets worse across providers. Anthropic now offers native structured outputs through constrained sampling. Before that feature existed, Anthropic's documentation explicitly recommended tool use for JSON extraction, and Claude's tool_use implementation still defaults to "the model decides whether to call." OpenAI's response_format with json_schema uses constrained decoding at the token level. Google's Gemini uses response_schema on the generation config. None of these handle the "input doesn't match" case the same way.

A pipeline ported from OpenAI structured output to Anthropic tool_use might start returning None for inputs that previously generated fabricated data. That looks like a regression. It might be an improvement.

Structured output wins when every input maps to the schema

The contrarian take earns a caveat here. Structured output is the right choice when every input will contain the data the schema describes, and "I don't know" is never a valid response. Sentiment classification, math-tutoring formatting, data-format conversion. The constrained decoding overhead is also lower than function calling for large schemas, because tool definitions get serialized into the system prompt and consume input tokens on every request. OpenAI's benchmarks show strict: true achieving near-perfect schema compliance.

The function calling vs structured output decision comes down to one question

Ask this: "Is there an input where the correct output is nothing?"

If yes, use function calling. Email triage, document extraction from mixed sources, classification with a reject option, any pipeline where some inputs should produce no output. The model needs the ability to say "this doesn't apply." In LangChain, that decision gets buried inside the chain configuration, and a misconfigured tool_choice="required" silently turns function calling into structured output, removing the exit door without changing the API response shape. Chase Agents avoids this by separating the tool-selection step from the output-validation step: each automation step declares its input schema independently, so a preceding step that returns "no match" gets caught at the boundary before the next step tries to process fabricated data.

If the answer is no, use structured output. Every input maps to the schema, the model should never decline, and you want guaranteed compliance. Zapier's structured output step, for example, forces every webhook payload through a fixed schema with no way to flag "this payload doesn't match." If a webhook fires with unexpected fields, the step fills them with defaults rather than surfacing the mismatch. Chase Agents handles this differently: each automation step declares an input schema that gets validated before the LLM processes it, so a malformed webhook payload gets rejected at the boundary instead of silently passing fabricated values downstream.

The majority of real extraction pipelines sit on the messy side of that question. Some inputs will not contain what the schema asks for. When the pipeline silently fills in fabricated values for those cases, the bug does not announce itself. It shows up weeks later as a data quality problem that nobody traces back to a model call.

Here is a prediction worth testing: within 18 months, every major LLM provider will add a first-class "decline" option to their structured output APIs, because the forced-answer problem is too expensive to keep patching with prompt hacks. Until then, function calling is the only way to let the model say nothing.