OpenAI's batch API saves 50% per token. The other 50% is engineering time.

LLM & Model Guides · By Caleb Sakala · April 22, 2026

When teams first discover the OpenAI batch API, the pitch sounds simple: submit requests in a JSONL file, wait up to 24 hours, pay half the token price. For GPT-5.4, that means $1.25 per million input tokens instead of $2.50, and $7.50 per million output tokens instead of $15.00. Run 10,000 classification calls a day, and the batch API turns a $150 daily bill into $75.

That math checks out. It is also incomplete.

The discount applies to tokens. Everything else, the formatting, the polling, the error recovery, the capacity planning, costs engineering hours. Whether those hours matter depends entirely on what the pipeline does and how many requests flow through it.

What the openai batch API does after you upload the file

The official docs describe the batch API as asynchronous processing with a 24-hour completion window. In practice, that description skips the interesting parts.

Every request in a batch must be formatted as a single JSON line in a .jsonl file. Each line needs a custom_id, the HTTP method, the URL path, and the full request body:

{"custom_id": "req-001", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-5.4-mini", "messages": [{"role": "user", "content": "Classify this support ticket: ..."}]}}

You upload this file, create a batch object pointing to it, and poll for completion. There is no webhook notification. There is no streaming. You check the batch status endpoint repeatedly until the state changes to completed, failed, or expired.

Completion times are unpredictable. OpenAI guarantees the 24-hour window but provides no minimum. A batch of 50 requests might finish in 90 seconds. A batch of 50,000 might take 18 hours. There is no priority mechanism and no way to estimate completion time before submission.

The enqueued token limit that breaks production pipelines

Each model has a cap on how many tokens you can have pending across all active batches simultaneously. For GPT-5.4, OpenAI sets that limit at 10 billion tokens. It sounds generous.

Sinaptia, a development consultancy that processes large document sets through the batch API, published a detailed account of hitting this wall (sinaptia.dev, June 2025). Their batches worked fine in testing with a few hundred requests. In production, with multiple pipelines running concurrently and submitting batches on overlapping schedules, they exceeded the enqueued token limit. Batches started failing, and the error message gave no clear indication that token capacity was the issue.

The fix required building a queue manager to track total enqueued tokens across every active batch before submitting new ones. That is a queue manager on top of a system that is supposed to be the queue manager. This kind of operational overhead never appears in the "50% savings" calculation.

Honestly? This is probably the single most useful thing to know upfront. The docs bury it, the tutorials skip it, and the error messages are unhelpful. If you are processing anything beyond toy volumes, build the token tracker before you need it.

For most teams, batch should not be the default

The 24-hour window works for offline analytics. Classify yesterday's support tickets overnight and route them in the morning. Generate embeddings for a document corpus that gets indexed weekly.

But "can wait until tomorrow" rules out most of the work teams actually want to batch. A user uploads a document and expects a summary within seconds. A sales team wants lead scores before the morning standup, and the batch they submitted at 11pm still shows in_progress at 8am. A CI pipeline generating test cases from specs cannot block for a full day.

The temptation is to submit small batches and hope they complete quickly. Sometimes they do. But there is no SLA on speed within the 24-hour window. Building production systems around "it usually finishes in a few minutes" creates the kind of fragility that surfaces at the worst possible time.

The cost math with all the variables

Take a concrete example. A team runs 10,000 GPT-5.4-mini classification calls per day, each with approximately 800 input tokens and 100 output tokens.

Synchronous API cost per day:

• Input: 10,000 x 800 = 8M tokens x $0.75/1M = $6.00
• Output: 10,000 x 100 = 1M tokens x $4.50/1M = $4.50
• Total: $10.50/day ($315/month)

Batch API cost per day:

• Input: 8M tokens x $0.375/1M = $3.00
• Output: 1M tokens x $2.25/1M = $2.25
• Total: $5.25/day ($157.50/month)

Monthly token savings: $157.50. That is real money for a bootstrapped team.

Now add the engineering cost. The JSONL generation pipeline, the polling logic with exponential backoff, the token limit tracker, the partial failure recovery system, the monitoring dashboard to catch batches that expire before completing. A senior developer at $150/hour needs roughly 8 to 12 hours to build this reliably. Call it $1,500 upfront plus ongoing maintenance.

At $157.50/month in savings, the break-even point is about 10 months, assuming nothing breaks and the pipeline needs zero changes. That math gets worse if daily volume drops or if the team needs to handle edge cases that surface after launch.

For teams routing high-volume classification or embedding work through automation platforms, the calculus changes. A procurement team using Chase Agents to sort incoming vendor invoices can drop a batch processing step into the overnight workflow: fire at midnight, collect at 6am, feed results into the approval queue. The JSONL formatting, polling, and error handling are managed by the automation step configuration. The $157.50/month savings arrives without the $1,500 engineering investment.

Error handling is where the real complexity lives

A synchronous API call fails, you retry it. A batch fails, the recovery process is more involved.

Partial failures happen regularly. A batch of 1,000 requests might complete with 980 successes and 20 failures. Each failure has its own error reason: content policy filters, malformed model responses, or occasional rate limit errors that should not apply to batch processing but sometimes do (documented across multiple OpenAI community forum threads from late 2025).

Recovering from partial failures requires parsing the output file line by line, matching custom_id values against the input to identify which requests failed, extracting those failed requests, resubmitting them as a new batch, waiting for that batch to complete, then merging both result sets into a single output. A reliable implementation takes 8 to 12 hours of focused engineering work, and it needs to run at 3am without human intervention.

This is where workflow automation earns its cost back. When a batch classification step fails inside a Chase Agents automation, the platform's error-recovery routing can catch the partial failure, extract the failed custom_id set, resubmit only those requests in a follow-up step, and merge the results before passing them downstream. That retry logic would take a developer a full day to build and test by hand.

When the openai batch api is worth it (and when it is not)

The openai batch api saves money on tokens. That claim is accurate and verifiable from OpenAI's published pricing page (openai.com/api/pricing). The 50% discount applies to every supported model.

The question worth asking is whether the token savings exceed the cost of building and maintaining the batch infrastructure for your specific volume. If monthly savings do not cover at least 10 hours of engineering time, the synchronous API is probably cheaper in total cost of ownership.

For a team processing 50,000+ calls per day, batch is almost certainly worth it regardless of overhead. For a team processing 500 calls per day, the synchronous API with prompt caching (which stacks with model-level discounts and does not require JSONL formatting) will likely save more money with less complexity. The batch API is a volume play, and the volume threshold is higher than most teams expect.