What is batch inference?

Batch inference runs many model requests as one asynchronous job instead of one call at a time. You upload a file of requests, they run within a completion window, and you download a file of results, which suits large jobs that do not need an immediate answer.

Concepts

When batch inference fits

Most model calls are real time: an application sends a request and waits for the answer. Many workloads do not need an answer within seconds. Classifying a product catalog, summarizing an archive of documents, extracting fields from thousands of records or generating outputs for an evaluation set can all run in the background and finish later.

Batch inference runs that kind of work as one job. You submit every request at once, check the job's status, and collect all the results together. Your code does not keep thousands of connections open or pace its own requests, and on some models batch requests cost less than the same requests sent in real time.

How the Batch API works

  1. Write a JSONL file. Put one request per line. Each line has a custom_id that is unique in the file (up to 256 characters) and a body with the same request you would send in real time. A line may also set method to POST and url to the batch endpoint. One file holds up to 50,000 requests.
  2. Upload it. Send the file to POST /v1/files with purpose set to batch.
  3. Create the batch. Call POST /v1/batches with the returned input_file_id. endpoint defaults to /v1/chat/completions, completion_window is 24h, and metadata holds your own labels.
  4. Poll its status. Call GET /v1/batches/{batch_id}. Its status moves through validating, in_progress and finalizing to completed, or ends as failed, expired or cancelled.
  5. Download the results. When the batch ends, fetch the files named by output_file_id and error_file_id with GET /v1/files/{file_id}/content.

The Batch API guide has complete examples, and the create batch reference lists every field.

Reading the results

The output file has one line for each request that succeeded, and the error file one line for each request that did not. Each line carries the custom_id you set, so match results to requests by that ID. A request that ran has a response with its status_code and body. A request that never ran has response set to null and an error code such as batch_expired or batch_cancelled.

A batch that has not finished when its 24-hour window ends expires, and its unfinished requests appear in the error file; submit those again in a new batch. POST /v1/batches/{batch_id}/cancel stops a batch early, and requests that already ran stay billed. Input files expire 7 days after upload, and output and error files 7 days after the batch ends, so download results before then.

Eligible models and billing

Only batch-eligible models can run in a batch. GET /v1/models reports batch_supported for each model, eligible model pages show a Batch API row, and a batch whose lines name other models is refused when you create it. The models listed on this page are the eligible ones.

Each line bills when it runs, at the model's list price less its batch_discount_pct. Some models run batch at a discount, and others at list price, where the value is 0. On a model with a batch discount, batch requests cannot use web search or built-in tools, so send those requests in real time. To estimate the cost of a job, count its tokens as described in what tokens are, and compare eligible text models.

Models available through the Batch API

8 models. Each model's page has its rate card, parameters and code.

ModelBatch APICategory
Mistral Small 4 Mistral AIAvailable, 35% off list priceText Generation
Nova Lite 1.0 AmazonAvailable, 35% off list priceText Generation
Nova Lite 2 AmazonAvailable, 35% off list priceText Generation
Nova Micro 1.0 AmazonAvailable, 35% off list priceText Generation
Nova Pro 1.0 AmazonAvailable, 35% off list priceText Generation
Seed 2.0 Lite ByteDanceAvailableText Generation
Seed 2.0 Mini ByteDanceAvailableText Generation
Seed 2.0 Pro ByteDanceAvailableText Generation

Frequently asked questions

How long does a batch take?

A batch runs within a 24-hour completion window. Requests that have not run when the window ends are returned in the error file as expired, so you can submit them again.

Is batch inference cheaper than real-time requests?

On some models. Each eligible model lists its batch discount, and a value of 0 means batch runs at the model's list price.

How many requests can one batch hold?

One input file holds up to 50,000 requests, one per line, each with its own custom_id.

Can I cancel a batch?

Yes. Cancelling stops the requests that have not run, which appear in the error file as cancelled. Requests that already ran stay billed.

Ready to use better endpoints?

Check out our pricing or reach out if you want your own model deployed on our stack.