Batch inference runs many model requests as one asynchronous job instead of one call at a time. You upload a file of requests, they run within a completion window, and you download a file of results, which suits large jobs that do not need an immediate answer.
Most model calls are real time: an application sends a request and waits for the answer. Many workloads do not need an answer within seconds. Classifying a product catalog, summarizing an archive of documents, extracting fields from thousands of records or generating outputs for an evaluation set can all run in the background and finish later.
Batch inference runs that kind of work as one job. You submit every request at once, check the job's status, and collect all the results together. Your code does not keep thousands of connections open or pace its own requests, and on some models batch requests cost less than the same requests sent in real time.
custom_id that is unique in the file (up to 256 characters) and a body with the same request you would send in real time. A line may also set method to POST and url to the batch endpoint. One file holds up to 50,000 requests.POST /v1/files with purpose set to batch.POST /v1/batches with the returned input_file_id. endpoint defaults to /v1/chat/completions, completion_window is 24h, and metadata holds your own labels.GET /v1/batches/{batch_id}. Its status moves through validating, in_progress and finalizing to completed, or ends as failed, expired or cancelled.output_file_id and error_file_id with GET /v1/files/{file_id}/content.The Batch API guide has complete examples, and the create batch reference lists every field.
The output file has one line for each request that succeeded, and the error file one line for each request that did not. Each line carries the custom_id you set, so match results to requests by that ID. A request that ran has a response with its status_code and body. A request that never ran has response set to null and an error code such as batch_expired or batch_cancelled.
A batch that has not finished when its 24-hour window ends expires, and its unfinished requests appear in the error file; submit those again in a new batch. POST /v1/batches/{batch_id}/cancel stops a batch early, and requests that already ran stay billed. Input files expire 7 days after upload, and output and error files 7 days after the batch ends, so download results before then.
Only batch-eligible models can run in a batch. GET /v1/models reports batch_supported for each model, eligible model pages show a Batch API row, and a batch whose lines name other models is refused when you create it. The models listed on this page are the eligible ones.
Each line bills when it runs, at the model's list price less its batch_discount_pct. Some models run batch at a discount, and others at list price, where the value is 0. On a model with a batch discount, batch requests cannot use web search or built-in tools, so send those requests in real time. To estimate the cost of a job, count its tokens as described in what tokens are, and compare eligible text models.
8 models. Each model's page has its rate card, parameters and code.
| Model | Batch API | Category |
|---|---|---|
| Mistral Small 4 Mistral AI | Available, 35% off list price | Text Generation |
| Nova Lite 1.0 Amazon | Available, 35% off list price | Text Generation |
| Nova Lite 2 Amazon | Available, 35% off list price | Text Generation |
| Nova Micro 1.0 Amazon | Available, 35% off list price | Text Generation |
| Nova Pro 1.0 Amazon | Available, 35% off list price | Text Generation |
| Seed 2.0 Lite ByteDance | Available | Text Generation |
| Seed 2.0 Mini ByteDance | Available | Text Generation |
| Seed 2.0 Pro ByteDance | Available | Text Generation |
A batch runs within a 24-hour completion window. Requests that have not run when the window ends are returned in the error file as expired, so you can submit them again.
On some models. Each eligible model lists its batch discount, and a value of 0 means batch runs at the model's list price.
One input file holds up to 50,000 requests, one per line, each with its own custom_id.
Yes. Cancelling stops the requests that have not run, which appear in the error file as cancelled. Requests that already ran stay billed.
Check out our pricing or reach out if you want your own model deployed on our stack.