API Reference: Data Plane

The data plane is where you send work. It speaks the OpenAI wire format, so an OpenAI SDK pointed at it works unmodified for the calls described here.

from openai import OpenAI

client = OpenAI(
    api_key="llmb_i_EXAMPLE_KEY_NOT_A_REAL_KEY",
    base_url="https://YOUR-HOST/v1",
)

Two things to know before anything else:

1. The model field names a task, not an LLM. A task is a named unit of work. We pick the model that runs it, and we may pick a different one next call. If you want one exact model, pin it with llmbench.forced_model (see the llmbench extension block ).
2. Streaming is refused, not ignored. stream: true returns a 400. There is no SSE code path. See Streaming .

Authentication

Authorization: Bearer <key>. Keys start with llmb_, then a one-letter scope tag:

Scopes are not interchangeable. A management key on /v1/chat/completions is a 403 key_plane_mismatch, not a 401 — the key is valid, it is just on the wrong plane. A missing, unknown, revoked or expired key is a 401 invalid_api_key.

The plane publishes its own OpenAPI schema at /v1/openapi.json, with an interactive docs view at /v1/docs.

POST /v1/chat/completions

What is identical to OpenAI

• The request envelope: model, messages, response_format.
• The response body: id, object: "chat.completion", created, model, choices[0].message, usage. There is exactly one choice and finish_reason is always "stop".
• usage includes prompt_tokens_details.cached_tokens and completion_tokens_details.reasoning_tokens.
• The error envelope: {"error": {"message", "type", "param", "code"}}, with type drawn from OpenAI’s own set so SDK string-matching keeps working.
• 429 responses carry Retry-After in seconds, so an SDK’s built-in retry works.

What differs

Messages

• messages must be non-empty and must contain a user message.
• At most one user message. A second one is a 400 unsupported_value (“Only a single user message is supported in v1”).
• assistant, tool and function roles are a 400 unsupported_value (“Multi-turn conversations are not supported in v1”).
• Every content must be a string. Content-parts arrays (the vision shape) are a 400 invalid_request_body.
• Multiple system messages are allowed; they are joined with a blank line between them.

Parameters we ignore, and parameters we refuse

Most top-level keys that are not model, messages, response_format, stream or llmbench are accepted, have no effect, and are listed back to you in x-llmbench-ignored-params (comma-separated). That includes the usual suspects:

temperature, top_p, max_tokens, max_completion_tokens, presence_penalty,
frequency_penalty, seed, stop, user, logit_bias, stream_options

These are refused with a 400 unsupported_parameter when they carry a meaningful value: tools, tool_choice, functions, function_call, logprobs, audio, modalities, web_search_options, and n when it is not 1.

Benign off-values are accepted rather than spuriously rejected, so a repointed OpenAI client that always sends tools: [], tool_choice: "none", logprobs: false or modalities: ["text"] keeps working. Those params are the exception to the reporting rule above: a benign off-value on one of them — and n: 1 — is accepted silently and is not listed in x-llmbench-ignored-params.

response_format

A JSON Schema using a construct we do not support returns 400 invalid_request_body with param: "response_format". Any other type value is also a 400.

The llmbench extension block

The OpenAI SDK merges extra_body into the JSON body root, so llmbench arrives as a top-level key. The block is strict: an unknown key is a 400, never a silent no-op.

resp = client.chat.completions.create(
    model="invoice-classifier",
    messages=[{"role": "user", "content": "…"}],
    extra_body={"llmbench": {
        "selection": {"strategy": "relative", "quality_threshold": 0.9},
        "trace_id": "job-4417",
    }},
)

There is no objective field. If you have older code sending one, it now returns 400 invalid_request_body with param: "llmbench" rather than being ignored.

llmbench.selection

selection narrows the task’s configured policy for one call. Anything you omit is inherited; nothing is persisted. It is strict — an unknown key is a 400.

strategy: "cost" applies no score gate — it is not “cheapest that is still good enough”. For that, send strategy: "relative" with a quality_threshold.

An empty or all-null selection object is treated as absent.

An unhonourable selection block is refused before any provider call, so a bad spec never costs you anything.

Response headers

Six headers are on every 200:

Nine more appear when applicable:

The three quality-* headers are emitted together or not at all. Absent means “not resolvable”, never “not met” — we would rather tell you nothing than tell you a promise was broken when we could not check.

The same values are mirrored into the response body under a top-level llmbench object, with the x-llmbench- prefix stripped and hyphens kept:

{
  "id": "chatcmpl-3f0c…",
  "object": "chat.completion",
  "model": "invoice-classifier",
  "choices": [{"index": 0, "finish_reason": "stop",
               "message": {"role": "assistant", "content": "…"}}],
  "usage": {"prompt_tokens": 812, "completion_tokens": 96, "total_tokens": 908,
            "prompt_tokens_details": {"cached_tokens": 0},
            "completion_tokens_details": {"reasoning_tokens": 0}},
  "llmbench": {
    "call-id": "b1f0…", "capability": "invoice-classifier",
    "model": "some-model-v2", "provider": "some-provider",
    "service-tier": "standard", "cost-usd": "0.00042100",
    "band": "MEDIUM", "phase": "exploit"
  }
}

The model that ran can change between calls

If a model’s output fails our format checks, or a model does not return an output at all, we try an alternative model for that call rather than failing it. That is why x-llmbench-model may differ from one call to the next for the same model slug. The one exception is forced_model: with a pin there is no alternative, and the call fails.

Streaming is refused

client.chat.completions.create(model="…", messages=[…], stream=True)

HTTP/1.1 400 Bad Request
x-llmbench-error-code: streaming_not_supported
x-llmbench-error-class: router
x-llmbench-diagnostic: 0a3d…  (32 hex chars)

{"error": {
  "message": "Streaming is not supported by this Service. Re-send the request with stream=false. (diagnostic: 0a3d…)",
  "type": "invalid_request_error",
  "param": "stream",
  "code": "streaming_not_supported"
}}

We refuse rather than quietly ignoring stream, so that a client written against a streaming API fails visibly instead of appearing to stream. The Terms of Service say the same thing contractually: neither synchronous nor batch mode streams.

Practical consequences:

• The check runs during translation, before any model is selected and before any provider call — so a refused streaming request costs no provider spend and produces no call record.
• It runs after the rate/quota gate, so it does consume one quota unit and one recorded inbound request.
• One check serves both lanes. On this endpoint it is the response; on a batch input line it becomes that line’s entry in the error file.

GET /v1/models

Lists the tasks you can put in the model field — not the underlying LLMs. Listing LLMs would hand you ids this plane rejects, which looks like it worked and then fails.

{"object": "list", "data": [
  {"id": "invoice-classifier", "object": "model", "created": 0, "owned_by": "organization"},
  {"id": "summarize-long-doc",  "object": "model", "created": 0, "owned_by": "llmbench"}
]}

• owned_by is llmbench for tasks we ship and organization for your own.
• created is always 0. We do not map a task’s creation time onto this field, and a fabricated one would be worse than an obvious zero.
• The list is fully drained, not paginated — the SDK calls it once and reads data.
• Inactive tasks are omitted. GET /v1/models/{id} returns the same 404 for an inactive task as for one that never existed.

Errors

Every error is an OpenAI-shaped envelope. type stays inside OpenAI’s recognized set (invalid_request_error, authentication_error, permission_error, rate_limit_error, insufficient_quota, server_error); the fine-grained vocabulary is in code.

Errors from /v1/chat/completions — and errors raised while validating a file upload or a batch submission — carry three headers:

A 429 also carries Retry-After in seconds. A failure that reached invocation also carries x-llmbench-request-id; failures refused before invocation — a quota 429, an unknown model, a malformed body, the streaming refusal — deliberately carry no request id, because no route was ever opened and inventing one would be worse than omitting it.

GET/DELETE on a file or a batch, and the cancel route, return the same JSON envelope without those headers and without the (diagnostic: …) suffix — that is where file_not_found, batch_not_found and batch_not_cancellable come from on a lookup. Errors raised by POST /v1/files and POST /v1/batches do carry the headers and the suffix — including file_not_found for an unknown input_file_id, batch_cannot_complete and the create-time insufficient_balance 429.

Any provider message echoed to you is first stripped of key-shaped tokens and truncated to 500 characters.

Reading x-llmbench-error-class

Codes emitted before any model runs

Codes mapped from an invocation failure

A no_eligible_model on a task that runs on your own keys usually means you have not vaulted a key for any provider in its pool; the message names the providers and the vault endpoint to use.

Batch API

The Batch API is /v1/files plus /v1/batches, in OpenAI’s shape.

POST   /v1/files                       upload a JSONL input file (purpose="batch")
GET    /v1/files/{file_id}             metadata
GET    /v1/files/{file_id}/content     raw JSONL bytes
DELETE /v1/files/{file_id}             delete the content
POST   /v1/batches                     create
GET    /v1/batches                     list  (after, limit; limit clamped to 1–100)
GET    /v1/batches/{batch_id}          retrieve
POST   /v1/batches/{batch_id}/cancel   cancel (idempotent)

Input files

Every non-empty line must parse as a JSON object at upload time; full per-line structural validation happens after the batch is created. A file with no request lines is rejected.

Each line:

{"custom_id": "row-0001", "method": "POST", "url": "/v1/chat/completions",
 "body": {"model": "invoice-classifier",
          "messages": [{"role": "user", "content": "…"}],
          "llmbench": {"max_wait_seconds": 3600}}}

• custom_id — required, a non-empty string of at most 256 characters, unique within the file.
• method must be POST; url must equal the batch’s endpoint.
• body is a chat-completions body, with the same rules and the same llmbench block as the interactive endpoint — plus max_wait_seconds, which is valid only here.

Creating a batch

endpoint must be "/v1/chat/completions"; anything else is a 400.

metadata is capped at 16 keys, keys at 64 characters, values at 512 characters.

Your open-batch cap comes from your own configured limit. 0 means unlimited, and an account with no limits row falls back to 5. Exceeding it is a 429 too_many_open_batches with Retry-After: 60.

A successful create returns 200 with status: "validating". The file is parsed after that, asynchronously — so most content problems surface as a batch that moves to failed, not as an error on the create call.

Delta 1 — every line in a batch must route identically

This is our largest departure from OpenAI’s Batch API. A batch is one group: one task, one selection spec, one pin. model, llmbench.selection and llmbench.forced_model must agree across every line. Everything else — messages, custom_id, prompt_vars, response_format — is free to vary.

A file where every line repeats the same values is fine, which is what an SDK naturally emits. Only genuine disagreement is refused. That refusal happens during validation, not on the create call: a file whose lines disagree on routing fails the whole batch (status: "failed"), with the offending line numbers in errors.data and in an error file.

You can hoist the shared values instead of repeating them, via a top-level llmbench block on the create call. This is ours, not OpenAI’s — a hoisted file will not run against real OpenAI — so it is offered and never required.

client.batches.create(
    input_file_id="file-EXAMPLE0000",
    endpoint="/v1/chat/completions",
    completion_window="24h",
    extra_body={"llmbench": {
        "model": "invoice-classifier",
        "system": "You classify invoices.",
        "selection": {"strategy": "relative", "quality_threshold": 0.9},
    }},
)

Hoistable keys are exactly model, system, selection, forced_model, response_format. A malformed hoist block — an unknown key, a typo’d selecton — is the one routing problem that does 400 on the create call itself, rather than failing the batch later: a bad shared value fails once here instead of routing ten thousand calls under the wrong policy.

A value present at both levels must match. A line may never override package routing. A hoisted file and a fully-specified one build byte-identical requests.

Delta 2 — completion_window is per-attempt, not total elapsed

completion_window accepts Nm / Nh, floored at 15 minutes, capped at 24 hours, default "24h". Out of range or unparseable is a 400 invalid_request_body with param: "completion_window".

It is a per-attempt budget. If the window passes without an output, we cancel that attempt and reschedule the line to an alternative model with a fresh full window, excluding the model that timed out. That repeats until the line succeeds, every eligible model has been tried, or the batch hits its absolute expiry.

A line can tighten its own budget with llmbench.max_wait_seconds, clamped to [900, completion_window]. It can never widen it.

Delta 3 — the hard 24-hour expires_at

Independently of completion_window, every batch gets an absolute expires_at of created + 24 hours. When that passes, every still-open line becomes an error-file line with code batch_expired and the message “request did not complete before the batch expired”.

At submit time, a batch whose optimistic estimate cannot finish inside that 24 hours is refused up front with 400 batch_cannot_complete. The message names the line count, the lane, the minimum minutes needed, and how much of your own work is already queued ahead of it.

Statuses

validating, failed, in_progress, finalizing, completed, expired, cancelling, cancelled. The terminal set is failed, completed, expired, cancelled.

Cancel is idempotent: cancelling an already-cancelling or cancelled batch returns 200. Cancelling a terminal batch returns 409 batch_not_cancellable.

The batch object

GET /v1/batches/{id} returns the stock OpenAI object — id, object, endpoint, errors, input_file_id, completion_window, status, output_file_id, error_file_id, the per-state timestamps, request_counts.{total,completed,failed}, metadata — plus a non-standard llmbench block when there is something to say (a submission-time estimate for a window we thought was threatened, and how the package actually ran). SDKs ignore it.

Structural per-line validation failures fail the whole batch (status: "failed") and appear both in errors.data — capped at 100 entries, each with a line number — and in an error file.

Result lines

Output-file lines:

{"id": "batch_req_1041", "custom_id": "row-0001",
 "response": {"status_code": 200, "request_id": "b1f0…",
              "body": {"…a normal chat.completion, including its own llmbench mirror…"}},
 "error": null}

The per-line llmbench mirror carries call-id, capability, model, provider, service-tier, cost-usd, and optionally ignored-params and band. service-tier is where the value batch shows up. The mirror carries no phase and no switch-in — a batch client never sees a live response, so this mirror is the only channel those values would have had, and only the ones that are meaningful after the fact are included.

Error-file lines carry the diagnostic in-body, in one of three shapes.

A line that got a status code — whether from an invocation or from a refusal made before one (a malformed line, a refused parameter, an unknown model, streaming) — keeps the response shape. request_id is null when no route was ever opened:

{"id": "batch_req_1042", "custom_id": "row-0002",
 "response": {"status_code": 502, "request_id": "c9aa…",
              "body": {"error": {"message": "…", "type": "server_error",
                                 "param": null, "code": "provider_error"}}},
 "error": null}

{"id": "batch_req_1043", "custom_id": "row-0003",
 "response": {"status_code": 400, "request_id": null,
              "body": {"error": {"message": "…", "type": "invalid_request_error",
                                 "param": "stream", "code": "streaming_not_supported"}}},
 "error": null}

A line the batch’s absolute expiry cut off has no status code, so it uses the bare error shape:

{"id": "batch_req_1044", "custom_id": "row-0004",
 "response": null,
 "error": {"message": "request did not complete before the batch expired",
           "type": "invalid_request_error", "param": null, "code": "batch_expired"}}

A whole-batch structural failure writes a third shape — no line id, no custom_id, and a line key instead of an OpenAI type:

{"id": null, "custom_id": null, "response": null,
 "error": {"code": "invalid_line", "line": 42, "message": "…", "param": null}}

Delta 4 — an unknown model on a batch line hard-fails

On the interactive endpoint an unrecognised model string can be routed to a similar existing task or minted as a new draft. On a batch line it does not. An unknown model fails that line with 404 model_not_found, so a bulk file of typos cannot mass-create tasks.

Delta 5 — batch.* webhook events go only to registered endpoints

When a batch reaches a terminal state, we queue one event per active registered webhook endpoint. Zero registered endpoints means zero events. There is no fallback delivery target — if you have not registered one, poll GET /v1/batches/{id}.

Endpoints are registered on the management plane, with a management key:

POST   /api/inference/v1/management/webhook-endpoints/              register
GET    /api/inference/v1/management/webhook-endpoints/              list (never re-shows secrets)
POST   /api/inference/v1/management/webhook-endpoints/{id}/rotate   new secret
DELETE /api/inference/v1/management/webhook-endpoints/{id}          deactivate

The signing secret is returned exactly once, on create and on rotate. It is not recoverable — rotate mints a new one.

The event body:

{"object": "event",
 "id": "evt_9f2c…",
 "type": "batch.completed",
 "created": 1756000000,
 "data": { "…exactly what GET /v1/batches/{id} returns…" }}

So the event carries status and file ids, not results. Fetch the output file yourself.

Deliveries are POSTed with Content-Type: application/json, X-Timestamp: <unix seconds> and X-Signature: sha256=<hex>. Verify by computing HMAC-SHA256 over the bytes "<timestamp>." + <raw body> with your endpoint’s secret, over the raw body, before parsing it.

Webhook delivery is best-effort and polling is authoritative. The Terms say so, and nothing about a batch finishing depends on a delivery succeeding. Build the poll path even if you use webhooks.

Rate limits and quotas

Two separate 429s, distinguished by code:

Both carry Retry-After in seconds, so an OpenAI SDK’s built-in retry handles the first one without changes. A third 429 exists on the batch surface — too_many_open_batches, described under Batches — and insufficient_balance is also a 429; neither is a rate limit.

The numbers

Read your own values, and your current counters, at GET /api/inference/v1/quota/. A null limit means unlimited. Your open-batch cap is not in that payload — it is returned by GET /api/inference/v1/batches/limits, because it carries the opposite null convention: there, 0 means unlimited.

qps_inference is one bucket per account, shared by every key you hold and every data-plane route. /v1/chat/completions, /v1/files, /v1/batches, the native /api/inference/v1/invocations, /api/inference/v1/batches and /api/inference/v1/packages, and a run from the console Playground all draw down the same 10 requests per second, because the counter is keyed on the account and not on the key or the route. Two keys on one account share 10 between them, as do one key used from four processes. Issuing a second key isolates a workload’s credentials, not its rate: a noisy key starves the account’s other keys, so size every client against the account’s number together.

The window is fixed, not sliding. The count resets on the wall-clock second, so a burst straddling one boundary can put 20 requests through in a few milliseconds. Size your client against the average, not against the boundary.

Retry-After is the window, not a measured wait — 1 for qps_inference, 3600 for control_mutations_per_hour. On the data plane the counter has usually rolled well before the second is up.

No tier caps your call volume or your spend. Those four columns exist and are published at GET /quota/, and no plan sets them: what decides whether a call is served is your prepaid balance, not a call counter.

A 400 raised inside the gateway — a bad body, a refused parameter, the streaming refusal — still consumes one quota unit and one recorded inbound request; the counter is an abuse control, not the meter. Authentication failures (401 / 403) and the two provider-key-header 400s (credential_mode_conflict, zero_custody_not_enabled) do not.

A note on cost while a task is exploring

Some tasks run more than one model while we establish which performs best. x-llmbench-phase reports the task’s current fan-out phase and x-llmbench-switch-in estimates the judged calls remaining before it switches to cost-optimised. Read exploit as “fan-out is not exploring right now” — it is not a statement about which rate the call was billed at.

While a task is exploring, every model call in the chain appears on your invoice, per model — including alternatives whose output was not returned to you, the evaluation calls that compare them, calls that failed, and our own retries after a provider error. Once the task reaches a stable published rate you are billed that rate instead and the exploration is ours to fund.

Because evaluation runs candidate models against each other, its share of your provider spend is materially higher than its share of your call count.

The full rules — including the three different failure-billing rules for a published rate, an exploring task, and your own keys — are in the pricing terms and the terms of service , which govern.