Migrating from OpenAI

The OpenAI-compatible plane is mounted so that an OpenAI SDK works against it unmodified. Three things change in your code: the base URL, the key, and what you put in model. Everything else about the request and response envelope is the same shape you already parse.

This page is honest about the parts that are not drop-in. Read the two sections marked “fails loudly” before you plan the migration — if your code streams or holds multi-turn conversations, that is real work, not a config change.

The 30-second version

Before:

from openai import OpenAI

client = OpenAI(api_key="sk-EXAMPLE-not-a-real-key")

resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {"role": "system", "content": "You extract invoice fields as JSON."},
        {"role": "user", "content": "Invoice text here..."},
    ],
    temperature=0.2,
)
print(resp.choices[0].message.content)

After:

from openai import OpenAI

client = OpenAI(
    api_key="llmb_i_EXAMPLEKEYPLACEHOLDERxxxxxxxxxxxxxxxxxxxxx",
    base_url="https://dtp.kapualabs.com/v1",
)

resp = client.chat.completions.create(
    model="invoice-extract",          # a TASK, not an LLM
    messages=[
        {"role": "system", "content": "You extract invoice fields as JSON."},
        {"role": "user", "content": "Invoice text here..."},
    ],
    temperature=0.2,                  # accepted and ignored, see below
)
print(resp.choices[0].message.content)

The response is a standard chat.completion object with one choice and finish_reason: "stop", plus one extra top-level llmbench key (section 5). Your existing parsing keeps working — SDKs ignore unknown fields.

1. Base URL and key

The compat plane is mounted at /v1/, so the full path is exactly POST /v1/chat/completions. Set base_url to https://dtp.kapualabs.com/v1 — your dashboard’s Overview page renders the exact value for your deployment, and the API host is not necessarily the same host as this documentation site.

Keys start with llmb_, then a one-letter scope tag, then a 43-character random secret:

Both use Authorization: Bearer <key>, which is what the SDK’s api_key sets. Scopes are not interchangeable: a management key on /v1/ is 403 key_plane_mismatch, not 401. A missing or unrecognised key is 401 invalid_api_key. Anything that does not start with llmb_ is rejected without a database lookup.

Only a SHA-256 digest of the key is stored. The plaintext is shown exactly once, at issue or rotation; it cannot be recovered afterwards. The dashboard shows a key as its first 7 characters and last 4 (llmb_i_…wxyz) — that hint is not the secret and will not authenticate.

2. model is a task, not an LLM

This is the conceptual change, and it is the one worth understanding before you write any code.

The Terms define a Capability as “a named unit of work you call — the value you put in the model field of an OpenAI-compatible request. It is not a specific model.” You name the job; the Service selects a model to run it, and may try an alternative if the first one fails our format checks or returns nothing at all. So "gpt-4o-mini" has no meaning here as a routing instruction, and the model that actually ran is reported back in a response header rather than chosen by you.

GET /v1/models therefore lists tasks, not LLMs:

for m in client.models.list().data:
    print(m.id, m.owned_by)
# invoice-extract   organization
# summarize-long    llmbench

owned_by is llmbench for tasks we ship and organization for your own. created is always 0 — no creation timestamp is carried on this projection, and a fabricated one would be worse than an obvious zero. Inactive tasks are omitted from the list, and GET /v1/models/{id} returns the same 404 for an inactive task as for one that never existed. The list is drained in full, not paginated.

You do not have to create a task first

On POST /v1/chat/completions an unrecognised model string is resolved in order: exact slug, then alias, then lazy onboarding — which either routes you to an existing task or mints a new draft named after your slug. The draft is servable immediately. The x-llmbench-routing header tells you which of the four happened:

Note that derived and minted both create a task and both count against your draft cap; only alias and exact serve an existing one.

There is a per-account daily cap on new drafts. Exceeding it fails as 404 model_not_found with (daily new-model limit reached for this tenant) in the message. A model string longer than 128 characters is a 400.

The one thing you must do before the first call

A lazily-minted task defaults to running on your provider credentials. Selection only considers providers you have an active vaulted key for, so with an empty vault the eligible pool is empty and the call fails 503 no_eligible_model. The message names the exact providers and the endpoint to fix it.

Vault a key first, with a management key:

curl -X PUT https://dtp.kapualabs.com/api/inference/v1/vault/providers/anthropic \
  -H "Authorization: Bearer llmb_m_EXAMPLEKEYPLACEHOLDERxxxxxxxxxxxxxxxxxxx" \
  -H "Content-Type: application/json" \
  -d '{"api_key": "sk-ant-EXAMPLE-not-a-real-key"}'

That stores the key, activates the provider for your account, and runs an auth-check ping. GET /api/inference/v1/vault/providers lists what you have (metadata only, never key material); DELETE revokes. Partial coverage is fine — one vaulted provider in the pool is enough to serve.

So the migration order is: register, vault at least one provider key, then call.

3. What fails loudly

These return an error rather than being quietly worked around. That is deliberate: a client written against a feature we do not have should break visibly, not appear to work.

Streaming is refused

{
  "error": {
    "message": "Streaming is not supported by this Service. Re-send the request with stream=false. (diagnostic: 0f3c…)",
    "type": "invalid_request_error",
    "param": "stream",
    "code": "streaming_not_supported"
  }
}

HTTP 400, with x-llmbench-error-code: streaming_not_supported and x-llmbench-error-class: router. There is no SSE code path anywhere in the routers, and the same refusal applies to a batch input line — where it becomes that line’s entry in the error file.

Terms of Service section 2 says the same thing contractually: “Neither mode streams, and we do not offer streaming at this time… setting it to true is refused with an error rather than served — in synchronous and in batch mode alike.”

The check runs inside translation, before a model is selected and before any provider call, so a refused streaming request costs no provider spend and produces no call record. It runs after the quota gate, so it does consume one quota unit and one recorded inbound request — exactly like any other 400.

Migration impact: if you stream for perceived latency, you lose that and there is no workaround on this plane. Plan for a single complete response. If you stream to get incremental usage frames, the full usage block is on the completed response instead.

Multi-turn conversations are not supported in v1

messages must be non-empty, contain at most one user message, and every content must be a string.

Multiple system messages are fine; they are joined with a blank line between them.

Migration impact: this is the biggest one for chat-shaped applications. A conversation loop that appends assistant turns to messages will 400 on its second call. You have to fold the history you want the model to see into the single user message yourself.

Parameters that are refused

These are 400 unsupported_parameter when they carry a meaningful value: tools, tool_choice, functions, function_call, logprobs, audio, modalities, web_search_options, and n when it is not 1.

Benign off-values are accepted rather than spuriously rejected — tools: [], tool_choice: "none", logprobs: false, modalities: ["text"] all pass, because a repointed SDK often sends them by default.

4. What is silently ignored — and how you find out

Sampling parameters are accepted and ignored: the task’s own configuration owns generation parameters.

The rule is simple and there is no allowlist to memorise: every parameter you send that is not in the refused set above is accepted and ignored. That covers the sampling knobs you are used to (temperature, top_p, max_tokens, seed, stop, logit_bias, user, …) and equally any undeclared parameter your client happens to add, such as stream_options.

It is not silent in practice: every parameter that was ignored is echoed back, comma-separated, in the x-llmbench-ignored-params response header, and mirrored into the response body. Log that header during a migration and you will see immediately which of your knobs stopped doing anything.

response_format is honoured:

5. What you gain

Response headers

Every 200 carries six headers, always:

And these when applicable: x-llmbench-request-id (the route id — quote it in a support request and use it to look the route up in your dashboard), x-llmbench-ignored-params, x-llmbench-routing, x-llmbench-band (LOW/MEDIUM/HIGH/RANKED — the evidence band of the served model on this task), x-llmbench-phase (explore/exploit/continuous), and x-llmbench-switch-in (an estimate of how many more judged calls remain before the task settles onto cost-optimised routing — omitted both when it is not computable and when the answer is zero).

Three quality headers report the bar and whether it was met: x-llmbench-quality-floor, x-llmbench-quality-met (true/false) and x-llmbench-quality-requested (true only if you set the bar rather than inheriting the task’s default). They are omitted together when the block cannot be resolved — absent means “not resolvable”, never “not met”.

Reading them with the OpenAI SDK:

raw = client.chat.completions.with_raw_response.create(
    model="invoice-extract",
    messages=[{"role": "user", "content": "Invoice text here..."}],
)
print(raw.headers["x-llmbench-model"], raw.headers["x-llmbench-cost-usd"])
completion = raw.parse()

The same values are also mirrored into the response body under a top-level llmbench key, with the x-llmbench- prefix stripped and hyphens kept. The SDK keeps unknown body fields, so you can read them off the parsed object without dropping to raw responses:

completion = client.chat.completions.create(model="invoice-extract", messages=[...])
print(completion.model_extra["llmbench"])
# {'call-id': '…', 'capability': 'invoice-extract', 'model': 'claude-…',
#  'provider': 'anthropic', 'service-tier': 'standard', 'cost-usd': '0.00042100',
#  'band': 'HIGH', 'phase': 'exploit'}

Note that the standard model field of the response echoes the slug you requested. The model that actually ran is in the headers and the mirror. If you have a dashboard that graphs resp.model, repoint it.

Per-call routing control: extra_body

The OpenAI SDK merges extra_body into the JSON body root, which is where the llmbench extension block lives. It has exactly eight fields and it is strict — an unknown key is a 400, never a silent no-op, so a typo cannot quietly do nothing:

selection narrows the task’s configured policy for one call. Anything you omit is inherited, and nothing is persisted:

resp = client.chat.completions.create(
    model="invoice-extract",
    messages=[{"role": "user", "content": "Invoice text here..."}],
    extra_body={"llmbench": {
        "selection": {"strategy": "relative", "quality_threshold": 0.9},
        "trace_id": "job-7741",
    }},
)

On a task we serve and price ourselves, exactly two fields are rejected: min_confidence here, and forced_model below. Two others, quality_threshold and model_group, are honoured while such a task is still gathering evidence and go inert once it matures — the request is served, the field is dropped, and the field name comes back in x-llmbench-ignored-params. That is never an error. The fan-out fields are not gated at either mode. selection is strict too, and an object that is entirely empty or all-null is treated as absent.

If you read an older draft of our docs mentioning an objective field: it does not exist. Sending it is a 400 invalid_request_body with param: "llmbench". Note also that strategy: "cost" is not the old cheapest, which kept a quality bar — the equivalent of that is strategy: "relative" with a quality_threshold.

To pin one exact LLM, use forced_model, subject to holding a provider credential for it. The Terms are explicit that pinning removes the fallback: if a pinned model fails, there is no alternative and the call fails.

6. Batch

The Batch API surface is close to OpenAI’s and mostly ports as-is. POST /v1/files (with purpose="batch"), GET /v1/files/{id}, GET /v1/files/{id}/content, DELETE /v1/files/{id}, POST /v1/batches, GET /v1/batches, GET /v1/batches/{id}, POST /v1/batches/{id}/cancel. Ids are OpenAI-shaped (file-…, batch_…), the batch object is the stock shape, and the status vocabulary is the familiar one (validating, failed, in_progress, finalizing, completed, expired, cancelling, cancelled).

Caps: input files are JSONL, at most 100 MB and 10,000 lines. endpoint must be /v1/chat/completions — anything else is a 400. Client metadata is capped at 16 keys, 64-character keys, 512-character values. If an open-batch cap is configured on your account, exceeding it is 429 too_many_open_batches with Retry-After: 60; with no cap configured, none applies.

Five differences worth planning for:

Every line in a batch must route identically. model, llmbench.selection and llmbench.forced_model must agree across the whole file. This is checked during validation, before any line runs — but it is not an exception from batches.create(): the create call returns 200 with status validating, and the batch then moves to status failed with heterogeneous_routing entries naming the line numbers in its error file. Poll the batch or read the error file rather than expecting the SDK call to raise. messages, prompt_vars, custom_id and response_format are free to vary — those are content. You can hoist the shared values to a package-level llmbench block on the create call (model, system, selection, forced_model, response_format) instead of repeating them on every line; a value present at both levels must match.

An unknown model on a batch line hard-fails with 404 model_not_found. Lazy onboarding is deliberately off on the batch lane, so a bulk file of typos cannot mass-create tasks. This differs from the interactive endpoint, where the same string would be onboarded.

completion_window is per-attempt, not total elapsed. It accepts Nm/Nh, floored at 15 minutes, capped at 24 hours, defaulting to "24h". If the window passes without an output, that attempt is cancelled and the line is rescheduled to a different model with a fresh full window, excluding the model that timed out. A line can tighten (never widen) its own budget with llmbench.max_wait_seconds.

There is a separate absolute 24-hour backstop. Every batch also carries an expires_at of created + 24 hours, independent of completion_window. Lines still open at that point become error-file lines with code batch_expired. At submit time, a batch whose estimate cannot finish inside that window is refused up front with 400 batch_cannot_complete, and the message names the line count, the lane, the minutes needed and any of your work already queued ahead of it.

Webhooks go only to registered endpoints. One event per active registered webhook endpoint when a batch reaches a terminal state; zero registered endpoints means zero events and no fallback delivery. Register them on the management plane at POST /api/inference/v1/management/webhook-endpoints/. Events are {"object": "event", "id": "evt_…", "type": "batch.<status>", "created": …, "data": <the batch object>} — a pointer, not results — signed with X-Timestamp and X-Signature: sha256=<hex> over "<timestamp>." + raw_body. The Terms state that webhook delivery is best-effort and that you can check status and results through the API at any time regardless; polling GET /v1/batches/{id} is the authoritative path.

Output lines are the familiar chat.completion shape, and each line’s response.body carries an llmbench mirror — but a narrower one than the interactive endpoint’s. It has call-id, capability, model, provider, service-tier and cost-usd, plus band and ignored-params when they are available. phase, switch-in, routing, request-id and the quality block are interactive-only; do not write batch parsing that expects them.

Error lines carry the OpenAI-shaped error either inside response.body.error (when the failure came back from an invocation) or at top level in error (when it was refused before invocation).

7. Errors

The envelope is unchanged: {"error": {"message", "type", "param", "code"}}. type stays inside OpenAI’s recognised set (invalid_request_error, authentication_error, permission_error, rate_limit_error, insufficient_quota, server_error) so SDK string-matching keeps working. The fine-grained vocabulary lives in code, which is ours and does not match OpenAI’s — if you branch on error.code, that is code to update.

Three headers ride errors from the chat-completions path — everything raised at the gateway, everything mapped back from an invocation, and the throttling and quota 429s:

• x-llmbench-diagnostic — a 32-hex id, also appended to the message as (diagnostic: …). Quote it in a support request.
• x-llmbench-error-class — whose fault it is. This is the fastest retry signal you have:

Note the shape of gateway in particular: it is not only admission control. A tools=[...], an n=2 or a second user message all arrive as x-llmbench-error-class: gateway with no Retry-After, and retrying them will never succeed.

• x-llmbench-error-code — the same value as error.code.

Two narrowings worth knowing:

Plain 404s carry the body only. GET /v1/models/{id}, GET /v1/files/{id} and GET /v1/batches/{id} return the OpenAI error body for a miss with no x-llmbench-* headers and no (diagnostic: …) suffix in the message. Do not write error handling that requires the diagnostic id to be present on every non-2xx.

x-llmbench-request-id only appears once a route was opened. A failure that reached invocation carries it. Failures refused before invocation — quota 429, unknown model, malformed body, the streaming refusal — deliberately carry none, because no route was ever opened and a fabricated id would be worse than an absent one.

429s also carry Retry-After in seconds, and OpenAI SDKs auto-retry 429 honouring it, so short waits self-heal without code changes.

8. Cost expectations

Do not assume the per-token economics you are used to carry over unchanged, and read the pricing terms rather than inferring from a single invoice line.

The part that surprises people: while a task is still establishing which model performs best, every model call in the chain appears on your invoice, per model — alternatives, the evaluation calls that compare them, failed calls, and our retries after a provider error. Once a task reaches a stable published rate you are billed that rate instead and the exploration is ours to fund. Terms section 11 adds the expectation-setting sentence: because evaluation runs candidate models against each other, its share of your provider spend is materially higher than its share of your call count.

Failure billing differs by charging mode, and each mode states its own rule:

9. Migration checklist

1. Issue an inference-scope key (llmb_i_…) and a management-scope key (llmb_m_…). Store the plaintext at issue time — you cannot get it back.
2. Vault at least one provider key with the management key, or your first call is a 503.
3. Set base_url to the value your dashboard shows, and api_key to the llmb_i_… key.
4. Grep your codebase for stream=True on this client. Every one of those call sites needs rewriting.
5. Grep for conversation loops that append to messages. One user message, no assistant turns.
6. Grep for tools=, functions=, n= greater than 1, logprobs=True, and image content parts. All refused.
7. Replace hardcoded LLM names in model with task slugs, or let the first call resolve one and check x-llmbench-routing — derived and minted both mean a task was created for you.
8. Log x-llmbench-ignored-params for a day. It tells you exactly which of your parameters stopped mattering.
9. Repoint anything that read resp.model to x-llmbench-model or completion.model_extra["llmbench"]["model"].
10. If you branch on error.code, map the new codes — or branch on x-llmbench-error-class instead, which is coarser and more stable.
11. For batch: confirm every line in a file routes identically, poll for the terminal status rather than relying on batches.create() to raise, and register a webhook endpoint if you want notifications.

The interactive wire contract is also discoverable from the server itself: the compat plane is mounted with its OpenAPI schema and docs view, including the stream field’s refusal description.