Prompt Variables and Capabilities

The value you put in model names a task, not an LLM. A task — a capability — is the durable half of a call: the instructions, the output shape, the validity rules, the policy that picks a model. This page is about defining and editing one.

There are two ways to get instructions into a call, and they behave differently. Read the next section before anything else on this page.

Two ways to supply a prompt

On the OpenAI-compatible endpoint your messages are the prompt. The service folds them into a transport prompt whose entire body is a single slot. A capability’s stored system/user templates are not read on that path at all — so editing them changes nothing about how /v1/chat/completions behaves.

The v1 fold accepts a single user message plus any number of system messages, which are joined with a blank line between them. Four things are refused with 400:

• an assistant, tool or function role — “Multi-turn conversations are not supported in v1.”
• a second user message — “Only a single user message is supported in v1.”
• no user message at all
• a non-string content (vision / content-parts are not supported in v1)

If you want a template the service stores and fills in — one prompt, many calls, only the variables changing — use the native invocation endpoint.

Prompt variables

A template is text with {placeholder} slots. prompt_vars supplies the values.

POST /api/inference/v1/invocations/
Authorization: Bearer llmb_i_EXAMPLE_KEY_NOT_A_REAL_KEY

{
  "capability_slug": "support-triage",
  "workflow_id": "ticket-88421",
  "system_prompt_name": "SUPPORT_TRIAGE_SYSTEM_PROMPT",
  "user_prompt_name": "SUPPORT_TRIAGE_USER_PROMPT",
  "prompt_vars": {
    "ticket_body": "My invoice shows two charges for March.",
    "product": "Billing"
  }
}

capability_slug, workflow_id, system_prompt_name and user_prompt_name are all required. workflow_id is a free-form correlation string of yours; it is recorded with the call.

Substitution rules

• The renderer substitutes only {name}, where name is letters, digits and underscores.
• Substitution is single-pass. Braces inside a value are never re-scanned, so a variable containing { or } — JSON, code, another prompt — is safe.
• A placeholder with no matching key is left in place as literal text. (It will not normally get that far; see the next section.)
• A lone {, a lone }, and { } with nothing between the braces, are left untouched.
• {{name}} is not a supported escape. The renderer matches the inner {name} and substitutes it, leaving {value} — the outer braces survive as literal text.

Write {name} and nothing else. The pre-flight check described below recognises two further reference forms, {{name}} and ${name}, when deciding which variables are required. The renderer does not substitute ${name} at all — so a template containing ${foo} makes foo mandatory on every call while never being filled in.

Missing and extra variables

Before any model is called, the service compares the placeholders in your two templates against the keys you supplied.

• Missing a required variable fails the call up front. The native endpoint returns 503 with error_category: "config_error", error_code: "prompt_config_invalid", and a message naming the missing, required and supplied sets. Nothing is sent to a provider.
• Extra keys that no placeholder uses are ignored.
• Naming a prompt that does not exist under your account fails the same way (prompt_config_invalid).

prompt_vars on the OpenAI-compatible endpoint

extra_body.llmbench.prompt_vars is accepted there too, but because your messages are passed through as a single value and substitution is single-pass, the values are not substituted into your message text. They do reach the validator context — a declarative validator rule can read a named entry (see citations_preserved below). If you want templated prompts, use the native endpoint.

Keys beginning llmbench_ are reserved and rejected with 400 invalid_request_body, param: "llmbench.prompt_vars".

Where your content lives, and where it must not

The published Terms draw the line this way: the Template is the reusable instruction text; prompt variables are “where your content lives”.

That distinction is not cosmetic. Under the Terms’ data-grant table (§6.1), prompt variables are never stored on the shared_anon and private tiers, and retained verbatim on free. Template text, prompt versions and registered schemas are retained on every tier, including private, and are not covered by erasure. The Terms therefore prohibit personal data inside a Template on any tier, and any personal data at all on free.

Read §6.2 alongside that table, because it qualifies “never stored” in three ways that matter if you are relying on it:

• Batch is the big one. A call submitted through the Batch interface cannot be served without keeping its rendered request, so a working copy lives on the call record’s operational metadata on every tier including private. It is stripped when the batch finalizes, with a daily sweep and a 72-hour in-flight backstop behind that, plus a hard 24-hour batch expiry. For batch traffic “never stored” means “not stored beyond the operational window” — hours to a couple of days.
• shared_anon retains a routing embedding of your request, which the Terms treat as content rather than metadata. private retains no embedding.
• Benchmark sets you upload are exempt and are stored and re-run on every tier, which is what you uploaded them for.

Practically: put the variable, per-call, potentially sensitive part in prompt_vars, never in template text. Never put an API key, password or provider credential in either.

Creating a capability

Management-scope key (llmb_m_…), on the management plane.

POST /api/inference/v1/management/capabilities/
Authorization: Bearer llmb_m_EXAMPLE_KEY_NOT_A_REAL_KEY
Content-Type: application/json

{
  "slug": "support-triage",
  "display_name": "Support ticket triage",
  "description": "Classify an inbound ticket and extract the account reference.",
  "prompts": {
    "system": {"content": "You triage support tickets. Answer only with the schema."},
    "user":   {"content": "Product: {product}\n\nTicket:\n{ticket_body}"}
  },
  "output_schema": {
    "type": "object",
    "additionalProperties": false,
    "required": ["category", "urgency"],
    "properties": {
      "category":  {"type": "string", "enum": ["billing", "bug", "howto", "other"]},
      "urgency":   {"type": "integer", "minimum": 1, "maximum": 5},
      "account_ref": {"type": ["string", "null"], "maxLength": 32}
    }
  },
  "objective": "cheapest_good_enough",
  "batch_eligible": true
}

Fields

evaluation_policy_slug and human_evaluation_policy_slug are 422 on this plane — those govern how much evaluation gets bought and when human review fires, and are not customer-settable. There is no output_schema_class_path on this API at all: you register a JSON Schema, never a Python import path.

The response echoes the resolved configuration, including the prompt names the prompts were written under (system_prompt_name / user_prompt_name) — you need those for every native invocation — plus the seeded cells (model + provider) and any dropped_models with a reason (unknown_model, not_vaulted, model_inactive, provider_inactive, missing_flag:<requirement>).

Create-time refusals worth knowing

• 422, no eligible model. The task would be created but unable to serve anything — every model you named was dropped. dropped_models says why. Nothing is created; the whole create rolls back.
• 422, BYOK not serviceable. The task runs on your credentials and you hold no active vaulted key for any provider in its pool. The message names the providers and the route to fix it (PUT /api/inference/v1/vault/providers/{slug}). Partial coverage is fine — one usable provider in the pool is enough.
• 422, managed requires a billing account. pricing_mode: "managed" needs a billing customer on the account.
• 404, a named policy slug does not exist.

Prompts

A capability owns exactly two prompt names, derived from its slug:

support-triage  ->  SUPPORT_TRIAGE_SYSTEM_PROMPT
                    SUPPORT_TRIAGE_USER_PROMPT

(uppercased, hyphens to underscores). Omit name in the prompts block and you get these; they are what the create/update response reports back.

You may supply your own name — but only if it is not already taken. A name already in use elsewhere in your account, outside this capability’s own namespace, is refused with 409, because writing it would repoint every other task that resolves by that name. The practical consequence: a custom-named prompt can be created once and is not editable through this route afterwards. Omit name unless you have a reason.

Read the live content of a prompt with:

GET /api/inference/v1/management/prompts/{name}

which returns content, content_hash, version_counter, template, is_active.

Editing a prompt

Send prompts again on a PATCH:

PATCH /api/inference/v1/management/capabilities/support-triage
{"prompts": {"user": {"content": "Product: {product}\n\nTicket text:\n{ticket_body}"}}}

One rule governs edits: the set of {placeholders} must not change. Renaming or dropping a slot silently decouples the template from its call sites — the slot renders as literal text and the model is asked about something it was never shown — so an edit that changes the placeholder set is refused and that prompt is left as it was. Change the wording freely; to change the variables, change the call site and the template together by creating a new capability, or add the new task under a new slug.

Send prompt edits on their own. Prompts are written before the rest of a patch, and the system prompt before the user prompt — so if a two-prompt edit is refused on the second, the first has already been written.

Editing content that hashes identically to the current content is a no-op. Every render records which prompt version ran, so a call is always traceable to exact instruction text.

PUT /api/inference/v1/management/prompts/{name} is not an edit route. Without a config:direct_edit grant it returns 403; with one it returns 405 and points at the governed release surface. Edit prompts through the capability PATCH instead.

Prompts are normalised per model

Before a call runs, a template may be rewritten for the specific model that will serve it. The rewrite is required to preserve your {placeholder} set exactly; if it does not, or if it fails for any reason, the original template is used. Recorded provenance always references your template, never the rewritten text. Your slots, and the values you supply for them, are unaffected either way.

Output schemas

A capability can carry a JSON Schema that every output must satisfy. Register it inline at create/update time, or register it separately and reference it.

POST /api/inference/v1/output-schemas/
{"name": "triage-result", "json_schema": { ... }}

Registration is idempotent by content: identical content returns 200 with the existing ref, new content returns 201. The ref looks like schema:triage-result@1.

A capability stores a pinned ref (@version). Registering a new version does not move a task onto it — repointing is an explicit capability update. That is deliberate: your task’s definition of a valid answer should not change under you.

The supported subset

Schemas are compiled into a validating model over a bounded subset. Anything outside it is rejected at registration, with a JSON pointer to the offending node — never at call time.

Supported: root type: "object"; string, integer, number, boolean, object, array; properties; required; description; string enum; ["T", "null"] unions; anyOf (including a {"type": "null"} branch for optionals); internal $ref into $defs / definitions; minLength / maxLength, minimum / maximum, minItems / maxItems; items as a single object.

Rejected: oneOf, allOf, not, if / then / else, patternProperties, dependentSchemas, propertyNames, unevaluatedProperties, contains; external $ref; recursive $ref; tuple-form items; non-string enum; open objects (additionalProperties must be false or absent).

Limits: nesting depth 5, 200 nodes, 256 KB.

A property not listed in required is optional and may come back null. If a field must always be present, list it in required.

Per-call overrides

• Native: output_schema_ref_override on the request body.
• OpenAI-compatible: response_format. {"type": "json_schema"} registers or resolves your schema for that call; {"type": "json_object"} returns raw text checked to be JSON-parseable; {"type": "text"} or absent uses the task’s own schema.

An unsupported construct in response_format returns 400 invalid_request_body with param: "response_format".

Validators

A schema decides whether an output is well-formed. A validator decides whether it is acceptable. Set one per task with default_result_validator at create time, or per call with result_validator_override (native).

A validator that rejects an output makes that attempt a failed attempt: the model is excluded and the call is retried on an alternative model — which is the failover the Terms §2 describe. Only when the retries are exhausted does the call fail, as 500 output_validation_failed on the OpenAI-compatible surface.

Resolution is fail-closed. A non-blank validator name that resolves to nothing fails the call before any provider is contacted rather than silently skipping validation — and the same check runs at write time, so a typo is refused when you set it, not discovered in production.

Built-ins

Name one directly in default_result_validator:

A small number of built-ins are operator-only; naming one of those is refused at write time.

Your own validators

POST /api/inference/v1/validators/
{
  "name": "triage-sane",
  "spec_type": "declarative",
  "spec": {
    "spec_version": 1,
    "mode": "all",
    "rules": [
      {"type": "enum", "field": "category",
       "values": ["billing", "bug", "howto", "other"]},
      {"type": "numeric_range", "field": "urgency", "min": 1, "max": 5},
      {"type": "forbidden_substring", "field": "*",
       "values": ["as an AI language model"], "case_insensitive": true},
      {"type": "regex_match", "field": "account_ref",
       "pattern": "^ACC-[0-9]{6}$", "allow_missing": true}
    ]
  }
}

The spec is compiled and validated when you register it, with a pointer (rules[2]) to any offending rule. Re-posting the same name updates it and bumps its version only when the content changes. POST /api/inference/v1/validators/{name}/deactivate removes it from resolution; GET / and GET /{name} read them back.

Rule types — all rules must pass (mode: "all" is the only mode in v1):

Field addressing is a dot-path through the parsed output’s objects (account.reference); "*" means the whole output serialised as JSON. Only object keys resolve — an element of an array cannot be addressed, so a path segment that is a list index never matches anything. A field that is absent fails the rule with field_missing:<path> unless the rule sets allow_missing: true.

Limits: spec_version must be 1; spec ≤ 32 KB; ≤ 32 rules; regex pattern ≤ 512 characters; the subject is truncated to 256 KB before substring and regex checks.

The named_builtin form wraps a built-in with parameters instead:

{"name": "promo-280", "spec_type": "named_builtin",
 "spec": {"builtin": "promo_char_limit", "params": {"limit": 280}}}

Variant axes

variant_axes stratifies a task’s quality evidence, so a model that is strong on one kind of input is not credited for another. Axes are ordered, and selection walks from the most specific key down to the aggregate, dropping the rightmost axis at each step.

{"variant_axes": ["domain"]}

then per call:

{"variant_values": {"domain": "finance"}}

Resolvers exist for domain, language, input_size_band, agent and tenant. A value you pass in variant_values always wins over the derived one — so an axis name with no resolver only ever has the value you send.

Adding an axis does not create per-variant evidence retroactively; the task keeps serving from the aggregate pool until the more specific level has evidence of its own.

Editing a capability

PATCH /api/inference/v1/management/capabilities/{slug}

Patchable: display_name, description, prompts, output_schema / output_schema_ref, variant_axes, models, objective / selection_policy_slug, rollout_policy_slug, fanout_policy_slug, batch_eligible, idempotency_ttl_seconds, pricing_mode, evaluation_criteria_override, is_active.

default_result_validator is not patchable. A PATCH carrying it returns 400, field not patchable: 'default_result_validator'. Set it at create, or override it per call with result_validator_override on the native endpoint.

Two behaviours to plan for:

A model-set change that would empty the pool is refused (422) with nothing applied. Removals bench a model rather than deleting it, so its history survives and re-adding it later is a one-line patch.

An edit to what the task measures takes it off the shared evidence pool. Changing the prompts, the judging rubric (evaluation_criteria_override), the output schema, the validator or the variant axes makes later quality verdicts non-comparable with everyone else’s, so from that point the task’s bands are computed from your account’s own verdicts alone. The response to the create or PATCH that caused it carries stats_pooled: false. This is the mechanism that makes your rubric mean something; it is not a downgrade, but it does mean a freshly edited task has thinner evidence for a while.

Reading it back

A slug that does not exist and a slug belonging to another account return the same 404.

What this costs

Defining a task costs nothing. Calling one is priced by the pricing terms , and the rule differs by charging mode.

pricing_mode: "byok" — the default. We charge one per-call fee for the request and nothing else, and only for a request that delivered; a request of yours that failed carries no fee from us (§1.4). Your provider is a separate matter, and it is not limited in that way: every model call we make to serve your request runs on your keys, so your provider bills you for the alternatives we try alongside the one we return, the evaluation calls that compare them, repairs of malformed output, and our retries after a provider error (§3.2). Exploration is a setting on the task — you can turn it off for a single request or for the task as a whole.

pricing_mode: "managed", while the task is still exploring. §1.2 applies as written: every model call in the chain appears on your invoice, per model — the alternatives, the evaluation calls that compare them, calls that failed, and our own retries after a provider error. Once that task reaches a published rate, §1.4’s rules take over: a request that does not deliver a result is not billed at all, and the retries, alternatives and evaluation calls are ours.

A validator rejection retries the call on another model (see Validators); which pricing clause covers that retry is set out in the pricing terms, not here.

Limits

Control-plane writes (create, patch, register, deactivate) are rate-limited at 300 per hour, per account — every management key you hold shares the one budget — and return 429 with Retry-After: 3600 when the limit is reached. Reads are not counted against that limit.