Skip to main content

Chats

LLM completions with optional persistent configuration, supporting both stateless and per-chat modes.

Overview​

All completions run through a single endpoint, POST /chat/completions, which names exactly one target:

  • Stateless (ai_provider_id) — OpenAI-compatible; pass the full provider configuration on every request. No setup required.
  • Per-chat (chat_id) — create a Chat resource once to store the AI provider, default instructions, and model; then pass only chat_id and the messages array per request.

The two are mutually exclusive, and a request naming neither — or both — is rejected with 400.

Both targets support SSE streaming via stream: true. To see a completion driven end to end through a provider-backed flow, follow Connect Third-Party LLMs - Step 6 (Start a conversation).

See the Permissions Reference for the IAM action strings for this module.

Data Model​

Chat​

FieldTypeDescription
idstringPublic ID prefixed with chat_
project_idstringPublic ID of the owning project
ai_provider_idstring | nullPublic ID of the pinned AI provider, or null when the chat pins none and inherits its project's default_model_route_id
namestringOptional human-readable name
instructionsstringOptional default system prompt applied to all completions — the same name an Agent uses
modelstringOptional model override (falls back to provider's default_model)
created_atstringISO 8601 creation timestamp
updated_atstringISO 8601 last-updated timestamp

Message​

Each message in the messages array sent to the completions endpoint:

FieldTypeDescription
roleuser | assistantIdentifies the author of the message. system is refused — see System Instructions
contentstringText body (use this or document_id, not both)
document_idstringPublic ID of a document — the server resolves its content before the call

Key Concepts​

System Instructions​

System content never travels as a message — one rule, on every SOAT surface. On a completion it goes in the instructions request field — the same name everywhere: a completion request, a Chat, an Agent — and a role: "system" entry in messages is refused with 400 SYSTEM_MESSAGE_NOT_ALLOWED.

The server sends the field to the provider as its instructions argument, which is the only place the underlying AI SDK accepts it — allowSystemInMessages defaults to false there and throws, because a system message inside a caller-supplied array is a prompt-injection vector. SOAT's wire contract is the same contract.

The same rule everywhere else: an agent's system prompt is its instructions field (Agents), and a conversation's stored history carries only user and assistant turns (Conversations) — all three refuse a system entry with the same 400.

Per-chat override​

A Chat stores instructions applied to every completion on it. A single call replaces them by supplying its own instructions. The Chat record is not modified.

The stored prompt applies only when the request carries none. The two are never merged: combining them would produce a prompt neither the chat nor the caller wrote.

AI Provider Resolution​

For per-chat completions the AI provider is taken from the Chat record, and the pin must name a provider in the chat's own project — one from another project answers 400 AI_PROVIDER_NOT_FOUND, indistinguishably from an id that exists nowhere. A chat created without ai_provider_id pins none and resolves through its project's default_model_route_id instead, which gives its completions ordered provider failover; model cannot be combined with that (each route target names its own), and omitting the provider returns 400 when the project has no default.

For a stateless completion ai_provider_id is passed directly in the request body and is required — that call belongs to no chat, so there is no chat binding and no default to inherit. It is still scoped to a project: the provider's own — see Authorization.

See AI Providers for the full list of supported providers and how secrets are resolved. For a worked example of creating a provider the Chat can reference, see Chat with an LLM - Step 3 (Create a local AI provider).

Authorization​

Both targets are gated on the same action, chats:CreateChatCompletion, each checked against the project the call belongs to:

TargetProject the check runs against
chat_idthe chat's project
ai_provider_idthe AI provider's project

A caller without the action on that project gets 403, before any provider call and before an SSE stream is opened — a refused streaming request is a JSON 403, never an error frame inside a 200 stream. An ai_provider_id that does not exist is still 404, which is resolved before the permission check.

Streaming​

Set stream: true in the request body to receive an SSE stream. Each event contains a JSON object with a choices[0].delta.content chunk. The stream ends with data: [DONE].

Upstream provider errors​

When the provider rejects the completion — an unavailable model, a refused credential — or cannot be reached, POST /api/v1/chat/completions answers 502 AI_PROVIDER_ERROR with the provider's own status and message in the error message:

{
  "error": {
    "code": "AI_PROVIDER_ERROR",
    "message": "Provider returned 404: model \"gemini-2.0-flash\" not found"
  }
}

This is the same mapping Agents generation applies, so probing which models a provider can actually serve gives an interpretable answer instead of a bare 500.

A streaming request cannot report this as a status code — its 200 and headers are written before the provider is called. The failure arrives as a terminal data: {"error": "..."} frame carrying the same message, and the stream then ends without a [DONE].

Document-Backed Messages​

A message may carry a document_id instead of inline content. The server fetches that document and uses its content field as the message body. jq-based selection of tool output (the output_path behavior) is handled by Agents.

Examples​

Create a chat​

soat create-chat \
  --project-id proj_ABC \
  --ai-provider-id aip_abc123 \
  --name "Support Assistant" \
  --instructions "You are a helpful support assistant."

Run a per-chat completion​

Once a Chat is stored, run completions against it by passing chat_id and the messages array — the AI provider, instructions, and model come from the Chat record.

A Chat stores configuration, not conversation history: no message sent to or returned from a completion is persisted, so send the full messages array on every call.

soat create-chat-completion \
  --chat_id chat_01 \
  --messages '[{"role":"user","content":"What can you help me with?"}]'

Run a stateless completion​

soat create-chat-completion \
  --ai-provider-id aip_abc123 \
  --instructions "You are a helpful assistant." \
  --messages '[{"role":"user","content":"Hello!"}]'