Skip to main content

AI Providers

Per-project LLM provider configurations: model slug, optional base URL, optional configuration, and an optional link to a Secret that supplies the API key.

Overview​

An AI provider is a named configuration that tells the system how to reach a specific LLM endpoint. A project can have several providers. When a provider is linked to a secret, the secret's decrypted value is passed as the API key when calling the LLM; the key is never exposed through the API. See Connect Third-Party LLMs - Step 4 (Create provider records).

See the Permissions Reference for the IAM action strings for this module.

Data Model​

FieldTypeDescription
idstringPublic identifier (e.g. aip_…)
project_idstringID of the owning project
secret_idstring | nullPublic ID of the linked secret, or null
namestringHuman-readable label
providerAiProviderSlugProvider slug (see below)
default_modelstringDefault model name sent to the provider API
base_urlstring | nullOverride base URL (optional, useful for self-hosted LLMs)
configobject | nullArbitrary provider-specific configuration object
created_atstringISO 8601 creation timestamp
updated_atstringISO 8601 last-updated timestamp

Where a provider record may point​

base_url and config decide the URL the server requests, so both are bounded.

azure, bedrock and vertex have no base_url: the SDK builds the endpoint out of the record — <location>-aiplatform.googleapis.com, bedrock-runtime.<region>.amazonaws.com, <resourceName>.openai.azure.com. A value carrying a dot, a slash or an @ would name a different server, reached with the record's credential. So config.location, config.project, config.region and config.resourceName must each be a single name (letters, digits and hyphens); anything else is refused with 400 VALIDATION_FAILED on create and update, and 400 AI_PROVIDER_MISCONFIGURED when such a record is used.

base_url is checked for shape: an absolute http/https URL with no username or password (link a secret for the credential). Whether the endpoint may be reached is the deployment's egress rule, evaluated per request against the resolved address: a base_url inside your own network is refused unless the operator lists it in TOOL_EGRESS_ALLOWED_HOSTS, as for an http tool's target.

A model listing the provider rejects answers MODEL_LISTING_FAILED with the provider's status. The provider's response body goes to the server log, not to the caller: the host that wrote it is one the record named.

Provider Slugs​

Valid values for the provider field:

SlugDescription
openaiOpenAI
anthropicAnthropic
googleGoogle Gemini
xaixAI (Grok)
groqGroq
ollamaOllama (local)
azureAzure OpenAI
bedrockAmazon Bedrock
vertexGoogle Vertex AI
gatewayGeneric API gateway
customCustom / self-hosted model

A local ollama provider needs no linked secret; it uses the server's OLLAMA_BASE_URL. See Chat with an LLM - Step 3 (Create a local AI provider).

Key Concepts​

Bedrock authentication​

The bedrock provider supports two authentication modes, determined by the shape of the linked secret's JSON value.

IAM credentials — accessKeyId, secretAccessKey, and optionally sessionToken. The client signs requests with AWS SigV4.

{
  "accessKeyId": "<aws-access-key-id>",
  "secretAccessKey": "<aws-secret-access-key>",
  "sessionToken": "<optional-session-token>"
}

Bedrock API key — apiKey only (format ABSK…). The client uses Bearer token authentication via AWS_BEARER_TOKEN_BEDROCK (Amazon Bedrock API keys).

{ "apiKey": "ABSK..." }

Important: Store the secret value as a JSON object (shown above) — the only form that supports IAM credentials. A bare ABSK… string is also accepted and treated as { "apiKey": "<value>" }.

If neither field is present the default AWS credential chain (environment variables, instance profile, etc.) would be used — the deployment's credentials — which a deployment allows only by setting AI_PROVIDER_ALLOW_AMBIENT_CREDENTIALS. Without it, a bedrock record that links no secret is refused at create and update with 400 VALIDATION_FAILED, and with 400 AI_PROVIDER_MISCONFIGURED when such a record is used.

The region field in config defaults to us-east-1. An apiKey in config (without a linked secret) also works for quick testing; link a secret in production.

Bedrock request path​

Bedrock models are built on the Converse API — one path for every vendor Bedrock serves, and the reason a cache breakpoint is spelled cachePoint on bedrock and cacheControl on anthropic.

Converse is the common denominator, so Anthropic-only features (tool search with defer_loading, model betas) are not reachable through it: those are InvokeModel, which the AI SDK exposes as a separate Bedrock path with full Anthropic parity.

Decision (pre-v1): when one of those features is wanted, Anthropic models on Bedrock move to InvokeModel and every other vendor stays on Converse. Until then nothing changes. Recorded now because the switch is observable to agents that already exist — the breakpoint spelling, which features resolve, and how tokens are accounted — and a v1 contract should not freeze Converse by default merely because no one wrote the choice down.

Vertex AI authentication​

The vertex provider reaches Gemini models through Google Vertex AI: google calls the Gemini Developer API with a plain API key; vertex calls a Google Cloud project's regional endpoint and bills through that project (your own GCP project, VPC, and quota).

As with bedrock, the authentication mode is determined by the shape of the linked secret's value.

Service account — store the JSON key file verbatim as the secret value. The key file names its project, so no extra configuration is needed:

{
  "type": "service_account",
  "project_id": "my-gcp-project",
  "client_email": "vertex@my-gcp-project.iam.gserviceaccount.com",
  "private_key": "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n"
}

Express-mode API key — store the key on its own (no JSON wrapper), or as { "apiKey": "AIza..." }. Vertex AI in express mode targets a global, project-less endpoint, so project and location are ignored for this mode.

Application Default Credentials — link no secret and the server falls back to ADC: GOOGLE_APPLICATION_CREDENTIALS, Workload Identity, the GCE/GKE metadata server, or a local gcloud auth application-default login. No key material is stored.

Those are the deployment's credentials, so this mode is available only where the operator set AI_PROVIDER_ALLOW_AMBIENT_CREDENTIALS. Without it, a vertex record that links no secret is refused at create and update with 400 VALIDATION_FAILED, and with 400 AI_PROVIDER_MISCONFIGURED when such a record is used. It is a setting for a single-tenant deployment: on any other, it lets one project's record generate on the account the server runs as.

Federating an AWS identity (SOAT on ECS or EC2)​

ADC also covers SOAT running on AWS reaching Vertex through workload identity federation: point GOOGLE_APPLICATION_CREDENTIALS at the configuration gcloud iam workload-identity-pools create-cred-config --aws writes (it holds no secret material) and SOAT exchanges the task's own AWS identity for a Google access token. SOAT supplies the AWS half from the AWS default credential chain rather than the file's credential_source, so an ECS task role (delivered on the container credentials endpoint, which google-auth-library cannot read) works and is not replaced by the EC2 instance role.

Requirements: AWS_REGION (or AWS_DEFAULT_REGION) must be set on the server process (otherwise generations fail with AI_PROVIDER_MISCONFIGURED), and the pool provider's attribute condition must admit whichever role the credential chain resolves to (on ECS, the task role). Applies only to the ADC path with an AWS-sourced external_account configuration.

The provider's config object accepts two fields:

FieldDefaultDescription
project—Google Cloud project ID. Falls back to the project_id of a service-account secret. Required otherwise.
locationus-central1Vertex region serving the model, e.g. europe-west4 or global.
{ "project": "my-gcp-project", "location": "europe-west4" }

config.project overrides the key file's project_id, so one service account can serve models from several projects. When no project can be resolved (no config.project, and either no secret or one without project_id), creating a generation fails with AI_PROVIDER_MISCONFIGURED (400).

An apiKey in config is accepted as an express-mode fallback when no secret is linked, as for bedrock.

Listing the models a provider can run​

GET /api/v1/ai-providers/{ai_provider_id}/models asks the provider which models it can run, using that record's configuration and credentials, and returns provider-native ids — the same strings default_model and an agent's model carry. Reachable models are a property of the credential, not the slug.

Each entry carries what the provider reports: id, and optionally display_name, vendor, input_modalities, output_modalities, streaming, lifecycle (active / legacy / deprecated) and inference_types. A lifecycle other than active still serves but should not be pinned by anything new. A Bedrock model whose inference_types offers only inference_profile must be invoked through a cross-region profile id.

The response is { data, total, limit, offset }. ?q= keeps models whose id or display_name contains the text, ignoring case (?q=sonnet), and limit (default 50, max 100) / offset page the matches; total counts them. Both apply to the catalogue the provider returned in one call. A request without limit gets the first 50 models.

ProviderListingCredential the listing uses
openai, groq, xai, gateway, customGET {base_url}/models, so a self-hosted or proxied endpoint works toothe linked secret — required
anthropicGET /v1/modelsthe linked secret — required
googleAI Studio's model listthe linked secret — required
vertexthe Google publisher models the config.location region servesthe linked service-account key, else ADC
bedrockListFoundationModels in the provider's config.regionthe linked secret's IAM keys or API key, else the AWS default credential chain
azure, ollamaunsupported — Azure lists deployments an operator named, and Ollama lists whatever was pulled onto that host, so neither answers "which models can this provider run"—

Listing resolves credentials exactly as generation does, so a record that can generate can list — the deployment's own credentials included, on the same terms: a bedrock or vertex record that links none cannot list either, unless the operator set AI_PROVIDER_ALLOW_AMBIENT_CREDENTIALS.

  • A Vertex record needs no config.project when its secret is a service-account key; the key file names its project. config.project still overrides it.
  • Vertex express mode cannot list. The publisher-model listing rejects API keys (Google answers 401 UNAUTHENTICATED, "API keys are not supported by this API") and an express-mode record holds no other credential. Listing returns MODEL_LISTING_UNSUPPORTED naming the reason rather than forwarding that 401.
  • The Vertex list is a per-location publisher catalogue, not a per-project reachability check. The listing is rooted at publishers/google, so config.location selects the endpoint and the credential's project decides only who is billed and quota'd. Every location Vertex serves can be listed, including the non-regional ones: global, where several current Gemini models are served and which 404s in a region, and the eu / us data-residency multi-regions. A listed model can still fail at generation time if that project cannot serve it.
  • The Vertex list is advisory. The endpoint publishes embedding, TTS and classification models next to Gemini with no field distinguishing them, so nothing is filtered out; a listed id can 404 at generation time. For vertex, input_modalities and output_modalities are never reported (the API has no modality field), streaming is never reported, and lifecycle is active only for a model Google marks GA — a preview or experimental model reports no lifecycle.

Errors: MODEL_LISTING_UNSUPPORTED (400) for azure, ollama, and Vertex express mode; AI_PROVIDER_MISCONFIGURED (400) when the record lacks what the listing needs (a Vertex project from either config.project or the key file, a Bedrock region, or, for the API-key providers above, a linked secret); MODEL_LISTING_FAILED (502) when the provider rejects the request or answers with something other than JSON — its own status and message are carried in the error message. Authorized by ai-providers:ListAiProviderModels on the provider's project.

Listing models before you hold credentials​

On a deployment that set AI_PROVIDER_ALLOW_AMBIENT_CREDENTIALS, a bedrock or vertex record with no linked secret lists with the server's own credentials, so a vendor's live catalogue can be browsed before any key is provisioned: create a credential-less record naming only the region (or GCP project) and list against it:

soat create-ai-provider \
  --project-id proj_ABC \
  --name "Bedrock Catalog" \
  --provider bedrock \
  --default-model anthropic.claude-3-5-sonnet-20241022-v2:0 \
  --config '{"region":"us-east-1"}'

soat list-ai-provider-models --ai-provider-id aip_01

The record supplies the region and the IAM scope; the credential comes from the server's instance role. Without that setting the record is refused at creation; give it a key that can call ListFoundationModels (or the Vertex publisher listing) and it lists against its own credential.

Price overrides​

A per-provider price override is a price-book row bound to a specific AI provider (an enterprise-negotiated rate, a gateway with markup) that wins over the global default when usage cost is computed for that provider. Manage them with:

Both are authorized by the caller's access to the provider's own project (ai-providers:GetAiProviderPrices / ai-providers:ManageAiProviderPrices), so one project never sees another's rates. The provider slug is taken from the AI provider itself; supply the model, rates, and effective_from. effective_from must be in the future once the (model, component) has a price row (past prices are immutable; ship corrections as new future-dated rows), but a first price for a (model, component) nothing prices yet may be dated now or earlier, so a generation before the row lands is not metered at zero permanently. A refused row is named in error.meta (provider, model, component, effective_from). See Usage - Pricing for how the effective price is chosen and frozen onto each meter.

Deleting a provider​

DELETE /api/v1/ai-providers/{ai_provider_id} classifies everything that references the provider into two kinds:

DependentKindBehavior
Chats, agentsLive referenceAlways block with 409. force does not override them — delete or repoint each resource first.
Model routes whose targets name the providerLive referenceAlways block with 409. A target references its provider by id inside the route's targets, so no foreign key protects it — the guard is explicit. Repoint or delete the route first.
Price overridesSoft dependentBlock with 409 unless force=true, which deletes the overrides (meaningless without the provider).
Usage/generation recordsSoft dependentBlock with 409 unless force=true, which unlinks them (nulls the provider FK), preserving the row and its as-billed receipt.

A delete with no dependents (or force=true and only soft dependents) returns 204. On a 409 the response carries error.code = "AI_PROVIDER_HAS_DEPENDENTS" and an error.meta describing what blocked it:

{
  "error": {
    "code": "AI_PROVIDER_HAS_DEPENDENTS",
    "message": "AI provider 'aip_01' is in use by 2 chat(s), 1 agent(s) ...",
    "meta": {
      "chatCount": 2, "chatIds": ["chat_01", "chat_02"],
      "agentCount": 1, "agentIds": ["agent_01"],
      "modelRouteCount": 0, "modelRouteIds": [],
      "priceOverrideCount": 0, "usageEventCount": 0,
      "forcible": false
    }
  }
}

forcible is true only when the block comes solely from soft dependents, so a force=true retry would succeed. The *Ids arrays sample up to 20 offending IDs; the *Count fields report the true totals.

Examples​

Create an AI provider​

soat create-ai-provider \
  --project-id proj_ABC \
  --name "OpenAI GPT-4o" \
  --provider openai \
  --default-model gpt-4o \
  --secret-id sec_01

List providers in a project​

soat list-ai-providers --project-id proj_ABC

Set a per-provider price override​

soat update-ai-provider-prices \
  --ai-provider-id aip_ABC \
  --prices '[{"model":"gpt-4o","input_price_per_m":5,"output_price_per_m":15,"effective_from":"2099-01-01T00:00:00.000Z"}]'