Connect Third-Party LLMs

This tutorial connects SOAT to hosted LLM providers such as xAI, OpenAI, Anthropic, and Amazon Bedrock, showing how Secrets, AI Providers, Agents, and Sessions work together for externally hosted models.

Prerequisites

SOAT running locally. Follow Quick Start if needed.
CLI installed and configured, or SDK set up. See CLI or SDK.
Server is at http://localhost:5047.
Valid credentials for at least one third-party provider.

CLI
SDK
curl

export SOAT_BASE_URL=http://localhost:5047

import { SoatClient } from '@soat/sdk';

export SOAT_BASE_URL=http://localhost:5047

Step 1 — Log in as admin

Admin is the built-in superuser role. It bypasses policy evaluation entirely. See Users for full authentication and user management details.

CLI
SDK
curl

ADMIN_TOKEN=$(soat login-user --username admin --password Admin1234! | jq -r '.token')
export SOAT_TOKEN=$ADMIN_TOKEN

const soat = new SoatClient({ baseUrl: 'http://localhost:5047' });

const { data: session, error } = await soat.users.loginUser({
  body: { username: 'admin', password: 'Admin1234!' },
});

if (error) throw new Error(JSON.stringify(error));

const adminSoat = new SoatClient({
  baseUrl: 'http://localhost:5047',
  token: session.token,
});

ADMIN_TOKEN=$(curl -s -X POST "$SOAT_BASE_URL/api/v1/users/login" \
  -H "Content-Type: application/json" \
  -d '{"username":"admin","password":"Admin1234!"}' | jq -r '.token')

Step 2 — Create a project

Every resource in SOAT lives inside a project. Create one to hold the provider and agent.

CLI
SDK
curl

PROJECT_ID=$(soat create-project --name "Hosted LLM Demo" | jq -r '.id')
echo "PROJECT_ID: $PROJECT_ID"

const { data: project, error } = await adminSoat.projects.createProject({
  body: { name: 'Hosted LLM Demo' },
});

if (error) throw new Error(JSON.stringify(error));

const PROJECT_ID = project.id;

PROJECT_ID=$(curl -s -X POST "$SOAT_BASE_URL/api/v1/projects" \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name":"Hosted LLM Demo"}' | jq -r '.id')

Step 3 — Store provider credentials as secrets

Secrets store sensitive values encrypted; providers reference them by ID. Create one secret per provider credential set.

CLI
SDK
curl

OPENAI_SECRET_ID=$(soat create-secret \
  --project-id "$PROJECT_ID" \
  --name "openai-api-key" \
  --value "sk-<your-openai-key>" | jq -r '.id')

ANTHROPIC_SECRET_ID=$(soat create-secret \
  --project-id "$PROJECT_ID" \
  --name "anthropic-api-key" \
  --value "sk-ant-<your-anthropic-key>" | jq -r '.id')

XAI_SECRET_ID=$(soat create-secret \
  --project-id "$PROJECT_ID" \
  --name "xai-api-key" \
  --value "xai-<your-xai-key>" | jq -r '.id')

BEDROCK_SECRET_ID=$(soat create-secret \
  --project-id "$PROJECT_ID" \
  --name "bedrock-credentials" \
  --value '{"accessKeyId":"<aws-access-key-id>","secretAccessKey":"<aws-secret-access-key>","sessionToken":"<optional-session-token>"}' | jq -r '.id')

const { data: openAiSecret } = await adminSoat.secrets.createSecret({
  body: {
    project_id: PROJECT_ID,
    name: 'openai-api-key',
    value: 'sk-<your-openai-key>',
  },
});

const { data: anthropicSecret } = await adminSoat.secrets.createSecret({
  body: {
    project_id: PROJECT_ID,
    name: 'anthropic-api-key',
    value: 'sk-ant-<your-anthropic-key>',
  },
});

const { data: xaiSecret } = await adminSoat.secrets.createSecret({
  body: {
    project_id: PROJECT_ID,
    name: 'xai-api-key',
    value: 'xai-<your-xai-key>',
  },
});

const { data: bedrockSecret } = await adminSoat.secrets.createSecret({
  body: {
    project_id: PROJECT_ID,
    name: 'bedrock-credentials',
    value:
      '{"accessKeyId":"<aws-access-key-id>","secretAccessKey":"<aws-secret-access-key>","sessionToken":"<optional-session-token>"}',
  },
});

OPENAI_SECRET_ID=$(curl -s -X POST "$SOAT_BASE_URL/api/v1/secrets" \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{\"project_id\":\"$PROJECT_ID\",\"name\":\"openai-api-key\",\"value\":\"sk-<your-openai-key>\"}" \
  | jq -r '.id')

XAI_SECRET_ID=$(curl -s -X POST "$SOAT_BASE_URL/api/v1/secrets" \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{\"project_id\":\"$PROJECT_ID\",\"name\":\"xai-api-key\",\"value\":\"xai-<your-xai-key>\"}" \
  | jq -r '.id')

Step 4 — Create provider records

Each provider points to a hosted model endpoint. See AI Providers for the full list of supported providers and configuration options. Choose the provider that matches your hosted model:

CLI
SDK
curl

# OpenAI
OPENAI_PROVIDER_ID=$(soat create-ai-provider \
  --project-id "$PROJECT_ID" \
  --name "OpenAI" \
  --provider "openai" \
  --default-model "gpt-4.1-mini" \
  --secret-id "$OPENAI_SECRET_ID" | jq -r '.id')

# Anthropic
ANTHROPIC_PROVIDER_ID=$(soat create-ai-provider \
  --project-id "$PROJECT_ID" \
  --name "Anthropic" \
  --provider "anthropic" \
  --default-model "claude-3-5-sonnet-latest" \
  --secret-id "$ANTHROPIC_SECRET_ID" | jq -r '.id')

# xAI
XAI_PROVIDER_ID=$(soat create-ai-provider \
  --project-id "$PROJECT_ID" \
  --name "xAI" \
  --provider "xai" \
  --default-model "grok-3-mini" \
  --secret-id "$XAI_SECRET_ID" | jq -r '.id')

# Bedrock (secret value can be JSON credentials, region can live in config)
BEDROCK_PROVIDER_ID=$(soat create-ai-provider \
  --project-id "$PROJECT_ID" \
  --name "Bedrock" \
  --provider "bedrock" \
  --default-model "anthropic.claude-3-5-sonnet-20240620-v1:0" \
  --secret-id "$BEDROCK_SECRET_ID" \
  --config '{"region":"us-east-1"}' | jq -r '.id')

const { data: openAiProvider } = await adminSoat.aiProviders.createAiProvider({
  body: {
    project_id: PROJECT_ID,
    name: 'OpenAI',
    provider: 'openai',
    default_model: 'gpt-4.1-mini',
    secret_id: openAiSecret.id,
  },
});

const { data: xaiProvider } = await adminSoat.aiProviders.createAiProvider({
  body: {
    project_id: PROJECT_ID,
    name: 'xAI',
    provider: 'xai',
    default_model: 'grok-3-mini',
    secret_id: xaiSecret.id,
  },
});

const { data: bedrockProvider } = await adminSoat.aiProviders.createAiProvider({
  body: {
    project_id: PROJECT_ID,
    name: 'Bedrock',
    provider: 'bedrock',
    default_model: 'anthropic.claude-3-5-sonnet-20240620-v1:0',
    secret_id: bedrockSecret.id,
    config: { region: 'us-east-1' },
  },
});

OPENAI_PROVIDER_ID=$(curl -s -X POST "$SOAT_BASE_URL/api/v1/ai-providers" \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{\"project_id\":\"$PROJECT_ID\",\"name\":\"OpenAI\",\"provider\":\"openai\",\"default_model\":\"gpt-4.1-mini\",\"secret_id\":\"$OPENAI_SECRET_ID\"}" \
  | jq -r '.id')

Step 5 — Create an agent

Once the provider exists, create an agent that points at it.

CLI
SDK
curl

AGENT_ID=$(soat create-agent \
  --project-id "$PROJECT_ID" \
  --ai-provider-id "$OPENAI_PROVIDER_ID" \
  --name "Hosted Assistant" \
  --instructions "You are a helpful assistant using a hosted LLM." \
  | jq -r '.id')

const { data: agent } = await adminSoat.agents.createAgent({
  body: {
    project_id: PROJECT_ID,
    ai_provider_id: openAiProvider.id,
    name: 'Hosted Assistant',
    instructions: 'You are a helpful assistant using a hosted LLM.',
  },
});

AGENT_ID=$(curl -s -X POST "$SOAT_BASE_URL/api/v1/agents" \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{\"project_id\":\"$PROJECT_ID\",\"ai_provider_id\":\"$OPENAI_PROVIDER_ID\",\"name\":\"Hosted Assistant\",\"instructions\":\"You are a helpful assistant using a hosted LLM.\"}" \
  | jq -r '.id')

Step 6 — Start a conversation

Create a session and send a message through the provider-backed agent.

CLI
SDK
curl

SESSION_ID=$(soat create-session --agent-id "$AGENT_ID" | jq -r '.id')

soat add-session-message \
  --session-id "$SESSION_ID" \
  --message "Summarize why model routing matters in one paragraph."

const { data: session2 } = await adminSoat.sessions.createSession({
  body: { agent_id: agent.id },
});

await adminSoat.sessions.addSessionMessage({
  path: { session_id: session2.id },
  body: { message: 'Summarize why model routing matters in one paragraph.' },
});

SESSION_ID=$(curl -s -X POST "$SOAT_BASE_URL/api/v1/sessions" \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{\"agent_id\":\"$AGENT_ID\"}" | jq -r '.id')

curl -s -X POST "$SOAT_BASE_URL/api/v1/sessions/$SESSION_ID/messages" \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"message":"Summarize why model routing matters in one paragraph."}'

What's next

Provider rotation: Create multiple provider records in the same project and switch agents between them.
Custom gateways: Use the gateway or custom provider types when you have an OpenAI-compatible upstream.
Production secrets: Rotate provider secrets by creating a new secret and updating the provider's secret_id.

Prerequisites​

Step 1 — Log in as admin​

Step 2 — Create a project​

Step 3 — Store provider credentials as secrets​

Step 4 — Create provider records​

Step 5 — Create an agent​

Step 6 — Start a conversation​

What's next​