Skip to main content

Ingest Images and Audio with Converters

Native file ingestion turns PDFs and text files into searchable Documents. This tutorial extends it to images and audio by routing each unsupported content_type to a converter through an Ingestion Rule — and demonstrates the two converter kinds side by side:

Both routes reuse the same chunk + embed pipeline, so the converted text ends up searchable like any other document.

Runs against mock providers — no keys needed

Every provider/tool call is directed at a base_url you configure, so the flow can run against stand-in servers instead of the real APIs. The tutorials test runner does exactly this via the mock-providers service in tests/docker-compose.tutorials.yml, which answers with canned text after verifying the received bytes match the checked-in fixtures byte-for-byte.

Prerequisites​

  • SOAT running locally. Follow the Quick Start guide.
  • New to SOAT? Read Key Concepts first.
  • For production hardening (storing provider keys as secrets), see Configuration.
  • CLI installed and configured, or SDK set up. See CLI or SDK.
  • Provider credentials for real runs: an OpenAI API key with access to a vision model (gpt-4o or similar), and an xAI API key with access to its speech-to-text endpoint. For provider setup patterns see Connect Third-Party LLMs. Neither key is needed when running against the mock providers described above.
  • The fixture files (receipt.png, meeting.mp3) are checked into the repo at packages/website/docs/tutorials/fixtures/. Run this tutorial from a clone of the SOAT repo at the repo root — $FIXTURES_DIR below points there by default.
export SOAT_BASE_URL=http://localhost:5047   # CLI, SDK, and curl — do NOT append /api/v1

# Provider endpoints and keys. The defaults are the real providers; each is
# overridable so the tutorial can also run against local mocks (see the tip above).
export OPENAI_BASE_URL="${OPENAI_BASE_URL:-https://api.openai.com/v1}"
export OPENAI_API_KEY="${OPENAI_API_KEY:-sk-your-openai-key}"
export XAI_BASE_URL="${XAI_BASE_URL:-https://api.x.ai/v1}"
export XAI_API_KEY="${XAI_API_KEY:-xai-your-key}"

# Where this tutorial's fixture files live — override if your clone (or the
# directory you copied fixtures/ into) is somewhere else.
export FIXTURES_DIR="${FIXTURES_DIR:-./packages/website/docs/tutorials/fixtures}"

Step 1 — Log in as admin​

Admin is the built-in superuser and bypasses policy evaluation. See Users for authentication details.

ADMIN_TOKEN=$(soat login-user --username admin --password Admin1234! | jq -r '.token')
export SOAT_TOKEN=$ADMIN_TOKEN

Step 2 — Create a project​

Every resource lives inside a project.

PROJECT_ID=$(soat create-project --name "Media Ingestion" | jq -r '.id')
echo "PROJECT_ID: $PROJECT_ID"

Part A — Images and scanned PDFs via an OpenAI agent converter​

You point an Ingestion Rule at an agent and SOAT sends the file to it as multimodal input with a fixed "extract all text" instruction — no request/response mapping to write. For images and scanned PDFs, an OpenAI vision model does OCR directly.

Step 3 — Store the OpenAI key as a secret​

The agent authenticates through an AI provider, and the provider reads its credentials from a Secret rather than an inline key — so the key is encrypted at rest and never returned in API responses.

OPENAI_SECRET_ID=$(soat create-secret \
  --project-id "$PROJECT_ID" \
  --name "openai-api-key" \
  --value "$OPENAI_API_KEY" | jq -r '.id')
echo "OPENAI_SECRET_ID: $OPENAI_SECRET_ID"

Step 4 — Create a vision AI provider​

Create an AI provider backed by OpenAI with a vision-capable default_model, reading its key from the secret above. base_url points at OpenAI (overridden to the mock in CI).

OPENAI_PROVIDER_ID=$(soat create-ai-provider \
  --project-id "$PROJECT_ID" \
  --name "OpenAI Vision" \
  --provider "openai" \
  --default-model "gpt-4o" \
  --base-url "$OPENAI_BASE_URL" \
  --secret-id "$OPENAI_SECRET_ID" | jq -r '.id')
echo "OPENAI_PROVIDER_ID: $OPENAI_PROVIDER_ID"

Step 5 — Create the OCR agent​

Create an agent whose only job is to transcribe what it sees. The instructions matter most here: keep the model from summarizing or commenting, so the document text is the raw extracted content.

OCR_AGENT_ID=$(soat create-agent \
  --project-id "$PROJECT_ID" \
  --ai-provider-id "$OPENAI_PROVIDER_ID" \
  --name "OCR Agent" \
  --instructions "Extract all text from the provided file verbatim. Return plain text only — no commentary, no summary, no markdown fences." \
  | jq -r '.id')
echo "OCR_AGENT_ID: $OCR_AGENT_ID"

Step 6 — Route images to the agent​

Create an Ingestion Rule mapping image/* to the agent with agent_id. Agent converters take the file directly, so there is no file_delivery to choose and no request shape to map.

soat create-ingestion-rule \
  --project-id "$PROJECT_ID" \
  --content-type-glob "image/*" \
  --agent-id "$OCR_AGENT_ID" \
  --chunk-strategy "whole" | jq '{id: .id, content_type_glob: .content_type_glob}'

Step 7 — (Optional) OCR fallback for scanned PDFs​

A scanned PDF has content_type: application/pdf but no text layer, so the native parser yields nothing. A rule matching application/pdf is consulted only when native extraction returns no text — see Ingestion Rules — Content-Type Matching. Pointing it at the same vision agent makes it a scanned-PDF fallback; born-digital PDFs still skip the converter. (To OCR every PDF regardless of its text layer, set native_extraction: skip on the rule.)

soat create-ingestion-rule \
  --project-id "$PROJECT_ID" \
  --content-type-glob "application/pdf" \
  --agent-id "$OCR_AGENT_ID" \
  --chunk-strategy "whole" | jq '{id: .id, content_type_glob: .content_type_glob}'

Step 8 — Ingest an image without naming a converter​

Upload an image as a File, then ingest it exactly like a PDF or text file. Nothing about the call names the agent or the rule — POST /documents/ingest resolves the matching rule from the file's content_type automatically. Uploading via base64 lets us set content_type explicitly to image/png, which is what drives routing. $FIXTURES_DIR/receipt.png is a small receipt image with real text for the model to OCR.

IMAGE_FILE_ID=$(soat upload-file-base64 \
  --project-id "$PROJECT_ID" \
  --filename "receipt.png" \
  --content-type "image/png" \
  --content "$(base64 -w0 "$FIXTURES_DIR/receipt.png")" | jq -r '.id')
echo "IMAGE_FILE_ID: $IMAGE_FILE_ID"

soat ingest-document \
  --project-id "$PROJECT_ID" \
  --file-id "$IMAGE_FILE_ID" \
  --path-prefix "/images/" \
  --wait true | jq -e '.status == "ready"'
# prints `true` once the image is OCR'd, chunked, and embedded
# (chunk_count is reported by `soat get-document-status`; Step 14 confirms the text is searchable)

That is the whole image path: a secret, a provider, an agent, and one rule. (Against a real OpenAI account the model occasionally returns a non-answer; re-ingest with soat reingest-document if .status comes back failed.)

Part B — Audio via an xAI tool converter​

xAI's speech-to-text REST API (POST /v1/stt) is not a chat-completions endpoint, so no agent can call it — the case tool converters are for: an http tool calls the API and a pipeline tool reshapes the response into the bare-string shape ingestion rules expect. The general pattern is documented in Ingestion Rules — Building a Tool Converter for a Third-Party API.

Step 9 — Store the xAI key as a secret​

Same pattern as Step 3. A tool's execute.headers references the Secret through a secret reference token, never a raw value (see Step 10).

XAI_SECRET_ID=$(soat create-secret \
  --project-id "$PROJECT_ID" \
  --name "xai-api-key" \
  --value "$XAI_API_KEY" | jq -r '.id')
echo "XAI_SECRET_ID: $XAI_SECRET_ID"

Step 10 — Create the speech-to-text tool​

Create an http tool pointed directly at xAI's /stt endpoint (overridden to the mock in CI via $XAI_BASE_URL). Two things make this work:

  • {{secret:...}} in execute.headers — the raw key is never stored on the tool and resolves only right before the outbound request. See Secrets — Secret References.
  • execute.body_mode: "multipart" — the endpoint requires multipart/form-data; the file field is base64-decoded and attached as a real file part. See Tools — Request Body Encoding.
STT_TOOL_ID=$(soat create-tool \
  --project-id "$PROJECT_ID" \
  --name "xai-stt" \
  --type http \
  --description "Transcribes audio via xAI's speech-to-text API" \
  --execute '{"url":"'"$XAI_BASE_URL"'/stt","method":"POST","body_mode":"multipart","headers":{"Authorization":"Bearer {{secret:'"$XAI_SECRET_ID"'}}"}}' \
  --parameters '{"type":"object","properties":{"file":{"type":"object"},"language":{"type":"string"}}}' \
  | jq -r '.id')
echo "STT_TOOL_ID: $STT_TOOL_ID"

Step 11 — Wrap it in a pipeline to extract the transcript​

xAI's /stt response is an object ({ "text": "...", ... }), not the bare string a tool converter requires. The pipeline's output of { "var": "steps.call.text" } resolves to that bare scalar directly.

An http tool's output_mapping could express this extraction directly on the xai-stt tool; the two-tool version below illustrates chaining tools, which output_mapping alone cannot do.

STT_CONVERTER_ID=$(soat create-tool \
  --project-id "$PROJECT_ID" \
  --name "xai-stt-converter" \
  --type pipeline \
  --description "Calls the xAI STT tool and extracts the transcript as a bare string" \
  --pipeline '{"steps":[{"id":"call","tool_id":"'"$STT_TOOL_ID"'","input":{"file":{"var":"input.file"},"language":{"var":"input.language"}}}],"output":{"var":"steps.call.text"}}' \
  | jq -r '.id')
echo "STT_CONVERTER_ID: $STT_CONVERTER_ID"

Step 12 — Route audio to the tool converter​

Map audio/* to the pipeline tool with tool_id — the counterpart of Step 6's agent_id. A transcript is one long block of text, so chunk it with the size strategy for sharper retrieval — see Documents — File Ingestion and Chunking. preset_parameters merges a fixed language into every call, the same way it would for any other tool.

soat create-ingestion-rule \
  --project-id "$PROJECT_ID" \
  --content-type-glob "audio/*" \
  --tool-id "$STT_CONVERTER_ID" \
  --file-delivery base64 \
  --preset-parameters '{"language":"en"}' \
  --chunk-strategy "size" \
  --chunk-size 1000 \
  --chunk-overlap 200 | jq '{id: .id, content_type_glob: .content_type_glob}'

Step 13 — Ingest audio the same way​

Same call shape as the image; the audio/* rule from Step 12 routes it to the tool converter — the caller never names a tool. See Documents. $FIXTURES_DIR/meeting.mp3 is a few seconds of real speech.

AUDIO_FILE_ID=$(soat upload-file-base64 \
  --project-id "$PROJECT_ID" \
  --filename "meeting.mp3" \
  --content-type "audio/mpeg" \
  --content "$(base64 -w0 "$FIXTURES_DIR/meeting.mp3")" | jq -r '.id')
echo "AUDIO_FILE_ID: $AUDIO_FILE_ID"

soat ingest-document \
  --project-id "$PROJECT_ID" \
  --file-id "$AUDIO_FILE_ID" \
  --path-prefix "/audio/" \
  --wait true | jq -e '.status == "ready"'
# prints `true` once the audio is transcribed, chunked, and embedded
# (chunk_count is reported by `soat get-document-status`; Step 14 confirms the text is searchable)

Step 14 — Search the converted content​

Both documents are chunked and embedded like any other. Query them through Knowledge — the OCR and transcript text is fully searchable, regardless of which converter kind produced it.

# The OCR'd receipt text is retrievable
soat search-knowledge \
  --project-id "$PROJECT_ID" \
  --query "total amount on the receipt" \
  --document-paths '["/images/"]' \
  --limit 3 | jq -e '[.results[].content] | join(" ") | test("Total amount")'

# The transcribed audio is retrievable
soat search-knowledge \
  --project-id "$PROJECT_ID" \
  --query "when is the launch scheduled" \
  --document-paths '["/audio/"]' \
  --limit 3 | jq -e '[.results[].content] | join(" ") | test("launch is next tuesday"; "i")'

Next steps​

Reach for an agent converter first when a multimodal LLM can do the job directly; reach for a tool converter for a dedicated non-LLM API or an async-callback background job. To support another modality (e.g. video), add one rule pointing at a converter — no server changes.