Skip to main content

Ingest Images and Audio with Converters

Native file ingestion handles PDFs and text. An Ingestion Rule routes any other content_type to a converter; this tutorial shows both converter kinds:

Both routes share the chunk + embed pipeline, so converted text is searchable like any Document.

Runs against mock providers

Every provider/tool call targets a base_url you configure. The tutorials runner uses the mock-providers service in tests/docker-compose.tutorials.yml, which answers with canned text after verifying the received bytes match the checked-in fixtures.

Prerequisites​

export SOAT_BASE_URL=http://localhost:5047   # CLI, SDK, and curl — do NOT append /api/v1

# Provider endpoints and keys. The defaults are the real providers; each is
# overridable so the tutorial can also run against local mocks (see the tip above).
export OPENAI_BASE_URL="${OPENAI_BASE_URL:-https://api.openai.com/v1}"
export OPENAI_API_KEY="${OPENAI_API_KEY:-sk-your-openai-key}"
export XAI_BASE_URL="${XAI_BASE_URL:-https://api.x.ai/v1}"
export XAI_API_KEY="${XAI_API_KEY:-xai-your-key}"

# Where this tutorial's fixture files live — override if your clone (or the
# directory you copied fixtures/ into) is somewhere else.
export FIXTURES_DIR="${FIXTURES_DIR:-./packages/website/docs/tutorials/fixtures}"

Step 1 — Log in as admin​

See Users.

ADMIN_TOKEN=$(soat login-user --username admin --password Admin1234! | jq -r '.token')
export SOAT_TOKEN=$ADMIN_TOKEN

Step 2 — Create a project​

Every resource lives inside a project.

PROJECT_ID=$(soat create-project --name "Media Ingestion" | jq -r '.id')
echo "PROJECT_ID: $PROJECT_ID"

Part A — Images and scanned PDFs via an OpenAI agent converter​

An Ingestion Rule pointed at an agent sends the file as multimodal input with a fixed "extract all text" instruction; a vision model does the OCR.

Step 3 — Store the OpenAI key as a secret​

The AI provider reads its credentials from a Secret: encrypted at rest, never returned in responses.

OPENAI_SECRET_ID=$(soat create-secret \
  --project-id "$PROJECT_ID" \
  --name "openai-api-key" \
  --value "$OPENAI_API_KEY" | jq -r '.id')
echo "OPENAI_SECRET_ID: $OPENAI_SECRET_ID"

Step 4 — Create a vision AI provider​

An OpenAI AI provider with a vision-capable default_model, key from the secret above; base_url is overridden to the mock in CI.

OPENAI_PROVIDER_ID=$(soat create-ai-provider \
  --project-id "$PROJECT_ID" \
  --name "OpenAI Vision" \
  --provider "openai" \
  --default-model "gpt-4o" \
  --base-url "$OPENAI_BASE_URL" \
  --secret-id "$OPENAI_SECRET_ID" | jq -r '.id')
echo "OPENAI_PROVIDER_ID: $OPENAI_PROVIDER_ID"

Step 5 — Create the OCR agent​

An agent whose instructions forbid summarizing or commenting, so the document text is the raw extracted content.

OCR_AGENT_ID=$(soat create-agent \
  --project-id "$PROJECT_ID" \
  --ai-provider-id "$OPENAI_PROVIDER_ID" \
  --name "OCR Agent" \
  --instructions "Extract all text from the provided file verbatim. Return plain text only — no commentary, no summary, no markdown fences." \
  | jq -r '.id')
echo "OCR_AGENT_ID: $OCR_AGENT_ID"

Step 6 — Route images to the agent​

An Ingestion Rule maps image/* to the agent via agent_id. Agent converters take the file directly: no file_delivery, no request shape.

soat create-ingestion-rule \
  --project-id "$PROJECT_ID" \
  --content-type-glob "image/*" \
  --agent-id "$OCR_AGENT_ID" \
  --chunk-strategy "whole" | jq '{id: .id, content_type_glob: .content_type_glob}'

Step 7 — (Optional) OCR fallback for scanned PDFs​

A scanned PDF has content_type: application/pdf and no text layer. A rule matching application/pdf is consulted only when native extraction returns no text (Ingestion Rules — Content-Type Matching), so pointing it at the vision agent OCRs scanned PDFs while born-digital PDFs skip the converter. native_extraction: skip on the rule OCRs every PDF.

soat create-ingestion-rule \
  --project-id "$PROJECT_ID" \
  --content-type-glob "application/pdf" \
  --agent-id "$OCR_AGENT_ID" \
  --chunk-strategy "whole" | jq '{id: .id, content_type_glob: .content_type_glob}'

Step 8 — Ingest an image without naming a converter​

Upload an image as a File, then ingest it like a PDF. POST /documents/ingest resolves the rule from the file's content_type; the base64 upload sets it explicitly to image/png. $FIXTURES_DIR/receipt.png is a small receipt image.

IMAGE_FILE_ID=$(soat upload-file-base64 \
  --project-id "$PROJECT_ID" \
  --filename "receipt.png" \
  --content-type "image/png" \
  --content "$(base64 -w0 "$FIXTURES_DIR/receipt.png")" | jq -r '.id')
echo "IMAGE_FILE_ID: $IMAGE_FILE_ID"

soat ingest-document \
  --project-id "$PROJECT_ID" \
  --file-id "$IMAGE_FILE_ID" \
  --path-prefix "/images/" \
  --wait true | jq -e '.status == "ready"'
# prints `true` once the image is OCR'd, chunked, and embedded
# (chunk_count is reported by `soat get-document-status`; Step 14 confirms the text is searchable)

Against a real OpenAI account the model occasionally returns a non-answer; re-ingest with soat reingest-document if .status is failed.

Part B — Audio via an xAI tool converter​

xAI's speech-to-text REST API (POST /v1/stt) is not a chat-completions endpoint, so it needs a tool converter: an http tool calls the API and a pipeline tool reshapes the response into the bare-string converter contract. Pattern: Ingestion Rules — Building a Tool Converter for a Third-Party API.

Step 9 — Store the xAI key as a secret​

As in Step 3. The tool's execute.headers will reference the Secret through a secret reference token (Step 10).

XAI_SECRET_ID=$(soat create-secret \
  --project-id "$PROJECT_ID" \
  --name "xai-api-key" \
  --value "$XAI_API_KEY" | jq -r '.id')
echo "XAI_SECRET_ID: $XAI_SECRET_ID"

Step 10 — Create the speech-to-text tool​

An http tool pointed at xAI's /stt endpoint (the mock in CI via $XAI_BASE_URL):

  • {{secret:...}} in execute.headers resolves right before the outbound request; the raw key is never stored on the tool (Secrets — Secret References).
  • execute.body_mode: "multipart" sends multipart/form-data; the file field is base64-decoded into a real file part (Tools — Request Body Encoding).
STT_TOOL_ID=$(soat create-tool \
  --project-id "$PROJECT_ID" \
  --name "xai-stt" \
  --type http \
  --description "Transcribes audio via xAI's speech-to-text API" \
  --execute '{"url":"'"$XAI_BASE_URL"'/stt","method":"POST","body_mode":"multipart","headers":{"Authorization":"Bearer {{secret:'"$XAI_SECRET_ID"'}}"}}' \
  --parameters '{"type":"object","properties":{"file":{"type":"object"},"language":{"type":"string"}}}' \
  | jq -r '.id')
echo "STT_TOOL_ID: $STT_TOOL_ID"

Step 11 — Wrap it in a pipeline to extract the transcript​

/stt returns an object ({ "text": "...", ... }), not the bare string a tool converter requires; the pipeline's output { "var": "steps.call.text" } extracts it. An http tool's output_mapping could do the same on xai-stt alone; the two-tool version shows chaining.

STT_CONVERTER_ID=$(soat create-tool \
  --project-id "$PROJECT_ID" \
  --name "xai-stt-converter" \
  --type pipeline \
  --description "Calls the xAI STT tool and extracts the transcript as a bare string" \
  --pipeline '{"steps":[{"id":"call","tool_id":"'"$STT_TOOL_ID"'","input":{"file":{"var":"input.file"},"language":{"var":"input.language"}}}],"output":{"var":"steps.call.text"}}' \
  | jq -r '.id')
echo "STT_CONVERTER_ID: $STT_CONVERTER_ID"

Step 12 — Route audio to the tool converter​

Map audio/* to the pipeline tool with tool_id. A transcript is one long block, so chunk it with the size strategy (Documents — File Ingestion and Chunking). preset_parameters merges a fixed language into every call.

soat create-ingestion-rule \
  --project-id "$PROJECT_ID" \
  --content-type-glob "audio/*" \
  --tool-id "$STT_CONVERTER_ID" \
  --file-delivery base64 \
  --preset-parameters '{"language":"en"}' \
  --chunk-strategy "size" \
  --chunk-size 1000 \
  --chunk-overlap 200 | jq '{id: .id, content_type_glob: .content_type_glob}'

Step 13 — Ingest audio the same way​

Same call shape as the image; the audio/* rule routes it (Documents). $FIXTURES_DIR/meeting.mp3 is a few seconds of real speech.

AUDIO_FILE_ID=$(soat upload-file-base64 \
  --project-id "$PROJECT_ID" \
  --filename "meeting.mp3" \
  --content-type "audio/mpeg" \
  --content "$(base64 -w0 "$FIXTURES_DIR/meeting.mp3")" | jq -r '.id')
echo "AUDIO_FILE_ID: $AUDIO_FILE_ID"

soat ingest-document \
  --project-id "$PROJECT_ID" \
  --file-id "$AUDIO_FILE_ID" \
  --path-prefix "/audio/" \
  --wait true | jq -e '.status == "ready"'
# prints `true` once the audio is transcribed, chunked, and embedded
# (chunk_count is reported by `soat get-document-status`; Step 14 confirms the text is searchable)

Step 14 — Search the converted content​

Both documents are chunked and embedded like any other; query them through Knowledge.

# The OCR'd receipt text is retrievable
soat search-knowledge \
  --project-id "$PROJECT_ID" \
  --query "total amount on the receipt" \
  --document-paths '["/images/"]' \
  --limit 3 | jq -e '[.results[].content] | join(" ") | test("Total amount")'

# The transcribed audio is retrievable
soat search-knowledge \
  --project-id "$PROJECT_ID" \
  --query "when is the launch scheduled" \
  --document-paths '["/audio/"]' \
  --limit 3 | jq -e '[.results[].content] | join(" ") | test("launch is next tuesday"; "i")'

Next steps​

Use an agent converter when a multimodal LLM can do the job; a tool converter for a non-LLM API or an async-callback background job. Another modality (video) is one more rule, no server changes.