Skip to main content

Ingestion Rules

An Ingestion Rule routes a file content_type to a converter Tool so that non-text files (images, audio, scanned PDFs) can be ingested into Documents.

Overview​

Native file ingestion only extracts text from PDFs (text layer), text/plain, and text/markdown. Anything else fails with FILE_PARSE_FAILED. An Ingestion Rule fills that gap: it maps a content_type glob (e.g. image/*, audio/mpeg, application/pdf) to a converter — either a Tool (http/mcp/soat/pipeline) that calls an external OCR, speech-to-text, or vision service, or an Agent with a multimodal model. When POST /documents/ingest receives a file whose type has no native extractor — or a PDF whose native extraction yields no text — it looks up the best-matching rule and invokes the converter to produce the document text; the existing chunk + embedding pipeline is unchanged.

Rules are per-project. SOAT does not perform OCR or transcription itself — the rule points at a tool or agent you configure, so you can use any API or model you like.

See the Permissions Reference for the IAM action strings for this module.

Data Model​

IngestionRule​

FieldTypeDescription
idstringPublic identifier prefixed with igr_
project_idstringID of the owning project
content_type_globstringGlob matched against the file's content_type (image/*, image/png, audio/mpeg, application/pdf)
tool_idstring | nullConverter tool (tool_…). Must be a server-callable type: http, mcp, soat, or pipeline. client tools are rejected. Mutually exclusive with agent_id.
agent_idstring | nullConverter agent (agent_…). The file is sent to the agent as multimodal input and its text output becomes the document content. Mutually exclusive with tool_id.
actionstring | nullOperation id, required for soat/mcp tool converters
preset_parametersobject | nullMerged into the tool input before invocation (tool converters only). Cannot contain the reserved keys file or callback, which ingestion injects.
native_extractionstringFor PDFs: first (default) converts only when native extraction yields no text; skip bypasses native extraction and converts every matching PDF. Ignored for non-native types.
file_deliverystringHow the file reaches a tool converter: base64 (default) or download_url
chunk_strategystring | nullOptional default chunk strategy (page/whole/size), overridable per ingest request
chunk_sizenumber | nullOptional default for the size strategy
chunk_overlapnumber | nullOptional default for the size strategy
metadataobject | nullArbitrary JSON metadata
created_atstringISO 8601 creation timestamp
updated_atstringISO 8601 last-updated timestamp

project_id + content_type_glob is unique within a project — one rule per glob. Exactly one of tool_id / agent_id must be set.

Key Concepts​

Content-Type Matching​

At ingest time, resolveIngestionRule picks the matching rule with the highest specificity: an exact type (image/png) beats a subtype wildcard (image/*), which beats a full wildcard (*/*).

Rules are consulted in two cases:

  1. Non-native content type — the file type has no built-in extractor.
  2. Empty native extraction — a native type produced no text. In particular, a rule matching application/pdf acts as an OCR fallback for scanned/image-only PDFs: the built-in unpdf extractor runs first, and only when it returns no text does ingestion invoke the converter. Born-digital PDFs with a text layer skip the converter, so there is no added cost for the common case.

When no rule matches, behavior is unchanged: a non-native type is rejected with UNSUPPORTED_FILE_TYPE (400), and an empty native extraction fails the document with FILE_PARSE_FAILED.

PDF Conversion Mode​

For PDFs, the native_extraction field on the matching application/pdf rule controls when the converter runs:

native_extractionBehaviorUse when
first (default)Native unpdf extraction runs first; the converter fires only for PDFs with no text layer.You only want to OCR scanned/image-only PDFs; born-digital PDFs stay on the fast native path.
skipNative extraction is bypassed; every matching PDF goes to the converter.The PDFs' text layer is unreliable or garbled and you want OCR applied to all of them.

native_extraction has no effect on non-native types (images, audio) — there is no native extractor to skip, so their converter always runs.

Converter: Tool or Agent​

A rule's converter is either a Tool or an Agent (exactly one):

  • Tool converter (tool_id) — ingestion calls the tool with the JSON contract below and reads text from its response. Best for audio, specialized OCR APIs, and long async jobs (the tool can defer via the callback).
  • Agent converter (agent_id) — ingestion sends the file to the agent as multimodal input with a fixed "extract all text / transcribe" instruction; the agent's text output becomes the document content. Zero extra infrastructure, but the agent's model must support the file's modality (a vision model for images and scanned PDFs; an audio-capable model for audio). The generation is awaited inline — there is no deferral/callback for agent converters.

:::caution Audio agent converters need a Chat Completions-compatible AI provider An agent whose AI provider uses the openai provider slug talks to OpenAI's Responses API, which does not accept audio input — an audio file routed to such an agent fails with CONVERTER_FAILED (AI_UnsupportedFunctionalityError: file part media type audio/...). To use an OpenAI audio-capable model (e.g. gpt-audio-mini) as an audio converter, register it with the custom provider slug and base_url pointed at https://api.openai.com/v1 instead — that path uses the Chat Completions API, which does support audio input. Vision (image) agent converters are unaffected. Many dedicated speech-to-text APIs (including xAI's) aren't chat-completions-shaped at all, in which case a tool converter is the better fit regardless — see Ingest Images and Audio with Converters for a worked example of both converter kinds. :::

Building a Tool Converter for a Third-Party API​

A tool converter does not require hosting a separate adapter service. An http tool can point its execute.url directly at a third-party API (OpenAI, xAI, …); a pipeline tool wrapping it reshapes the request and response using the same JSON Logic mapping every other pipeline step uses — cat to build values like a base64 data: URI, var to extract a nested response field into the shape below. Point IngestionRule.tool_id at the pipeline tool. See Tools — pipeline for the mapping syntax.

Since the Converter Tool Contract accepts a bare string as its simplest output shape, the wrapping pipeline's output can be a single var expression rather than an object — it resolves directly to that scalar:

{
  "steps": [
    {
      "id": "call",
      "tool_id": "tool_stt_http",
      "input": { "file": { "var": "input.file" } }
    }
  ],
  "output": { "var": "steps.call.text" } // resolves to a bare string, e.g. "All the extracted text"
}

Hold the third-party API key in a Secret and embed a secret reference in the http tool's execute.headers — e.g. {"Authorization": "Bearer {{secret:sec_01HXYZ}}"}. Never paste the raw key into the tool config: the config is echoed back by GET /tools/{id}, while the {{secret:...}} token resolves only at call time and the raw value is never returned by any API response.

Not every third-party API accepts JSON. Many audio (speech-to-text) and specialized OCR endpoints require multipart/form-data and reject a JSON body outright (e.g. xAI's POST /v1/stt). For those, set execute.body_mode: "multipart" on the http tool: pass the file through as the { content_type, filename, data_base64 } shape ingestion already provides and it is decoded and attached as a real file part, alongside any scalar fields (model name, language, …). This keeps the "no separate adapter service" property — the http tool calls the multipart API directly.

Converter Tool Contract​

A tool converter is called via the same server-side path as every other tool call, with a fixed input shape, and must return one of three output shapes.

Input built by ingestion:

{
  "file": {
    "id": "file_01",
    "filename": "scan.png",
    "content_type": "image/png",
    "size": 20480,
    "data_base64": "iVBORw0KGgo…",        // when file_delivery = base64
    "download_url": "https://…/files/file_01/download?token=…" // when file_delivery = download_url
  },
  "callback": {                            // lets long-running tools defer their result
    "url": "https://…/api/v1/documents/doc_01/ingestion-callback",
    "token": "…"
  }
  // preset_parameters are merged in at the top level
}

Output — the tool may return either extracted text or a deferral:

"All the extracted text"                              // wrapped as a single page
{ "pages": [{ "text": "page 1", "page_number": 1 }] } // paged (e.g. OCR per page)
{ "status": "pending" }                                // long-running deferral — see below

Any other shape fails the document with CONVERTER_OUTPUT_INVALID; a tool error fails it with CONVERTER_FAILED. { "status": "pending" } is only honored for a tool converter ingested in the default async mode (see Synchronous vs Async (Callback) Conversion) — an agent converter, or a synchronous ingest request (?async=false), fails with CONVERTER_FAILED instead, since neither can wait for a later callback.

File Delivery​

file_delivery controls how the file bytes reach the tool's external API:

ModeBehaviorUse when
base64 (default)Ingestion downloads the file and passes data_base64 in the tool inputSmall files; provider-agnostic; works with any storage backend. Note: the whole file is loaded into memory and the request body.
download_urlIngestion passes a short-lived signed download_url; the tool/API fetches itLarge files (long audio, high-resolution images/scans) where base64 is impractical, and providers that accept a remote URL

Synchronous vs Async (Callback) Conversion​

A converter tool that returns text (or { pages }) directly is synchronous — ingestion continues to chunk and embed inline. An agent converter is always synchronous: its generation is awaited inline and it has no deferral path.

A tool that returns { status: "pending" } is asynchronous — but only when the document is being ingested in the default async mode (POST /documents/ingest without ?async=false). The document stays in processing while the external job runs, then the tool (or the service it wires) delivers the result to the Documents module's ingestion-callback endpoint — see Deliver an async converter result in the API reference for its path, query token, and request schema.

The callback's document ID and token come from the callback block ingestion injected into the tool's input (see Converter Tool Contract); its body uses the same output contract as a synchronous converter, adapted for a JSON body (a single page is { "text": "..." } rather than a bare string, since a top-level JSON string is not a valid HTTP JSON body).

The callback is authorized by a single-use, signed token scoped to that document and ingestion attempt — not by an IAM action, since the external converter is not a SOAT user. It is accepted (204) only while that attempt is still processing; a replayed callback, a callback for a superseded attempt (after re-ingest), or one that arrives after the stall timeout already failed the document is rejected with 409 INGESTION_CALLBACK_CONFLICT. An invalid or mismatched token is rejected with 401 INGESTION_CALLBACK_INVALID_TOKEN. Once a valid result arrives, ingestion runs the normal chunk + embed tail and marks the document ready.

If a synchronous ingest request (?async=false) or an agent converter encounters { status: "pending" }, the document fails immediately with CONVERTER_FAILED — neither can wait for a callback that may arrive arbitrarily later. Design a tool that might defer to only do so under async ingestion.

A document awaiting a callback for longer than CONVERSION_STALL_TIMEOUT_MS is auto-failed with CONVERSION_TIMEOUT (see Configuration). This is the converter-specific counterpart of the stuck-ingestion recovery in the documents module — both use the same lazy "recover on next read" mechanism (checked whenever the document is fetched, not a background cron), but the callback path additionally guards against a callback racing the timeout sweeper: the two finish a conversion via an atomic compare-and-set that only one can win, so a legitimate callback that arrives just as the sweeper fires is never silently dropped — it either wins outright or, if the sweeper already won, is rejected with a clear 409 rather than corrupting the failed state.

Failure Reasons​

Converter-related failure_reason values that can appear on a failed document (alongside the existing FILE_PARSE_FAILED, INGESTION_TIMEOUT):

failure_reasonMeaning
CONVERTER_FAILEDThe converter tool/agent call errored, an agent converter returned an async deferral (unsupported), or a tool converter returned an async deferral during synchronous ingestion (?async=false)
CONVERTER_OUTPUT_INVALIDThe tool (or callback) returned an unrecognized output shape
CONVERSION_TIMEOUTAn async conversion did not call back within CONVERSION_STALL_TIMEOUT_MS

Configuration​

Environment VariableRequiredDescription
CONVERSION_STALL_TIMEOUT_MSNoHow long (ms) a document may await an async converter callback before being auto-failed with CONVERSION_TIMEOUT. Defaults to 30 minutes. Separate from, and typically longer than, INGESTION_STALL_TIMEOUT_MS (default 5 minutes).

Examples​

Create an ingestion rule​

soat create-ingestion-rule \
  --project-id proj_ABC \
  --content-type-glob "image/*" \
  --tool-id tool_ocr \
  --file-delivery base64 \
  --chunk-strategy whole

OCR fallback for scanned PDFs (agent converter)​

A rule matching application/pdf fires only when the native extractor returns no text (a scanned PDF). Point it at a vision agent — no external adapter needed. Born-digital PDFs skip it.

soat create-ingestion-rule \
  --project-id proj_ABC \
  --content-type-glob "application/pdf" \
  --agent-id agent_vision \
  --chunk-strategy whole

Ingest an image using the rule​

Once a matching rule exists, ingest a non-native file exactly like any other — ingestion routes it to the converter automatically.

# Upload the image, then ingest — the image/* rule handles conversion
FILE_ID=$(soat upload-file --project-id proj_ABC --file ./scan.png | jq -r '.id')

soat ingest-document \
  --project-id proj_ABC \
  --file-id "$FILE_ID" \
  --path-prefix /scans/

List rules​

soat list-ingestion-rules --project-id proj_ABC