Skip to main content

Ingestion Rules

An Ingestion Rule routes a file content_type to a converter Tool so that non-text files (images, audio, scanned PDFs) can be ingested into Documents.

Overview​

Native file ingestion only extracts text from PDFs (text layer), text/plain, and text/markdown. Anything else fails with FILE_PARSE_FAILED. An Ingestion Rule fills that gap: it maps a content_type glob (e.g. image/*, audio/mpeg, application/pdf) to a converter — either a Tool (http/mcp/builtin/pipeline) that calls an external OCR, speech-to-text, or vision service, or an Agent with a multimodal model. When POST /documents/ingest receives a file whose type has no native extractor — or a PDF whose native extraction yields no text — it looks up the best-matching rule and invokes the converter to produce the document text; the existing chunk + embedding pipeline is unchanged.

Rules are per-project. SOAT does not perform OCR or transcription itself — the rule points at a tool or agent you configure, so you can use any API or model you like. In the engine & algorithms pattern, a converter tool is the knowledge engine's bring-your-own-algorithm seam: the contract below is the boundary, and everything downstream (chunking, embedding, retrieval) treats your converter's pages exactly like natively extracted ones.

See the Permissions Reference for the IAM action strings for this module.

Data Model​

IngestionRule​

FieldTypeDescription
idstringPublic identifier prefixed with igr_
project_idstringID of the owning project
content_type_globstringGlob matched against the file's content_type (image/*, image/png, audio/mpeg, application/pdf)
tool_idstring | nullConverter tool (tool_…). Must be a server-callable type: http, mcp, builtin, or pipeline. client tools are rejected. Mutually exclusive with agent_id.
agent_idstring | nullConverter agent (agent_…). The file is sent to the agent as multimodal input and its text output becomes the document content. Mutually exclusive with tool_id.
actionstring | nullOperation id, required for builtin/mcp tool converters
preset_parametersobject | nullMerged into the tool input before invocation (tool converters only). Cannot contain the reserved keys file or callback, which ingestion injects. A key the converter tool itself pins in its own preset_parameters stays pinned — the tool's value wins over the rule's.
native_extractionstringFor PDFs: first (default) converts only when native extraction yields no text; skip bypasses native extraction and converts every matching PDF. Ignored for non-native types.
file_deliverystringHow the file reaches a tool converter: base64 (default) or download_url
chunk_strategystring | nullOptional default chunk strategy (page/whole/size), overridable per ingest request
chunk_sizenumber | nullOptional default for the size strategy
chunk_overlapnumber | nullOptional default for the size strategy
metadataobject | nullArbitrary JSON metadata
created_atstringISO 8601 creation timestamp
updated_atstringISO 8601 last-updated timestamp

project_id + content_type_glob is unique within a project — one rule per glob. Exactly one of tool_id / agent_id must be set.

Key Concepts​

Content-Type Matching​

At ingest time, resolveIngestionRule picks the matching rule with the highest specificity: an exact type (image/png) beats a subtype wildcard (image/*), which beats a full wildcard (*/*).

Rules are consulted in two cases:

  1. Non-native content type — the file type has no built-in extractor.
  2. Empty native extraction — a native type produced no text. In particular, a rule matching application/pdf acts as an OCR fallback for scanned/image-only PDFs: the built-in unpdf extractor runs first, and only when it returns no text does ingestion invoke the converter. Born-digital PDFs with a text layer skip the converter, so there is no added cost for the common case.

When no rule matches, behavior is unchanged: a non-native type is rejected with UNSUPPORTED_FILE_TYPE (400), and an empty native extraction fails the document with FILE_PARSE_FAILED.

PDF Conversion Mode​

For PDFs, native_extraction on the matching application/pdf rule controls when the converter runs: first (default) runs native unpdf extraction first and converts only PDFs with no text layer; skip bypasses native extraction so every matching PDF is converted (use when the text layer is unreliable). It has no effect on non-native types — their converter always runs.

Converter: Tool or Agent​

A rule's converter is either a Tool or an Agent (exactly one):

  • Tool converter (tool_id) — ingestion calls the tool with the JSON contract below and reads text from its response. Best for audio, specialized OCR APIs, and long async jobs (the tool can defer via the callback).
  • Agent converter (agent_id) — ingestion sends the file to the agent as multimodal input with a fixed "extract all text / transcribe" instruction; the agent's text output becomes the document content. Zero extra infrastructure, but the agent's model must support the file's modality (a vision model for images and scanned PDFs; an audio-capable model for audio). The generation is awaited inline — there is no deferral/callback for agent converters.
Audio agent converters need a Chat Completions-compatible AI provider

An agent whose AI provider uses the openai provider slug talks to OpenAI's Responses API, which does not accept audio input — an audio file routed to such an agent fails with CONVERTER_FAILED (AI_UnsupportedFunctionalityError: file part media type audio/...). To use an OpenAI audio-capable model (e.g. gpt-audio-mini) as an audio converter, register it with the custom provider slug and base_url pointed at https://api.openai.com/v1 instead — that path uses the Chat Completions API, which does support audio input. Vision (image) agent converters are unaffected. Many dedicated speech-to-text APIs (including xAI's) aren't chat-completions-shaped at all, in which case a tool converter is the better fit regardless — see Ingest Images and Audio with Converters for a worked example of both converter kinds.

Building a Tool Converter for a Third-Party API​

A tool converter does not require a separate adapter service. An http tool can point execute.url directly at a third-party API, and a pipeline tool wrapping it reshapes request and response with the usual JSON Logic mapping; point IngestionRule.tool_id at the pipeline tool. Because the Converter Tool Contract accepts a bare string, the pipeline's output can be a single var expression (e.g. { "var": "steps.call.text" }).

Hold the third-party API key in a Secret and embed a secret reference in the http tool's execute.headers rather than pasting the raw key. For APIs that require multipart/form-data (many speech-to-text and OCR endpoints), set execute.body_mode: "multipart" on the http tool — the { content_type, filename, data_base64 } file shape ingestion provides is decoded and attached as a real file part.

Converter Tool Contract​

A tool converter is called via the same server-side path as every other tool call, with a fixed input shape, and must return one of three output shapes.

Input built by ingestion:

{
  "file": {
    "id": "file_01",
    "filename": "scan.png",
    "content_type": "image/png",
    "size": 20480,
    "data_base64": "iVBORw0KGgo…",        // when file_delivery = base64
    "download_url": "https://…/files/file_01/download?token=…" // when file_delivery = download_url
  },
  "callback": {                            // lets long-running tools defer their result
    "url": "https://…/api/v1/documents/doc_01/ingestion-callback",
    "token": "…"
  }
  // preset_parameters are merged in at the top level
}

Output — the tool may return either extracted text or a deferral:

"All the extracted text"                              // wrapped as a single page
{ "pages": [{ "text": "page 1", "page_number": 1 }] } // paged (e.g. OCR per page)
{ "status": "pending" }                                // long-running deferral — see below

Any other shape fails the document with CONVERTER_OUTPUT_INVALID; a tool error fails it with CONVERTER_FAILED. { "status": "pending" } is only honored for a tool converter ingested in the default async mode (see Synchronous vs Async (Callback) Conversion) — an agent converter, or a synchronous ingest request (?wait=true), fails with CONVERTER_FAILED instead, since neither can wait for a later callback.

File Delivery​

file_delivery controls how the file bytes reach the tool's external API:

ModeBehaviorUse when
base64 (default)Ingestion downloads the file and passes data_base64 in the tool inputSmall files; provider-agnostic; works with any storage backend. Note: the whole file is loaded into memory and the request body.
download_urlIngestion passes a short-lived signed download_url; the tool/API fetches itLarge files (long audio, high-resolution images/scans) where base64 is impractical, and providers that accept a remote URL

Synchronous vs Async (Callback) Conversion​

A converter tool that returns text (or { pages }) directly is synchronous — ingestion continues to chunk and embed inline. An agent converter is always synchronous: its generation is awaited inline and it has no deferral path.

A tool that returns { status: "pending" } is asynchronous — but only when the document is being ingested in the default async mode (POST /documents/ingest without ?wait=true). The document stays in processing while the external job runs, then the tool (or the service it wires) delivers the result to the Documents module's ingestion-callback endpoint — see Deliver an async converter result in the API reference for its path, query token, and request schema.

The callback's document ID and token come from the callback block ingestion injected into the tool's input (see Converter Tool Contract); its body uses the same output contract as a synchronous converter, adapted for a JSON body (a single page is { "text": "..." } rather than a bare string, since a top-level JSON string is not a valid HTTP JSON body).

The callback is authorized by a single-use, signed token scoped to that document and ingestion attempt — not by an IAM action, since the external converter is not a SOAT user. It is accepted (204) only while that attempt is still processing; a replayed callback, a callback for a superseded attempt (after re-ingest), or one that arrives after the stall timeout already failed the document is rejected with 409 INGESTION_CALLBACK_CONFLICT. An invalid or mismatched token is rejected with 401 INGESTION_CALLBACK_INVALID_TOKEN. Once a valid result arrives, ingestion runs the normal chunk + embed tail and marks the document ready.

If a synchronous ingest request (?wait=true) or an agent converter encounters { status: "pending" }, the document fails immediately with CONVERTER_FAILED — neither can wait for a callback that may arrive arbitrarily later. Design a tool that might defer to only do so under async ingestion.

A document awaiting a callback for longer than CONVERSION_STALL_TIMEOUT_MS is auto-failed with CONVERSION_TIMEOUT (see Configuration) — the converter-specific counterpart of stuck-ingestion recovery. A callback racing the timeout is settled by an atomic compare-and-set: it either wins outright or is rejected with 409, never silently dropped.

Failure Reasons​

Converter-related failure_reason values that can appear on a failed document (alongside the existing FILE_PARSE_FAILED, INGESTION_TIMEOUT):

failure_reasonMeaning
CONVERTER_FAILEDThe converter tool/agent call errored, an agent converter returned an async deferral (unsupported), or a tool converter returned an async deferral during synchronous ingestion (?wait=true)
CONVERTER_OUTPUT_INVALIDThe tool (or callback) returned an unrecognized output shape
CONVERSION_TIMEOUTAn async conversion did not call back within CONVERSION_STALL_TIMEOUT_MS

Configuration​

Environment VariableRequiredDescription
CONVERSION_STALL_TIMEOUT_MSNoHow long (ms) a document may await an async converter callback before being auto-failed with CONVERSION_TIMEOUT. Defaults to 30 minutes. Separate from, and typically longer than, INGESTION_STALL_TIMEOUT_MS (default 5 minutes).

Examples​

Create an ingestion rule​

soat create-ingestion-rule \
  --project-id proj_ABC \
  --content-type-glob "image/*" \
  --tool-id tool_ocr \
  --file-delivery base64 \
  --chunk-strategy whole

To create an agent-converter rule instead, pass --agent-id in place of --tool-id (e.g. a vision agent on application/pdf as an OCR fallback for scanned PDFs). Ingesting a matching file needs nothing special — POST /documents/ingest routes it to the converter automatically; see Documents.

List rules​

soat list-ingestion-rules --project-id proj_ABC