Skip to main content

Ingestion Rules

An Ingestion Rule routes a file content_type to a converter Tool so that non-text files (images, audio, scanned PDFs) can be ingested into Documents.

Overview​

Native file ingestion extracts text only from PDFs (text layer), text/plain, and text/markdown; anything else fails with FILE_PARSE_FAILED. An Ingestion Rule maps a content_type glob (image/*, audio/mpeg, application/pdf) to a converter: a Tool (http/mcp/builtin/pipeline) calling an external OCR, speech-to-text or vision service, or an Agent with a multimodal model. When POST /documents/ingest receives a file with no native extractor, or a PDF whose native extraction yields no text, the best-matching rule's converter produces the document text; chunking and embedding are unchanged.

Rules are per-project. SOAT performs no OCR or transcription itself. In the engine & algorithms pattern, the converter contract is the knowledge engine's bring-your-own-algorithm seam.

The same shape applied to turns instead of files is a memory rule: a selector and a pluggable handler, owned by the memory store the facts land in.

See the Permissions Reference for the IAM action strings for this module.

Data Model​

IngestionRule​

FieldTypeDescription
idstringPublic identifier prefixed with igr_
project_idstringID of the owning project
content_type_globstringGlob matched against the file's content_type (image/*, image/png, audio/mpeg, application/pdf)
tool_idstring | nullConverter tool (tool_…). Must be a server-callable type: http, mcp, builtin, or pipeline. client tools are rejected. Mutually exclusive with agent_id.
agent_idstring | nullConverter agent (agent_…). The file is sent to the agent as multimodal input and its text output becomes the document content. Mutually exclusive with tool_id.
actionstring | nullOperation id, required for builtin/mcp tool converters
preset_parametersobject | nullMerged into the tool input before invocation (tool converters only). Cannot contain the reserved keys file or callback, which ingestion injects. A key the converter tool itself pins in its own preset_parameters stays pinned — the tool's value wins over the rule's.
native_extractionstringFor PDFs: first (default) converts only when native extraction yields no text; skip bypasses native extraction and converts every matching PDF. Ignored for non-native types.
file_deliverystringHow the file reaches a tool converter: base64 (default) or download_url
chunk_strategystring | nullOptional default chunk strategy (page/whole/size), overridable per ingest request
chunk_sizenumber | nullOptional default for the size strategy
chunk_overlapnumber | nullOptional default for the size strategy
metadataobjectCaller-owned annotations on the rule — see Tags and metadata
created_atstringISO 8601 creation timestamp
updated_atstringISO 8601 last-updated timestamp

project_id + content_type_glob is unique. Exactly one of tool_id / agent_id must be set. A glob carries at most 4 wildcards and 255 characters; longer is refused with INGESTION_RULE_VALIDATION_FAILED.

Key Concepts​

Content-Type Matching​

resolveIngestionRule picks the most specific match: exact type (image/png) beats subtype wildcard (image/*) beats full wildcard (*/*).

Rules are consulted when:

  1. Non-native content type — no built-in extractor.
  2. Empty native extraction — a rule matching application/pdf is an OCR fallback for scanned PDFs: the built-in unpdf extractor runs first and the converter only when it returns no text.

With no matching rule, a non-native type is rejected with UNSUPPORTED_FILE_TYPE (400) and an empty native extraction fails with FILE_PARSE_FAILED.

PDF Conversion Mode​

native_extraction on the matching application/pdf rule: first (default) runs unpdf first and converts only PDFs with no text layer; skip converts every matching PDF (for unreliable text layers). No effect on non-native types.

Converter: Tool or Agent​

Exactly one of:

  • Tool converter (tool_id): ingestion calls the tool with the JSON contract below and reads text from its response. Suits audio, specialized OCR APIs and long async jobs (the tool can defer via the callback). The tool may be another project's, reached through an accepted share; once the share is suspended or revoked, the document fails with CONVERTER_FAILED.
  • Agent converter (agent_id): ingestion sends the file as multimodal input with a fixed "extract all text / transcribe" instruction; the agent's text output becomes the content. The model must support the modality (vision for images and scanned PDFs, audio-capable for audio). The generation is awaited inline; no deferral/callback.

When the converter is deleted​

A rule outlives its converter. Deleting a tool or agent that one of its own project's rules converts with is 409 TOOL_HAS_DEPENDENTS / AGENT_HAS_DEPENDENTS with meta.ingestion_rule_count, until force=true; another project's rule, converting through a share, never blocks the delete. The rule keeps its tool_id or agent_id, and every document it matches fails with CONVERTER_FAILED until PATCH /api/v1/ingestion-rules/{ingestion_rule_id} names another converter.

Audio agent converters need a Chat Completions-compatible AI provider

An AI provider with the openai slug uses OpenAI's Responses API, which rejects audio input: the document fails with CONVERTER_FAILED (AI_UnsupportedFunctionalityError: file part media type audio/...). Register an audio-capable model (e.g. gpt-audio-mini) under the custom slug with base_url https://api.openai.com/v1 (Chat Completions) instead. Vision converters are unaffected. Speech-to-text APIs that are not chat-completions-shaped (including xAI's) need a tool converter; see Ingest Images and Audio with Converters.

Building a Tool Converter for a Third-Party API​

No adapter service is needed: an http tool points execute.url at the third-party API, a pipeline tool wrapping it reshapes request and response with JSON Logic, and IngestionRule.tool_id names the pipeline tool. Since the Converter Tool Contract accepts a bare string, the pipeline's output can be a single var (e.g. { "var": "steps.call.text" }).

Hold the API key in a Secret and use a secret reference in execute.headers. For multipart/form-data APIs, set execute.body_mode: "multipart"; the { content_type, filename, data_base64 } file shape is decoded and attached as a file part.

Converter Tool Contract​

A tool converter is called via the normal server-side tool path with a fixed input and one of three output shapes.

Input built by ingestion:

{
  "file": {
    "id": "file_01",
    "filename": "scan.png",
    "content_type": "image/png",
    "size": 20480,
    "data_base64": "iVBORw0KGgo…",        // when file_delivery = base64
    "download_url": "https://…/files/file_01/download?token=…" // when file_delivery = download_url
  },
  "callback": {                            // lets long-running tools defer their result
    "url": "https://…/api/v1/documents/doc_01/ingestion-callback",
    "token": "…"
  }
  // preset_parameters are merged in at the top level
}

Output — the tool may return either extracted text or a deferral:

"All the extracted text"                              // wrapped as a single page
{ "pages": [{ "text": "page 1", "page_number": 1 }] } // paged (e.g. OCR per page)
{ "status": "pending" }                                // long-running deferral — see below

Any other shape fails the document with CONVERTER_OUTPUT_INVALID; a tool error with CONVERTER_FAILED. { "status": "pending" } is honored only for a tool converter in the default async mode (see Synchronous vs Async (Callback) Conversion); an agent converter or a ?wait=true request fails with CONVERTER_FAILED.

File Delivery​

file_delivery controls how the file bytes reach the tool's external API:

ModeBehaviorUse when
base64 (default)Ingestion downloads the file and passes data_base64 in the tool inputSmall files; provider-agnostic; works with any storage backend. Note: the whole file is loaded into memory and the request body.
download_urlIngestion passes a short-lived signed download_url; the tool/API fetches itLarge files (long audio, high-resolution images/scans) where base64 is impractical, and providers that accept a remote URL

Synchronous vs Async (Callback) Conversion​

A tool returning text (or { pages }) is synchronous: chunking and embedding continue inline. An agent converter is always synchronous.

A tool returning { status: "pending" } is asynchronous, only under default async ingestion (POST /documents/ingest without ?wait=true). The document stays in processing until the tool (or its service) posts the result to the ingestion-callback endpoint; see Deliver an async converter result for path, query token and schema. Document ID and token come from the callback block in the tool input (Converter Tool Contract); the body uses the synchronous output contract adapted for JSON (a single page is { "text": "..." }, not a bare string).

The callback is authorized by a single-use signed token scoped to the document and ingestion attempt, not an IAM action. Accepted (204) only while that attempt is processing; a replay, a superseded attempt (after re-ingest), or arrival after the stall timeout is rejected with 409 INGESTION_CALLBACK_CONFLICT; a bad token with 401 INGESTION_CALLBACK_INVALID_TOKEN. A valid result runs the chunk + embed tail and marks the document ready.

?wait=true requests and agent converters fail immediately with CONVERTER_FAILED on { status: "pending" }; a tool should defer only under async ingestion.

A document awaiting a callback longer than CONVERSION_STALL_TIMEOUT_MS is failed with CONVERSION_TIMEOUT (see Configuration), the counterpart of stuck-ingestion recovery. A callback racing the timeout is settled by atomic compare-and-set: it wins or gets 409, never silently dropped.

Failure Reasons​

Converter-related error.code values on a failed document (alongside FILE_PARSE_FAILED, INGESTION_TIMEOUT, INGESTION_FAILED):

error.codeMeaning
CONVERTER_FAILEDThe converter tool/agent call errored, an agent converter returned an async deferral (unsupported), or a tool converter returned an async deferral during synchronous ingestion (?wait=true)
CONVERTER_OUTPUT_INVALIDThe tool (or callback) returned an unrecognized output shape
CONVERSION_TIMEOUTAn async conversion did not call back within CONVERSION_STALL_TIMEOUT_MS

Who may act on an ingestion rule​

Every route that acts on one ingestion rule is authorized against that ingestion rule's SRN, srn:<project_id>:ingestionRule:<ingestion_rule_id>, not against the project. A policy may therefore name the ingestion rules it covers:

{
  "statement": [
    {
      "effect": "Allow",
      "action": ["ingestion-rules:GetIngestionRule", "ingestion-rules:UpdateIngestionRule"],
      "resource": ["srn:proj_V1StGXR8Z5jdHi6B:ingestionRule:igr_V1StGXR8Z5jdHi6B"]
    }
  ]
}

The SRN type is spelled ingestionRule, in camelCase, where every other type is snake_case. srn:<project_id>:ingestion_rule:… matches nothing — renaming it would change a published contract, so it is tracked on its own.

Refusals keep the shapes IAM defines: a read the caller may not perform is 404 (an ingestion rule it may not see does not announce itself), a write is 403, and a credential scoped to another project is 403 API_KEY_PROJECT_SCOPE. A write on a resource in a project none of the caller's policies name is 404 too — the same answer their read would get, so a refusal never confirms existence across a tenant boundary.

Listing ingestion rules stays project-scoped: GET /api/v1/ingestion-rules asks whether the caller may list ingestion rules in a project at all, so a policy that names individual ingestion rules grants no listing.

Configuration​

Environment VariableRequiredDescription
CONVERSION_STALL_TIMEOUT_MSNoHow long (ms) a document may await an async converter callback before being auto-failed with CONVERSION_TIMEOUT. Defaults to 30 minutes. Separate from, and typically longer than, INGESTION_STALL_TIMEOUT_MS (default 5 minutes).

Examples​

Create an ingestion rule​

soat create-ingestion-rule \
  --project-id proj_ABC \
  --content-type-glob "image/*" \
  --tool-id tool_ocr \
  --file-delivery base64 \
  --chunk-strategy whole

For an agent converter pass --agent-id instead of --tool-id. POST /documents/ingest routes matching files to the converter automatically; see Documents.

List rules​

soat list-ingestion-rules --project-id proj_ABC