Skip to main content

Documents

The Documents module stores documents with per-chunk embedding vectors for semantic search across project content.

Overview​

A Document is backed by a File and scoped to a project. Its content is split into DocumentChunks, each with its own embedding vector, for cosine-similarity search without an external vector database.

Two creation paths:

Ids are prefixed doc_; the internal primary key is never returned.

A corpus many agents write into is covered end to end in Single Source of Truth.

See the Permissions Reference for the IAM action strings for this module.

Data Model​

Document​

FieldTypeDescription
idstringPublic identifier prefixed with doc_
file_idstringID of the underlying File record
project_idstringID of the owning project
pathstring | nullLogical path within the project (e.g. /reports/q1.txt). Also used as the resource ID segment in path-based SRNs.
filenamestringOriginal filename
content_typestringMedia type of the source file the document was ingested from (e.g. application/pdf). Absent when the underlying file is gone.
sizenumberFile size in bytes
statusstringIngestion lifecycle state: pending → processing → ready | failed, plus withdrawn. Plain-text documents are always ready.
versionnumberContent version, starting at 1 — see Versioning
titlestring | nullHuman-readable title (auto-set to filename for PDF ingestion)
metadataobject | nullCaller-supplied JSON, read by the server only to judge it against the project's metadata schemas — see Tags and metadata. Stored as JSON, so the structure written is the structure held; null on an update clears it. Key casing is preserved verbatim — unlike other response fields, metadata keys are not converted between snake_case and camelCase. Ingestion progress (chunk_count, total_pages) and failure info (error) live on GET /documents/:id/status instead — see Polling Ingestion Status.
tagsobject | nullKey-value string tags
contentstring | nullThe document's text exactly as written; for an ingested binary (PDF), the extracted text, each window's overlap counted once — only present in GET /documents/:id responses
chunk_strategystringThe chunk strategy the document was last (re-)ingested with (page | whole | size). Absent when the default (whole) was used — the key is omitted rather than sent as null.
chunk_sizenumberWindow size in characters used when chunk_strategy is size. Absent otherwise.
chunk_overlapnumberOverlap in characters between consecutive windows used when chunk_strategy is size. Absent otherwise.
created_atstringISO 8601 creation timestamp
updated_atstringISO 8601 last-updated timestamp

DocumentChunk (internal)​

Chunks are not exposed directly; they are used for embedding-based search. content on GET /documents/:id is the stored text, not the chunks.

FieldTypeDescription
chunk_indexnumberZero-based position of the chunk within the document
page_numbernumber | nullSource page number (PDF ingestion only)
contentstringText of this chunk
embeddingvectorpgvector embedding — stored but never returned

Path Field​

path defaults to /<filename>, or to /<document_id>.txt for a document created from inline content with neither a path nor a filename — a document with no path is reachable only by its id, since a prefix filter is a LIKE and never matches null. That document's filename is the same leaf, so the key and the download name agree.

Paths are absolute (start with /) and normalized (. and .. resolved); project_id + path is unique. PATCH /documents/{document_id} accepts path to move a document, and null to clear it — a cleared document is reachable by id alone.

A path another file in the project holds is 409 NAME_CONFLICT on create, on ingest and on a move, the same answer the files API gives. A refused move changes nothing, content included.

Platform-written documents​

A document the platform writes on your behalf — today, each conversation message — is filed under the reserved root at /.system/conversations/{conversation_id}/{document_id}.txt.

It is read-only: PATCH, a path move and a tag write are refused with 400 RESERVED_PATH, and the owning module deletes it. A read by id returns it as normal, but a list or a knowledge search leaves it out unless the request names a directory inside the root.

Writing your own document under /.system/ is refused with the same code, and so is any tag key starting with system. — those record which conversation, actor, agent and role a row came from, and are the platform's to write (400 RESERVED_TAG_KEY). See what a turn is stamped with.

Listing a Directory​

GET /api/v1/documents accepts path_prefix to return one directory:

soat list-documents --project-id proj_ABC --path-prefix /reports/

The prefix is a path boundary, not a substring: /reports matches /reports/q1.txt, never /reports-archive/q1.txt. reports, /reports and /reports/ are the same filter; % and _ are literal. / selects the whole project apart from the reserved root, which only a prefix naming it returns. The filter runs in SQL with the policy filter, so total and pagination stay accurate per group.

Key Concepts​

Async File Ingestion​

POST /api/v1/documents/ingest returns 202 Accepted by default; the document is created with status: pending and extraction + embedding run in the background. Poll GET /api/v1/documents/:id until status is ready or failed.

?wait=true blocks until completion and returns 201 Created with status: ready (or failed). See Synchronous & Asynchronous Execution for the platform-wide wait contract. A file larger than SYNC_INGESTION_MAX_BYTES (default 10 MB) is rejected with 413 FILE_TOO_LARGE_FOR_SYNC; ingest it in background mode instead.

Tags​

Key-value string pairs set at creation or ingestion, or via the tag sub-endpoints, and matched by soat:ResourceTag/<key>. GET /api/v1/documents filters by pair with ?tags=key:value (repeatable, all must match); Knowledge search applies the same pairs to chunks. See IAM — Tags.

Put a field in tags if a policy should read it or it identifies the row; otherwise put it in metadata.

Polling Ingestion Status​

GET /documents/:id returns the full chunk content, which can be megabytes. GET /api/v1/documents/:id/status returns only the lifecycle fields:

{
  "id": "doc_V1StGXR8Z5jdHi6B",
  "status": "processing",
  "chunk_count": 7,
  "total_chunks": 12,
  "total_pages": 12,
  "progress": 58,
  "error": null
}

Field semantics (they change with status):

FieldMeaning
statuspending → processing → ready | failed
chunk_countChunks currently indexed — a live count. It is 0 while pending, grows during processing, and equals the final total once ready.
total_chunksPlanned total number of chunks, known once chunking begins (null until then). The denominator for progress.
total_pagesSource pages extracted. null until extraction has run (i.e. until ready/failed); null is not the same as zero pages.
progressPercentage chunk_count / total_chunks. 0 while pending, climbs while processing (capped at 99), 100 when ready, null when failed or not yet computable.
errorThe failure_reason (e.g. FILE_PARSE_FAILED, INGESTION_TIMEOUT). Only set when status is failed; otherwise null.

Ingestion Events​

Every path that settles an ingestion (pipeline, async converter callback, stall sweeper) emits a terminal event on the project event bus, deliverable through a webhook:

EventEmitted whenExtra data field
documents.ingestedThe document reached status: ready and its chunks are queryablechunk_count — the final number of chunks indexed
documents.ingest_failedThe ingestion settled in status: failederror — the same reason GET /documents/:id/status reports

Both carry the document in data in REST shape. A re-ingest emits a fresh event each time it settles; a document stuck in processing emits nothing until the stall sweeper fails it (see Stuck Ingestion Recovery).

Stuck Ingestion Recovery​

A document left in processing (or pending) by a dead worker is self-recovered: when read via GET /documents/:id or GET /documents/:id/status after no progress for INGESTION_STALL_TIMEOUT_MS (default 5 minutes), it transitions to failed with error = INGESTION_TIMEOUT. Re-process it with the re-ingest endpoint below.

Re-ingesting a Document​

POST /api/v1/documents/:id/ingest re-runs ingestion against the stored source file: chunks are discarded and the document reset to status: pending. Use it to recover a failed document or re-chunk with a different chunk_strategy. It accepts the same chunk_strategy / chunk_size / chunk_overlap fields and ?wait= toggle as POST /documents/ingest, returning 202 (default) or 201 (?wait=true).

Lifecycle states:

StatusMeaning
pendingEnqueued; background worker has not started yet
processingActively extracting pages, chunking, and generating embeddings
readyFully indexed; content and chunk embeddings are available for search
failedProcessing encountered an error. The error field on GET /documents/:id/status describes it
withdrawnTaken out of listings and knowledge search, with its chunks dropped — see Withdrawal

error values: FILE_PARSE_FAILED (no extractable text and no matching converter rule), FILE_NOT_FOUND, INGESTION_TIMEOUT (see Stuck Ingestion Recovery); with an Ingestion Rule, also CONVERTER_FAILED, CONVERTER_OUTPUT_INVALID, and CONVERSION_TIMEOUT.

Embedding concurrency is bounded (default 5 simultaneous requests).

Versioning​

A document's content and annotations are versioned by the same append-only archive as agent versions. Version 1 is written on create; every write that changes the content, title, path, metadata, tags or chunk configuration increments version and archives the state it replaced. Re-writing the state the document already holds archives nothing, so two version numbers never denote the same content — which is what lets a run cite a version to say what it read. A knowledge search result carries the document_version its chunk belongs to, and a generation records it for every chunk its agent's retrieval injected (retrieval).

A write may name the version it is changing (expected_version, or an If-Match header) and is refused with 409 VERSION_CONFLICT when the document has moved on — see Concurrent Writes. A refused write leaves the document exactly as it was — stored text, search chunks and path — and so does the loser of two concurrent writes that read the same version.

OperationEndpoint
List versions, newest firstGET /api/v1/documents/{document_id}/versions
Fetch one versionGET /api/v1/documents/{document_id}/versions/{version}
Roll back to a versionPOST /api/v1/documents/{document_id}/versions/{version}/restore

Restore appends. Restoring v1 of a document at v2 writes v1's content as v3, so a run citing v2 still resolves. It runs through the ordinary update path, so the content is re-chunked and re-embedded; restoring the state the document already holds is a no-op.

A snapshot is the whole state, not a diff. What a restore has to reproduce is what a run read, which is a read by version, and a diff chain would have to be replayed to answer that.

Withdrawal​

POST /api/v1/documents/{document_id}/withdraw takes a document out of every default read while keeping its history. The withdrawal is archived as a tombstone version carrying no content, so one mechanism answers what a document holds now and there is no second lifecycle flag for a reader to miss.

  • It leaves GET /api/v1/documents; ?include_withdrawn=true shows it again.
  • Its chunks are dropped from the index, so it leaves knowledge search entirely. A withdrawn document therefore costs a live search nothing, and include_withdrawn has no meaning there.
  • It stays readable by id, which is what makes its history readable and its restore possible.
  • Withdrawing an already-withdrawn document is 409 DOCUMENT_ALREADY_WITHDRAWN.

Restoring any content version brings it back: the content is re-chunked, the document returns to listings and search, and a documents.restored event fires. Restoring the tombstone itself is 400 — it holds no content, so restore the version before it.

Withdrawal does not apply under /.system/: a platform-written document keeps the owning module's lifecycle. DELETE stays what it is — permanent, and it removes the backing file.

Metadata schemas​

A project may declare what a document's metadata must satisfy under a path prefix, so a corpus many writers share reads as structured data rather than as whatever each writer happened to attach. The declaration is a metadata schema:

POST /api/v1/metadata-schemas
{
  "project_id": "proj_abc",
  "resource_type": "document",
  "path_prefix": "/reports",
  "schema": {
    "type": "object",
    "required": ["quarter"],
    "properties": {
      "quarter": { "type": "string", "enum": ["Q1", "Q2", "Q3", "Q4"] },
      "owner": { "type": "string" }
    }
  }
}

A write that stores metadata violating the declaration in force is refused with 400 VALIDATION_FAILED, and error.meta names the metadata_schema_id, the resource_type and the path_prefix that refused it. Every door is judged the same way — POST /api/v1/documents, PATCH /api/v1/documents/{document_id}, a document resource in a formation, and a restore, which is a write like any other and can be refused by a declaration written after the version it restores.

  • A prefix is a path boundary. /reports governs /reports/q1.txt and never /reports-archive/q1.txt.
  • The longest matching prefix decides, and it decides alone: a nested prefix replaces the outer rule rather than adding to it, which is what lets one corner of a corpus be different.
  • No bag is an empty bag. A create that omits metadata is judged as {}, so a required field refuses it, exactly as the dry run below reports. Clearing metadata with "metadata": null is judged the same way. POST /api/v1/documents/ingest carries no bag: the document it files is judged when metadata is first written to it.
  • A move is a write. Repathing a document into a prefix whose schema its metadata — or its lack of any — does not satisfy is refused; a write that touches neither path nor metadata is not re-judged, so tightening a schema does not freeze the documents already stored.
  • The reserved root cannot be governed. A /.system prefix is refused at declaration — a platform-written document carries no caller metadata.

POST /api/v1/metadata-schemas/validate answers what a write would be told without writing, which is what a batch import wants before it starts.

Relations​

A document can say what it is to another document: POST /api/v1/documents/{document_id}/relations asserts a typed edge, and GET /api/v1/documents/{document_id}/relations pages the edges that document asserts.

TypeWhat the asserting document claims
derived_fromIt was produced from the other document
supersedesIt replaces the other document
citesIt refers to the other document

The kinds are a declared set, not free text: a corpus many agents write into is read by consumers that act on an edge, and derived-from, derivedFrom and derived from are three edges to a reader and one to whoever typed them.

  • An edge is owned by the document it leaves. Asserting one is a write of that document (documents:UpdateDocument) and leaves the other untouched, so an agent can record what its own report derives from without making another report claim anything.
  • Both documents live in one project. An edge that left the project would be visible from one end and invisible from the other, since every surface that reads it is project-scoped.
  • Asserting the same edge twice is 409. An edge is a fact; the second assertion is the same fact.
  • ?related_to=doc_… finds neighbours on either side — what a document points at, and what points at it. A document with no relations narrows the listing to nothing.
  • GET /api/v1/documents/{document_id}/relations lists the edges a document asserts, oldest first, in a { data, total, limit, offset } envelope (limit default 50, max 100). A single-document read carries no edges.
  • DELETE /api/v1/documents/{document_id}/relations/{relation_id} retracts an edge; both documents stay as they are.

NDJSON export​

GET /api/v1/documents/export streams a project's documents as newline-delimited JSON, oldest first, one document object per line — the corpus as a file, for archiving it or moving it into another system.

  • project_id is required; path_prefix narrows it exactly as it narrows the listing.
  • The rows are the rows GET /api/v1/documents returns for the same caller: a policy that hides a document hides it here, and withdrawn documents and the reserved /.system/ root are left out.
  • The response streams and pages internally, so a project's size is not the request's size.
  • Authorized by documents:ExportDocuments, separate from documents:ListDocuments.

The export is not an MCP tool (an unbounded stream has no JSON projection); tools use the paged list-documents.

Metadata filters​

GET /api/v1/documents narrows by the bag with ?metadata=, a JSON object url-encoded into one parameter. Knowledge search reads the same object in its request body, so a filter written for one holds for the other.

?metadata={"quarter":"Q1","revision":{"gte":3,"lt":11},"status":{"in":["draft","final"]}}
FormMeaning
"field": valueThe bag holds that field with exactly that value
{ "in": [...] }Any one of the listed values
{ "gt": v }, { "gte": v }, { "lt": v }, { "lte": v }Ordering; bounds on one field combine
  • The match is exact. {"revision": 3} and {"revision": "3"} are different filters, which is why the filter travels as JSON rather than as key:value pairs.
  • Ordering compares by the operand's type. gt/gte/lt/lte order the field against the value given: a number orders numerically, a string lexicographically. Any field, any scope. An operand with no ordering — a boolean, null, a list, an object — is 400 VALIDATION_FAILED with error.meta.field naming it.
  • A range never matches another type. A document whose field holds text is not below every number; it is not comparable, so it is left out of both directions rather than failing the query.
  • A filter narrows what the caller may already see. It is applied on top of the IAM policy, never instead of it.

File Ingestion and Chunking​

POST /api/v1/documents/ingest ingests a file uploaded via POST /api/v1/files/upload. Format is detected from content_type:

Content typeHow the source text is extracted
application/pdfParsed page-by-page; blank pages are dropped. If no text is extracted (e.g. a scanned PDF), ingestion falls back to a converter tool when an Ingestion Rule matches application/pdf.
text/plainRead as a single source page
text/markdownRead as a single source page
other (image/*, audio/*, …)Converted to text by the tool named in the matching Ingestion Rule, then chunked normally

A content type with no built-in extractor and no matching Ingestion Rule is rejected with UNSUPPORTED_FILE_TYPE (400).

file_id is unique across documents; ingesting an already-ingested file returns 409 FILE_ALREADY_INGESTED. Use Re-ingesting a Document to re-chunk or recover it; upload a new copy to ingest the same source under another path.

Extracted text is split by chunk_strategy:

  • chunk_strategy: page (default): one chunk per source page with page_number set (PDF only; non-paged sources yield one chunk).
  • chunk_strategy: whole: a single chunk, source text joined by newlines.
  • chunk_strategy: size: fixed-size character windows with overlap, chunk_size (default 1000) and chunk_overlap (default 200). Page attribution is dropped.

POST /api/v1/documents accepts the same options with default strategy whole.

Embeddings are computed concurrently across chunks; an embedding failure is non-fatal and the chunk is stored without a vector. chunk_count on GET /documents/:id/status can differ from total_pages: whole gives 1, size depends on text length.

The last-used chunk_strategy / chunk_size / chunk_overlap are persisted on the document, so a Formation document resource re-plan converges to a no-op. Changing a formation document's chunk_strategy re-chunks the stored text on the next update-formation.

Path-Based SRNs​

Policies can target documents by path; the server evaluates both the id-based and path-based SRN. Worked example: Agent SOAT Tools and Preset Parameters — Step 4 (Create documents):

SRN formMatches
srn:proj_ABC:document:doc_XYZSpecific document by ID
srn:proj_ABC:document:/reports/q1.txtDocument at the exact path /reports/q1.txt
srn:proj_ABC:document:/reports/*All documents under /reports/
srn:proj_ABC:document:*All documents in the project (id wildcard)
*All resources in the project

List and search apply policy filters in SQL, so pagination counts are accurate. SRN syntax: IAM Reference.

Project ID Resolution​

project_id is optional; when omitted, accessible projects are resolved from the caller's policies (an API key is scoped to its project). A supplied project_id the policies do not grant returns 403 Forbidden. See IAM — Authorization Model.

Configuration​

Environment VariableRequiredDescription
FILES_STORAGE_DIRYesDirectory where .txt files are written (shared with Files)
EMBEDDING_PROVIDERYesEmbedding backend: ollama, openai, or bedrock
EMBEDDING_MODELYesModel name, e.g. qwen3-embedding:0.6b
EMBEDDING_DIMENSIONSYesVector dimensions — must match the model output, e.g. 1024, and be at most 2000
OLLAMA_BASE_URLNoOllama server URL, defaults to http://localhost:11434
SYNC_INGESTION_MAX_BYTESNoMax file size (bytes) allowed for synchronous ingestion (?wait=true). Larger files return 413. Defaults to 10485760 (10 MB).
INGESTION_STALL_TIMEOUT_MSNoHow long (ms) a document may stay in pending/processing with no progress before it is auto-failed with INGESTION_TIMEOUT. Defaults to 300000 (5 min).

Ollama setup: ollama pull qwen3-embedding:0.6b, then set EMBEDDING_PROVIDER=ollama, EMBEDDING_MODEL=qwen3-embedding:0.6b, EMBEDDING_DIMENSIONS=1024, and (if not local) OLLAMA_BASE_URL.

Examples​

Create a document​

soat create-document \
  --project-id proj_ABC \
  --filename q1-report.txt \
  --path /reports/q1-report.txt \
  --content "Q1 revenue was \$1.2M..."

Ingest a file​

Upload via POST /api/v1/files/upload, then call POST /api/v1/documents/ingest with the returned file_id. Works for PDFs and text/* files alike.

# Step 1: upload the file (PDF, .txt, or .md). The CLI sends the bytes
# base64-encoded; for a large file, use the presigned-token flow instead
# (see the Files module).
FILE_ID=$(soat upload-file-base64 \
  --project-id proj_ABC \
  --content "$(base64 -w0 ./report.pdf)" \
  --filename report.pdf \
  --content-type application/pdf | jq -r '.id')

# Step 2: ingest — one chunk per page (default)
soat ingest-document \
  --project-id proj_ABC \
  --file-id "$FILE_ID" \
  --path-prefix /reports/

Filter a listing by metadata​

GET /api/v1/documents takes the filter as one url-encoded JSON object. Ordering operators need the field's type, so project_id names the project whose metadata schemas declare it.

soat list-documents \
  --project-id proj_V1StGXR8Z5jdHi6B \
  --metadata '{"quarter":"Q1","revision":{"gte":3}}'