Skip to main content

Files

File upload, download, metadata management, and deletion over a pluggable storage backend (local filesystem, S3, or GCS).

Overview​

Files are associated with a project and persisted through the configured storage backend — local filesystem, S3, or GCS. Every file record exposes a public id; the internal database primary key is never returned. File metadata is tracked in PostgreSQL, while the physical location and backend selection are system-managed and not exposed through the API (see Configuration).

See the Permissions Reference for the IAM action strings for this module.

Data Model​

FieldTypeDescription
idstringPublic identifier
prefixstringDirectory within the project (e.g. /assets). Optional on write; defaults to / (root). Read-only on the record (derived from path).
filenamestringOriginal / download name and the key's leaf segment (e.g. logo.png). Optional on write; defaults to the uploaded file's name.
pathstring | nullRead-only. Full key = prefix + / + filename (e.g. /assets/logo.png). Unique per project; the file's identity and the resource ID segment in path-based SRNs.
content_typestringMIME type
sizenumberFile size in bytes
metadatastringArbitrary JSON string for custom metadata
project_idstringID of the owning project
created_atstringISO 8601 creation timestamp
updated_atstringISO 8601 last-updated timestamp

You address a file with two write fields, mirroring an S3 object: a prefix (the directory, like an S3 prefix — defaults to /) and a filename (the leaf name, like the tail of an S3 key). The server combines them into the read-only path = prefix + / + filename — the file's full key (akin to an S3 object key). path is normalized at write time, and project_id + path is unique within a project; it is the file's identity and the target of path-based policy SRNs. To move a file, change its prefix; to rename it, change its filename — either rebuilds path. Creating or uploading a file at a prefix + filename that already resolves to an existing path in the project returns 409 NAME_CONFLICT. Storage backend selection (local/s3/gcs) and the physical on-disk location are system-managed and not exposed through the API — see Configuration.

Key Concepts​

Storage Backends​

The physical location of a file's bytes is handled by a storage provider, selected at runtime with FILES_STORAGE_PROVIDER (default local). The backend is transparent to the API: the same endpoints, records, and download flow work identically regardless of where the bytes live. Each file records which backend stored it, so reads and deletes always route back to the correct provider even if the active backend is later changed.

ProviderFILES_STORAGE_PROVIDERWhere bytes live
Local filesystemlocal (default)A project-scoped directory tree under FILES_STORAGE_DIR
S3 / S3-compatibles3Objects in the bucket named by FILES_S3_BUCKET

Both backends use the same logical object layout, {projectPublicId}/{category}/{fileId}{ext}:

SegmentDescription
projectPublicIdPublic project ID (e.g. proj_ABC) — isolates files by project
categoryDerived from the first segment of the file's logical path (e.g., /traces/foo.json → traces/)
fileIdThe file's public ID
extFile extension from the original filename

If a file has no path, the category defaults to files/. For the local backend this becomes a path under FILES_STORAGE_DIR; for S3 it becomes the object key (optionally namespaced by FILES_S3_KEY_PREFIX):

# local:   {FILES_STORAGE_DIR}/{projectPublicId}/{category}/{fileId}{ext}
/data/files/proj_1a123a/traces/trace_abc123.json
/data/files/proj_1a123a/documents/doc_xyz.md

# s3:      s3://{FILES_S3_BUCKET}/{FILES_S3_KEY_PREFIX}/{projectPublicId}/{category}/{fileId}{ext}
s3://my-bucket/proj_1a123a/traces/trace_abc123.json

Traces persist their raw step payloads as files in the traces/ category; see it end to end in Debug Session, Generation, and Trace History - Step 6 (Download raw trace steps).

Path-Based SRNs​

Policies can target files by their logical path rather than their id. When a file has a path set, the server evaluates both the id-based SRN and the path-based SRN:

SRN formMatches
soat:proj_ABC:file:file_XYZSpecific file by ID
soat:proj_ABC:file:/assets/logo.pngFile at the exact path /assets/logo.png
soat:proj_ABC:file:/exports/*All files under /exports/
soat:proj_ABC:file:*All files in the project (id wildcard)

The list endpoint applies policy filters at the SQL level — the database returns only rows the caller is permitted to see. See IAM for full SRN syntax and policy authoring guidance, or walk through scoping a read-only policy to files in Permissions in Practice - Step 7 (Verify permissions with file operations).

Upload Tokens (decoupled uploads)​

Upload tokens provide a two-step upload flow — the local-storage equivalent of an S3 presigned URL — usable from any client (SDK, CLI, curl, or an MCP agent):

  1. Request a token — POST /api/v1/files/presigned-url returns a single-use upload_token, an upload_url, and an expires_at (15-minute lifetime). This step is authenticated and requires files:UploadFile. By default upload_url is relative (e.g. /api/v1/files/upload/upt_xxx); when the server is configured with SOAT_BASE_URL, it is returned as a fully-qualified absolute URL so clients and MCP agents can POST to it without knowing the server base URL in advance — see Configuration.
  2. Upload the content — POST /api/v1/files/upload/{token} writes the file and returns the standard file record. This endpoint requires no bearer credential — the token is the credential — and accepts either multipart/form-data (field file) or JSON with a base64 content field.

Because the two steps are decoupled, the party that authorizes the upload (step 1) need not be the party that transfers the bytes (step 2) — the token can be handed to a browser, a worker, or a CLI to complete the upload directly over HTTP.

The token is invalidated after a single successful upload. Subsequent uploads return 409; expired tokens return 410; unknown tokens return 404.

Large files via MCP​

This flow is what makes large uploads possible through MCP. The upload-file-base64 tool requires the full base64 content as a single tool-call parameter, and payloads larger than ~100 KB are truncated before they reach the agent's tool call. With upload tokens, step 1 (create-presigned-url) is a small request with a small response that always fits.

Both steps are exposed as MCP tools (create-presigned-url and upload-file-with-token). However, the MCP payload limit still applies to step 2 when the bytes travel as a tool-call argument — so for large files the agent should perform step 2 out-of-band instead, using whatever non-MCP HTTP capability its runtime provides — a shell (e.g. curl), a fetch/HTTP tool, or a direct SDK call. The bytes then travel over plain HTTP and never become a tool-call argument, so the MCP payload limit never applies.

For large files, use multipart/form-data and stream the file straight from disk so it is never held as one big in-memory string — do not use the base64 content field, which would just reintroduce a large payload:

# Step 1 returned upload_url = /api/v1/files/upload/upt_xxx
curl -F "file=@/path/to/large-report.pdf" "$BASE_URL/api/v1/files/upload/upt_xxx"

An agent whose runtime has no out-of-band HTTP path (a pure LLM with only MCP tools and no shell, fetch, or SDK) cannot perform step 2 — but such an agent has no way to move a large file through any mechanism regardless. The token flow assumes the agent can make an ordinary HTTP request outside of MCP.

Configuration​

Environment VariableRequiredDescription
FILES_STORAGE_PROVIDERNoStorage backend: local (default) or s3. Selects where new files are written.
FILES_STORAGE_DIRFor localAbsolute path to the directory where uploaded files are stored. Must be writable by the server process. Required when the provider is local.
FILES_S3_BUCKETFor s3Name of the S3 bucket that stores file objects. Required when the provider is s3.
FILES_S3_REGIONNoAWS region of the bucket. Falls back to AWS_REGION if unset.
FILES_S3_KEY_PREFIXNoKey prefix prepended to every object, to namespace files within a shared bucket (e.g. soat/).
FILES_S3_ENDPOINTNoCustom endpoint URL for S3-compatible stores (e.g. MinIO, Cloudflare R2). Omit for AWS S3.
FILES_S3_FORCE_PATH_STYLENoSet to true to use path-style bucket addressing (required by some S3-compatible stores).
SOAT_BASE_URLNoPublic base URL of the server (e.g. https://api.example.com). When set, the presigned-URL flow returns an absolute upload_url; otherwise the URL is relative. A trailing slash is trimmed.

AWS credentials for the s3 backend are resolved through the standard AWS SDK credential chain (AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY, a shared profile, or an instance/task role).

When running the local backend via Docker, mount a volume at FILES_STORAGE_DIR to persist files across container restarts:

services:
  server:
    image: soat-server
    environment:
      FILES_STORAGE_DIR: /data/files
    volumes:
      - files-data:/data/files

volumes:
  files-data:

To use S3 instead, set the provider and bucket (no volume needed):

services:
  server:
    image: soat-server
    environment:
      FILES_STORAGE_PROVIDER: s3
      FILES_S3_BUCKET: my-soat-files
      FILES_S3_REGION: us-east-1

Examples​

Upload a file (base64)​

soat upload-file-base64 \
  --project-id proj_ABC \
  --content-base64 "iVBORw0KGgo..." \
  --prefix /assets \
  --filename logo.png

Upload a file via an upload token​

# Step 1 — request a single-use token
TOKEN=$(soat create-presigned-url \
  --project-id proj_ABC \
  --content-type application/pdf \
  --prefix /documents \
  --filename report.pdf | jq -r .upload_token)

# Step 2 — upload the content directly (no payload limit)
soat upload-file-with-token \
  --token "$TOKEN" \
  --content "$(base64 -w0 report.pdf)"

List files in a project​

soat list-files --project-id proj_ABC