# Files

> File upload, download, metadata, and deletion over a pluggable storage backend — local filesystem, S3, or GCS.

# Files

File upload, download, metadata management, and deletion over a pluggable storage backend (local filesystem, S3, or GCS).

## Overview

Files are associated with a project and persisted through the configured storage backend — local filesystem, S3, or GCS. Every file record exposes a public `id`; the internal database primary key is never returned. File metadata is tracked in PostgreSQL, while the physical location and backend selection are system-managed and not exposed through the API (see [Configuration](#configuration)).

> See the [Permissions Reference](../permissions.md) for the IAM action strings for this module.

## Related Tutorials

- [Debug Session, Generation, and Trace History - Step 6 (Download raw trace steps)](/docs/tutorials/debug-session-generation-trace-history#step-6---download-raw-trace-steps-using-file_id)
- [Orchestrate a Sonnet - Step 8 (Read the persisted poem document)](/docs/tutorials/orchestrate-a-sonnet#step-8--read-the-persisted-poem-document)
- [Permissions in Practice - Step 7 (Verify permissions with file operations)](/docs/tutorials/permissions#step-7--verify-permissions)

## Data Model

| Field          | Type                     | Description                                                                                                         |
| -------------- | ------------------------ | ------------------------------------------------------------------------------------------------------------------- |
| `id`           | string                   | Public identifier                                                                                                   |
| `prefix`       | string                   | Directory within the project (e.g. `/assets`). Optional on write; defaults to `/` (root). Read-only on the record (derived from `path`). |
| `filename`     | string                   | Original / download name and the key's leaf segment (e.g. `logo.png`). Optional on write; defaults to the uploaded file's name. |
| `path`         | string \| null           | **Read-only.** Full key = `prefix` + `/` + `filename` (e.g. `/assets/logo.png`). Unique per project; the file's identity and the resource ID segment in path-based SRNs. |
| `content_type` | string                   | MIME type                                                                                                           |
| `size`         | number                   | File size in bytes                                                                                                  |
| `metadata`     | string                   | Arbitrary JSON string for custom metadata                                                                           |
| `project_id`   | string                   | ID of the owning project                                                                                            |
| `created_at`   | string                   | ISO 8601 creation timestamp                                                                                         |
| `updated_at`   | string                   | ISO 8601 last-updated timestamp                                                                                     |

`path` is normalized at write time and unique per project; it is the file's identity and the target of path-based policy SRNs. To **move** a file, change its `prefix`; to **rename** it, change its `filename` — either rebuilds `path`. Writing to a `prefix` + `filename` that resolves to an existing `path` in the project returns `409 NAME_CONFLICT`.

## Key Concepts

### Storage Backends

The physical location of a file's bytes is handled by a **storage provider**, selected at runtime with `FILES_STORAGE_PROVIDER` (default `local`). The backend is transparent to the API: the same endpoints, records, and download flow work identically regardless of where the bytes live. Each file records which backend stored it, so reads and deletes always route back to the correct provider even if the active backend is later changed.

| Provider | `FILES_STORAGE_PROVIDER` | Where bytes live |
| -------- | ------------------------ | ---------------- |
| Local filesystem | `local` (default) | A project-scoped directory tree under `FILES_STORAGE_DIR` |
| S3 / S3-compatible | `s3` | Objects in the bucket named by `FILES_S3_BUCKET` |

Both backends use the same logical object layout, `{projectPublicId}/{category}/{fileId}{ext}`:

| Segment           | Description                                                                                        |
| ----------------- | -------------------------------------------------------------------------------------------------- |
| `projectPublicId` | Public project ID (e.g. `proj_ABC`) — isolates files by project                                    |
| `category`        | Derived from the first segment of the file's logical `path` (e.g., `/traces/foo.json` → `traces/`) |
| `fileId`          | The file's public ID                                                                               |
| `ext`             | File extension from the original filename                                                          |

Every writer builds this key the same way, so a new storage backend inherits the layout rather than defining its own. `ext` describes the stored bytes: a [document](/docs/modules/documents)'s text object is always `.txt`, whatever the document is named.

If a file has no `path`, the category defaults to `files/`. For the local backend this becomes a path under `FILES_STORAGE_DIR`; for S3 it becomes the object key (optionally namespaced by `FILES_S3_KEY_PREFIX`):

```
# local:   {FILES_STORAGE_DIR}/{projectPublicId}/{category}/{fileId}{ext}
/data/files/proj_1a123a/traces/trace_abc123.json
/data/files/proj_1a123a/documents/doc_xyz.md

# s3:      s3://{FILES_S3_BUCKET}/{FILES_S3_KEY_PREFIX}/{projectPublicId}/{category}/{fileId}{ext}
s3://my-bucket/proj_1a123a/traces/trace_abc123.json
```

Traces persist their raw step payloads as files in the `traces/` category; see it end to end in [Debug Session, Generation, and Trace History - Step 6 (Download raw trace steps)](/docs/tutorials/debug-session-generation-trace-history#step-6---download-raw-trace-steps-using-file_id).

### Path-Based SRNs

Policies can target files by their logical `path` rather than their `id`. When a file has a `path` set, the server evaluates **both** the id-based SRN and the path-based SRN:

| SRN form                              | Matches                                   |
| ------------------------------------- | ----------------------------------------- |
| `srn:proj_ABC:file:file_XYZ`         | Specific file by ID                       |
| `srn:proj_ABC:file:/assets/logo.png` | File at the exact path `/assets/logo.png` |
| `srn:proj_ABC:file:/exports/*`       | All files under `/exports/`               |
| `srn:proj_ABC:file:*`                | All files in the project (id wildcard)    |

The list endpoint applies policy filters at the SQL level — the database returns only rows the caller is permitted to see. See [IAM](./iam.md) for full SRN syntax and policy authoring guidance, or walk through scoping a read-only policy to files in [Permissions in Practice - Step 7 (Verify permissions with file operations)](/docs/tutorials/permissions#step-7--verify-permissions).

### Upload Tokens (decoupled uploads)

Upload tokens provide a two-step upload flow — the local-storage equivalent of an S3 presigned URL — usable from any client (SDK, CLI, curl, or an MCP agent):

1. **Request a token** — [`POST /api/v1/files/presigned-url`](/docs/api/files/create-presigned-url) returns a single-use `upload_token`, an `upload_url`, and an `expires_at` (15-minute lifetime). This step is authenticated and requires `files:UploadFile`. By default `upload_url` is **relative** (e.g. `/api/v1/files/upload/upt_xxx`); when the server is configured with `SOAT_BASE_URL`, it is returned as a **fully-qualified absolute URL** so clients and MCP agents can POST to it without knowing the server base URL in advance — see [Configuration](#configuration).
2. **Upload the content** — [`POST /api/v1/files/upload/{token}`](/docs/api/files/upload-file-with-token) writes the file and returns the standard file record. This endpoint requires **no bearer credential** — the token is the credential — and accepts either `multipart/form-data` (field `file`) or JSON with a base64 `content` field.

Because the two steps are decoupled, the party that authorizes the upload (step 1) need not be the party that transfers the bytes (step 2) — the token can be handed to a browser, a worker, or a CLI to complete the upload directly over HTTP.

The token is invalidated after a single successful upload. Subsequent uploads return `409`; expired tokens return `410`; unknown tokens return `404`.

### Downloading from a tool

[`GET /api/v1/files/{file_id}/download`](/docs/api/files/download-file) streams the raw bytes and is a REST/SDK/CLI operation only — raw bytes have no JSON form, so it is not offered as an MCP or `builtin` tool action. Use `download-file-base64`, which returns the same content as a base64 string in a normal JSON response. Large files are subject to the client's tool-call payload limit, so an agent should fetch the download URL out-of-band with whatever HTTP capability its runtime provides.

#### Large files via MCP

MCP tool-call payloads larger than ~100 KB are truncated, so `upload-file-base64` cannot carry a large file. Use the token flow instead: step 1 (`create-presigned-url`, exposed as an MCP tool) is always small; perform step 2 **out-of-band** — via a shell (`curl`), a `fetch`/HTTP tool, or a direct SDK call — using `multipart/form-data` streamed from disk, not the base64 `content` field:

```bash
# Step 1 returned upload_url = /api/v1/files/upload/upt_xxx
curl -F "file=@/path/to/large-report.pdf" "$BASE_URL/api/v1/files/upload/upt_xxx"
```

## Configuration

| Environment Variable | Required | Description                                                                                             |
| -------------------- | -------- | ------------------------------------------------------------------------------------------------------- |
| `FILES_STORAGE_PROVIDER` | No   | Storage backend: `local` (default) or `s3`. Selects where new files are written. |
| `FILES_STORAGE_DIR`  | For `local` | Absolute path to the directory where uploaded files are stored. Must be writable by the server process. Required when the provider is `local`. |
| `FILES_S3_BUCKET`    | For `s3` | Name of the S3 bucket that stores file objects. Required when the provider is `s3`. |
| `FILES_S3_REGION`    | No       | AWS region of the bucket. Falls back to `AWS_REGION` if unset. |
| `FILES_S3_KEY_PREFIX` | No      | Key prefix prepended to every object, to namespace files within a shared bucket (e.g. `soat/`). |
| `FILES_S3_ENDPOINT`  | No       | Custom endpoint URL for S3-compatible stores (e.g. MinIO, Cloudflare R2). Omit for AWS S3. |
| `FILES_S3_FORCE_PATH_STYLE` | No | Set to `true` to use path-style bucket addressing (required by some S3-compatible stores). |
| `FILE_UPLOAD_MAX_BYTES` | No    | Ceiling on a multipart upload, in bytes. Defaults to `26214400` (25 MB). A larger body is refused with `UPLOAD_TOO_LARGE` (`413`) while it is still streaming, so nothing is buffered or stored. |
| `SOAT_BASE_URL`      | No       | Public base URL of the server (e.g. `https://api.example.com`). When set, the presigned-URL flow returns an absolute `upload_url`; otherwise the URL is relative. A trailing slash is trimmed. |

AWS credentials for the `s3` backend are resolved through the standard AWS SDK credential chain (`AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY`, a shared profile, or an instance/task role).

When running the `local` backend via Docker, mount a volume at `FILES_STORAGE_DIR` to persist files across container restarts:

```yaml
services:
  server:
    image: soat-server
    environment:
      FILES_STORAGE_DIR: /data/files
    volumes:
      - files-data:/data/files

volumes:
  files-data:
```

To use S3 instead, set the provider and bucket (no volume needed):

```yaml
services:
  server:
    image: soat-server
    environment:
      FILES_STORAGE_PROVIDER: s3
      FILES_S3_BUCKET: my-soat-files
      FILES_S3_REGION: us-east-1
```

## Examples

### Upload a file (base64)

<Tabs groupId="client">
<TabItem value="cli" label="CLI" default>

```bash
soat upload-file-base64 \
  --project-id proj_ABC \
  --content "iVBORw0KGgo..." \
  --prefix /assets \
  --filename logo.png
```

</TabItem>
<TabItem value="sdk" label="SDK">

```ts

const soat = new SoatClient({ baseUrl: 'https://api.example.com', token: 'sk_...' });

const { data, error } = await soat.files.uploadFileBase64({
  body: {
    project_id: 'proj_ABC',
    content: 'iVBORw0KGgo...',
    prefix: '/assets',
    filename: 'logo.png',
  },
});
if (error) throw new Error(JSON.stringify(error));
```

</TabItem>
<TabItem value="curl" label="curl">

```bash
curl -X POST https://api.example.com/api/v1/files/upload-base64 \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{
    "project_id": "proj_ABC",
    "content": "iVBORw0KGgo...",
    "prefix": "/assets",
    "filename": "logo.png"
  }'
```

</TabItem>
</Tabs>

### Upload a file via an upload token

<Tabs groupId="client">
<TabItem value="cli" label="CLI" default>

```bash
# Step 1 — request a single-use token
TOKEN=$(soat create-presigned-url \
  --project-id proj_ABC \
  --content-type application/pdf \
  --prefix /documents \
  --filename report.pdf | jq -r .upload_token)

# Step 2 — upload the content directly (no payload limit)
soat upload-file-with-token \
  --token "$TOKEN" \
  --content "$(base64 -w0 report.pdf)"
```

</TabItem>
<TabItem value="sdk" label="SDK">

```ts
const { data: token } = await soat.files.createPresignedUrl({
  body: {
    project_id: 'proj_ABC',
    content_type: 'application/pdf',
    prefix: '/documents',
    filename: 'report.pdf',
  },
});

const { data, error } = await soat.files.uploadFileWithToken({
  path: { token: token!.upload_token! },
  body: { content: base64Content },
});
if (error) throw new Error(JSON.stringify(error));
```

</TabItem>
<TabItem value="curl" label="curl">

```bash
# Step 1 — request a token
TOKEN=$(curl -s -X POST https://api.example.com/api/v1/files/presigned-url \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{"project_id":"proj_ABC","prefix":"/documents","filename":"report.pdf"}' | jq -r .upload_token)

# Step 2 — upload the file (token is the credential, no Authorization header)
curl -X POST "https://api.example.com/api/v1/files/upload/$TOKEN" \
  -F "file=@report.pdf"
```

</TabItem>
</Tabs>

### List files in a project

<Tabs groupId="client">
<TabItem value="cli" label="CLI" default>

```bash
soat list-files --project-id proj_ABC
```

</TabItem>
<TabItem value="sdk" label="SDK">

```ts
const { data, error } = await soat.files.listFiles({
  query: { project_id: 'proj_ABC' },
});
if (error) throw new Error(JSON.stringify(error));
```

</TabItem>
<TabItem value="curl" label="curl">

```bash
curl https://api.example.com/api/v1/files?project_id=proj_ABC \
  -H "Authorization: Bearer <token>"
```

</TabItem>
</Tabs>