Skip to main content

Govern Metadata with Schemas

metadata is a free bag: whatever a writer attaches is what the next reader gets. In a corpus several agents and people write into, that is how one writer's quarter becomes another's period, and the reader finds out at query time.

A metadata schema states the structure where every writer sees it. This tutorial declares one for /reports, watches a bad write be refused, checks a batch before sending it, and narrows the rule for one corner of the corpus.

By the end you will have:

WhatWhy it matters
A declaration governing /reportsEvery writer is judged against one rule, whichever door they came through
A refused writeThe corpus cannot acquire a document that violates it
A dry runA batch importer learns what it would be told, before sending anything
A nested declarationOne corner of the corpus can be different without loosening the rest

Prerequisites​

  • SOAT running locally at http://localhost:5047 (Quick Start).
  • jq on your PATH.
  • Key Concepts for projects, documents and the admin credential.
  • Documents for the path model this builds on.
export SOAT_BASE_URL=http://localhost:5047

Step 1 — Log in and create a project​

A declaration is project-scoped: it governs writes into the project it belongs to and nothing else.

ADMIN_TOKEN=$(soat login-user --username admin --password Admin1234! | jq -r '.token')
export SOAT_TOKEN=$ADMIN_TOKEN

PROJECT_ID=$(soat create-project --name "Quarterly Reports" | jq -r '.id')
echo "project: $PROJECT_ID"

Step 2 — Declare what a report must carry​

The declaration names three things: the resource_type it governs, the selector that type is addressed by, and the schema itself. A document is selected by path_prefix, because a path is what a document is filed under.

/reports is matched on a path boundary: it covers /reports/q1.txt and never /reports-archive/q1.txt.

REPORT_SCHEMA_ID=$(soat create-metadata-schema \
  --project-id "$PROJECT_ID" \
  --resource-type document \
  --path-prefix /reports \
  --schema '{"type":"object","required":["quarter","owner"],"properties":{"quarter":{"type":"string","enum":["Q1","Q2","Q3","Q4"]},"owner":{"type":"string","minLength":1}}}' \
  | jq -r '.id')

echo "schema: $REPORT_SCHEMA_ID"

The schema is compiled as it is stored, so one JSON Schema cannot parse is refused here rather than kept as a rule that silently governs nothing.

Step 3 — Write a report that satisfies it​

Nothing about the write changes: the declaration is enforced inside POST /api/v1/documents, not by anything the caller has to remember to call.

soat create-document \
  --project-id "$PROJECT_ID" \
  --path /reports/q1.txt \
  --content "Revenue grew 12% quarter over quarter." \
  --metadata '{"quarter":"Q1","owner":"finance"}' | jq '{id, path, metadata}'

Step 4 — Watch a bad write be refused​

A misspelled quarter and a missing owner. The refusal names the declaration that produced it, so a writer knows which rule to read rather than guessing which of its fields was wrong.

# → expect-fail
soat create-document \
  --project-id "$PROJECT_ID" \
  --path /reports/q2.txt \
  --content "Revenue was flat." \
  --metadata '{"quarter":"Quarter 2"}'
{
  "code": "VALIDATION_FAILED",
  "message": "metadata does not satisfy the schema declared for '/reports': (root) must have required property 'owner'; /quarter must be equal to one of the allowed values",
  "meta": {
    "metadata_schema_id": "mdschema_V1StGXR8Z5jdHi6B",
    "resource_type": "document",
    "path_prefix": "/reports"
  }
}

Two more things this refuses, for the same reason:

  • Clearing the bag. "metadata": null on PATCH /api/v1/documents/{document_id} is a write of the metadata, so a required field refuses it.
  • Moving a document in. Repathing a document into /reports is judged against /reports, because the move is what changed which rule applies.

A document that carries no metadata is judged as carrying an empty bag, so under /reports it is refused for the missing quarter. POST /api/v1/documents/ingest carries no bag at all: the document it files is judged when metadata is first written to it.

Step 5 — Check a batch before sending it​

An importer with a thousand rows wants to know what it would be told, without writing anything. POST /api/v1/metadata-schemas/validate answers exactly that: which declaration governs the path, and the error the write would be refused with.

soat validate-metadata \
  --project-id "$PROJECT_ID" \
  --path /reports/q3.txt \
  --metadata '{"quarter":"Q3"}' | jq
{
  "valid": false,
  "resource_type": "document",
  "metadata_schema_id": "mdschema_V1StGXR8Z5jdHi6B",
  "path_prefix": "/reports",
  "error": {
    "code": "VALIDATION_FAILED",
    "message": "metadata does not satisfy the schema declared for '/reports': (root) must have required property 'owner'",
    "meta": {
      "metadata_schema_id": "mdschema_V1StGXR8Z5jdHi6B",
      "resource_type": "document",
      "path_prefix": "/reports"
    }
  }
}

It reports and refuses nothing. The enforcement stays in the document write path, because a check a writer has to call is advisory — and the writer who skips it is the one the rule exists for.

Step 6 — Make one corner of the corpus different​

Legal reports carry a reviewing counsel that ordinary reports do not. Declare a second schema for the nested prefix: the longest matching prefix decides, and it decides alone (Metadata Schemas — a declaration is a row). A nested declaration replaces the outer rule rather than adding to it, which is what lets this corner differ instead of only ever being stricter.

soat create-metadata-schema \
  --project-id "$PROJECT_ID" \
  --resource-type document \
  --path-prefix /reports/legal \
  --schema '{"type":"object","required":["counsel"],"properties":{"counsel":{"type":"string","minLength":1}}}' \
  | jq '{id, path_prefix}'

# Governed by /reports/legal, so `counsel` alone is enough — `owner` is the
# outer rule's requirement and the outer rule does not apply here.
soat create-document \
  --project-id "$PROJECT_ID" \
  --path /reports/legal/q1-review.txt \
  --content "No material litigation exposure this quarter." \
  --metadata '{"counsel":"external"}' | jq '{path, metadata}'

Step 7 — Tighten the rule without touching what is stored​

Reports now need a published_at. Replace the schema on the existing declaration with PATCH /api/v1/metadata-schemas/{metadata_schema_id}: the next write of the fields it governs is judged against it, and the documents already stored are left alone.

soat update-metadata-schema \
  --metadata-schema-id "$REPORT_SCHEMA_ID" \
  --schema '{"type":"object","required":["quarter","owner","published_at"],"properties":{"quarter":{"type":"string","enum":["Q1","Q2","Q3","Q4"]},"owner":{"type":"string","minLength":1},"published_at":{"type":"string"}}}' \
  | jq '{id, path_prefix}'

# The Q1 report stored in Step 3 still reads back exactly as written.
soat list-documents --project-id "$PROJECT_ID" --path-prefix /reports \
  | jq '[.data[] | {path, metadata}]'

# A new report without the new field is refused.
# → expect-fail
soat create-document \
  --project-id "$PROJECT_ID" \
  --path /reports/q4.txt \
  --content "Revenue grew 4%." \
  --metadata '{"quarter":"Q4","owner":"finance"}'

Step 8 — Read the contract, and stop enforcing it​

GET /api/v1/metadata-schemas is what a new writer reads to learn what the project expects. Deleting a declaration stops governing new writes; the documents already stored keep the metadata they hold, because the rule governed writes rather than rows.

soat list-metadata-schemas --project-id "$PROJECT_ID" \
  | jq '[.data[] | {resource_type, path_prefix, required: .schema.required}]'

soat delete-metadata-schema --metadata-schema-id "$REPORT_SCHEMA_ID"

# Free again: /reports/legal still has its own rule, /reports has none.
soat create-document \
  --project-id "$PROJECT_ID" \
  --path /reports/q4.txt \
  --content "Revenue grew 4%." \
  --metadata '{"quarter":"Q4"}' | jq '{path, metadata}'

What you built​

A corpus whose structure is stated once and enforced everywhere it is written:

RuleWhere it holds
A prefix is a path boundary/reports covers /reports/q1.txt, never /reports-archive/q1.txt
The longest matching prefix decides, alone/reports/legal replaces /reports for what it covers
No bag is an empty bagA create without metadata and a clearing null are both judged as {}
Tightening applies forwardStored documents are untouched; the next write of those fields is judged
The check is not the enforcementvalidate reports; the document write path refuses

The same declaration governs a document resource in a formation, so a template can ship a corpus and the rule governing it together.

Today document is the resource type the registry enforces. A type joins it in the change that gives its write path the same gate — the declaration's shape does not change, only which selector it carries.

Next steps​