Skip to main content

Evaluations

SDK methods for the Evaluations module, accessed via soat.evaluations. See the Evaluations module docs for permissions and data model, and each method's REST page for the full request/response schema.

soat.evaluations.listDatasets()​

List datasets

  • Method: GET
  • Path: /api/v1/datasets
await soat.evaluations.listDatasets({
  query: { /* + optional */ },
});

→ Request & response schema

soat.evaluations.createDataset()​

Create a dataset

  • Method: POST
  • Path: /api/v1/datasets
await soat.evaluations.createDataset({
  body: { name, /* + optional */ },
});

→ Request & response schema

soat.evaluations.getDataset()​

Get a dataset

  • Method: GET
  • Path: /api/v1/datasets/{dataset_id}
await soat.evaluations.getDataset({
  path: { dataset_id },
});

→ Request & response schema

soat.evaluations.updateDataset()​

Update a dataset

  • Method: PUT
  • Path: /api/v1/datasets/{dataset_id}
await soat.evaluations.updateDataset({
  path: { dataset_id },
  body: { /* + optional */ },
});

→ Request & response schema

soat.evaluations.deleteDataset()​

Delete a dataset

  • Method: DELETE
  • Path: /api/v1/datasets/{dataset_id}
await soat.evaluations.deleteDataset({
  path: { dataset_id },
});

→ Request & response schema

soat.evaluations.listDatasetItems()​

List dataset items

  • Method: GET
  • Path: /api/v1/datasets/{dataset_id}/items
await soat.evaluations.listDatasetItems({
  path: { dataset_id },
  query: { /* + optional */ },
});

→ Request & response schema

soat.evaluations.createDatasetItem()​

Add a dataset item

  • Method: POST
  • Path: /api/v1/datasets/{dataset_id}/items
await soat.evaluations.createDatasetItem({
  path: { dataset_id },
  body: { input, /* + optional */ },
});

→ Request & response schema

soat.evaluations.createDatasetItemFromGeneration()​

Curate a dataset item from a generation

  • Method: POST
  • Path: /api/v1/datasets/{dataset_id}/items/from-generation
await soat.evaluations.createDatasetItemFromGeneration({
  path: { dataset_id },
  body: { generation_id, /* + optional */ },
});

→ Request & response schema

soat.evaluations.updateDatasetItem()​

Update a dataset item

  • Method: PUT
  • Path: /api/v1/datasets/{dataset_id}/items/{item_id}
await soat.evaluations.updateDatasetItem({
  path: { dataset_id, item_id },
  body: { /* + optional */ },
});

→ Request & response schema

soat.evaluations.deleteDatasetItem()​

Delete a dataset item

  • Method: DELETE
  • Path: /api/v1/datasets/{dataset_id}/items/{item_id}
await soat.evaluations.deleteDatasetItem({
  path: { dataset_id, item_id },
});

→ Request & response schema

soat.evaluations.listEvals()​

List evals

  • Method: GET
  • Path: /api/v1/evals
await soat.evaluations.listEvals({
  query: { /* + optional */ },
});

→ Request & response schema

soat.evaluations.createEval()​

Create an eval

  • Method: POST
  • Path: /api/v1/evals
await soat.evaluations.createEval({
  body: { name, agent_id, dataset_id, scorers, /* + optional */ },
});

→ Request & response schema

soat.evaluations.getEval()​

Get an eval

  • Method: GET
  • Path: /api/v1/evals/{eval_id}
await soat.evaluations.getEval({
  path: { eval_id },
});

→ Request & response schema

soat.evaluations.updateEval()​

Update an eval

  • Method: PUT
  • Path: /api/v1/evals/{eval_id}
await soat.evaluations.updateEval({
  path: { eval_id },
  body: { /* + optional */ },
});

→ Request & response schema

soat.evaluations.deleteEval()​

Delete an eval

  • Method: DELETE
  • Path: /api/v1/evals/{eval_id}
await soat.evaluations.deleteEval({
  path: { eval_id },
});

→ Request & response schema

soat.evaluations.listEvalRuns()​

List eval runs

  • Method: GET
  • Path: /api/v1/evals/{eval_id}/runs
await soat.evaluations.listEvalRuns({
  path: { eval_id },
  query: { /* + optional */ },
});

→ Request & response schema

soat.evaluations.startEvalRun()​

Start an eval run

  • Method: POST
  • Path: /api/v1/evals/{eval_id}/runs
await soat.evaluations.startEvalRun({
  path: { eval_id },
  body: { /* + optional */ },
});

→ Request & response schema

soat.evaluations.getEvalRun()​

Get an eval run

  • Method: GET
  • Path: /api/v1/evals/{eval_id}/runs/{eval_run_id}
await soat.evaluations.getEvalRun({
  path: { eval_id, eval_run_id },
});

→ Request & response schema

soat.evaluations.listEvalResults()​

List eval run results

  • Method: GET
  • Path: /api/v1/evals/{eval_id}/runs/{eval_run_id}/results
await soat.evaluations.listEvalResults({
  path: { eval_id, eval_run_id },
  query: { /* + optional */ },
});

→ Request & response schema

soat.evaluations.cancelEvalRun()​

Cancel an eval run

  • Method: POST
  • Path: /api/v1/evals/{eval_id}/runs/{eval_run_id}/cancel
await soat.evaluations.cancelEvalRun({
  path: { eval_id, eval_run_id },
});

→ Request & response schema