Evals API

19 operations. Authentication, errors and pagination work as described in the REST API overview.

Compare two runs of one prompt, case by case

GET/api/v1/evals/compareToken optional

compareEvalRuns parameters
Name	Type	Description
`a` queryrequired	`string`
`b` queryrequired	`string`

Responses: 200 OK400, 404, 429

Response fields

compareEvalRuns response fields
Name	Type	Description
`a`required	`object`
`b`required	`object`
`rows`required	`object[]`
`truncated`required	`boolean`

Example

curl 'https://incantory.ai/api/v1/evals/compare' \
  -H "Authorization: Bearer $INCANTORY_TOKEN"

A prompt's eval datasets

GET/api/v1/evals/datasetsToken optional

Public datasets for everyone; every dataset for the prompt owner.

listEvalDatasets parameters
Name	Type	Description
`prompt` queryrequired	`string`	owner/slug

Responses: 200 OK304, 400, 404, 429

Response fields

listEvalDatasets response fields
Name	Type	Description
`items`required	`object[]`
`canManage`required	`boolean`

Example

curl 'https://incantory.ai/api/v1/evals/datasets' \
  -H "Authorization: Bearer $INCANTORY_TOKEN"

Create a dataset on your prompt

POST/api/v1/evals/datasetsToken required · scope evals:write

Request body

createEvalDataset body fields
Name	Type	Description
`prompt`required	`string`	owner/slug of the prompt the dataset belongs to
`name`required	`string`
`description`	`string`
`visibility`	`"public" \| "unlisted" \| "private"`	Default public

Responses: 201 OK400, 401, 403, 404, 409, 410, 429

Response fields

createEvalDataset response fields
Name	Type	Description
`id`required	`string`
`prompt`required	`string`
`name`required	`string`
`description`required	`string`
`visibility`required	`"public" \| "unlisted" \| "private"`
`caseCount`required	`number`
`createdAt`required	`string`
`updatedAt`required	`string`

Example

curl -X POST 'https://incantory.ai/api/v1/evals/datasets' \
  -H "Authorization: Bearer $INCANTORY_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"prompt":"alice/haiku","name":"name"}'

Get a dataset

GET/api/v1/evals/datasets/{id}Token optional

getEvalDataset parameters
Name	Type	Description
`id` pathrequired	`string`

Responses: 200 OK400, 404, 429

Response fields

getEvalDataset response fields
Name	Type	Description
`id`required	`string`
`prompt`required	`string`
`name`required	`string`
`description`required	`string`
`visibility`required	`"public" \| "unlisted" \| "private"`
`caseCount`required	`number`
`createdAt`required	`string`
`updatedAt`required	`string`
`canManage`required	`boolean`

Example

curl 'https://incantory.ai/api/v1/evals/datasets/{id}' \
  -H "Authorization: Bearer $INCANTORY_TOKEN"

Rename / describe / change visibility

PATCH/api/v1/evals/datasets/{id}Token required · scope evals:write

updateEvalDataset parameters
Name	Type	Description
`id` pathrequired	`string`

Request body

updateEvalDataset body fields
Name	Type	Description
`name`	`string`
`description`	`string`
`visibility`	`"public" \| "unlisted" \| "private"`

Responses: 200 OK400, 401, 403, 404, 409, 429

Response fields

updateEvalDataset response fields
Name	Type	Description
`id`required	`string`
`prompt`required	`string`
`name`required	`string`
`description`required	`string`
`visibility`required	`"public" \| "unlisted" \| "private"`
`caseCount`required	`number`
`createdAt`required	`string`
`updatedAt`required	`string`

Example

curl -X PATCH 'https://incantory.ai/api/v1/evals/datasets/{id}' \
  -H "Authorization: Bearer $INCANTORY_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{}'

Delete a dataset (soft)

DELETE/api/v1/evals/datasets/{id}Token required · scope evals:write

deleteEvalDataset parameters
Name	Type	Description
`id` pathrequired	`string`

Responses: 204 OK400, 401, 403, 404, 429

Example

curl -X DELETE 'https://incantory.ai/api/v1/evals/datasets/{id}' \
  -H "Authorization: Bearer $INCANTORY_TOKEN"

List a dataset's cases

GET/api/v1/evals/datasets/{id}/casesToken optional

listEvalCases parameters
Name	Type	Description
`id` pathrequired	`string`
`cursor` query	`string`
`limit` query	`integer (1–500)`

Responses: 200 OK304, 400, 404, 429

Response fields

listEvalCases response fields
Name	Type	Description
`items`required	`object[]`
`nextCursor`required	`string \| null`

Example

curl 'https://incantory.ai/api/v1/evals/datasets/{id}/cases' \
  -H "Authorization: Bearer $INCANTORY_TOKEN"

Add cases (batch)

POST/api/v1/evals/datasets/{id}/casesToken required · scope evals:write

replace: true deletes the existing cases first. A dataset holds at most 10,000 cases.

addEvalCases parameters
Name	Type	Description
`id` pathrequired	`string`

Request body

addEvalCases body fields
Name	Type	Description
`cases`required	`object[]`
`replace`	`boolean`	Delete the existing cases first

Responses: 201 OK400, 401, 403, 404, 413, 422, 429

Response fields

addEvalCases response fields
Name	Type	Description
`added`required	`number`
`caseCount`required	`number`

Example

curl -X POST 'https://incantory.ai/api/v1/evals/datasets/{id}/cases' \
  -H "Authorization: Bearer $INCANTORY_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"cases":[{"input":{}}]}'

Edit a case

PATCH/api/v1/evals/datasets/{id}/cases/{caseId}Token required · scope evals:write

updateEvalCase parameters
Name	Type	Description
`id` pathrequired	`string`
`caseId` pathrequired	`string`

Request body

updateEvalCase body fields
Name	Type	Description
`input`	`object`
`expected`	`any`
`tags`	`string[]`

Responses: 200 OK400, 401, 403, 404, 429

Response fields

updateEvalCase response fields
Name	Type	Description
`id`required	`string`
`input`required	`object`
`expected`required	`any`
`tags`required	`string[]`
`position`required	`number`

Example

curl -X PATCH 'https://incantory.ai/api/v1/evals/datasets/{id}/cases/{caseId}' \
  -H "Authorization: Bearer $INCANTORY_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{}'

Delete a case

DELETE/api/v1/evals/datasets/{id}/cases/{caseId}Token required · scope evals:write

deleteEvalCase parameters
Name	Type	Description
`id` pathrequired	`string`
`caseId` pathrequired	`string`

Responses: 204 OK400, 401, 403, 404, 429

Example

curl -X DELETE 'https://incantory.ai/api/v1/evals/datasets/{id}/cases/{caseId}' \
  -H "Authorization: Bearer $INCANTORY_TOKEN"

Import cases from CSV or JSONL

POST/api/v1/evals/datasets/{id}/importToken required · scope evals:write

CSV: header row of variable names plus optional expected and tags (;-separated) columns. JSONL: {input, expected?, tags?} or a flat object of variables per line. 422 invalid_import names the line.

importEvalCases parameters
Name	Type	Description
`id` pathrequired	`string`

Request body

importEvalCases body fields
Name	Type	Description
`format`required	`"csv" \| "jsonl"`
`text`required	`string`
`replace`	`boolean`

Responses: 201 OK400, 401, 403, 404, 413, 422, 429

Response fields

importEvalCases response fields
Name	Type	Description
`added`required	`number`
`caseCount`required	`number`

Example

curl -X POST 'https://incantory.ai/api/v1/evals/datasets/{id}/import' \
  -H "Authorization: Bearer $INCANTORY_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"format":"csv","text":"text"}'

Runs reported for a prompt (optionally one version or dataset)

GET/api/v1/evals/runsToken optional

listEvalRuns parameters
Name	Type	Description
`prompt` queryrequired	`string`	owner/slug
`version` query	`integer (…–9007199254740991)`
`datasetId` query	`string`
`cursor` query	`string`
`limit` query	`integer (1–100)`

Responses: 200 OK304, 400, 404, 429

Response fields

listEvalRuns response fields
Name	Type	Description
`items`required	`object[]`
`nextCursor`required	`string \| null`

Example

curl 'https://incantory.ai/api/v1/evals/runs' \
  -H "Authorization: Bearer $INCANTORY_TOKEN"

Create a run for a prompt version

POST/api/v1/evals/runsToken required · scope evals:write

Only the prompt owner (or an admin) may report runs: a passing run opens requirePassingEval labels. The version defaults to the label, else the latest. model is a catalog slug (unknown names are kept as free text). Errors use prompt_not_found / version_not_found / dataset_not_found.

Request body

createEvalRun body fields
Name	Type	Description
`prompt`required	`string`	owner/slug
`version`	`integer (…–9007199254740991)`	Version number; default: the label, else the latest version
`label`	`string`
`model`	`string`	Model slug from GET /api/v1/models; unknown names are kept as free text
`datasetId`	`string`
`metadata`	`object`
`runner`	`"ci" \| "sdk" \| "manual"`	Default: ci for API tokens, manual for the web UI
`passThreshold`	`number (0–1)`	Pass rate the run needs to pass (default 1.0)
`commitSha`	`string`
`externalUrl`	`string (uri)`	Link to the CI job

Responses: 201 OK400, 401, 403, 404, 410, 429

Response fields

createEvalRun response fields
Name	Type	Description
`id`required	`string`
`prompt`required	`string`
`version`required	`number`
`dataset`required	`object \| null`
`model`required	`object \| null`
`modelText`required	`string \| null`
`runner`required	`"ci" \| "sdk" \| "manual"`
`status`required	`"running" \| "completed" \| "failed"`
`passed`required	`boolean \| null`
`summary`required	`object`
`startedAt`required	`string`
`finishedAt`required	`string \| null`
`commitSha`required	`string \| null`
`externalUrl`required	`string \| null`
`createdBy`required	`string \| null`

Example

curl -X POST 'https://incantory.ai/api/v1/evals/runs' \
  -H "Authorization: Bearer $INCANTORY_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"prompt":"alice/haiku"}'

Get a run (status, verdict, summary)

GET/api/v1/evals/runs/{id}Token optional

getEvalRun parameters
Name	Type	Description
`id` pathrequired	`string`

Responses: 200 OK400, 404, 429

Response fields

getEvalRun response fields
Name	Type	Description
`id`required	`string`
`prompt`required	`string`
`version`required	`number`
`dataset`required	`object \| null`
`model`required	`object \| null`
`modelText`required	`string \| null`
`runner`required	`"ci" \| "sdk" \| "manual"`
`status`required	`"running" \| "completed" \| "failed"`
`passed`required	`boolean \| null`
`summary`required	`object`
`startedAt`required	`string`
`finishedAt`required	`string \| null`
`commitSha`required	`string \| null`
`externalUrl`required	`string \| null`
`createdBy`required	`string \| null`
`resultCount`required	`number`
`canManage`required	`boolean`

Example

curl 'https://incantory.ai/api/v1/evals/runs/{id}' \
  -H "Authorization: Bearer $INCANTORY_TOKEN"

Finish a run, pin or unpin its verdict

PATCH/api/v1/evals/runs/{id}Token required · scope evals:write

passed pins the verdict (audited); null unpins it. status: "failed" marks a crashed run (never passes).

updateEvalRun parameters
Name	Type	Description
`id` pathrequired	`string`

Request body

updateEvalRun body fields
Name	Type	Description
`status`	`"running" \| "completed" \| "failed"`
`passed`	`boolean \| null`	Pin the verdict (null clears the pin; the server recomputes)
`passThreshold`	`number (0–1)`
`externalUrl`	`string (uri) \| null`

Responses: 200 OK400, 401, 403, 404, 429

Response fields

updateEvalRun response fields
Name	Type	Description
`id`required	`string`
`prompt`required	`string`
`version`required	`number`
`dataset`required	`object \| null`
`model`required	`object \| null`
`modelText`required	`string \| null`
`runner`required	`"ci" \| "sdk" \| "manual"`
`status`required	`"running" \| "completed" \| "failed"`
`passed`required	`boolean \| null`
`summary`required	`object`
`startedAt`required	`string`
`finishedAt`required	`string \| null`
`commitSha`required	`string \| null`
`externalUrl`required	`string \| null`
`createdBy`required	`string \| null`

Example

curl -X PATCH 'https://incantory.ai/api/v1/evals/runs/{id}' \
  -H "Authorization: Bearer $INCANTORY_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{}'

Delete a run (audited)

DELETE/api/v1/evals/runs/{id}Token required · scope evals:write

deleteEvalRun parameters
Name	Type	Description
`id` pathrequired	`string`

Responses: 204 OK400, 401, 403, 404, 429

Example

curl -X DELETE 'https://incantory.ai/api/v1/evals/runs/{id}' \
  -H "Authorization: Bearer $INCANTORY_TOKEN"

A run's results in report order

GET/api/v1/evals/runs/{id}/resultsToken optional

listEvalResults parameters
Name	Type	Description
`id` pathrequired	`string`
`cursor` query	`string`
`limit` query	`integer (1–500)`

Responses: 200 OK304, 400, 404, 429

Response fields

listEvalResults response fields
Name	Type	Description
`items`required	`object[]`
`nextCursor`required	`string \| null`

Example

curl 'https://incantory.ai/api/v1/evals/runs/{id}/results' \
  -H "Authorization: Bearer $INCANTORY_TOKEN"

Append results (batch, ≤1000)

POST/api/v1/evals/runs/{id}/resultsToken required · scope evals:write

Each batch recomputes the run summary and verdict from all its results (verdict: passRate over results with a boolean passed ≥ the run's passThreshold, default 1.0) and marks the run completed. A pass on a catalog model adds a "verified on" row.

appendEvalResults parameters
Name	Type	Description
`id` pathrequired	`string`

Request body

appendEvalResults body fields
Name	Type	Description
`results`required	`object[]`

Responses: 200 OK400, 401, 403, 404, 413, 422, 429

Response fields

appendEvalResults response fields
Name	Type	Description
`accepted`required	`number`
`run`required	`object`

Example

curl -X POST 'https://incantory.ai/api/v1/evals/runs/{id}/results' \
  -H "Authorization: Bearer $INCANTORY_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"results":[{}]}'

Per-version score summary (latest run per model × dataset)

GET/api/v1/evals/scoresToken optional

getEvalScores parameters
Name	Type	Description
`prompt` queryrequired	`string`	owner/slug

Responses: 200 OK400, 404, 429

Response fields

getEvalScores response fields
Name	Type	Description
`prompt`required	`string`
`versions`required	`object[]`

Example

curl 'https://incantory.ai/api/v1/evals/scores' \
  -H "Authorization: Bearer $INCANTORY_TOKEN"