Experiments

A dataset is a stored set of tickets. A simulation judges that dataset with a pack’s published version. The API is available. The Experiments item in the app is disabled and labeled on the roadmap.

Create datasets and start simulations over the API. The screen at /experiments does not open.

Datasets

POST /api/datasets with { "name": "May refunds" } requires packs:write and returns { id, name }. GET /api/datasets requires packs:read and returns { items: [{ id, name, createdAt, ticketCount }] }. Add tickets with POST /api/datasets/{datasetId}/tickets. Tickets are redacted before they are stored. A dataset holds at most 250 tickets. Going past the cap is 400 with dataset is capped at 250 tickets.

JSON
CSV

curl https://judged.systems/api/datasets/DATASET_ID/tickets \
  -X POST \
  -H "x-api-key: $JUDGED_API_KEY" \
  -H "content-type: application/json" \
  -d '{
    "tickets": [
      {
        "subject": "Charged twice for May",
        "customerMessage": "I was billed twice and want one refunded."
      }
    ]
  }'

The body is text/csv. The header row must include subject and customerMessage. externalId is optional.

curl https://judged.systems/api/datasets/DATASET_ID/tickets \
  -X POST \
  -H "x-api-key: $JUDGED_API_KEY" \
  -H "content-type: text/csv" \
  --data-binary $'subject,customerMessage,externalId\nCharged twice,I was billed twice,48213\n'

A header without those columns is 400 with csv header must include subject,customerMessage.

201 returns { "inserted": 1, "total": 1 }.

Simulations

POST /api/simulations requires simulations:run:

{ "datasetId": "DATASET_ID", "packId": "PACK_ID" }

202 returns { "id": "..." } before the run finishes. The pack must be published, or the response is 409. GET /api/simulations/{id} returns the run:

Field	Meaning
`status`	`queued`, `running`, `complete`, or `failed`
`error`	Set when `status` is `failed`, otherwise null
`total` / `done`	Tickets in the dataset, and how many have a stored evaluation
`reviewRate`	Share of finished rows whose status is `review`
`avgLatencyMs`	Mean latency of finished rows
`totalTokens`	Input plus output tokens recorded on those rows
`evaluations`	`{ id, status, ticketHash }` for each stored row

GET /api/simulations lists runs with dataset and pack names. Each evaluation is also a normal evaluation with source simulation.

Get started

Packs

Connect

Operate

Reference

Datasets

Simulations

​Datasets

​Simulations

Datasets

Simulations