Ticket
A ticket is one support request. Every transport uses this shape. You map your own fields onto it. There is no connector for a specific helpdesk.
Strings are trimmed. Empty messages are dropped. Then the ticket is redacted.
An array of messages is one conversation. It is not a batch of tickets. One judge call is one ticket.
Pack
A pack is a name plus a document of questions. Each question is independent. The model answers all of them on the same ticket.
Question ids match
^[a-z][a-z0-9_]{0,63}$ and stay stable across versions. The model does not see the id as a name. It sees name and instructions.
How to write them: Questions. How answers become a result: Results.
Draft and published version
A draft can change. A published version cannot.
Publish copies the draft into a new version and makes that version live. Restore copies a published version back onto the draft. Compare lists question ids that were added, removed, or changed. See Packs.
Evaluation
An evaluation is the stored judgment. It keeps:- the pack id and the version id that ran
- the redacted ticket and a hash of that projection
- the answers, review reasons, model id, usage, and latency
- the source:
playground,api,mcp,webhook, orsimulation
status.
Label
A label is a human answer for one question on one evaluation. The evaluation response includesagreement for that question:
- Choice. The label option equals the selected option.
- Score. The label level index equals the model score rounded to the nearest level, clamped to the criteria.
- Boolean. The label is true when the model probability is at least
0.5.
Result
result.status is accept, review, or reject. Pack thresholds produce accept or review. A failed model call is stored as review with the error kind in reviewReasons. The current threshold rules do not emit reject.