Skip to main content
A workflow defines behavior. An evaluation defines whether that behavior is correct for a dataset. Models, prompts, and steps change; the evaluation is the check that tells you whether the output still matches the expected result. A workflow that works on one document is not done until it passes against a representative dataset. Three pieces work together: datasets, evaluators, and experiments. Used end to end, they turn quality from a spot check into a repeatable score. See Eval-first development.

​
Datasets

A dataset is a set of examples. Each example is an input (the arguments and files the workflow receives) plus, optionally, the expected output (the ground-truth result). Datasets are portable folder archives, so you can build them locally and push them, or pull and edit them.
This is the automation dataset archive format for both workflows and agents. See Build a dataset for the full walkthrough.

​
Evaluators

An evaluator scores a run’s actual output against the example’s expected output and returns a pass or a number. Define them in evaluators.yaml. Three types ship in the box:
  • exact-diff does a JSON deep-diff against the expected output. Use it when the output should match exactly. Author comparison with a per-field rules map: each key selects a path ($ for the full output), {} uses inherited platform defaults, and options such as numericTolerance, order: unordered, items: at-least | at-most | exactly, and allowExtraFields apply at that path and inherit down the tree unless a descendant overrides them. Unordered arrays of objects may also set matchBy on that array path; it requires order: unordered on the same rule and is not inherited. Identity type and value must both match (numeric 10 differs from string '10'). Array order / items / matchBy belong on lineItems, not on the item path lineItems[]. Ordered arrays keep historical prefix matching: items: at-least requires expected to match actual from index 0 (trailing actual extras only). Set ignore: true on a descendant rule to exclude volatile evidence, citations, timestamps, or other captured golden fields from both sides of the comparison.
  • llm-judge uses an LLM to score the output against a rubric, with a pass threshold. Use it when “correct” is fuzzy (summaries, free text).
  • custom-script runs a typed TypeScript scoring function in the sandbox. Use it when correctness is a calculation you can write. The function receives expected and actual envelopes (expected.data / actual.data; actual may also include files) and includes a required : number return annotation, matching transform.script.

​
Experiments

An experiment runs a workflow version across the whole dataset and collects the scores in one place. Run two experiments on two versions and compare them to see what a change improved or broke.
Some CLI output and older command help call this identifier a batchId. The REST API and SDKs call the same value an experimentId. See Evaluate a workflow for the end-to-end flow. When production review uncovers a new miss, use Review production runs to correct it and promote it into the dataset.