Skip to main content
Once you have a dataset, you can score a workflow across every example and compare versions. This is how you tell whether a change made the workflow better or worse.

​
1. Define evaluators

Evaluators score actual output against each example’s expected output. Put them in evaluators.yaml:
  • exact-diff for output that should match exactly.
  • llm-judge for fuzzy correctness using explicit evaluation criteria.
  • custom-script for correctness you can compute in typed TypeScript.
passThreshold is workflow-level: the weighted mean across evaluators must reach it for the run to pass.

​
Per-field exact-diff rules

Under config.rules, each key is a path into the workflow output. $ targets the full output; {} selects a path with inherited platform defaults. An ancestor rule covers its subtree, and more specific descendants override inherited settings.
  • order: unordered on an array path compares items by value, not position. Matches always use distinct item slots; items independently controls whether missing expected or additional actual items are allowed.
  • matchBy on an unordered array of objects pairs items by exact identity (sku or [country, sku]) before comparing the paired objects. It requires order: unordered on the same rule and is not inherited by nested arrays. A matching identity with a wrong field reports that field path. Identity type and value must both match: numeric 10 differs from string '10'; there is no coercion or numeric tolerance. Omit it to keep structural matching, including legitimate duplicate objects. With items: at-least, extra actual items may omit or duplicate identities; with at-most, missing expected identities are allowed but every actual item being checked needs a unique valid identity; exactly validates both sides.
  • items: at-least requires every expected item and allows additional actual items. items: at-most allows missing expected items but rejects unexpected actual items. items: exactly permits neither. The default is at-least.
  • order is independent. Ordered arrays use positional prefix matching: at-least requires expected to match actual from index 0 (only trailing actual extras allowed); at-most requires actual to match expected from index 0 (only trailing expected extras allowed); exactly is equal-length positional comparison. unordered ignores positions. Set order, items, and matchBy on the array path (lineItems), not the item path (lineItems[]).
  • allowExtraFields: false on an object path (often someArray[]) rejects extra object keys in actual output.
  • numericTolerance sets an absolute tolerance for numeric fields at that path. When omitted, comparison keeps the historical relative epsilon.
  • ignore: true removes that path and its descendants from both expected and actual output. Use it under a selected parent to discard volatile evidence, citations, timestamps, or other fields captured in golden output. It cannot be combined with other settings on the same rule. If every rule is ignored, exact-diff compares the full output minus those paths.
Use lineItems for the array, lineItems[] for each element, and lineItems[].unitPrice for a field inside every element. Extract steps may add _grounding metadata; exact-diff strips it automatically before comparison. Configs authored before per-field rules continue to run unchanged. Do not mix legacy top-level options with rules in the same evaluator. Validate and push:

​
2. Run an experiment

An experiment runs the workflow across the whole dataset and collects every score in one batch:
--wait blocks until the experiment finishes and exits non-zero if any example failed, so it works in CI. Some CLI help still calls the experiment id a batchId; it is the same identifier.

​
3. Compare versions

Change the workflow, push a new version, run a second experiment, then diff the two experiments:
The comparison shows which examples improved, regressed, or stayed the same.

​
Tips

  • llm-judge scores have variance. Average across runs or raise passThreshold deliberately rather than chasing single-run noise.
  • Keep expected outputs in the dataset so exact-diff and llm-judge have ground truth to compare against.