> ## Documentation Index
> Fetch the complete documentation index at: https://docs.eigenpal.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Define typed agent evaluations

> Score EigenPal agent outputs with typed callbacks, exact comparisons and LLM judges, then choose a weighted or custom pass rule while datasets stay in storage.

Add evaluation logic to an agent with `.defineEvaluator(name, evaluator)`, then
finish with one `.defineEvalPass(...)`. The output schema supplies callback types
and autocomplete. Keep datasets and their documents in platform storage.

## Register scores and a pass rule

```ts agent.ts theme={null}
import { exactDiff, llmJudge, weightedScore } from '@eigenpal/automations';
import reviewer from './base.ts';

const scored = reviewer
  .defineEvaluator(
    'fields',
    exactDiff({
      select: (output) => ({ total: output.total }),
      options: { rules: { total: { numericTolerance: 0.01 } } },
    })
  )
  .defineEvaluator(
    'meaning',
    llmJudge({
      prompt: 'Score whether the actual summary preserves the expected invoice facts.',
      select: ({ actual, expected }) => ({
        actual: actual.summary,
        expected: expected?.summary,
      }),
    })
  )
  .defineEvaluator('nonnegative', ({ actual }) => Number(actual.total >= 0));

export const reviewed = scored.defineEvalPass(
  weightedScore({
    weights: { fields: 2, meaning: 1, nonnegative: 1 },
    threshold: 0.9,
  })
);

export default reviewed;
export const filesystemReviewer = reviewed.withFilesystem({ cwd: null });
```

Here, `base.ts` is the invoice agent from the [quickstart](/sdks/automations).
The filesystem variant reuses the completed evaluation logic. Every chain returns
a new definition; the original stays unchanged.

Use literal, unique evaluator names. Names start with a letter and contain letters,
digits, `_` or `-`, up to 64 characters. Register at most 32 evaluators.
TypeScript rejects duplicate names, unknown score names and misspelled output fields.

Once you add an evaluator, finish the chain with one pass rule before compiling
or releasing. After the pass, evaluator registration stays closed;
`.withFilesystem()` remains available. Agents with no evaluators need no pass rule.

## Custom score callbacks

A callback receives readonly `{ actual, expected }` and returns a finite number
from 0 to 1. `actual` follows the output schema. `expected` is optional and deeply
partial because a dataset example may omit fields. Use optional chaining such as
`expected?.supplier?.taxId` when your schema has nested expected fields.

Raw JSON Schemas and absent output schemas infer `unknown`; use Zod for full
inference. Conditional JSON Schema constraints such as `if`, `not` and dependent
schemas are rejected for partial expectations instead of being weakened.

## exactDiff

`exactDiff({ select, options? })` applies the same synchronous selector to actual
and expected output, then compares the projections. The selector receives a
readonly partial output, so handle missing nested values on both calls.

Return selected JSON data. Use an object to give comparison results readable
labels. Optional chaining prevents a JavaScript exception; it does not make a
missing expected field match. Missing, undefined or empty expected selections
produce evaluation errors. Explicit `null`, `false`, zero and empty strings
remain values to compare.

`options.rules` maps selected projection paths to comparison rules:

* `ignore: true`: ignore a path; do not combine with other rule settings.
* `order: 'ordered' | 'unordered'`: array ordering.
* `items: 'at-least' | 'at-most' | 'exactly'`: item-count policy.
* `matchBy`: item identity path or path array; requires unordered comparison.
* `allowExtraFields`: allow extra object fields.
* `numericTolerance`: nonnegative absolute numeric difference.

Paths infer from the selected shape up to four levels deep. Use `$` for the whole
projection. Global `paths`, `unorderedPaths`, `numericTolerance` and
`allowExtraFields` are also available; do not combine them with `rules`.

Omitted options preserve the platform comparator's defaults, including extra
object fields and positional-prefix array matching. Use `items: 'exactly'` and
`allowExtraFields: false` where extra values must fail.

## llmJudge

`llmJudge({ prompt, select, model? })` takes a prompt from 10 to 131072 characters.
Its synchronous selector receives typed `{ actual, expected }` and returns only
the JSON data you want to disclose to the judge. Optional `model` selects a
configured organization model; omission uses the configured selection.

The platform calls the judge outside the callback, through its model connection.
Selectors do not receive credentials. Expected dataset answers stay out of ordinary
agent input. Defining or compiling the judge does not make a model request;
running an evaluation does and uses evaluation billing.

## Choose one pass decision

For a weighted mean, use `weightedScore({ weights, threshold })`. In the example,
the result is `(fields * 2 + meaning + nonnegative) / 4`; the inclusive threshold
is `0.9`. Only listed scores contribute. Weights must be finite, nonnegative and
have a positive finite total. The threshold must be between 0 and 1.

For custom logic, replace the weighted pass above with a predicate:

```ts theme={null}
export default scored.defineEvalPass(
  ({ fields, meaning, nonnegative }) => nonnegative === 1 && (fields === 1 || meaning >= 0.95)
);
```

The predicate sees only registered score names and must return a boolean. There
is no implicit weighting for predicates. Studio can display weighted decisions
as weights/thresholds; custom predicates display as source with the verdict and
individual scores. Errors, timeouts, invalid scores and missing selected data
prevent an overall pass rather than being skipped.

## Run evaluations

Evaluations work in both agent modes. Ordinary inference does not invoke them.
Use the existing dataset/experiment features in Studio, the
[API client evaluation methods](/sdks/typescript/evaluation), or the CLI to run
experiments. You do not import datasets into TypeScript or commit documents to Git.

Model/turn/time guards cover inference and tools. Evaluation callbacks have
[separate fixed execution limits](/sdks/automations/callbacks#execution-limits).
General conversational follow-up APIs and `.asTool()` are not part of this SDK.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.