Define typed agent evaluations

Add evaluation logic to an agent with .defineEvaluator(name, evaluator), then finish with one .defineEvalPass(...). The output schema supplies callback types and autocomplete. Keep datasets and their documents in platform storage.

Register scores and a pass rule

agent.ts

import { exactDiff, llmJudge, weightedScore } from '@eigenpal/automations';
import reviewer from './base.ts';

const scored = reviewer
  .defineEvaluator(
    'fields',
    exactDiff({
      select: (output) => ({ total: output.total }),
      options: { rules: { total: { numericTolerance: 0.01 } } },
    })
  )
  .defineEvaluator(
    'meaning',
    llmJudge({
      prompt: 'Score whether the actual summary preserves the expected invoice facts.',
      select: ({ actual, expected }) => ({
        actual: actual.summary,
        expected: expected?.summary,
      }),
    })
  )
  .defineEvaluator('nonnegative', ({ actual }) => Number(actual.total >= 0));

export const reviewed = scored.defineEvalPass(
  weightedScore({
    weights: { fields: 2, meaning: 1, nonnegative: 1 },
    threshold: 0.9,
  })
);

export default reviewed;
export const filesystemReviewer = reviewed.withFilesystem({ cwd: null });

Here, base.ts is the invoice agent from the quickstart. The filesystem variant reuses the completed evaluation logic. Every chain returns a new definition; the original stays unchanged. Use literal, unique evaluator names. Names start with a letter and contain letters, digits, _ or -, up to 64 characters. Register at most 32 evaluators. TypeScript rejects duplicate names, unknown score names and misspelled output fields. Once you add an evaluator, finish the chain with one pass rule before compiling or releasing. After the pass, evaluator registration stays closed; .withFilesystem() remains available. Agents with no evaluators need no pass rule.

Custom score callbacks

A callback receives readonly { actual, expected } and returns a finite number from 0 to 1. actual follows the output schema. expected is optional and deeply partial because a dataset example may omit fields. Use optional chaining such as expected?.supplier?.taxId when your schema has nested expected fields. Raw JSON Schemas and absent output schemas infer unknown; use Zod for full inference. Conditional JSON Schema constraints such as if, not and dependent schemas are rejected for partial expectations instead of being weakened.

exactDiff

exactDiff({ select, options? }) applies the same synchronous selector to actual and expected output, then compares the projections. The selector receives a readonly partial output, so handle missing nested values on both calls. Return selected JSON data. Use an object to give comparison results readable labels. Optional chaining prevents a JavaScript exception; it does not make a missing expected field match. Missing, undefined or empty expected selections produce evaluation errors. Explicit null, false, zero and empty strings remain values to compare. options.rules maps selected projection paths to comparison rules:

ignore: true: ignore a path; do not combine with other rule settings.
order: 'ordered' | 'unordered': array ordering.
items: 'at-least' | 'at-most' | 'exactly': item-count policy.
matchBy: item identity path or path array; requires unordered comparison.
allowExtraFields: allow extra object fields.
numericTolerance: nonnegative absolute numeric difference.

Paths infer from the selected shape up to four levels deep. Use $ for the whole projection. Global paths, unorderedPaths, numericTolerance and allowExtraFields are also available; do not combine them with rules. Omitted options preserve the platform comparator’s defaults, including extra object fields and positional-prefix array matching. Use items: 'exactly' and allowExtraFields: false where extra values must fail.

llmJudge

llmJudge({ prompt, select, model? }) takes a prompt from 10 to 131072 characters. Its synchronous selector receives typed { actual, expected } and returns only the JSON data you want to disclose to the judge. Optional model selects a configured organization model; omission uses the configured selection. The platform calls the judge outside the callback, through its model connection. Selectors do not receive credentials. Expected dataset answers stay out of ordinary agent input. Defining or compiling the judge does not make a model request; running an evaluation does and uses evaluation billing.

Choose one pass decision

For a weighted mean, use weightedScore({ weights, threshold }). In the example, the result is (fields * 2 + meaning + nonnegative) / 4; the inclusive threshold is 0.9. Only listed scores contribute. Weights must be finite, nonnegative and have a positive finite total. The threshold must be between 0 and 1. For custom logic, replace the weighted pass above with a predicate:

export default scored.defineEvalPass(
  ({ fields, meaning, nonnegative }) => nonnegative === 1 && (fields === 1 || meaning >= 0.95)
);

The predicate sees only registered score names and must return a boolean. There is no implicit weighting for predicates. Studio can display weighted decisions as weights/thresholds; custom predicates display as source with the verdict and individual scores. Errors, timeouts, invalid scores and missing selected data prevent an overall pass rather than being skipped.

Run evaluations

Evaluations work in both agent modes. Ordinary inference does not invoke them. Use the existing dataset/experiment features in Studio, the API client evaluation methods, or the CLI to run experiments. You do not import datasets into TypeScript or commit documents to Git. Model/turn/time guards cover inference and tools. Evaluation callbacks have separate fixed execution limits. General conversational follow-up APIs and .asTool() are not part of this SDK.

Overview

Agent authoring

TypeScript API client

Python API client

Define typed agent evaluations

Register scores and a pass rule

Custom score callbacks

exactDiff

llmJudge

Choose one pass decision

Run evaluations

​Register scores and a pass rule

​Custom score callbacks

​exactDiff

​llmJudge

​Choose one pass decision

​Run evaluations

Register scores and a pass rule

Custom score callbacks

exactDiff

llmJudge

Choose one pass decision

Run evaluations