$ gauge run
✓ evals/router.eval.md › password-reset (1293ms)
  ✓ contains "auth"
  ✓ llm-rate "routes without asking a clarifying question" >= 0.8
✓ evals/router.eval.md › refund (1170ms)
  ✓ equals "billing"
✗ evals/summarize.eval.md (980ms)
  ⚠ regression — score dropped 0.91 → 0.62
  ✗ llm-judge "faithful to the source" — judge failed: invents a statistic

2 passed, 1 failed, 3 total

Why gauge

Quickstart

From zero to a passing eval in three steps.

1. Set your API key

gauge reads keys from the environment (or a .env file in the working directory).

# OpenAI, Anthropic, or both
export OPENAI_API_KEY=sk-...
export ANTHROPIC_API_KEY=sk-ant-...

2. Write an eval

Create evals/router.eval.md. Frontmatter is config; the body is the prompt.

---
provider: openai/gpt-4o-mini
# scenario = the setup (BDD "Given"): the agent + shared context
scenario:
  system: |
    You are a support router. Reply with one lowercase category slug
    (auth, billing, technical, other). No punctuation.
  vars:
    input: "I forgot my password"
assert:
  - contains: "auth"
  - llm-judge: "Routes correctly without asking a clarifying question"
---

User request: {{input}}

3. Run it

$ npx gauge-eval run

✓ evals/router.eval.md (1293ms)
  ✓ contains "auth"
  ✓ llm-judge "Routes correctly without asking a clarifying question"

1 passed, 0 failed, 1 total

Eval format

One file is one eval: a prompt, the variables it needs, and the assertions its output must satisfy.

Two shapes are supported. Markdown with YAML frontmatter — best when the prompt is real prose:

---
provider: anthropic/claude-opus-4-8
# scenario is optional — it sets up the world before the prompt runs
scenario:
  system: You are a patient science teacher.
vars:
  topic: photosynthesis
assert:
  - contains: "sunlight"
---

Explain {{topic}} to a 10-year-old in two sentences.

The optional scenario block is the setup step — the agent system prompt and shared context. See Scenarios for the full picture; skip it for a bare prompt.

…or a single YAML document — handy for matrix-heavy, prose-light cases (the prompt lives in a prompt: field):

provider: openai/gpt-4o
prompt: "Classify sentiment (pos/neg): {{text}}"
assert:
  - regex: "/^(pos|neg)$/"

Variables

{{name}} placeholders in the prompt are filled from vars. A missing variable is an error, not a silent blank — so typos surface immediately.

Scenarios

If you like BDD, the pieces line up: a scenario is the Given, the prompt body is the When, and assert is the Then. The scenario sets up the world before the prompt runs.

system becomes the model's system / agent instruction; vars form the base layer that spec- and case-level vars merge over. Both are rendered with the case's vars, so a scenario can interpolate {{...}} too.

---
provider: anthropic/claude-opus-4-8
scenario:
  system: You are a terse support router. Queues: {{queues}}.
  vars:
    queues: "auth, billing, other"
assert:
  - regex: "/^(auth|billing|other)$/"
---

Classify: {{input}}

Var precedence

Last wins: scenario.vars → spec vars → case vars. A scenario sets the shared defaults; individual cases override what they need.

How system reaches the model

It maps to each provider's native system role — an OpenAI/Azure system message, Anthropic's top-level system, Gemini's systemInstruction, Ollama's system. The exec provider receives it as the GAUGE_SYSTEM environment variable, so stdin stays the bare prompt.

Providers

Where the prompt runs. Use the vendor/model shorthand, or the object form for providers that need more.

The exec provider — test any harness, any language

This is the escape hatch that makes gauge work for tooling built on top of harnesses. The rendered prompt is written to the command's stdin; its stdout becomes the output. Your harness stays in whatever language it's written in.

---
provider:
  type: exec
  command: "python my_agent.py --tool-router"
assert:
  - llm-judge: "Selected the search tool, not the calculator"
---

Find me flights to Lisbon under $600.

Any OpenAI-compatible host

Most vendors speak the OpenAI chat API. Point the openai-compat object form at any base URL and name the env var holding the key — DeepSeek, Groq, Together, Fireworks, and more, without waiting for a built-in shorthand.

provider:
  type: openai-compat
  baseUrl: https://api.deepseek.com/v1
  model: deepseek-chat
  apiKeyEnv: DEEPSEEK_API_KEY

Assertions

Each assertion is a one-key object. A case passes when every assertion passes.

Deterministic checks

assert:
  - equals: "billing"
  - contains: "refund"
  - regex: "/^(auth|billing|other)$/"   # JS has no (?i) — use /.../i

LLM-as-judge

When correctness is fuzzy, let a model grade it. llm-judge is a binary gate; llm-rate returns a graded 0–1 score with a threshold — useful for tracking drift over time.

assert:
  - llm-judge: "Faithful to the source; invents no facts"
  - llm-rate:
      rubric: "Clear enough for a 10-year-old"
      min: 0.8

Structured output & similarity

For agents that emit JSON — tool calls, structured plans — validate the shape with json-schema. For fuzzy text targets, similarity scores embedding cosine against a reference.

assert:
  - json-schema:
      type: object
      required: [tool, args]
      properties:
        tool: { enum: [web_search, calculator] }
  - similarity:
      reference: "The capital of France is Paris."
      min: 0.8

Matrix cases

One prompt, many inputs. Spec-level vars and assert are shared; each case merges its own vars over them and appends its own assertions.

---
provider: openai/gpt-4o-mini
assert:
  - regex: "/^(auth|billing|technical|other)$/"   # every case
cases:
  - name: password-reset
    vars: { input: "I forgot my password" }
    assert: [{ equals: "auth" }]
  - name: refund
    vars: { input: "I want a refund" }
    assert: [{ equals: "billing" }]
---

Classify the request into one category slug: {{input}}

✓ evals/router.eval.md › password-reset (1293ms)
  ✓ regex /^(auth|billing|technical|other)$/
  ✓ equals "auth"
✓ evals/router.eval.md › refund (1170ms)
  ✓ equals "billing"

2 passed, 0 failed, 2 total

CLI & reporters

A few commands, a handful of flags, exit codes that mean something.

$ gauge init                    # scaffold gauge.config.yaml + example eval
$ gauge run [paths...] [-r tty|json|junit|html] [-f <substr>] [-u] [-c <n>] [--cache]
$ gauge watch [paths...]        # re-run on change
$ gauge report                  # reprint last run — with cost + baseline diffs

Reporters

Regression baselines

The feature that turns a test runner into an eval framework: catch the day your prompt quietly gets worse.

$ gauge run -u        # snapshot this run as the baseline
$ gauge run           # later runs compare against it

A case regresses — and fails the run — when:

• it was passing in the baseline and now fails,
• its score drops by more than 0.05, or
• (for assertion-free cases) its output changes.

⚠ 1 regression(s) vs baseline.

✗ evals/summarize.eval.md (980ms)
  ⚠ regression — score dropped 0.91 → 0.62

0 passed, 1 failed, 1 total   exit 1

Every run is also written to .gauge/last-run.json for gauge report. Add .gauge/ to your .gitignore — or commit baseline.json if you want the baseline versioned with the code.

Config & caching

Optional gauge.config.yaml in the working directory sets defaults. CLI flags always win.

paths: [evals]
reporter: tty           # tty | json | junit | html
judge: openai/gpt-4o-mini
filter: ""
concurrency: 5
cache: false

Response caching

With --cache (or cache: true), identical (model, prompt) calls are served from .gauge/cache. Two wins: re-runs are near-instant, and runs become deterministic — which pairs well with baselines.

$ time gauge run --cache   # first run — live API
... 2.96s
$ time gauge run --cache   # second run — cache hit
... 0.07s

Custom providers & scorers

Use gauge as a library and register your own vendor or assertion with one function.

A custom provider

import { registerProvider } from "gauge-eval";

registerProvider("myllm", () => ({
  vendor: "myllm",
  async complete({ model, prompt }) {
    const output = await callMyModel(model, prompt);
    return { output, latencyMs: 0 };
  },
}));

Then reference it in any spec:

provider: myllm/some-model

A custom scorer

The key you register becomes the assertion name. Return { pass, score, label, message }.

import { registerScorer } from "gauge-eval";

registerScorer("word-count-under", (value, output) => {
  const limit = value as number;
  const n = output.split(/\s+/).filter(Boolean).length;
  return {
    pass: n <= limit, score: n <= limit ? 1 : 0,
    label: `word-count-under ${limit}`,
    message: n <= limit ? "" : `${n} words`,
  };
});

assert:
  - word-count-under: 50

The library also exports the runner, parser, scorers, and store, so you can build gauge into a larger workflow:

import { parseSpec, runAll } from "gauge-eval";

const spec = parseSpec("x.eval.md", source);
const results = await runAll([spec], { concurrency: 8 });

Testing an agent harness

gauge was built for the layer most eval tools ignore: the tooling wrapped around a model — routers, tool-selectors, retrievers, multi-step agents.

The pattern

• Expose a text-in / text-out entry point in your harness (a subcommand, a script).
• Point the exec provider at it. gauge feeds the prompt on stdin, reads the result on stdout.
• Assert on behavior — the tool it picked, the route it chose, the faithfulness of the answer.
• Baseline it so a prompt tweak that breaks tool selection fails CI instead of shipping.

Example: a tool-router regression suite

---
provider:
  type: exec
  command: "node dist/agent.js --print-tool"
cases:
  - name: weather-needs-search
    vars: { q: "what's the weather in Lisbon?" }
    assert: [{ equals: "web_search" }]
  - name: math-needs-calc
    vars: { q: "what is 4.2% of 1,900?" }
    assert: [{ equals: "calculator" }]
---

{{q}}