Back to the platform

Workflow Benchmarks and Regression

Change a workflow only when it still finds what it found before.

Noether is benchmarks and regression testing for AI workflows. Every version of a workflow runs against reference datasets whose answers are known, and a candidate is compared with the version in production before anyone relies on it.

Workflow Benchmarks and Regression

Proves that a new version of a workflow still finds what the last one found, before anyone relies on it.

Benchmarks and regression testing for AI workflows. Every version of a workflow runs against reference datasets whose answers are known, and is scored on what it finds, what it misses and what it costs. A candidate version is compared with the version in production before it is published, so a change that loses a finding is caught before it ships. Workflows come first: the same datasets and benchmarks are built next for agents and for models.

A workflow changes only when the measurements show it still finds what it found before.

Capabilities

What Noether measures, and how

An AI workflow is changed often: a new prompt, a new model, a new step. Noether answers one question before each change goes into production: does it still find what it found before?

Reference datasets with known answers

Each dataset is a small application with problems planted in it. It carries the exact list of what a workflow must report, and of the problems that only look real, which it must not report, so a result is checked against an answer rather than judged by opinion.

Datasets that change like code

A dataset changes through a reviewed change, like code. Every score names the version of the dataset and the version of the workflow it came from, so two runs are compared on the same ground.

Every version on the same datasets

Each version of a workflow runs on the same datasets, in isolated environments, and is scored on its raw results: what it found, what it missed and what it reported that is not there.

A candidate against the version in production

Before a new version is published, it is compared with the version in production: the findings it keeps, the ones it loses and the ones it gains, side by side.

What each benchmark run costs

Each benchmark run states what it cost, so a version that finds the same at a lower price, or more at a higher one, is a choice made with the numbers in hand.

Kept apart from your customers

A benchmark run is kept apart from the work your customers see. A version under test changes nothing for them until someone decides to publish it.

How it works

From a dataset to a published version

Four steps. The last one is a decision taken by a person, never a side effect of installing new code.

  1. Write down what must be found

    A dataset holds a small application with planted problems, the list of findings a workflow must report and the problems that only look real, which it must not report.

  2. Run each version on the same datasets

    The version in production and the candidate run on the same datasets, each in an isolated environment, and both are scored on their raw results.

  3. Compare the two

    Findings kept, lost and gained, and the cost of each run. A change that loses a finding shows up here, before it goes into production.

  4. Publish by decision

    A version is published only when someone decides to publish it, with the comparison in front of them.

A version that has not been published reaches no customer, however many times it has run.

Built for

Built for the people who change a workflow and answer for its results

Engineering

Teams that build AI workflows

Change a prompt, a model or a step and know, before it goes into production, what the change gained and what it lost.

Quality

The people who approve a new version

Decide on a new version from a comparison against known answers, not from a demonstration on the cases it happens to get right.

Leadership

Those who answer for the results

When a customer asks whether the new version still finds what the old one found, the answer is a measurement that already exists.

Why Noether

Why Noether?

Named after Emmy Noether, the mathematician whose theorem of 1918 ties each symmetry of a physical system to a quantity that the system conserves. A regression test asks the same kind of question of software: when the workflow changes, what must stay the same?

Workflows come first, because they change the most often and a lost finding costs the most there.

The same datasets and benchmarks are built next for agents and for models: datasets and benchmarks of every kind, wherever a change has to prove it kept what mattered.

What must stay the same, written down

The findings a workflow has to keep are listed in the dataset, so a change is measured against them rather than against a memory of how it used to behave.

Measured on raw results

A version is scored on what it actually reported, one finding at a time, and on what that cost.

Published by decision

Publishing a version is an explicit act taken after the benchmark, and never a consequence of installing new code.

Our experts with the product

Our experts build the first datasets with your team, train the people who will keep them, and hand the practice over to them.

Measure a workflow before you rely on it

Tell us which workflow you change most often, and we will show you what a benchmark of it looks like.

Meet an expert