Email Classifier

Critical

segmented by category5 runsemail_classifiersample data

Latest

74%

-20 pts vs baseline

Baseline

94%

trusted bar

Latest run

Latest evaluation

Email Classifier #5 · claude-haiku-4-5 · Dataset v1 · Jul 18, 2026

Critical

74%pass rate

20 pts below baseline.

14 of 54 cases failing.

40 passed14 failed

Inspect this run

Quality by check

category_match — mean score74%

summary_quality — mean score100%

Quality by category

general8%

technical87%

account93%

billing100%

What this means

Quality has dropped badly enough to block a merge.

Merge blocked

Latest run: 40 of 54 test cases passed (74%). 14 gave a wrong answer.
Baseline: 20 points below the trusted baseline (74% vs 94%).
Trend: Quality has declined across the last 5 runs (94% → 74%).
Regressions: 5 measures got worse than the baseline, all 5 of them badly enough to block a merge. The biggest drop is in the category match check in “general”, down 83 points.
Weak spot: Failures are concentrated in one place: category “general” passes only 8% of its 12 cases, against 74% overall. Fixing that one area would move the number most.

What to do next

Investigate before merging. CI runs the same comparison and exits non-zero, so this will block the merge until the regression is fixed or a new baseline is deliberately promoted.

A merge with this run would be blocked.

Health over time

5 runs · 94% → 74%

Baseline

94%

Email Classifier #1 · claude-haiku-4-5 · Dataset v1 · Jul 18, 2026

Needs attention

5 metrics regressed against baseline.

Pass rate-20 pts
category_match — mean score-20 pts
category_match — pass rate-20 pts

Root-cause analysis

Recent runs

#574%#494%#396%#294%#1baseline94%