EvalGate

Email Classifier

Critical
segmented by category5 runsemail_classifiersample data

Latest

74%

-20 pts vs baseline

Baseline

94%

trusted bar

Latest run

Golden dataset

v1

Human-authored golden set for customer support email classification. 54 cases spanning all four categories with deliberate edge cases: ambiguous intent, typos, very short messages, mixed-language text, and sarcasm.

54 hand-labeled cases across 4 categories · segmented by category

By difficulty

easy16
hard16
medium22

By category

technical15
account14
billing13
general12
diff
cat

54 of 54 cases

ec-001
billingeasy

Hi, I was charged $49 twice this month for the same subscription. Can you refund the duplicate charge?

Clear duplicate-charge billing case.

ec-002
billingeasy

Could you please send me a copy of my invoice for April? I need it for expense reporting.

Invoice request.

ec-003
billingeasy

My card was declined when you tried to renew my plan. How do I update my payment method?

Payment failure / payment method update.

ec-004
billingmedium

Why did my monthly price jump from $20 to $35 without any notice? That seems unfair.

Pricing complaint.

ec-005
billingeasy

I applied the promo code WELCOME20 at checkout but the discount was not applied to my order.

Coupon not applied.

ec-006
billingmedium

Do you provide a proper VAT invoice with your company tax ID for EU customers?

Tax/VAT invoice question.

ec-007
billinghard

Refund?

Edge: extremely short message; intent inferred as billing/refund.

ec-008
billingmedium

i think i was billd twice for my subscripton this monthh, can u check my acount charges plz

Edge: typos. Mentions 'account' but the intent is about charges (billing).

ec-009
billinghard

Hola, me cobraron dos veces this month and I only have one subscription. Please help.

Edge: mixed Spanish/English. Double-charge billing issue.

ec-010
billinghard

Wonderful. Charged $99 for a plan I cancelled last week. Truly a flawless system you have here.

Edge: sarcastic tone. Underlying issue is an erroneous charge (billing).

ec-011
billingmedium

Can I switch from monthly to annual billing, and would I get the annual discount immediately?

Billing cycle change.

ec-012
billingeasy

I see a pending charge of $1 on my statement from your company. What is that for?

Card verification charge inquiry.

ec-013
billingmedium

Please stop charging me. I want a full refund for the last three months immediately.

Refund demand; primary intent is monetary (billing) even though it implies cancellation.

ec-014
technicaleasy

Every time I try to export a report to PDF the app freezes and then closes. I'm on the latest version on Windows 11.

Crash on export.

ec-015
technicalmedium

Your status page says everything is fine but I'm getting a 500 error on every page load right now.

Server error / possible outage.

ec-016
technicalmedium

Our integration suddenly started returning 401 Unauthorized even though the API key hasn't changed.

API auth error; technical despite mentioning a key (not account login).

ec-017
technicaleasy

The dashboard takes almost 30 seconds to load lately. It used to be instant. Is something wrong?

Performance degradation.

ec-018
technicaleasy

How do I connect the Slack integration? I can't find where to add the webhook in settings.

How-to / configuration question.

ec-019
technicalmedium

the app keps freezng when i clic the export buton on my mac, plz fix asap

Edge: typos. Clear technical freeze bug.

ec-020
technicalhard

It's broken.

Edge: extremely short and vague; defaulting to technical as a malfunction report.

ec-021
technicalhard

Bonjour, the export feature ne fonctionne pas since the last update, it just shows a blank screen.

Edge: mixed French/English. Technical regression after update.

ec-022
technicalhard

Amazing, your 'new' update deleted all of my saved reports. Brilliant engineering, really.

Edge: sarcastic. Underlying technical data-loss bug.

ec-023
technicalmedium

My webhook endpoint isn't receiving any events even though I enabled them an hour ago.

Webhook delivery issue.

ec-024
technicalmedium

After the latest release my charts stopped rendering and the console shows a JavaScript error.

Front-end rendering bug.

ec-025
technicaleasy

Is there a way to bulk import contacts from a CSV file? I have about 5000 rows.

How-to / usage question.

ec-026
technicalmedium

The mobile app won't sync with my desktop data. I tap refresh but nothing updates.

Sync failure.

ec-027
accounteasy

I reset my password an hour ago but now I get 'invalid credentials' on every login attempt.

Login failure after reset.

ec-028
accounteasy

I forgot my password and the reset email never arrives. I've checked spam too.

Password reset email not arriving.

ec-029
accountmedium

My two-factor authentication codes are all being rejected even though my phone clock is correct.

2FA access issue.

ec-030
accounteasy

I'd like to change the email address associated with my account to a new work email.

Profile change.

ec-031
accountmedium

Please delete my account and erase all of my personal data in accordance with GDPR.

Account closure / data deletion.

ec-032
accountmedium

My account got locked after a few wrong password attempts. How can I unlock it?

Account lockout.

ec-033
accountmedium

I noticed a login from a country I've never visited. Can you check if my account was compromised?

Security / suspicious login.

ec-034
accountmedium

How do I add a teammate to my account and give them edit access to our projects?

Access / team management.

ec-035
accountmedium

cant acces my acount, the password dont work anymore and i didnt change it

Edge: typos. Access/login problem.

ec-036
accounthard

Locked out.

Edge: extremely short; account lockout inferred.

ec-037
accounthard

No puedo iniciar sesion, my password reset link says it has already expired.

Edge: mixed Spanish/English. Account access issue.

ec-038
accounthard

Love being locked out of my own account for absolutely no reason. Fantastic security, thanks.

Edge: sarcastic. Account lockout underneath.

ec-039
accountmedium

Can I transfer ownership of the workspace to another admin before I leave the company?

Ownership transfer / access.

ec-040
generaleasy

Hey team, love the product! Do you happen to offer a partner program for agencies?

Partnership inquiry.

ec-041
generaleasy

Just wanted to say your support team helped me last week and they were fantastic. Thank you!

Positive feedback.

ec-042
generalmedium

I'm a journalist writing about productivity tools. Could someone from your team do a short interview?

Press / media inquiry.

ec-043
generaleasy

Do you have any open positions on your engineering team? I couldn't find a careers page.

Careers question.

ec-044
generalhard

Quick question.

Edge: extremely short and contentless; defaults to general.

ec-045
generalmedium

do u gys have a referal programm? a freind told me i could get credit for invitng peple

Edge: typos. General referral-program inquiry.

ec-046
generalhard

hi

Edge: minimal content; general.

ec-047
generalhard

Ciao! Just wanted to say grazie mille, your team was super helpful and the app is great.

Edge: mixed Italian/English. Positive feedback.

ec-048
generalhard

Oh sure, like anyone there actually reads feedback. Anyway, the app is fine I guess. Bye.

Edge: sarcastic. General feedback with no actionable request.

ec-049
generalmedium

You should really add a dark mode. My eyes would thank you during late-night sessions.

Feature suggestion; general feedback rather than a technical fault.

ec-050
generaleasy

Are you running any webinars or live demos this month? I'd like to attend one with my team.

Event / webinar question.

ec-051
technicalhard

I paid for the Premium plan yesterday but the premium features still don't show up in my account.

Edge: ambiguous between billing (paid) and technical (entitlement not applied). Payment succeeded, so primary issue is features not unlocking = technical.

ec-052
accountmedium

Please cancel my subscription effective today; I no longer need the service.

Edge: cancellation. Per category rules, plan cancellation/closure is treated as account.

ec-053
generalhard

Is the Pro plan worth it over the Basic plan? Mainly curious about the difference in features.

Edge: ambiguous between billing (plans/pricing) and general (pre-sales question). It is an exploratory comparison, not a charge issue, so general.

ec-054
technicalhard

The invoice download button gives me a 404 error, so now I can't get the receipt I need.

Edge: ambiguous between billing (invoice/receipt) and technical (404 error). The blocker is a broken button, so the primary issue is technical.