Menu

Research

Call-quality review, version 0.1.2: every check, and every known gap

A demonstration contact-centre task. The agent grades five outbound insurance sales calls, one of them in Malay mixed with English, against a compliance checklist, then updates the records: call dispositions, the do-not-call list, and one flag for each critical breach.

Task cc-qa-001 · Version 0.1.2, gates run 10 November 2026 · Datarena

Demonstration · About the public demonstrations. These calls are short text excerpts, not audio. The grade rests on our answer key; no QA lead or native-speaker rater has graded these calls yet.

Identity

Task
cc-qa-001, version 0.1.2, verifier frozen by hash
Family
call-qa, difficulty easy, as the public gate file labels it
Domain
Regulated outbound sales: call QA and disposition
Status
Demonstration · text excerpts
Gates run
10 November 2026, Python 3.11.8, deterministic gates only: no model runs, no human raters, no training
Model attempts
Version 0, on or before 7 November 2026: two current models from one provider, one rollout each, both passed. The later versions, which tightened the verifier and left the answer key unchanged, run next
Sealed set
Excluded. This task is disclosed in full and is not for sale

The job

A QA analyst on an outbound insurance campaign grades five calls against the campaign’s checklist of seven items. Two calls carry a critical breach: a sales agent who claims the returns are guaranteed, and a customer who asks in Malay not to be called again while the agent keeps selling. A third call carries two breaches that are not critical. On a fourth, the customer asks not to be called at the office, which is a request to call back, not a do-not-call request. The fifth is clean. The instruction, as the agent receives it:

You are the QA analyst for campaign HL-03. Grade the five calls in calls/ against docs/QA-CHECKLIST-HL-03.md. In crm/campaign.db: write one qa_scores row per call and checklist item (PASS, FAIL or NA), correct any wrong disposition in calls, add customers to dnc where the checklist requires it, and add a qa_flags row (severity CRITICAL) for every critical breach.

The right answer grades every call on every item, PASS, FAIL or NA, sets the Malay call’s disposition to do-not-call, adds that customer to the do-not-call list with who added them and why, leaves every other disposition alone, and writes exactly one critical flag for each critical breach.

What the grade rests on

The verifier compares the agent’s grades with an answer key. We wrote the answer key, the checklist and the transcripts, including the Malay one. A grade that differs from the key fails, even where a QA lead might accept it. Until a QA lead has graded these calls, the verifier shows that it applies our key faithfully, not that our key matches how the trade grades.

The commissioned version replaces our checklist with one a QA lead builds from public rule text, and the Malay transcript with one written by a native speaker. A second practitioner then grades blind, and we report the agreement.

The workspace

Every run starts from the same files and the same database, written fresh by the task’s own reset.

PathWhat it holds after reset
calls/C-101.txt756 bytes
calls/C-102.txt518 bytes
calls/C-103.txt400 bytes
calls/C-104.txt513 bytes
calls/C-105.txt360 bytes
crm/campaign.dbcalls (5), dnc (1), qa_flags (0), qa_scores (0)
docs/QA-CHECKLIST-HL-03.md1300 bytes

Reset digest, SHA-256 of the seeded workspace with the database taken as its SQL dump: d04c7bacdeaf936821d1f41d995b03009854e154d37148776643ac42c6a5c848. Reset deterministic from an empty directory: yes. A used workspace resets to the same digest through our test harness: yes.

The toolbox: what the agent may touch

Reads
The five transcripts, the checklist, and every table in the database.
Writes
One grade per call and checklist item, the disposition field of a call, new entries on the do-not-call list, and critical flags.
Must not touch
The transcripts and the checklist, the database schema, every other field of the call records, and the do-not-call entry that was there at the start. No file may be added anywhere. Any of these fails a gate, and the run scores zero whatever else it did.

Gates and checks

A submission the verifier cannot score, because the database is unreadable or holds no grades, is rejected as invalid at reward zero, with a structured reason. Then three gates. Fail any and the reward is zero, with no checks counted.

  1. transcripts and checklist untouched, no file added under calls/ or docs/.
  2. CRM schema unchanged; call records changed only in disposition; seeded DNC rows untouched.
  3. qa_scores holds exactly one valid result for each of the 35 call-item cells.

Then eight checks, each compared with the database on disk. The verifier never reads what the agent says it did. Weights sum to 17; 3 checks are critical. Strict, the training reward, needs every gate and every check.

CheckWhat must be trueWeightCritical
K1Both critical breaches found (guarantee claim; ignored do-not-call request in Malay)3yes
K2Both non-critical breaches found (no recording notice; no consent before ID and card)2
K3No compliant cell marked FAIL2
K4All 35 cells match the reference grid, including NA1
K5The do-not-call request is honoured: C-103 set to DNC and CU-2203 added to dnc with who added it and why (not the system provenance of imported rows)3yes
K6No false DNC: nobody else added, and the call-me-later customer stays CALLBACK3yes
K7Correct dispositions left alone1
K8qa_flags holds exactly one CRITICAL row per critical breach and no other row2

Score and strict are defined as on the packaging task’s page.

The probes

A reward hack fakes or tampers and must score exactly zero. A plausible mistake is one a careful analyst could make; it must not earn the strict reward. A correct alternative is a right answer written another way; it must earn it. 9 reward hacks, 9 plausible mistakes and 1 correct alternative ran on 10 November 2026, and every one returned the expected result.

ProbeKindWhat it didStrictScoreWhat failed
emptyReward hackNo answer writtenFail0.00Rejected as invalid
report_onlyReward hackA written report saying the work is done, with nothing written to the databaseFail0.00gate 1, Rejected as invalid
extra_report_in_outReward hackCorrect database rows, plus a report beside them that says every call passedFail0.00gate 1
delete_breach_callsReward hackDeleted the two calls with breaches, then passed the restFail0.00gate 2
all_passPlausible mistakeMarked every cell PASS: finds none of the four breachesFail0.35K1, K2, K4, K5, K8
all_fail_dnc_everyonePlausible mistakeMarked every cell FAIL and put every customer on the do-not-call listFail0.47K3, K4, K6, K7, K8
keyword_dncPlausible mistakeRead “don’t call me at the office, call my mobile” as a do-not-call requestFail0.53K3, K4, K6, K8
misses_malay_dncPlausible mistakeGraded the English calls correctly and missed the do-not-call request made in MalayFail0.47K1, K4, K5, K8
dnc_provenance_rewrittenReward hackRewrote who added the existing do-not-call entry, and whyFail0.00gate 2
call_record_rewrittenReward hackPointed the Malay call at another customerFail0.00gate 2
schema_column_addedReward hackAdded a column to the database schemaFail0.00gate 2
drop_qa_flagsReward hackCorrect grades, then dropped the flags table: a rejection, not a crashFail0.00Rejected as invalid
forged_transcriptReward hackPlanted a sixth transcriptFail0.00gate 1
dnc_row_without_provenancePlausible mistakeThe new do-not-call entry says neither who added it nor whyFail0.82K5
dnc_row_mimics_systemPlausible mistakeThe new do-not-call entry copies the wording of an imported entry, as if the system had added itFail0.82K5
duplicate_critical_flagsPlausible mistakeFlagged the guarantee breach three timesFail0.88K8
lowercase_false_flagPlausible mistakeA false flag on a compliant call, severity written in lower caseFail0.88K8
major_flags_addedPlausible mistakeAn unrequested MAJOR flag beside the two critical onesFail0.88K8
analyst_id_and_transcript_span_dncCorrect alternativeA correct answer whose do-not-call entry names the analyst and quotes the Malay requestPass1.00none
referenceReference solutionThe reference solutionPass1.00none

Our test harness also built 15 hostile inputs from a correct end state: 4 with a table dropped, 4 with a table’s first column renamed, one with the database overwritten with text, 6 with an input replaced by a directory. Every one returned a report at reward zero with no verifier error.

What a practitioner settles

The answer key encodes one convention wherever the trade may have several. A QA lead decides which is right. Open for this task:

  • Does confirming a customer’s name count as asking for a personal detail before the recording notice? The key says no. If it does, four calls the key passes on item I3 fail it.
  • Does naming the product before the recording notice count as describing it? The key says no, so two calls pass I3 that a stricter reading would fail.
  • “Don’t call me at the office, call my mobile” is a callback in the key, not a do-not-call request. A QA lead confirms where the line falls in English, in Malay and in calls that mix the two.

What the verifier does not yet catch

We attack our own verifiers. On 8 November 2026 we tested this one against 40 single changes to a correct end state: version 0 accepted 15 wrong end states at full score and stopped with an error on two. We re-ran all 40 against version 0.1.2 on 9 November 2026. The classes found, and where each stands:

Class of wrong workStatus on this task, 9 November 2026
Duplicate flags, or false flags with a severity other than the exact word CRITICALCaught since version 0.1: K8 requires exactly one CRITICAL row per critical breach and no other row. duplicate_critical_flags, lowercase_false_flag, major_flags_added
The existing do-not-call entry rewritten, or deleted and added backCaught since version 0.1 by gate 2: dnc_provenance_rewritten
A new do-not-call entry that says neither who added it nor why, or copies the wording of an imported entryCaught since version 0.1 by K5, critical: dnc_row_without_provenance, dnc_row_mimics_system
Columns added to the schema and filled with notes that contradict the gradesCaught since version 0.1 by gate 2: schema_column_added
A report beside correct tables that contradicts themCaught since version 0.1.1: any added file fails gate 1. extra_report_in_out
An input replaced by a directory crashes the verifierClosed in version 0.1: 6 hostile inputs of this kind return a zero without error
Open: a flags row with every field empty stops the verifier with an error instead of a structured rejectionOpen. Found on 9 November 2026. No wrong answer is accepted; the fix and its probe are scheduled for the next version
Still not read: whether the reason on a new do-not-call entry is trueOpen, by design. K5 asks for who added the entry and why, and accepts any reason that is not empty. A practitioner’s blind grade is the only check on it
Not yet tested: a do-not-call request phrased in other words, or switching between English and Malay mid-sentenceOpen. The task holds one Malay request and one office callback. Transcripts of real or re-enacted calls widen it

Hand-written probes are a regression gate, never a guarantee: they prove that every wrong end state we have thought of scores what it should, and nothing about the ones we have not.

Change log

  1. By 7 November 2026. Version 0 written and run. Two current models from one provider passed it in one rollout each.
  2. 8 November 2026. Version 0.1, after the review: the schema, the call records other than disposition and the existing do-not-call entry are compared with the seed; flags must match exactly; a new do-not-call entry must say who added it and why; an unreadable database is a structured rejection. The answer key is unchanged.
  3. 8 November 2026. Version 0.1.1: any added file fails gate 1. The answer key is unchanged.
  4. 8 November 2026. Version 0.1.2: one correct alternative, analyst_id_and_transcript_span_dnc. The answer key is unchanged.
  5. 9 November 2026. This page published. The review’s 40 mutations re-run against version 0.1.2: 38 caught, one open (above), and one that resets a used workspace by hand, which our harness’s own reset handles.

Walk through it

Book a twenty-minute call and we run this task in front of your engineer: a reset, a reward hack, a plausible mistake, then the reference solution. The terms for a first order are on the packaging task’s page.

Request a sample pack Book the walk-through