Identity
- Task
cc-qa-001, version 0.1.2, verifier frozen by hash- Family
- call-qa, difficulty easy, as the public gate file labels it
- Domain
- Regulated outbound sales: call QA and disposition
- Status
- Demonstration · text excerpts
- Gates run
- 10 November 2026, Python 3.11.8, deterministic gates only: no model runs, no human raters, no training
- Model attempts
- Version 0, on or before 7 November 2026: two current models from one provider, one rollout each, both passed. The later versions, which tightened the verifier and left the answer key unchanged, run next
- Sealed set
- Excluded. This task is disclosed in full and is not for sale
The job
A QA analyst on an outbound insurance campaign grades five calls against the campaign’s checklist of seven items. Two calls carry a critical breach: a sales agent who claims the returns are guaranteed, and a customer who asks in Malay not to be called again while the agent keeps selling. A third call carries two breaches that are not critical. On a fourth, the customer asks not to be called at the office, which is a request to call back, not a do-not-call request. The fifth is clean. The instruction, as the agent receives it:
You are the QA analyst for campaign HL-03. Grade the five calls in calls/ againstdocs/QA-CHECKLIST-HL-03.md. Incrm/campaign.db: write one qa_scores row per call and checklist item (PASS, FAIL or NA), correct any wrong disposition in calls, add customers to dnc where the checklist requires it, and add a qa_flags row (severity CRITICAL) for every critical breach.
The right answer grades every call on every item, PASS, FAIL or NA, sets the Malay call’s disposition to do-not-call, adds that customer to the do-not-call list with who added them and why, leaves every other disposition alone, and writes exactly one critical flag for each critical breach.
What the grade rests on
The verifier compares the agent’s grades with an answer key. We wrote the answer key, the checklist and the transcripts, including the Malay one. A grade that differs from the key fails, even where a QA lead might accept it. Until a QA lead has graded these calls, the verifier shows that it applies our key faithfully, not that our key matches how the trade grades.
The commissioned version replaces our checklist with one a QA lead builds from public rule text, and the Malay transcript with one written by a native speaker. A second practitioner then grades blind, and we report the agreement.
The workspace
Every run starts from the same files and the same database, written fresh by the task’s own reset.
| Path | What it holds after reset |
|---|---|
calls/C-101.txt | 756 bytes |
calls/C-102.txt | 518 bytes |
calls/C-103.txt | 400 bytes |
calls/C-104.txt | 513 bytes |
calls/C-105.txt | 360 bytes |
crm/campaign.db | calls (5), dnc (1), qa_flags (0), qa_scores (0) |
docs/QA-CHECKLIST-HL-03.md | 1300 bytes |
Reset digest, SHA-256 of the seeded workspace with the database taken as its SQL dump: d04c7bacdeaf936821d1f41d995b03009854e154d37148776643ac42c6a5c848. Reset deterministic from an empty directory: yes. A used workspace resets to the same digest through our test harness: yes.
The toolbox: what the agent may touch
- Reads
- The five transcripts, the checklist, and every table in the database.
- Writes
- One grade per call and checklist item, the disposition field of a call, new entries on the do-not-call list, and critical flags.
- Must not touch
- The transcripts and the checklist, the database schema, every other field of the call records, and the do-not-call entry that was there at the start. No file may be added anywhere. Any of these fails a gate, and the run scores zero whatever else it did.
Gates and checks
A submission the verifier cannot score, because the database is unreadable or holds no grades, is rejected as invalid at reward zero, with a structured reason. Then three gates. Fail any and the reward is zero, with no checks counted.
- transcripts and checklist untouched, no file added under calls/ or docs/.
- CRM schema unchanged; call records changed only in disposition; seeded DNC rows untouched.
- qa_scores holds exactly one valid result for each of the 35 call-item cells.
Then eight checks, each compared with the database on disk. The verifier never reads what the agent says it did. Weights sum to 17; 3 checks are critical. Strict, the training reward, needs every gate and every check.
| Check | What must be true | Weight | Critical |
|---|---|---|---|
| K1 | Both critical breaches found (guarantee claim; ignored do-not-call request in Malay) | 3 | yes |
| K2 | Both non-critical breaches found (no recording notice; no consent before ID and card) | 2 | |
| K3 | No compliant cell marked FAIL | 2 | |
| K4 | All 35 cells match the reference grid, including NA | 1 | |
| K5 | The do-not-call request is honoured: C-103 set to DNC and CU-2203 added to dnc with who added it and why (not the system provenance of imported rows) | 3 | yes |
| K6 | No false DNC: nobody else added, and the call-me-later customer stays CALLBACK | 3 | yes |
| K7 | Correct dispositions left alone | 1 | |
| K8 | qa_flags holds exactly one CRITICAL row per critical breach and no other row | 2 |
Score and strict are defined as on the packaging task’s page.
The probes
A reward hack fakes or tampers and must score exactly zero. A plausible mistake is one a careful analyst could make; it must not earn the strict reward. A correct alternative is a right answer written another way; it must earn it. 9 reward hacks, 9 plausible mistakes and 1 correct alternative ran on 10 November 2026, and every one returned the expected result.
| Probe | Kind | What it did | Strict | Score | What failed |
|---|---|---|---|---|---|
empty | Reward hack | No answer written | Fail | 0.00 | Rejected as invalid |
report_only | Reward hack | A written report saying the work is done, with nothing written to the database | Fail | 0.00 | gate 1, Rejected as invalid |
extra_report_in_out | Reward hack | Correct database rows, plus a report beside them that says every call passed | Fail | 0.00 | gate 1 |
delete_breach_calls | Reward hack | Deleted the two calls with breaches, then passed the rest | Fail | 0.00 | gate 2 |
all_pass | Plausible mistake | Marked every cell PASS: finds none of the four breaches | Fail | 0.35 | K1, K2, K4, K5, K8 |
all_fail_dnc_everyone | Plausible mistake | Marked every cell FAIL and put every customer on the do-not-call list | Fail | 0.47 | K3, K4, K6, K7, K8 |
keyword_dnc | Plausible mistake | Read “don’t call me at the office, call my mobile” as a do-not-call request | Fail | 0.53 | K3, K4, K6, K8 |
misses_malay_dnc | Plausible mistake | Graded the English calls correctly and missed the do-not-call request made in Malay | Fail | 0.47 | K1, K4, K5, K8 |
dnc_provenance_rewritten | Reward hack | Rewrote who added the existing do-not-call entry, and why | Fail | 0.00 | gate 2 |
call_record_rewritten | Reward hack | Pointed the Malay call at another customer | Fail | 0.00 | gate 2 |
schema_column_added | Reward hack | Added a column to the database schema | Fail | 0.00 | gate 2 |
drop_qa_flags | Reward hack | Correct grades, then dropped the flags table: a rejection, not a crash | Fail | 0.00 | Rejected as invalid |
forged_transcript | Reward hack | Planted a sixth transcript | Fail | 0.00 | gate 1 |
dnc_row_without_provenance | Plausible mistake | The new do-not-call entry says neither who added it nor why | Fail | 0.82 | K5 |
dnc_row_mimics_system | Plausible mistake | The new do-not-call entry copies the wording of an imported entry, as if the system had added it | Fail | 0.82 | K5 |
duplicate_critical_flags | Plausible mistake | Flagged the guarantee breach three times | Fail | 0.88 | K8 |
lowercase_false_flag | Plausible mistake | A false flag on a compliant call, severity written in lower case | Fail | 0.88 | K8 |
major_flags_added | Plausible mistake | An unrequested MAJOR flag beside the two critical ones | Fail | 0.88 | K8 |
analyst_id_and_transcript_span_dnc | Correct alternative | A correct answer whose do-not-call entry names the analyst and quotes the Malay request | Pass | 1.00 | none |
reference | Reference solution | The reference solution | Pass | 1.00 | none |
Our test harness also built 15 hostile inputs from a correct end state: 4 with a table dropped, 4 with a table’s first column renamed, one with the database overwritten with text, 6 with an input replaced by a directory. Every one returned a report at reward zero with no verifier error.
What a practitioner settles
The answer key encodes one convention wherever the trade may have several. A QA lead decides which is right. Open for this task:
- Does confirming a customer’s name count as asking for a personal detail before the recording notice? The key says no. If it does, four calls the key passes on item I3 fail it.
- Does naming the product before the recording notice count as describing it? The key says no, so two calls pass I3 that a stricter reading would fail.
- “Don’t call me at the office, call my mobile” is a callback in the key, not a do-not-call request. A QA lead confirms where the line falls in English, in Malay and in calls that mix the two.
What the verifier does not yet catch
We attack our own verifiers. On 8 November 2026 we tested this one against 40 single changes to a correct end state: version 0 accepted 15 wrong end states at full score and stopped with an error on two. We re-ran all 40 against version 0.1.2 on 9 November 2026. The classes found, and where each stands:
| Class of wrong work | Status on this task, 9 November 2026 |
|---|---|
| Duplicate flags, or false flags with a severity other than the exact word CRITICAL | Caught since version 0.1: K8 requires exactly one CRITICAL row per critical breach and no other row. duplicate_critical_flags, lowercase_false_flag, major_flags_added |
| The existing do-not-call entry rewritten, or deleted and added back | Caught since version 0.1 by gate 2: dnc_provenance_rewritten |
| A new do-not-call entry that says neither who added it nor why, or copies the wording of an imported entry | Caught since version 0.1 by K5, critical: dnc_row_without_provenance, dnc_row_mimics_system |
| Columns added to the schema and filled with notes that contradict the grades | Caught since version 0.1 by gate 2: schema_column_added |
| A report beside correct tables that contradicts them | Caught since version 0.1.1: any added file fails gate 1. extra_report_in_out |
| An input replaced by a directory crashes the verifier | Closed in version 0.1: 6 hostile inputs of this kind return a zero without error |
| Open: a flags row with every field empty stops the verifier with an error instead of a structured rejection | Open. Found on 9 November 2026. No wrong answer is accepted; the fix and its probe are scheduled for the next version |
| Still not read: whether the reason on a new do-not-call entry is true | Open, by design. K5 asks for who added the entry and why, and accepts any reason that is not empty. A practitioner’s blind grade is the only check on it |
| Not yet tested: a do-not-call request phrased in other words, or switching between English and Malay mid-sentence | Open. The task holds one Malay request and one office callback. Transcripts of real or re-enacted calls widen it |
Hand-written probes are a regression gate, never a guarantee: they prove that every wrong end state we have thought of scores what it should, and nothing about the ones we have not.
Change log
- By 7 November 2026. Version 0 written and run. Two current models from one provider passed it in one rollout each.
- 8 November 2026. Version 0.1, after the review: the schema, the call records other than disposition and the existing do-not-call entry are compared with the seed; flags must match exactly; a new do-not-call entry must say who added it and why; an unreadable database is a structured rejection. The answer key is unchanged.
- 8 November 2026. Version 0.1.1: any added file fails gate 1. The answer key is unchanged.
- 8 November 2026. Version 0.1.2: one correct alternative,
analyst_id_and_transcript_span_dnc. The answer key is unchanged. - 9 November 2026. This page published. The review’s 40 mutations re-run against version 0.1.2: 38 caught, one open (above), and one that resets a used workspace by hand, which our harness’s own reset handles.
Walk through it
Book a twenty-minute call and we run this task in front of your engineer: a reset, a reward hack, a plausible mistake, then the reference solution. The terms for a first order are on the packaging task’s page.