THE STUDIO / LIVE FIRE 04
FACILITATOR
Live Fire · Scored Drill · On the Record
THE STUDIO/LIVE FIRE 04/COMPANION TO MODULE 04 — THE OPERATOR

THE
BURN

The console is live and the meter is running. Five situations from the operator's chair — a loop eating the daily budget, a frontier model doing a small model's job, a context window carrying thirty stale documents, a deadline priced at double, and one workload where the cheap model is the most expensive thing you run. Route, throttle, or kill.

FORMAT  SCORED DRILL / OPERATOR CONSOLE
AUDIENCE  OPERATORS, PLATFORM OWNERS, FINOPS, CIO OFFICE
DOCTRINE  COST PER OUTCOME, NOT COST
RUNTIME  ~25 MIN

Press → or swipe to begin. The record starts when you do.

SCENE 01/THE BRIEFING

Agents don't feel pain.
Budgets do.

Each console shows a live readout — tokens, retries, unit costs, quality metrics — and four operator actions. One action is the operator's move; the others are the reflexes that burn money or quality. The arithmetic is on screen: do it. And carry one number above all others — not cost, but cost per correct outcome.

“A stale flag on one of eight servers. Forty-five minutes. Four hundred and forty million dollars. Automation at production speed does not wait for the meeting.”Module 04 — The Operator · on Knight Capital, August 2012
RULE 01 — THE READOUT

Everything you need is in the console: rates, volumes, quality deltas. The right call is computable. One attempt.

RULE 02 — THE NUMBER

The operator's metric is cost per correct outcome. Spend is not the objective; waste and quality-loss are both failures of it.

RULE — THE WAGER

Before lodging, stake your conviction. MEASURED: +1 right, 0 wrong. HIGH CONVICTION: +2 right, −1 wrong. The call is the skill; calibration is the discipline.

RULE — THE CHAIN

Every selection, stake, and call is timestamped and hash-chained in your browser — the append-only invariant used for agent audit trails. Your run is itself evidence.

Facilitator Note

Group format: put the console on screen and give the room ninety seconds of silent arithmetic before any discussion. Vote on the action, then the stake. Console Five will split finance from risk — let it. That split is the module's thesis: the cheapest model can be the most expensive thing you run.

SCENE 02/CONSOLE ONE
ON THE CLOCK 00:00

The Loop

ALERT 06:14 UTC — Reconciliation task RECON-77 has been running six hours. The budget dashboard went amber at 05:40. Read the console. Make the call.
OPERATOR CONSOLE — TASK RECON-77LIVE
Console readoutruntime 6h 14m · tool call fetch_statement → HTTP 400, INVALID DATE RANGE, every attempt · retries: 312 and counting, no backoff · context resent per attempt: 61,400 tokens (full task history) · burn last hour: $212 · daily cap $500 — 80% consumed by 06:14 · task priority: routine, due 17:00
CONVICTION —
312 identical failures is not a reasoning problem.
Facilitator Key — Console One

Call: Option C. The arithmetic: 312 × 61,400 tokens ≈ 19M tokens spent re-asking a broken question. The room's instinct to “let it finish” is the reflex this console exists to kill.

Discussion: does any agent in your shop currently run without a retry ceiling and a per-task budget? If the answer takes more than ten seconds, the answer is yes.

SCENE 03/CONSOLE TWO
ON THE CLOCK 00:00

The Wrong Lane

REVIEW 09:00 — The nightly support-ticket triage job hit the monthly cost review. Read the console. Make the call.
OPERATOR CONSOLE — JOB TRIAGE-NIGHTLYMONTHLY REVIEW
Console readouttask: 6-label classification of support tickets · volume 42,000/night · current model: frontier tier at ~$0.11/ticket → $4,620/night, ~$138k/month · benchmark run: small model vs frontier on 2,000-ticket sample — agreement 98.4%, disagreements concentrated in one ambiguous label pair · small model unit cost ~$0.007/ticket → $294/night
CONVICTION —
98.4% agreement is a fact. Use it.
Facilitator Key — Console Two

Call: Option B. ~93% cost reduction with a tripwire. The harness matters more than the routing: without it, every downgrade is a leap of faith; with it, it's an experiment with an alarm.

Discussion: list your three highest-volume AI jobs. For how many does anyone know the small-model agreement rate? That number is the price of not benchmarking.

SCENE 04/CONSOLE THREE
ON THE CLOCK 00:00

The Context Hoarder

REVIEW 14:30 — The research assistant's per-turn cost has quadrupled since launch. Nobody changed the model. Read the console. Make the call.
OPERATOR CONSOLE — ASSISTANT RES-DESKCOST ANOMALY
Console readoutsession context: 178,000 tokens, resent in full every turn · composition: 92% is 30 documents attached across prior, completed tasks · per-turn cost $2.90 · volume ~400 turns/day → ~$1,160/day · citation analysis: last 200 turns referenced 4 of the 30 documents · quality complaints: none · latency: rising with context length
CONVICTION —
The context is a bill that arrives every single turn.
Facilitator Key — Console Three

Call: Option C. The 40x creep happened without anyone changing anything — that's the signature of context sediment. The week of quality-watch after the change is part of the call, not an afterthought.

Discussion: who in the room knows the current average context size of their longest-running assistant? Sediment grows precisely where nobody is looking.

SCENE 05/CONSOLE FOUR
ON THE CLOCK 00:00

The Batch Blindspot

PLANNING 11:00 — Legal wants 480,000 archived contracts classified by month-end, eleven days out. The job was set up on the realtime endpoint because that's what the team always uses. Read the console. Make the call.
OPERATOR CONSOLE — JOB CONTRACT-CLASS11 DAYS TO DEADLINE
Console readoutvolume: 480,000 documents, archive (no user waiting on any single result) · current setup: realtime endpoint, business-hours execution · projected cost: ~$38,000 · projected completion: day 10 of 11 · provider batch endpoint: 50% unit discount, 24h turnaround SLA per batch · throughput check: full volume fits in 8 batch submissions with 3 days of slack
CONVICTION —
No human is waiting on any single result. Price accordingly.
Facilitator Key — Console Four

Call: Option B. Identical model and output at half price with more deadline slack — the rare free lunch, left on the table purely by default settings.

Discussion: audit reflex for the room — which of your recurring jobs have a human waiting on individual results, and which are on realtime anyway?

SCENE 06/CONSOLE FIVE
ON THE CLOCK 00:00

The False Economy

ESCALATION 16:45 — Three months ago the fraud narrative-review agent was downgraded to the mini model, saving $3,100 a month. Risk has now escalated the quality trend. Read the console. Make the call.
OPERATOR CONSOLE — AGENT FRAUD-NARRATIVERISK ESCALATION
Console readouttask: narrative review of flagged cases for missed fraud patterns · volume: 900 cases/month · downgrade saving: $3,100/month · missed-pattern rate: 1.2% before → 4.7% after (validated on a 400-case back-test) · delta: ~31 additional missed cases/month · loss on the relevant missed-case class: ~$240,000 average · human QA capacity: 60 cases/month maximum
CONVICTION —
Do the expected-loss arithmetic before you defend the saving.
Facilitator Key — Console Five

Call: Option B. Pair this deliberately with Console Two: same doctrine, opposite direction. Routing down with evidence and routing up with evidence are one skill.

Discussion: which of your workloads was downgraded on cost grounds in the last year — and does a back-tested quality delta exist for any of them?

SCENE 07/THE SCORE
0 / 5
PENDING
CONVICTION BOOK: 0 / 10  ·  —

Complete the five scenarios to receive your rating.

SCENE 08/THE DOCTRINE

Cost per correct outcome.
Both directions.

Five consoles, five operator disciplines. Carry these forward:

01

Loops are budget leaks. Retry ceilings, backoff, minimal resent context, per-task budgets. Agents don't feel pain — kill fast, patch, rerun.

02

Route by capability, prove with a harness. The premium lane is for premium problems, and the standing eval — not the expensive default — is what protects quality.

03

Context is a bill, not a bin. Everything in the window ships every turn. Summarize, window, retrieve on citation, re-baseline.

04

Urgency belongs to the task, not the endpoint. “Who is waiting on this result?” is worth half the invoice on archive-scale work.

05

Sometimes the operator's move is to spend more. A saving that buys missed fraud is not a saving. The number is cost per correct outcome — defend it in both directions.

Knight Capital needed forty-five minutes to prove what agents now prove faster: automation at production speed does not wait for the meeting. The operator's chair exists so someone is doing the arithmetic while the meter runs. Run this drill with your platform owners and your finance partners in the same room — Console Five is written to divide them, and the division is the lesson.

NEXT DRILL → LF-05 THE FORGERY← LF-03 THE RECONSTRUCTIONTHE SYSTEM

THE STUDIO — LIVE FIRE 04 · COMPANION TO MODULE 04: THE OPERATOR

TitleGUIDED NARRATION
Copied