The real cost of
a completed AI task.

CHAPTER 01Count the work that survives review

A model invoice cannot tell a founder whether an AI feature makes money. The invoice records consumption. The business needs to know how much it spends delivering an outcome that a customer will accept.

Those quantities can move in different directions. A cheaper response may require more correction. An expensive response may remove a manual step. A fast answer may arrive before the system has checked whether it is allowed to take the action the customer requested. Choosing between these systems requires a definition of finished work.

Consider a supplier invoice entering an accounts-payable workflow. Extracting the total is one operation. Matching it to a purchase order, checking the supplier, identifying a duplicate, and deciding whether the invoice needs approval are separate operations. A system that extracts the total correctly has not necessarily completed the workflow. A system that schedules an unauthorized payment has failed even if every extracted field is accurate.

For this example, the accepted outcome could be: a valid invoice is matched and routed according to policy, or an exception is sent to the correct reviewer with sufficient evidence. That definition allows a safe escalation to count as successful routing. It does not allow that escalation to count as fully automated payment processing. Both measures can be useful, provided they remain separate.

Before comparing model prices, write the acceptance criteria. Record which actions are allowed, which checks are mandatory, how exceptions should be handled, and who decides whether the result is acceptable. An economic comparison built on changing acceptance criteria will confuse a reduction in standards with an improvement in efficiency.

CHAPTER 02Define completion before comparing systems

A task ends when the system stops working. A completed task meets the requirements of the person or organisation that requested it. The difference is small in wording and large in measurement. A conversation can end with a confident answer while the underlying request remains unresolved. An API can return successfully after doing something the requester did not authorise. A document can be grammatically correct and still omit the fact that would change a decision.

For this article, a completed task is one that meets declared acceptance criteria within the permitted workflow. The word “accepted” remains useful in the measurement: it names the decision that admits an outcome into the denominator. Acceptance should be grounded in evidence about the result, rather than in the model's own statement that it has finished. This definition applies to service workflows, research, document handling, and software changes. It does not make their acceptance criteria interchangeable.

Separate the result from the route

Return to the invoice example. An outcome specification might require the correct supplier, amount, currency, purchase-order match, and routing decision. A process specification might require access checks, explicit approval above a threshold, and a durable record of the action. The first describes the final state. The second constrains how the system may reach it. Both belong in acceptance when the route matters to the business.

A system could arrive at the right payment amount by reading a document it was not permitted to access. It could obtain the right approval from the wrong person. It could submit a correct payment twice and leave one duplicate pending. Checking only the final answer would miss these failures. Checking only the final database state may also miss an intermediate disclosure or an action later reversed by another process.

The original τ-bench paper acknowledges this distinction. Its reward combines checks on the final database and required information in user-facing responses. The authors explicitly note that reward 1 can be insufficient for complete success because a policy violation, such as missing confirmation, may escape those checks. The analysis in this article therefore uses “benchmark pass” for reward-positive records, not “independently verified business completion.” Yao et al., τ-bench, Section 3, 2024.

That limitation is useful when designing a pilot. Every checker has a scope. A schema validator checks structure. A database invariant checks a property of state. A citation checker checks whether a reference supports a claim under its rubric. No single check should inherit responsibility for properties it cannot observe. The acceptance specification should identify which requirement each checker covers and which requirements still need an independent review.

Choose the unit that matches the promise

If the product promises to draft a response for an employee, an accepted draft may be the correct unit. If it promises to resolve a support request, acceptance needs to include the resolution. If it promises to prepare a code change for review, the unit differs from a change that has passed review and shipped. A product can deliver value at any of these stages. The cost claim needs to describe the stage it actually serves.

A single customer request can also contain several tasks. “Change my flight and update the hotel booking” is one utterance but at least two external operations. Counting each operation as a separate success can inflate a completion rate when the customer needs both. Counting the combined request alone can hide useful partial progress. Keep a parent request identifier and record the component outcomes beneath it. Report the combined result when that is the commercial promise.

The same issue appears in coding. A patch that passes tests may still require documentation, migration work, or a security review. A merged change may later be reverted. Those observations do not make passing tests or merging meaningless; they establish different milestones. A coding-specific version of this study would need its own observation window and measures of rework. The airline and retail traces here cannot supply those answers.

Make disagreement visible

People will disagree about whether a task is complete. An engineer may accept a response that an operations specialist rejects for missing context. Two reviewers may interpret an ambiguous policy differently. Treat those disagreements as information about the acceptance specification before treating them as model errors. If the organisation cannot describe a consistent standard, the evaluation will reflect that inconsistency.

Use a small calibration set to let reviewers apply the proposed rubric independently. Compare the disputed cases, revise ambiguous instructions, and preserve examples of acceptable alternatives. Do this before using the same rubric to decide a deployment. After the evaluation starts, changes to the rubric should produce a new version. Otherwise, the team can unknowingly improve its measured score by relaxing what it counts as success.

A dispute record need not be elaborate. It can name the task, the requirement, the two decisions, the evidence each reviewer used, and the final resolution. Its purpose is to make future decisions consistent. For open-ended research, more than one answer may deserve acceptance. The rubric should allow that variety while still requiring accurate claims, relevant coverage, and citations that support the argument.

CHAPTER 03What published results do—and do not—price

Anthropic reported that its multi-agent research configuration improved performance by 90.2% over a single-agent configuration on an internal research evaluation. The comparison used an Opus 4 lead with Sonnet 4 subagents against single-agent Opus 4. The same report describes multi-agent token use at about fifteen times that of chat interactions. These figures have different comparison groups; dividing one by the other would not produce a valid cost-effectiveness estimate. Neither figure establishes the economics of an accounts-payable workflow. Anthropic, June 13, 2025.

METR's early-2025 randomized study involved 16 experienced open-source developers and 246 issues. Allowing AI tools increased completion time by 19% in that setting. The authors explicitly caution against generalizing to most developers or future tools. METR, July 10, 2025.

In February 2026, METR described selection and measurement problems in its follow-up study. Developers' willingness to participate, their choice of submitted tasks, and concurrent agent use complicated the estimates. METR considered the newer data an unreliable measure of the current productivity effect. Treating the earlier slowdown as a timeless property of AI would ignore this update. METR, February 24, 2026.

For a buyer, these reports help identify what to measure: the task population, the comparison system, the acceptance standard, and the work outside the model call. They do not supply a universal productivity multiplier to insert into a financial forecast.

CHAPTER 04Newer evidence: a 2026 benchmark snapshot

The current reference point is Sierra's August 2026 submission metadata for GPT-5.6-sol and Claude Opus 5. These are newer reported evaluations, not reruns performed for this article. Both submissions use the banking_knowledge domain, AllTools retrieval, benchmark version 1.0.1, a GPT-5.2 user simulator, and four trials. Their evaluation dates and reasoning configurations differ, so the comparison is a description of those submitted systems rather than an isolation of the model alone.

Sierra reports a single-attempt pass rate of 46.91% and a four-run all-pass rate of 27.84% for GPT-5.6-sol at xhigh reasoning, evaluated on 22 July 2026. For Claude Opus 5 at max reasoning, evaluated on 3 August 2026, it reports 48.71% and 31.96%, respectively. These percentages belong to this banking configuration. They are not general success rates for either model. GPT-5.6-sol submission. Claude Opus 5 submission.

Both records leave the cost field null. Consequently, this snapshot cannot establish which configuration is cheaper per completed task. The reported pass-rate difference should not be converted into a financial advantage. Nor does this article infer statistical significance from the published aggregate percentages. A comparable billing study needs the underlying resource records, the acceptance definition, and an explicit analysis of uncertainty.

The newer snapshot and the historical trace analysis answer different questions. The snapshot shows what Sierra reports for two newer configurations. The historical files allow this article to reproduce an interaction-volume calculation from individual records. Their domains, benchmark generations, and evaluation conditions differ. Placing the results on a single model-progress curve would be misleading; no such trend is claimed here.

The GPT-4o and Claude 3.5 material below is retained only as a historical accounting example. It should not guide a present-day model purchase. Its contribution is the reproducible method for retaining failed attempts in the ledger. That method survives model changes, while its old performance numbers do not become current merely because the article has been updated.

CHAPTER 05An audit of 1,980 historical agent attempts

The first analysis for this article uses four public trajectory files from Sierra's original τ-bench repository. Together they contain 1,980 attempts across two configurations and two domains. Within each configuration, the airline tasks cover 50 distinct task identifiers and the retail tasks cover 115. The GPT-4o files contain four trials per task; the files labelled sonnet-35-new contain eight. These are repeated benchmark attempts, not 1,980 independent customer workflows.

The original paper and the later repository files are different evidence objects. This article computes scores from the frozen files; it does not claim to reproduce every number in the paper. Publication date, repository revision, and run configuration should travel with any quoted result. A score that differs from an earlier table is a reason to inspect those versions before concluding that a model improved or regressed.

The source is frozen at commit 59a200c6d575d595120f1cb70fea53cef0632f6b. The accompanying script downloads only these four files and the repository's MIT licence, records content hashes, checks unique task/trial pairs, and computes the figures from the records. It makes no model calls. This is a reanalysis of published traces, not a fresh evaluation of the models. Frozen source repository.

The repository now warns that these tasks are outdated and directs readers to its successor. That matters: this analysis cannot establish how current agents perform, and defects in the historical benchmark may affect its reward labels. Its narrower purpose is to show what happens to the accounting when unsuccessful attempts remain in the ledger. The models, tasks, and reported outcomes belong to this archived experiment. Repository notice.

A successful attempt does not account for the whole workload

In the GPT-4o airline file, 84 of 200 attempts received reward 1. Across all attempts, the traces contain 2,454 assistant messages. Of these, 1,625 occur in attempts that received reward 0. Failed attempts therefore account for 66.2% of recorded assistant messages, even though they produce none of the reward-positive outcomes in the denominator.

The average is 12.27 assistant messages per attempt. Dividing the same total by the 84 successful attempts produces 29.21 messages per successful attempt. The second number includes messages spent on failures. It is not the average length of a successful conversation. Those successful conversations contain 829 messages, or 9.87 each. These three numbers answer different questions; a dashboard that labels all of them “cost per task” would be ambiguous.

The ratio between workload messages per success and messages per attempt is the reciprocal of the success rate. Here, 1 / 0.42 is approximately 2.38. That multiplier follows from the choice of denominator. It does not prove that a particular deployment will cost 2.38 times its prototype, nor that the failures can be fixed by spending that much more.

In the GPT-4o retail file, 278 of 460 attempts pass. The traces contain 6,564 assistant messages, of which 2,778 belong to failures. The failed-message share is 42.3%. Workload messages per success are 23.61, compared with 14.27 per attempt. The same accounting method produces a different picture on the other domain.

The Sonnet-labelled airline file contains 184 successes in 400 attempts and 4,674 assistant messages. Failed attempts account for 62.9% of those messages. Its retail file contains 637 successes in 920 attempts and 12,624 assistant messages, with 32.5% of messages in failures. These are descriptions of the recorded configurations. The files alone do not isolate which differences arise from the model, prompts, task difficulty, simulator behaviour, or other execution conditions.

Assistant messages are a count, not a bill

An assistant message may contain a short question or a long structured tool call. Counting each as one does not make their token consumption equal. Input context can also grow during a conversation. A short output late in a long session may require more input processing than a long output near the start.

None of these 1,980 records includes a top-level total_cost field. Some records contain info.user_cost, but a simulated user's expenditure is not the agent's complete bill. Using that field as production inference cost would mislabel the data. The repository's agent implementation accumulates response costs internally; the existence of that code does not restore a missing value to a historical output file. Agent implementation at the frozen commit.

For this iteration, the charts use assistant messages as an observable measure of interaction volume. They do not estimate token use, dollars, wall-clock time, or human effort. A dollar-cost study needs different records or additional instrumentation. That missing information is part of the finding: even detailed public traces may not be sufficient to reconstruct unit economics.

Repeated reliability asks a different question

A single-attempt success rate answers how often the recorded attempts passed. A repeatability measure asks how often the same task succeeds across several attempts. The distinction matters when a product must work consistently for recurring customer requests.

For the GPT-4o airline data, ten of the fifty tasks passed all four observed trials: 20%. Thirty-six tasks passed at least once: 72%. The 42% attempt-level success rate lies between those values, but none is a substitute for the others. The “at least once” result assumes access to multiple outcomes and a way to identify success. A deployed system may have neither.

Comparing “all recorded trials passed” across four-run and eight-run files would impose a different standard on each configuration. Instead, this analysis computes the mean probability that four distinct recorded trials, selected uniformly without replacement for a task, all passed. For a task with n trials and s successes, the estimator is C(s,4) / C(n,4), averaged equally across tasks. When there are exactly four trials, it reduces to the fraction of tasks that pass all four.

This produces approximately 20.0% for GPT-4o airline, 38.3% for GPT-4o retail, 22.4% for the Sonnet-labelled airline file, and 46.2% for its retail file. These are finite-record estimates. They do not establish independence between retries or predict how a future model will perform.

A rerun is also different from a recovery policy. An operational retry may carry forward information from the failure, change a tool, seek approval, or stop after an irreversible action. The benchmark's repeated trials cannot tell us whether such a policy is safe or economical. Testing recovery requires recording state and actions across the recovery sequence itself.

What this changes in a buying decision

A procurement pilot should retain failed attempts and measure the resources they consume. It should also define which outcomes a human or program can verify before the system acts. Otherwise, a reported “success after retries” may depend on an evaluator the buyer does not actually possess.

Ask for the raw counts behind any efficiency claim. How many tasks entered the cohort? How many attempts were made? Which outcomes passed the acceptance criteria? What fraction required intervention? Which resource fields were captured, and which were inferred? A vendor may have good answers. The goal is to make the arithmetic reproducible rather than negotiate over an attractive percentage.

The historical dataset supports a limited conclusion: counting only successful conversations omits a substantial share of recorded interaction volume in these files. It does not establish the financial value of replacing a human workflow. That requires a matched baseline, reliable cost observations, and an acceptance standard that reflects the actual job.

CHAPTER 06Build a cost model that can be reconciled

The most useful cost model starts with records that can be joined. A task identifier connects the customer request to attempts. Attempt identifiers connect model calls, tool operations, and execution events. Review records connect human work to the task that required it. Billing records connect measured consumption to a provider's charge. If these relationships are missing, a detailed spreadsheet can still be an estimate assembled from incompatible totals.

Use two time views. The operational view groups events by when they happened and helps explain this month's bill. The cohort view groups spending by the tasks that generated it and helps explain cost per completed task. A task opened on the last day of a month may incur most of its review cost in the next month. Neither view is wrong, but mixing this month's spending with only this month's completed tasks can distort the ratio when the backlog changes.

Keep quantities before converting to currency

For each model call, preserve whatever usage categories the provider actually returns, along with the provider, model identifier, and request identifier. Do not assume that similarly named categories across providers have identical billing semantics. Store the raw usage payload or a versioned representation where permitted, then apply a documented pricing rule. When information is absent, mark it absent instead of estimating it silently from message length.

This article deliberately does not include a table of current model prices. The historical traces do not contain the required agent usage records, so current prices would create an appearance of precision without recovering the original bill. A future billing analysis should state whether it uses invoiced cost, the prices effective when a run occurred, or a later repricing scenario. Those are three different questions and can lead to different conclusions.

The same care applies to tool charges. A search request, document conversion, hosted browser, database query, or storage operation may be metered independently. A tool call in the transcript is not necessarily one billable event. The tool may retry internally, use cached results, or invoke another service. The task ledger needs the charge at the service boundary where it is observed, with enough linkage to avoid counting it twice.

Execution resources need explicit treatment too. If a worker is rented for a fixed period, its cost does not disappear while it is idle. A per-second calculation may describe active execution accurately while omitting reserved capacity. For a shared worker pool, document how idle time is allocated or report it separately. Otherwise, the total of task-level costs will not reconcile to the amount paid for the system.

Distinguish marginal and allocated cost

Marginal cost asks what additional spending a further unit of work causes under the current capacity arrangement. Allocated cost assigns shared expenses to units according to a chosen rule. A system can have low marginal cost and substantial allocated cost. That is common whenever shared capacity or engineering work supports many requests, but the relevant amounts must be measured for the particular system.

Suppose a hypothetical service spends $2,000 per month on a fixed execution pool and another $0.10 on direct usage for each task. At 10,000 tasks, the simple allocation is $0.30 per task before considering failures: $0.20 from the fixed pool and $0.10 from usage. At 20,000 tasks, it becomes $0.20 if the same pool handles the volume. The apparent improvement comes from spreading fixed cost, not from a more efficient model.

That calculation has a capacity assumption. If the higher volume requires a second pool, the fixed component per task returns to $0.20. A smooth spreadsheet curve can therefore conceal a step in actual spending. Show the capacity boundary and state what happens when it is crossed. Do not extrapolate a fixed-cost allocation indefinitely beyond the workload the system has demonstrated it can handle.

Engineering salaries present a related question. A pilot may require substantial one-time integration work, while ongoing maintenance is smaller or larger depending on the system. Separating implementation from recurring operation helps a buyer understand the commitment. It does not justify excluding implementation from an investment decision. Report the initial expense, the recurring expense, and the time horizon used to compare them.

Reconcile before optimising

Before ranking configurations, take a closed observation period and compare the ledger total with the corresponding bills and recorded human effort. Identify unassigned costs, duplicated charges, missing requests, and expenses that belong to a different workload. Keep a reconciliation difference visible. A model that explains most spending can still be useful, provided the unexplained amount and its likely causes remain explicit.

Reconciliation is also a check on the scope of the claim. If a vendor reports inference cost while the buyer asks about operating cost, both parties can be discussing accurate numbers that answer different questions. Name the numerator in the chart title. “Model API cost per benchmark pass” is narrower and more inspectable than “cost per task.” “Direct operating cost per customer-accepted request” requires additional evidence.

The historical trace study stops at interaction volume because that is what it can recover. It does not estimate a financial saving. Its practical contribution is to expose how the denominator and failure accounting change the interpretation of a recorded workload. A complete cost study should preserve that discipline even when richer billing information becomes available.

CHAPTER 07Work through a complete hypothetical cohort

The following example is constructed to explain the accounting. It is not a reported customer deployment, an industry average, or a conversion of the τ-bench data. Every monetary amount and operational count in it is an assumption. Keeping the example separate from the observed results allows a reader to change the assumptions without changing what the benchmark actually showed.

Imagine 1,000 incoming requests in a fixed cohort. Six hundred complete automatically. Another 200 complete after human review. One hundred are rejected under the acceptance criteria, and 100 remain unresolved when the window closes. The completed-task denominator is 800. The automatic-completion rate is 60%, while the total completion rate is 80%. Neither should be presented as the other.

Assume model calls, tools, and execution consume $1,000 across the entire cohort. Reviewers handle 250 requests, including some that do not ultimately complete, at an average of six minutes each. At an illustrative labour rate of $40 per hour, review consumes 25 hours and costs another $1,000. Direct cost for the cohort is $2,000. Dividing by 800 completed tasks yields $2.50 per completed task.

Notice that 250 reviewed requests and 200 review-assisted completions are compatible. Some reviews can confirm that a request should be rejected or remains unresolved. If the spreadsheet multiplied human time only by successful reviews, it would remove legitimate work from the numerator. The reviewer did not work for free because the task remained incomplete.

Compare like with like

Now construct a manual baseline that handles the same incoming cohort and reaches the same 800 accepted outcomes. Suppose it requires eight minutes per incoming request at the same illustrative $40 hourly rate. The baseline consumes approximately 133.33 hours and costs $5,333.33, or $6.67 per completed task. Under these assumptions, the AI-assisted workflow has lower direct operating cost.

The comparison is deliberately conditional. If the manual baseline actually completes more requests, delivers better quality, or handles cases the AI workflow excludes, the two ratios do not establish equivalent service. A credible pilot must measure those differences rather than assume them away. The example holds them fixed to isolate the arithmetic, not because real implementations will naturally match.

Add $3,000 of shared monthly platform and maintenance cost allocated to this cohort. The AI-assisted total becomes $5,000, or $6.25 per completed task. Its advantage over the hypothetical manual baseline is now only $333.33 for the cohort. The narrow difference leaves little room for omitted expenses. A conclusion drawn from the original $2.50 figure alone would miss the importance of the cost boundary.

Increase average review time from six to twelve minutes while leaving all other assumptions unchanged. Human review now costs $2,000, so direct cost rises to $3,000. With the same $3,000 shared allocation, the total reaches $6,000, or $7.50 per completed task. The ordering reverses. This does not establish that human review is usually the largest expense; it shows that review time is a decision-sensitive variable in this particular model.

Find the threshold worth measuring

The break-even review time can be calculated from these assumptions. Start with the $5,333.33 manual baseline, subtract $1,000 in nonhuman direct spending and $3,000 in shared allocation, and $1,333.33 remains available for review. Across 250 reviewed requests at $40 per hour, that allows eight minutes per reviewed request. Below that value, the model favours the AI-assisted option; above it, it favours the manual baseline.

A useful pilot would therefore measure review time carefully around that threshold, while checking that completion quality and volume remain comparable. Spending equal effort refining every input would be wasteful. If a variable can reverse the decision within a plausible range, it deserves better measurement. If it barely changes the result over a broad range, rougher estimation may be adequate for this stage.

The threshold is not permanent. A different workload, labour rate, shared allocation, or completion rate changes it. Its value is that it converts a vague debate about whether AI is cheaper into a concrete question about a measurable quantity. The team can then collect evidence that resolves the decision instead of accumulating more generic market statistics.

Keep unfinished work visible

The 100 unresolved requests create an additional obligation. Some may complete later, some may be cancelled, and some may require expensive intervention. The current ratio is a snapshot at the declared observation boundary. It should not quietly become the final lifetime cost of the cohort. Report the unresolved count and revisit the cohort when its outcomes mature.

There are several reasonable ways to handle this. A team can wait until every request reaches a terminal state, report a fixed-age cohort, or publish provisional figures with later revisions. The choice depends on the workflow. What matters is consistency and disclosure. Changing the observation rule whenever unresolved tasks make the result look worse would make comparisons unreliable.

For a buyer, the workbook accompanying a pilot should allow these assumptions to be changed individually. For a founder, the same workbook can identify where better tooling or product scope would materially improve economics. For an investor, it can distinguish a demonstrated operating advantage from one that depends on unmeasured future improvements. None of those uses requires pretending the hypothetical numbers are empirical evidence.

CHAPTER 08Retries change coverage, cost, and risk

A retry is a decision to spend more resources after an earlier attempt has failed or become uncertain. That decision can be sensible, but it needs a reason. A temporary service error may disappear. A missing document may become available. A second model may handle a task the first could not. Repeating an unauthorised action or an ambiguous write can instead make the situation worse.

The historical repeats in this article do not represent a deployed retry policy. They are separate recorded trials of benchmark tasks. The distinction matters because a real recovery step inherits state, elapsed time, and possible side effects from what happened before. It may also have information about the failure that an independent benchmark trial does not receive. Its economics must be evaluated as a sequence, not reconstructed by selecting favourable completed runs after the fact.

An idealised retry calculation

Consider a hypothetical task with independent success probability p on every attempt and identical cost c per attempt. Assume a perfect success detector, no harmful side effects, and a maximum of k attempts. Stop as soon as an attempt succeeds. Under these assumptions, the probability of eventual success is 1 - (1-p)^k. The expected attempt count is 1 + (1-p) + ... + (1-p)^(k-1), which equals [1 - (1-p)^k] / p when p is positive.

Multiply the expected attempts by c, then divide by the probability of completion. The expected workload spending per completed task is c / p. In this idealised model, allowing more retries improves completion coverage but does not improve that particular cost ratio. The additional successes are accompanied by proportionate additional spending. This is an algebraic consequence of the assumptions, not a measured property of deployed agents.

Set p to 0.5, c to one illustrative dollar, and k to three. Completion probability is 0.875. Expected attempts are 1.75. Expected spending per incoming task is $1.75, and spending per completed task is $2. With one attempt, the corresponding figures are 0.5, one attempt, $1, and the same $2 per completed task. More customers receive a completed result, but the ratio alone conceals the higher budget required to serve the cohort.

Now remove the assumptions one by one. If later attempts use longer context, c is no longer constant. If failures identify hard tasks, their conditional probability of success may be lower than p. If recovery introduces useful information, it may be higher. If a reviewer must determine success, the detector adds cost and error. If an attempt creates an irreversible side effect, another attempt may not even be a permissible action.

The practical lesson is to measure the conditional value of the next attempt. Ask what changes after a failure and why the next step is expected to help. A fallback model, a corrected tool argument, and a request for missing customer information are different interventions. Combining them under “retry rate” makes it harder to learn which one earns its cost.

A timeout does not tell you whether a write happened

Suppose an agent requests a booking and receives no response before its deadline. The booking may have failed, may still be processing, or may have succeeded while the response was lost. Generating a second booking request with a new identity can create a duplicate. This uncertainty exists at the service boundary; the language model cannot resolve it through confidence or explanation alone.

Amazon's Builders' Library describes using caller-provided request identifiers to recognise duplicate intent and discusses parameter mismatches and late-arriving requests. Its example is infrastructure APIs, not an AI-agent benchmark. The relevant mechanism is a stable operation identity that the service can enforce when a request is repeated. Malcolm Featonby, Making retries safe with idempotent APIs.

In an agent workflow, preserve that identity outside the model's free-form memory. Associate it with the intended operation and inspect the service's result before deciding whether to issue a new operation. If the user changes the requested action, that is a new decision with its own validation. Reusing an identifier for different intent can be as confusing as generating a new one for the same intent.

Idempotency does not mean every business effect is automatically reversible or that every third-party tool implements the same guarantees. A workflow still needs to understand the actual API contract. Where duplicate protection is absent, the recovery path may require reconciliation or human intervention. Include that work in the cost model rather than presenting safe recovery as a free property of the orchestration layer.

Limit retries at the system level

A local retry rule can also interact with other retry rules. The agent may retry a tool, the tool client may retry a request, and a downstream service may retry its dependency. A single incoming task can then produce more work than any one layer's counter suggests. Instrument the boundary between these layers and identify which layer owns recovery for a given failure.

Google's SRE chapter on overload describes per-request and per-client retry budgets, and warns about retry amplification across service layers. It provides operational examples for distributed systems rather than a universal setting for AI products. The transferable design question is how the service bounds recovery work when many requests fail together. Google SRE, Handling Overload, retry guidance.

For a pilot, record the stop reason: completed, attempt limit reached, time budget exhausted, prohibited action, unavailable dependency, or escalation. A low spending figure achieved by stopping early is different from one achieved by completing efficiently. A high completion rate achieved with unbounded recovery is different from one that fits a predictable budget. The decision needs both the service outcome and the expenditure required to reach it.

CHAPTER 09Human review and automated judging belong in the experiment

Review can make an AI-assisted workflow valuable. It can also conceal the work needed to make an output usable. The relevant question is what the reviewer must do. Approving a correctly assembled result after checking a few critical fields is different from reconstructing the entire answer from source material. Both may be labelled “human in the loop,” but their time requirements and failure modes differ.

Observe the review process directly in the pilot. Record time spent inspecting, correcting, searching for missing evidence, communicating with the requester, and recovering from tool errors. Preserve whether the reviewer saw the AI output before forming an independent view. A reviewer who begins from the model's answer is performing a different task from one who solves the problem independently and then compares results.

Review design affects the work being measured

An interface can expose the source for each field, highlight changes, show the proposed external action, and separate uncertainty from completed checks. These choices can make review more targeted. They can also lead a reviewer to overlook omitted information if the interface presents only what the model chose to include. Evaluate the interface with cases containing missing evidence and plausible-looking errors, not only correct outputs.

For the invoice workflow, a reviewer should be able to inspect the original amount, the extracted amount, the purchase-order match, and the proposed routing action. A summary that merely says “all checks passed” supplies less evidence. If the reviewer has to open several systems to verify the claim, the resulting effort belongs to the workflow's cost even if it occurs outside the AI application's browser tab.

A safe escalation may be an excellent result. It prevents an unsupported action and directs the task to someone who can resolve it. Count it as an escalation with its own outcome, rather than automatically as an AI failure or an autonomous success. Product goals differ: a triage assistant may be designed to escalate, while a fully automated service may be priced around a different promise.

This is why an acceptance rate without an intervention rate is incomplete. Two systems can complete the same fraction of requests while requiring very different staffing. Conversely, a system with more escalation may avoid a costly class of mistakes. The evaluation should preserve enough detail to show that trade-off rather than compressing it into a single score.

A model judge is another measured component

For open-ended outputs, a model can help evaluate relevance, coverage, or adherence to a rubric. The question is whether its decisions agree with the acceptance standard on the actual workload. A judge's fluent explanation is not itself evidence that the decision is correct. Treat the judge as a component with a version, prompt, inputs, and measurable error patterns.

Zheng and colleagues examined model judges in MT-Bench and Chatbot Arena, including position, verbosity, and self-enhancement biases. Their study also found strong agreement with human preferences under the evaluated conditions. That combination supports testing model-assisted evaluation, while giving no warrant to assume equivalent reliability on every enterprise workflow. Zheng et al., Judging LLM-as-a-Judge, Sections 3–4, 2023.

A practical validation set should include examples where a response is persuasive but wrong, terse but sufficient, correct with an unsupported explanation, and incomplete in a way that changes the decision. If two outputs are compared, test whether changing their order changes the verdict. Keep any reference answer or source evidence available to the judge consistent with the evaluation design.

Report false acceptance separately from false rejection. A judge that rejects correct work increases review or recovery effort. A judge that accepts incorrect work can inflate the measured completion rate and allow mistakes downstream. Which error matters more depends on the workflow. A single agreement percentage can hide an unacceptable pattern in the cases that matter most.

Calibration must survive changes

A model upgrade, revised rubric, new document format, or changed user population can invalidate earlier calibration. Keep a fixed set of diagnostic examples so that changes can be compared, and add fresh cases from observed disagreements. Do not tune repeatedly on the final evaluation set and then describe the resulting score as an independent estimate.

Human adjudication also needs versioning. If reviewers learn new policy interpretations during the pilot, old decisions may need to be revisited. This is not a reason to avoid learning. It is a reason to distinguish performance under the original rubric from performance under the revised one. A correction history makes the result easier to trust than a table whose labels change without explanation.

The cost of evaluation should remain visible. Model judging adds inference and execution. Human calibration adds specialist time. Disagreement resolution adds investigation. These expenses can be justified by better decisions, but they are part of the system that produces accepted outcomes. Counting generation alone while relying on an unpriced evaluation process understates the work required to deliver the product.

CHAPTER 10Context, retrieval, and caching change what an attempt contains

A model call receives more than the latest user sentence. Depending on the application, its input can include policy instructions, tool descriptions, retrieved documents, conversation history, and intermediate results. Two calls counted as one attempt can therefore carry very different workloads. This is another reason the message counts in the historical analysis cannot be converted into a bill without usage data.

When investigating a costly workflow, inspect the composition of the input. Separate information needed for the current decision from material carried forward by default. A long history may contain useful constraints, obsolete instructions, duplicate tool output, or irrelevant retrieval results. Removing text is not automatically an improvement: a shorter input that drops a necessary policy can reduce cost while increasing unacceptable actions.

Test information use, not context capacity alone

The Lost in the Middle study varied the placement of relevant information in multi-document question answering and key-value retrieval tasks. It found position-sensitive performance in the models it examined. Those historical results do not establish a fixed weakness in every current model; they provide a controlled way to test whether input organisation changes an application's outcome. Liu et al., Lost in the Middle, Sections 2–3, 2023.

For an invoice assistant, an analogous experiment could hold the invoice and acceptance rules constant while varying irrelevant attachments or the location of a relevant policy. For a research assistant, it could add plausible distractor documents while keeping the required answer unchanged. These would be new experiments, not results already contained in this article. The purpose would be to identify whether extra input helps, does nothing, or makes the decision less reliable.

Retrieval introduces another possible failure point. The needed evidence may never reach the model. An evaluation that scores only the final answer cannot always distinguish a retrieval miss from an interpretation error. Save the retrieved document identifiers and versions so that failures can be inspected. If the source changed after the run, preserve enough provenance to recover what the system actually saw.

Permissions should be applied before a document becomes usable evidence in the workflow. A result can be accurate and still rely on material the requester should not see. A benchmark of answer quality alone does not establish that access control works. Test permission boundaries separately and include those requirements in acceptance when the product operates across users or organisations.

Cache the right object

Caching can refer to several different things: a retrieved document, a tool result, a model input prefix, or a completed application response. These objects have different validity conditions. A document cache may remain useful until the source changes. A permission-sensitive response may become invalid when access changes. A booking availability result may be obsolete before the next customer request.

For each cache, name the key, expiry condition, and invalidation trigger. Record whether a run used cached or fresh information. A comparison between two configurations can otherwise confound a model change with a warmer cache. If the pilot's cache state is unlike the expected deployment, the measured resource use may not transfer even when the code is identical.

Do not treat cache hits as free unless the relevant costs are actually absent. Storage, retrieval, validation, and invalidation can all require work. More importantly, a stale result can trigger downstream correction. The useful measure is the effect on completed outcomes and total spending under the chosen validity rules. A higher cache-hit rate is an intermediate observation, not a business result by itself.

The architectural changes worth testing are specific: remove duplicate context while preserving rules, retrieve fewer but more relevant documents, validate freshness before acting, or reuse a safe intermediate result. Each change should have a baseline and an expected failure mode. “Use more context” and “add caching” are too broad to identify what improved the workflow or where its new risks sit.

CHAPTER 11Design a pilot that can change your mind

A useful pilot is a comparison between plausible operating choices on a defined workload. It should be capable of producing a result that leads the team to stop, narrow the scope, or choose a different architecture. If every possible result can be described as encouraging, the pilot is not resolving a decision. It is generating material for a presentation.

Begin with a short decision statement. For example: should this team route a specified class of requests through an AI-assisted workflow instead of its current process? Name the class, the current process, the acceptance criteria, and the constraints on cost, delay, and unacceptable actions. The statement should be narrow enough that the available evidence can answer it. “Does AI improve our business?” cannot be resolved by a small task-level experiment.

Choose a baseline people would actually use

A baseline should be a credible alternative. It might be the existing manual process, a deterministic rules engine, a simpler model workflow, or a vendor product. Comparing a carefully tuned agent with an intentionally weak prompt answers little about a buying decision. Comparing with the current process without recording its quality can also be misleading if that process has substantial errors of its own.

Give each alternative the information and tools appropriate to its intended use, then document the differences. A human with access to an internal policy database should not be compared with an agent denied that information unless that restriction reflects the real deployment. If the proposed agent requires a new data integration, include the work required to provide it rather than crediting the model with the benefit of previously unavailable data.

Separate adaptation from evaluation. Let the team improve prompts, tools, and interfaces using a development set. Freeze the chosen configuration before measuring its performance on a held-out set. If the result exposes a defect that must be fixed, record the change and evaluate the new version on fresh or properly reserved cases. Repeatedly inspecting the same held-out cases gradually turns them into development material.

Define the task population

Write down where tasks come from, which are eligible, and which are excluded. Exclusions can be justified by scope, permissions, or feasibility. They become a problem when the final claim forgets them. A system tested only on short English-language requests should not inherit a claim about long multilingual requests because they enter the same support queue.

Record characteristics that may affect difficulty before looking at outcomes. These could include the number of requested operations, document length, missing information, need for external writes, or policy ambiguity. The categories should reflect the workload rather than a desire for a tidy taxonomy. Their purpose is to reveal whether an average hides an important part of the service.

Selection can change after tools become available. METR's follow-up developer study describes participants becoming less willing to work without AI and selectively submitting tasks. That warning concerns its specific study, but it illustrates why the population entering an experiment deserves as much attention as the model version. A later sample may differ from an earlier one even when the recruitment description looks similar. METR, February 2026 update, selection effects.

For a company pilot, retain a count of eligible tasks that did not enter the experiment and the reasons where known. If staff choose when to use the agent, compare that selected workload with the wider queue before extrapolating. Voluntary adoption can be commercially meaningful while still making causal estimates harder. The analysis should distinguish evidence of preference from evidence of improved performance.

Measure at the right level

Where practical, compare alternatives on matched tasks. For workflows involving human participants, consider whether task assignment can be randomised without compromising the service. If the same person performs the same task twice, learning can affect the second attempt. If different people handle each condition, differences in experience can affect the result. The design should address the relevant source of variation instead of relying on a large-looking record count.

Repeated attempts on the same task are useful for studying reliability, but they do not create new independent task types. A confidence interval that treats every repeat as unrelated can overstate how much the study knows about performance across tasks. If uncertainty is estimated by resampling, the resampling unit should reflect the dependence structure. For repeated tasks, that often means keeping a task's related observations together rather than scattering them as independent rows.

The current trace analysis does not publish confidence intervals because it describes the supplied finite records and does not establish a sampling model for a wider customer population. That is a deliberate boundary. More sophisticated statistics would not repair unknown representativeness or outdated task definitions. A later study can add uncertainty estimates when its target population and sampling assumptions are explicit.

Measure outcomes after enough time has passed to observe relevant corrections. A support answer accepted immediately may lead to a reopened request. A code change merged today may require repair next week. A workflow with delayed consequences needs a follow-up window that reflects those consequences. Report what the window can capture and what remains outside it, rather than treating absence of a recorded complaint as proof of lasting success.

Decide what would count as a win

Set the decision criteria before seeing the final results. A system might need to preserve completion quality, reduce direct cost, meet a latency requirement, and avoid a specified class of unauthorised action. Some of these are hard constraints; others are trade-offs. Making that distinction explicit prevents a strong average score from compensating for a failure the business cannot accept.

Do not collapse all errors into an average monetary penalty unless the valuation is justified. Some failures are reversible and cheap. Others are difficult to detect or unacceptable regardless of their estimated frequency. A decision can use cost comparisons within a set of systems that already satisfy required controls. It does not need to assign a speculative dollar value to every possible incident.

At the end of the pilot, report the result against the original criteria, then explain any newly discovered issue that changes the decision. A missed target is useful information. It may identify a narrower task class that works, a review bottleneck worth fixing, or an integration that dominates the cost. The pilot has served its purpose if the next decision is better informed, even when the outcome is not deployment.

CHAPTER 12Budget for slow and expensive cases

An average can describe the whole cohort accurately while concealing the tasks that determine staffing or capacity. A request requiring a long investigation may be rare in the sample but operationally important. The right response is to inspect the distribution and the cases behind it, rather than replace the average with whichever percentile makes the product look strongest.

Record elapsed completion time and active human handling time separately. For elapsed time, specify the start and end events. Does the clock begin when the customer submits the request, when a worker starts, or when the first model call begins? Does it stop at generation, review, or final execution? A fast model response can coexist with a slow customer outcome if most time is spent in a queue.

For spending, retain per-task totals that include all attempts and interventions. Report the median, a high percentile appropriate to the sample size, and the maximum with its context. A maximum from a small pilot is an observation, not a credible upper bound. A percentile based on only a handful of cases in the upper tail should not be presented with excessive precision. Explain how many records support it and whether unresolved work is excluded.

Separate expensive successes from expensive failures. An expensive successful task may be worth serving because its outcome is valuable. An expensive failure may reveal a missing stop rule, an unavailable dependency, or an unsupported task class. The same aggregate cost can call for different engineering changes depending on where it occurs. Inspect representative traces before deciding whether to shorten context, change routing, or increase the budget.

A spending cap also changes the product. If a task is stopped after reaching its limit, record that stop as an outcome of the capped policy. Do not reuse the uncapped system's completion rate in the capped system's cost forecast. The same principle applies to deadlines: a response that would have succeeded after the customer left is not a timely completion under a deadline-based acceptance rule.

For the historical files in this article, no latency distribution is reported. The selected analysis does not establish reliable start and end timestamps for customer completion. Adding a latency chart would therefore require new evidence. The cost ledger proposed here includes those fields so that a later dataset can support the analysis, while the current figures remain confined to the observations actually available.

CHAPTER 13Architecture choices should survive an economic comparison

The article's accounting does not imply that the smallest model or simplest workflow is always preferable. A more expensive configuration can be worthwhile if it completes valuable work within the required constraints. The test is whether the additional capability survives a comparison that includes its full resource use, failure handling, and operating burden.

Consider three possible designs for a document workflow. One model produces a draft. A second design adds retrieval and deterministic validation. A third uses several agents to gather evidence and critique a combined result. These are proposals, not an ordering from primitive to advanced. The right choice depends on which parts of the task need additional capability and whether the result is worth the added work.

Change one mechanism at a time where possible

If a new system changes the model, prompt, retrieval corpus, tool implementation, and review interface together, a better score does not identify the cause. It can still establish that the combined system performs better under the test, but it gives weaker guidance about what to preserve or simplify. A focused comparison can isolate the contribution of one mechanism when doing so is practical.

For example, compare the same generator with and without a deterministic check for missing required fields. Then inspect which failures the check catches and what happens after rejection. If rejection simply restarts the entire workflow, the added check may improve correctness while increasing spending. If it returns a precise repair instruction, recovery may be cheaper. The experiment should include that recovery behaviour, not stop at the check's detection rate.

A multi-agent comparison needs similar care. Equal task sets do not imply equal budgets. Give the single-agent baseline a reasonable opportunity to use the available tools and spending allowance, and report the actual resources each configuration consumes. If the multi-agent version is allowed more time or more context, name that difference. Its advantage may still be valuable, but it should not be attributed solely to coordination.

The Anthropic report discussed earlier is a useful first-party engineering account, not a controlled experiment for every workload. Its internal performance comparison and its token-use comparison have different baselines. A buyer should carry that caution into its own testing: state the comparison group beside each result and avoid combining percentages whose denominators differ.

Include the cost of change

Operating cost includes more than the cost of one stable run. A policy changes, a tool schema evolves, a model version is retired, or a source system changes its document format. The team must detect the effect, update the implementation, and re-establish confidence in the result. A configuration that is cheap to execute but hard to validate after changes can impose a substantial maintenance burden.

Sculley and colleagues' Hidden Technical Debt in Machine Learning Systems examines system-level dependencies, configuration, and maintenance risks around ML components. It is a 2015 engineering paper, not a measurement of present-day agent costs. Its relevance here is the need to inspect the surrounding system and its dependencies when evaluating an apparently small model change. Sculley et al., 2015.

For a proposed workflow, identify the dependencies whose changes require retesting. Keep examples of the checks needed after a policy update, a provider change, or a tool modification. Record the effort during the pilot when such changes occur. Where no change occurs, estimate the maintenance requirement as an explicit assumption rather than claiming it has been observed.

A simpler system may be easier to inspect, but simplicity is not a substitute for meeting the task requirements. A rules-only solution that cannot handle essential variation may shift work to staff. A complex system may be justified when that variation is valuable to handle. The comparison should include the work displaced into adjacent processes, so that architectural simplicity in one component does not become hidden complexity elsewhere.

Scope can be a better intervention than another agent

A product may serve a well-defined subset of requests automatically and route the rest to an existing process. That can be a useful design if the boundary is reliable and the commercial promise is clear. The evaluation should measure whether the system identifies that boundary correctly, how much traffic lies inside it, and what happens to requests outside it.

Narrowing scope should not be concealed by reporting only the accepted subset. Show the whole incoming population and the portion the product handles. A system that completes 95% of a small, selected category may be valuable, but it is different from one that completes 95% of the entire queue. The buyer needs both the within-scope performance and the coverage of the proposed scope.

This approach can also make research more useful. Instead of asking whether an agent is generally reliable, ask which kinds of requests it handles under a specific policy, tool set, and budget. That result is less portable as a slogan, but more portable as a method. Another team can reproduce the conditions, inspect the boundary, and decide whether its own workload is similar enough to justify a pilot.

CHAPTER 14What founders and investors can reasonably infer

A technical result becomes commercially useful when it changes an assumption about the product. Lower resource use may improve contribution margin. More consistent completion may make a workflow sellable to customers who previously required manual handling. Better evidence capture may reduce review effort. Each inference needs a connection to the actual product rather than an automatic translation from benchmark score to company value.

The trace study here supports no estimate of market size, revenue growth, or valuation. It does not contain customer contracts or complete operating costs. It can still sharpen diligence by showing how apparently straightforward performance claims depend on task definitions, failed work, and missing instrumentation. Those questions are relevant before accepting a company's own cost model.

Ask for a reconciliation, not a single impressive run

A founder should be able to explain how a representative cohort moves from incoming requests to completed outcomes. The explanation should identify automatic completions, interventions, rejections, and unresolved work. It should connect those states to the spending necessary to produce them. A demonstration can show a capability; the cohort explains how consistently and economically that capability is delivered.

An investor can request this evidence without demanding access to sensitive customer content. Aggregated counts, anonymised traces, documented acceptance criteria, and reconciled cost categories may be sufficient for an initial review. The data-sharing arrangement must fit the company's obligations. Missing evidence should remain an open diligence question, rather than becoming an invitation to invent representative figures.

Ask how the reported workload compares with the intended customer population. Are the pilot users unusually experienced? Are they submitting only tasks they believe the system can solve? Are integrations maintained manually behind the scenes? Is the product receiving favourable capacity pricing that may not persist? Each answer changes the conditions under which the reported economics apply.

These are not reasons to dismiss an early product. Early systems often rely on manual work and limited scope while their builders learn. The distinction is between a demonstrated result and a plan to improve it. A company can make a credible case for the latter by identifying the bottleneck, showing why a proposed change should address it, and measuring progress against a stable baseline.

Separate saved effort from captured value

Reducing time spent on a task does not automatically reduce payroll or create revenue. The organisation may use the released capacity to serve more customers, improve quality, clear a backlog, or simply reduce overtime. Those outcomes can be valuable, but they are different from an immediate cash saving. The product's value proposition should name the mechanism through which the benefit is realised.

The same applies to employee time. A tool that saves scattered minutes may not create a contiguous block that can be reassigned. A tool that removes an entire recurring process may change staffing or service capacity more directly. A task-level study can measure the first-order time effect, while a broader operational study is needed to establish what the organisation does with it.

Pricing introduces another distinction. A vendor may charge per seat, per request, per completed task, or through a fixed contract. The buyer's cost and the vendor's operating cost are not the same. A technically efficient product can still be expensive for a buyer under a particular contract, while a generously priced pilot can conceal weak vendor economics. Review both sides when the decision requires it.

For a founder, the most useful outcome of this analysis is a list of assumptions that deserve instrumentation. For an investor, it is a way to distinguish observed operating evidence from forecast improvements. For an engineering leader, it is a method for comparing designs without allowing failed work or unpriced review to disappear. The article's figures support those questions; they do not answer them for every company.

CHAPTER 15Read this study as a reproducible accounting example

The strongest claim in the current analysis is deliberately narrow: in the supplied historical traces, unsuccessful attempts account for a substantial fraction of recorded assistant messages. The figures can be recovered from the files and the script. They show how including failed attempts changes the description of the workload. They do not show actual financial waste, because an assistant message has no fixed price and a failed attempt may still produce information useful in another workflow.

The analysis is exploratory. The dataset was selected because it was public, inspectable, and contained repeated outcomes. The metric was chosen after inspecting the available fields. This is not a preregistered study with a hypothesis fixed before data access. A future confirmatory experiment should freeze its protocol before examining the final outcomes and distinguish that result from this exploratory analysis.

What a replication would need to preserve

To reproduce these numbers, preserve the source commit, four file hashes, reward rule, grouping by task identifier, and definition of an assistant message. Changing any of these can change the results. A reproduction should report its own environment and compare its output with the saved result file. If it obtains different values, inspect input versions and filtering rules before attributing the difference to model behaviour.

The script uses complete files and checks for duplicate task/trial pairs and binary reward values. It does not manually remove difficult cases. It also does not verify that every recorded message corresponds to a billable model call, or that the historical reward implementation captures all business requirements. Those limitations are preserved in the chart labels and surrounding discussion because they affect what a reader can infer.

The four-run estimator is checked independently by enumerating all combinations of four trials within each task and comparing the mean all-pass fraction with the formula. This validates the arithmetic used for these finite records. It does not validate the assumption that future trials will be independent or drawn from the same distribution. Arithmetic verification and external validity are separate obligations.

What would materially strengthen the next version

The next dataset should add recorded agent usage or billing while retaining task outcomes and repeat identifiers. It should permit a clear separation between the agent, any simulated user, tools, and evaluation costs. A dataset containing only total spending for the entire experiment would help less than records that can be joined at the attempt level. The latter would allow the analysis to locate which failures consume resources and how costs vary across tasks.

A stronger study would also re-evaluate acceptance beyond the existing reward where the underlying records allow it. That could reveal cases that passed a narrow checker but violated a process requirement. Such a review needs a documented rubric and, where judgment is involved, independent adjudication. It should not quietly overwrite the original reward; both the source label and the revised assessment should remain available.

A production claim would require further evidence still: a representative workload, a credible baseline, actual review effort, complete cost boundaries, and an observation window for delayed corrections. Until those exist, this article is a technical reference for designing the measurement and a reproducible analysis of one historical dataset. Its usefulness depends on keeping that scope intact as the manuscript grows.

A reader who wants to apply the method can begin with one closed cohort. Name the acceptance criteria, retain every attempt, reconcile the spending, and inspect the unresolved cases. If those four pieces cannot be assembled, the first engineering task is instrumentation. If they can, the team has a basis for a concrete decision about the next model, tool, reviewer, or workflow change.

CHAPTER 16How to reproduce and challenge this iteration

Run python3 analyse.py in the article's research folder with Python 3 and internet access for the first download. Subsequent runs use the saved raw files. The output contains file hashes, source URLs, record counts, observed success rates, the four-run estimator, and interaction totals. No API credentials or paid model calls are needed.

The analysis checks for duplicate task/trial pairs, binary rewards, and uniform repeat counts within each file. It does not independently regrade the conversations or correct known task defects. It also does not estimate confidence intervals: these charts describe the supplied finite records, and the task sample is not established as representative of customer demand. An inference to a larger population would need an explicit sampling model.

Useful challenges include identifying a mislabeled reward, testing whether a different observable resource measure changes the result, and supplying complete cost-bearing traces under a compatible licence. The next empirical increment should add a dataset with recorded agent usage or billing, rather than decorate the current message counts with assumed prices.

CHAPTER 17References and how to use them

The references below support specific mechanisms or reported findings. They are not interchangeable endorsements of the article's cost model. The worked dollar examples are this article's hypothetical calculations. The historical trace figures are local calculations from the frozen files. Sources were checked on 7 September 2026; model and product capabilities should not be inferred from a paper's age alone.

• Yao, Shinn, Razavi and Narasimhan — τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (2024). Read Section 3 for reward construction and the distinction between pass^k and pass@k. The paper explicitly limits what reward 1 guarantees. Its original experiments are not the same object as the later repository snapshot used here.
• Sierra — frozen historical trajectories. These four files are the numerical source for this article. Use the saved hashes and analysis script to reproduce the tables. The repository's current warning says the original tasks are outdated.
• Sierra — tool-calling agent implementation at the same commit. This code accumulates response costs internally. It does not make missing historical cost fields recoverable.
• Anthropic — How we built our multi-agent research system (13 June 2025). A first-party engineering report. Its internal evaluation comparison and token-volume comparison use different baselines; they cannot be combined into a universal efficiency ratio.
• Becker, Rush, Barnes and Rein — METR's early-2025 developer productivity study (10 July 2025). Read the population, assignment method, and explicit nonclaims with the reported 19% slowdown. The article uses this as a bounded study, not a claim about all developers.
• Becker, Rush, Cunningham, Rein and Mahamud — METR's experiment-design update (24 February 2026). Read alongside the earlier study. Selection and time-measurement problems limit interpretation of the follow-up results.
• Zheng and colleagues — Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (2023, version 4). Sections 3 and 4 cover judge limitations and agreement evaluation. Human-preference agreement in this setting is not a certificate of correctness for enterprise actions.
• Liu and colleagues — Lost in the Middle: How Language Models Use Long Contexts (2023, version 3). The controlled question-answering and retrieval experiments motivate testing input organisation. They do not establish a fixed performance penalty for current models.
• Featonby — Making retries safe with idempotent APIs, Amazon Builders' Library. Read the discussion of client request identifiers, changed parameters, and late requests when designing recovery for external writes.
• Google SRE — Handling Overload. The retry-budget discussion explains how local recovery can amplify system load. The examples are operational guidance, not recommended default settings for every AI application.
• Sculley and colleagues — Hidden Technical Debt in Machine Learning Systems (2015). A systems-maintenance perspective on ML dependencies. It supports examining the surrounding architecture, not a numerical forecast of agent maintenance cost.

• Sierra — GPT-5.6-sol submission, 4 August 2026. Reported banking results, evaluated 22 July; cost is null.
• Sierra — Claude Opus 5 submission, 4 August 2026. Reported banking results, evaluated 3 August; cost is null.