An AI agent can produce an answer, a code change or a finished-looking report without delivering what the recipient can actually use. Imagine a report that is finished but cannot be opened by
An AI agent can produce an answer, a code change or a finished-looking report without delivering what the recipient can actually use.
Original AI-generated conceptual illustration for CoinRank; not a depiction of a real system.
Imagine a report that is finished but cannot be opened by its intended recipient. An AI agent can produce an answer, a code change or a finished-looking report without delivering what the recipient can actually use. The gap is not necessarily a failure of intelligence. It may be an untested assumption about what “done” means, who accepts the result or what still has to happen outside the agent’s workspace.
The useful question is not whether every task should run without a person. It is whether the agent makes the whole job easier—including review, handoffs and verification—or simply moves work into a less visible place.
In his October 31 HackerTube post, Andrej Karpathy describes rebuilding a video-viewing interface around recommendations found in Hacker News comments. He says the project took roughly two hours. His account also identifies manual setup of an API key, a server and a domain, performed alongside the agent’s work.
That is a creator’s report, not an independently timed experiment or a security audit. It nevertheless offers a useful starting point for analysis: the person asking for the tool knew what kind of viewing experience he wanted to replace.
The distinction matters when a tool leaves its creator’s hands. A personal preference can guide one person’s acceptance; a shared workflow needs an explicit recipient, an agreed standard and a way to verify the result. HackerTube does not establish enterprise delivery performance. It helps frame the question: whose definition of “usable” is the agent trying to satisfy?
For evaluating a delegated task, I would separate three checkpoints rather than collapse them into one success label.
The artifact exists. There is a report, patch, spreadsheet or other output at a known location and version. This says something was produced—not that it is correct or accessible to its intended recipient.
The result is accepted. The responsible person or process has checked the output against agreed criteria. A passing automated test may be one criterion, but it need not cover completeness, maintainability, relevance or suitability for a particular use.
Delivery is verified. The authorized destination contains the accepted result, and the intended recipient can use it. For a hypothetical report, that could mean confirming the correct version is accessible—not merely that a file was created. The evidence required depends on the task; some tasks end at acceptance and do not need a separate deployment.
These are proposed reporting distinctions, not a new benchmark. The stopping condition should be agreed before work begins. If a necessary approval is missing, stopping and identifying the next responsible person can be correct behavior. It should be recorded as a valid boundary, not mislabeled as delivery of the final result.
A March 2026 METR research note illustrates the distinction. Four maintainers across three repositories reviewed 296 AI-generated patches covering 95 issues, alongside previously merged human patches used to calibrate review differences. Automated test success did not reliably imply hypothetical maintainer acceptance.
The qualifications are important. These were single-attempt patches from sampled 2024–2025 agents, without a feedback-and-revision round. Reviewers were making hypothetical decisions, not actually merging or deploying the code. They lacked CI and were asked to disregard testing requirements. The publication is a research note with less review than METR’s research posts.
This does not establish a universal agent failure rate or a permanent capability limit. It shows why an evaluation result must be interpreted through its endpoint. A grader and a recipient can ask different questions of the same output.
For an operating team, the implication is practical: define what the next owner will check before declaring the workflow complete. Otherwise, an apparent improvement in generation speed can leave the acceptance work unchanged and unmeasured.
METR’s July 2025 randomized study involved 16 experienced developers and 246 tasks in familiar open-source repositories. In that February–June 2025 setting, access to AI tools increased completion time by 19%. Completion meant developers considered their work ready for review, not that it had been merged or deployed. This was a particular population using tools of that period, primarily Cursor with Claude 3.5/3.7 Sonnet—not a verdict on current agents.
Its February 2026 update is essential context. METR found that participant and task selection, changed compensation and concurrent agent use made its later experiment unreliable for estimating current productivity effects. The researchers considered greater acceleration plausible but could not reliably estimate the magnitude from those data.
The lesson is not to reuse the old slowdown as today’s headline. It is to be precise about the clock. Elapsed time, active human effort and waiting are different quantities. They can overlap.
Consider a hypothetical review queue: an agent waits while a person completes unrelated work. Counting the entire wait as that person’s labor would exaggerate cost. Conversely, recording only the agent’s run time would hide active minutes spent clarifying the request, checking the result and repairing an error. Neither measure, alone, describes the whole workflow.
A defensible comparison keeps task scope and acceptance quality consistent. Without a comparable baseline, a team can describe what happened; it cannot yet claim a productivity gain or loss.
Project Vend’s second-phase report, published by Anthropic in December 2025, provides a different setting: an agent-assisted physical retail operation. The system gained tools and procedures for customer records, inventory and checking prices and delivery information before commitments. The report describes more realistic quotes and waiting times.
Human support remained explicit. The agent had no interface for making purchases and checked with a human before buying; staff supported physical delivery, stocking and difficult customer situations. Models, prompts and tools changed together, so the report cannot isolate a single cause of business improvement or demonstrate labor-adjusted autonomous profitability.
A more accurate commitment may arrive later than an optimistic one. That is not automatically a worse outcome. Necessary authorization, quality review and physical work should have named owners and clear handoffs rather than be treated as defects to conceal.
There is a separate question about what an agent is technically allowed to do. CoinRank’s analysis of approval, permissions and isolation addresses that boundary. Here the question is different: after a permitted action, what evidence shows the requested result was accepted and delivered?
A team does not need a sweeping autonomy score to investigate this. It can start with one bounded task and record the following:
This is a proposed observation method, not a validated measurement instrument. It should expose uncertainty, not manufacture a precise total from incomplete records. A manual or alternative-workflow comparison is useful only if its scope and quality threshold are comparable.
My view is that the important shift is from supervising activity to defining acceptable outcomes. The aim is not zero human involvement. It is to make the remaining involvement purposeful, visible and proportionate to the task.
The cited cases cannot be pooled into one delivery-success statistic: they measure different things, under different conditions. They do suggest a better question than whether an agent looked busy or announced success. Who can use the result now—and what work, if any, is still waiting for someone else?
Sources checked November 2, 2026, at 9:33 a.m. Eastern Time (GMT−05:00).
〈AI Agent Task Completion: What Counts as Delivered?〉這篇文章最早發佈於《CoinRank》。
