When Code Gets Cheap, Judgment Gets Expensive
Agentic Coding Speeds Up the Typing. The Evidence Suggests the Scarce Work Moves to Specifying, Checking and Maintaining.
Ask a room of engineers how much faster AI coding tools make them and the answers tend to come back as multipliers, sometimes two, sometimes ten for the right kind of task. From the keyboard those numbers feel accurate. A function that used to take twenty minutes of typing now appears in a few seconds, and that change is hard to miss.
The controlled evidence is more uneven. In an early lab experiment, developers given an AI pair programmer, a tool that suggests code as the person types, finished a small web server task about 56 percent faster than a control group [1]. Field trials at three large companies, with several thousand developers doing their ordinary work, found a smaller and noisier gain in completed tasks, and the less experienced developers benefited most [2]. A randomized trial with experienced open-source maintainers, working in large repositories they knew well, pointed the other way. With AI tools allowed, their tasks took about 19 percent longer, even though the same developers believed afterwards that the tools had sped them up [3].
One way to reconcile those results is to stop treating productivity as a single dial. Writing code is one station in a longer line. Before it comes deciding what to build and describing it precisely enough for someone, or something, else to build it. After it comes checking what came back and living with it for years. AI tools have made the writing station far cheaper. So far they have done much less for the others. When one station on an assembly line speeds up, the line does not run ten times faster. The queue moves to the next slowest station. For software, the evidence suggests that station is human judgment, and the skills gaining value are the ones that work there.
That framing picks up where The Meta-Engineer, an earlier Thinkata series, left off. Don't Vibe, Architect looked at how professional developers plan, delegate and validate agent work, and The Edge of the Underdefined asked which parts of engineering stay human as agents improve. This piece comes at the same shift from the measurement side, asking what the productivity numbers actually capture and what they leave out.
From Autocomplete to Delegation
The tools have changed shape, and that matters for where the queue forms. Early assistants completed a line or a block. Agentic coding tools, systems that can read a repository, edit several files, run the tests and retry on failure without a person typing each step, take a task description and return a finished change. One measurement effort tracks the length of task an AI system can complete on its own, expressed as how long the same task takes a skilled human. That horizon has been doubling roughly every seven months since 2019 [4]. As it grows, more of the human's day shifts from typing alongside the tool to handing it a job and judging the result.
Don't Vibe, Architect described two ways developers work with agents, acceleration on familiar tasks where the goal is speed, and exploration of unfamiliar territory where the goal is learning. That split matters for the productivity numbers. Large speedups live in acceleration, because the person already holds the answer and only needs it typed. Exploration depends on being able to tell which of several plausible options is actually right, and that check runs at human speed.
One study logged what programmers spent their time on during assisted sessions. Verifying suggestions took more of the session than writing new functionality did, and checking plus editing the tool's output together took up a large share of the total [5]. From a systems perspective that is close to what a pipeline model predicts. Generation got cheap, but every generated line still passes through a reviewer, and the reviewer still runs at human speed.
This also offers one reading of the open-source slowdown. A small, self-contained lab task is nearly pure acceleration, because the spec is clear and the context fits in a paragraph. A mature repository is full of unwritten conventions, old decisions and edge cases that live in the maintainers' heads. In that setting the time saved on typing can go straight back into explaining context to the tool and checking whether it was respected.
The Pen Does More of the Work
A stack of technical drawings becomes something buildable only after someone has gone over each sheet, circling the dimension that does not match and the hole that will not line up. Agent-written pull requests seem to follow the same path. Across several hundred pull requests produced with an agentic coding tool in open-source projects, most were eventually merged, yet close to half of the merged ones needed human revisions first, most often for bugs, documentation and project-specific conventions [6]. The agent drafted the change. Someone who knew the project still had to go over it.
The Bill Arrives Later
What happens when review does not keep pace? A difference-in-differences study, a method that compares projects adopting a tool with similar projects that did not, before and after adoption, followed open-source projects that started using an AI-first code editor [7]. Development velocity jumped after adoption and then faded. Static analysis warnings, automated flags for likely bugs and risky patterns, rose and stayed high, and so did measures of code complexity. The authors' modeling points to that accumulated complexity as a major factor in the later slowdown. In effect, the early speed was borrowed against future maintenance.
Security shows a similar pattern with a human twist. In a controlled experiment, participants with an AI assistant wrote less secure code than those without one, yet were more likely to believe their code was secure [8]. Participants who trusted the assistant less, and who rephrased and adjusted their prompts more actively, introduced fewer vulnerabilities. What separated them looks like calibrated distrust, a sense of when output that looks finished has not really been checked.
Both findings sit on the same gap the open-source trial exposed, the distance between how productive AI-assisted work feels and what it delivers once someone has to own it. That gap is plausibly where engineering skill now earns most of its value.
Skills Gaining Weight
If the scarce work has moved toward specification and verification, the skills that matter are likely to move with it. None of them is new, and Don't Vibe, Architect found experienced developers already working this way, planning before they delegate and validating everything that comes back. What seems to be changing is their share of the job, and how early in a career they are needed.
A Task Written Down Is a Task That Can Be Checked
A pen on a blank page forces choices that a quick request in conversation can skip. Specification, describing a change precisely enough that someone without the surrounding context could carry it out, has long been a senior engineering skill. With agents it becomes an everyday one. The quality of the task description puts a ceiling on the quality of the change, and a spec that names its acceptance tests also hands the reviewer something concrete to check against.
Decomposition sits right beside it. Agents do well on tasks with clear boundaries, and the length of task they can handle, while growing, is still finite [4]. Splitting a feature into pieces that each fit inside that horizon, and that can each be verified on their own, is a design skill closer to architecture than to typing.
A less obvious skill is shaping the environment the agent works in. Work on SWE-agent, a research system that lets a language model fix real bugs in real repositories, found that the design of the agent-computer interface, the set of commands the agent uses to search, view and edit files and to run tests, had a large effect on how often it succeeded [9]. The same underlying model did markedly better with tools built around its limitations. For a working team, the equivalent is the test suite, the lint rules, the written project instructions and the fast feedback loops that let an agent catch its own mistakes before a person has to. Engineers who build that scaffolding raise the ceiling on every task anyone delegates afterwards.
Then there is review, which now covers a larger share of the work and asks something slightly different of the reviewer. An agent does not get tired or cut corners on purpose, but it also has no memory of which shortcuts the team agreed to avoid. The reviewer has to hold the architecture in mind and ask whether a change that passes its tests also fits the system. The Edge of the Underdefined argued that this kind of taste lives in the fit between code and human intentions, which is part of why it resists automation. The complexity findings above suggest it is also easy to lose under delivery pressure [7].
Where Judgment Comes From
A harder question follows about how that judgment gets built in the first place. The Edge of the Underdefined raised the practitioner worry about junior engineers shipping code nobody on the team understands, and asked how judgment develops if the routine work that used to train it goes away. Controlled evidence on that question has started to appear. A randomized experiment on developers learning an unfamiliar programming library found that those working with AI assistance scored worse afterwards on conceptual understanding, code reading and debugging, without a significant gain in speed on average [10]. Participants who handed the coding over entirely saw some speedup but learned the least.
The same study found that the cost was not spread evenly. Several interaction patterns preserved learning, including asking only conceptual questions, and generating code and then asking the assistant to explain it [10]. What those patterns share is that the person kept doing the thinking and used the tool to check or extend it.
Set that beside the field trials, where less experienced developers gained the most output from AI tools [2], and a tension appears. The engineers with the most to gain in short-term output may also be the ones whose long-term skill formation is most exposed. A team pushing for multiplied output from its newest engineers could find, a few years on, that it has fewer people able to do the review and architecture work the agents depend on.
Looking Ahead
Across these studies the multiplier looks real at the station where code is typed, and much smaller, sometimes negative, once the work includes understanding a large system, verifying the result and maintaining it afterwards. That suggests a practical way to read any 10x claim, which is to ask which station was measured and over what time span.
For individual engineers, the evidence points toward skills that resemble what strong senior engineers already practice, clear written specs, careful decomposition, investment in tests and tooling, skeptical review and a feel for architecture, now exercised at higher volume and much earlier in a career. For teams, it suggests measuring outcomes over a longer horizon than a sprint, since the velocity gains and the complexity costs arrive on different schedules.
The Edge of the Underdefined closed on the idea that engineers move to wherever meaning is still underdefined. The productivity evidence adds a practical footnote. Working at that edge seems to depend on judgment built through close contact with code, and the early results on learning-preserving interaction patterns hint that the same tools can either support that contact or erode it, depending on how they are used. Whether that holds outside a single experiment, and across a whole career, is a question the field has only begun to study.
References
- S. Peng et al., "The Impact of AI on Developer Productivity: Evidence from GitHub Copilot," arXiv, 2023, [Online]
- Z. Cui et al., "The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers," Management Science, Articles in Advance. DOI: [Online]
- J. Becker et al., "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity," arXiv, 2025, [Online]
- T. Kwa et al., "Measuring AI Ability to Complete Long Software Tasks," arXiv, 2025, [Online]
- H. Mozannar et al., "Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming," arXiv, 2022, [Online]
- M. Watanabe et al., "On the Use of Agentic Coding: An Empirical Study of Pull Requests on GitHub," arXiv, 2025, [Online]
- H. He et al., "Speed at the Cost of Quality: How Cursor AI Increases Short-Term Velocity and Long-Term Complexity in Open-Source Projects," arXiv, 2025, [Online]
- N. Perry et al., "Do Users Write More Insecure Code with AI Assistants?," in Proc. ACM SIGSAC Conference on Computer and Communications Security (CCS'23), 2023, pp. 2785–2799. arXiv: [Online]
- J. Yang et al., "SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering," arXiv, 2024, [Online]
- J. H. Shen and A. Tamkin, "How AI Impacts Skill Formation," arXiv, 2026, [Online]
The Part You Meant to Keep
AI help tends to look good in the moment and cost something later. The evidence so far offers no single cutoff, and the line seems to move with what a person wants to keep. Could a cognitive offloading scale find it?
The Map and the Walk
Language models learn by predicting human text, while reinforcement learning learns from the outcomes of its own attempts. A look at how the two methods differ, what each struggles with on its own, and why chat assistants, reasoning models and robots increasingly rely on both.
Working on something related?
Thinkata builds production AI systems of the kind these articles discuss. A 20-minute call is the fastest way to find out whether it fits.