Find the Needle: What AI Actually Does to Software Engineering (as of May/2026)

By far the greatest danger of Artificial Intelligence is that people conclude too early that they understand it. Eliezer Yudkowsky, Artificial Intelligence as a Positive and Negative Factor in Global Risk

The Shift Is Real. The Marketing Is Not

Three years ago, AI in software engineering meant autocomplete. You typed, it suggested. Most developers ignored it half the time.

Today, AI coding agents open pull requests without being asked. They take a GitHub issue, analyse the codebase, write the fix, run the tests, and flag it for human review. Goldman Sachs is running hundreds of these agents alongside 12,000 human engineers. Accenture rolled Claude Code to tens of thousands of developers. Deloitte — 470,000 employees. This is not pilot territory anymore.

But here is the gap that will cost organisations money: the demo is not the deployment.

Goldman Sachs CIO Marco Argenti has been explicit about this. His projection of 3–4x productivity uplift is paired with a clear framing — supervised agents, bounded tasks, human approval on every pull request. A hybrid workforce, not a replacement. That distinction matters more than any benchmark score.

The organisations treating the Devin launch demo as their deployment model are the ones Gartner is counting in its projection that over 40% of agentic AI projects will be cancelled by end of 2027 — primarily due to ROI and governance failures.

What Got Us Here — Three Generations, Not One

This did not happen in a straight line. Three distinct capability generations compressed into roughly five years. Each one changed what the technology could do. More importantly, each changed what it meant to govern it.

Generation 1 — Autocomplete (2021–2023): The Pair Programmer

GitHub Copilot launched in 2021, built on OpenAI's Codex — GPT-3 fine-tuned on public code. The mental model was simple: human types, AI suggests the next line. Useful. Not transformative. Copilot had 20 million users by 2026 and over 90% of the Fortune 100 as customers. What it could not do was understand your codebase as a system. It saw functions, not architecture.

Generation 2 — Conversational Coding (2023–2024): The Assistant

Copilot X introduced a chat interface in early 2023, transitioning from Codex to GPT-4. The interaction changed: human asks, AI explains, generates whole files, summarises pull requests. The shift in underlying capability that enabled this was not just scale — it was the expansion of context windows from 8,000 to 200,000 tokens. A model could now hold an entire module in working memory, not just a function. Reasoning started becoming possible at the file level.

Late 2023 added a critical dimension most enterprise leaders ignored: open-source models — DeepSeek Coder, Code Llama, StarCoder — began closing the gap with frontier commercial models. By January 2025, DeepSeek-R1 reached reasoning parity with OpenAI's o1 at MIT licence. The same capability that cost millions to train became freely deployable. Governance became harder the moment the frontier became open.

Generation 3 — Agentic Execution (2024–2026): The Agent

March 2024 was the visible inflection point. Cognition released Devin, capable of opening a terminal, browsing documentation, running tests, and iterating without being asked for each step. The category shifted from suggestion to execution.

What enabled this was a convergence of three things happening simultaneously:

Reasoning models. Claude 3.7 Sonnet, released in February 2025, was the first model with "extended thinking" — the ability to reason through complex, multi-step problems before generating output, not just pattern-match from training data. OpenAI's o3 and o4-mini followed. This is architecturally different from generation 1 and 2. These models can hold a constraint in mind, reason against it, and revise — closer to how a senior engineer approaches an ambiguous problem than how autocomplete works.

Tool use and execution loops. Models gained verified, reliable access to terminal commands, file systems, browsers, and CI pipelines. The planning-execution-verification loop became possible: AI plans a multi-step change, executes it across files, runs the test suite, reads the failure, revises, retries. This self-healing capability is what separates an agent from an assistant. An assistant provides code and stops. An agent stays until the tests pass.

Context at repository scale. Context windows reached 1–2 million tokens in commercial deployments, with experimental models exceeding 10 million. That is the difference between reasoning over a function and reasoning over your entire codebase — undocumented dependencies, internal naming conventions, three-year-old architectural decisions — in a single session. Whole-system reasoning without fragmentation became real, not theoretical.

On the horizon — Multi-Agent Orchestration

Anthropic's Opus 4.5 and 4.6 releases introduced "Agent Teams": specialised agents — architect, coder, QA tester — working under a coordinating orchestrator. Gartner recorded a 1,445% surge in multi-agent system inquiries between Q1 2024 and Q2 2025. The frameworks are proliferating: LangGraph, CrewAI, OpenAI Agents SDK, Google ADK. None has emerged as the standard. Most production deployments called "multi-agent" are still hierarchical — one orchestrator, several specialists. Peer-level multi-agent orchestration across a full SDLC remains experimental in 2026. But the direction is set.

The METR time-horizon study captures what all of this means in aggregate: the autonomous-task duration AI agents can complete with 50% reliability has been doubling every seven months since 2019, accelerating to every four months through 2024–25.

The pace is not slowing.

But the numbers need context before you act on them. A model scoring 88% on SWE-bench Verified scores roughly 46% on SWE-bench Pro — the contamination-resistant version of the same benchmark. A March 2026 preprint found that nearly 20% of "solved" patches from top-ranked agents pass the test suite without actually fixing the underlying bug. The leaderboard has become a marketing instrument. The vendors optimising for the headline score and the vendors selling you on it are, increasingly, the same people. Treat benchmark claims as directional signals, not procurement criteria.

What AI Actually Does Well Today

Not what it will do. What it does reliably, right now, in production — with a human review gate.

Bounded ticket work. Dependency bumps, framework upgrades, language migrations, test coverage expansion, documentation. Well-defined input, clear verification anchor. This is where Devin's 67% pull-request merge rate comes from. The tasks are scoped, the expected output is known, and a failing test tells the agent when it is wrong.

Legacy code comprehension. This is the underutilised ROI most enterprises are sitting on. With million-token context windows, AI can analyse undocumented 500GB codebases, map dependencies, generate interactive call graphs, and flag architectural constraints that have never been written down. McKinsey estimates AI accelerates migration timelines by 40–50%. Most organisations are not using it here because the use case feels unglamorous. That is the needle.

Developer experience acceleration. AI achieves roughly 83% test coverage versus 54% with traditional manual approaches. Code review augmentation — surfacing issues for human triage — is already standard in high-performing engineering teams.

What AI does not do reliably: architecturally novel features, ambiguous or evolving requirements, production system design, security-sensitive code paths, anything where the verification anchor is a human judgment rather than a passing test. Cognition's own 2025 performance review on Devin framed it honestly — "senior-level at codebase understanding, but junior at execution." 85% failure rate on complex, ambiguous work without human intervention.

Know the difference before you set the expectation.

The Other Side of the Argument

The evidence for AI's transformative impact on software engineering is real. So is the evidence against the hype. Any executive making deployment decisions needs both.

The productivity data is more contested than your vendors admit.

The METR randomised controlled trial — the only RCT-grade study in the entire literature — found that AI made experienced developers working in mature codebases 19% slower in early 2025. That same cohort believed they were 20% faster. The perception-reality gap is the most important finding in the study, and almost no one leads with it.

McKinsey and GitHub's controlled studies show up to 2x speedups. But their conditions matter: bounded tasks, novel code, low existing tribal knowledge. METR's conditions also matter: experienced engineers, mature high-quality codebases, complex tasks. Both findings are true. Context determines which one applies to your organisation.

DORA 2025 — the most rigorous industry-wide analysis, covering 5,000 technology professionals — found that AI adoption positively correlates with throughput and negatively correlates with software-delivery stability. More PRs merged. More incidents. Faros AI's telemetry across 10,000+ developers found 98% more PRs merged — and time-in-PR-review up 441%. Faster input into a system that cannot absorb it faster is not delivery improvement. It is pipeline pressure.

The benchmark numbers are not what they appear.

SWE-bench Verified scores reached 88% by April 2026. Contamination-resistant SWE-bench Pro tells a different story: a model scoring 80.9% on Verified scores roughly 46% on Pro. A model at 55% Verified scores 23% on Pro. The gap between the leaderboard and the harder benchmark is 30–35 points. That is the number to cite the next time a vendor pitches you a score.

The replacement narrative does not match employment data.

US Bureau of Labor Statistics data shows software developer employment is down less than 0.5% since 2019. Software developer job postings on Indeed were up 11% year-over-year as of early 2026. BLS projects 15% growth in software developer employment through 2034.

The compression is real but targeted. Stanford's Digital Economy Lab found early-career workers in high-AI-exposure roles fell 6% from late 2022 to mid-2025, while workers aged 30+ in the same roles grew 6–13%. The story is not replacement. It is stratification. Junior roles are compressing; senior roles are gaining leverage.

The Stack Overflow Developer Survey — 135,000+ respondents — found that 46% of developers do not trust AI-generated output. Favourable views of AI tools dropped from 70% to 60% between 2024 and 2025. Adoption is near-universal. Trust is not.

The honest position: the bullish and bearish cases are both defensible from the data. Anyone presenting only one side is selling something.

Three Things You Should Already Be Doing

1. Redeploy senior engineers as orchestrators, not individual contributors.

The productivity differential between senior and junior engineers using AI is structural, not marginal. Senior engineers capture disproportionate gains because they have the architectural judgment to verify, redirect, and govern AI output. Junior engineers often lack the context to catch what AI gets wrong on the first pass.

This is not a reason to eliminate junior headcount. It is a reason to redesign how senior engineers spend their time — less writing, more governing. Less individual output, more fleet orchestration. The organisations doing this are seeing compounding returns. The ones still measuring AI impact as "lines of code per developer" are missing it.

2. Put AI on your legacy estate before your greenfield projects.

Most AI coding investments go into new feature development. Most of the unrealised value is in legacy code comprehension and bounded migration. If your architecture team cannot tell you, in an hour, what breaks if you touch a specific legacy module — AI-powered codebase analysis is the fastest path to that knowledge. The ROI is not glamorous. It is real.

3. Treat every AI-generated code path as a security event.

Veracode tested over 100 large language models and found 45% of AI-generated code samples introduced OWASP Top 10 vulnerabilities. That rate did not meaningfully improve across three successive generations of frontier models. Apiiro documented a 10x surge in monthly security findings between December 2024 and June 2025 in enterprise deployments. In August–November 2025, Anthropic publicly disclosed that a threat actor used agentic coding tools to automate 80–90% of operations in espionage campaigns against approximately 30 organisations.

AI coding agents with shell access and repo write permissions are now the most privileged identities in many engineering organisations. If security review is not a hard gate before AI-generated code reaches production, it is not a risk you are managing — it is a risk you have accepted without knowing it.

Three Risks You Cannot Governance Away Later

The junior pipeline problem will not show up in your 2026 P&L.

Optimising junior engineering headcount downward in 2026 is financially rational and strategically dangerous. Stanford's data is clear: early-career workers in the most AI-exposed roles experienced a 16% relative decline in employment after controlling for firm-level factors. The engineers you do not hire or develop in 2026 are the senior engineers you will not have in 2031. That is not recoverable quickly, and it will not be visible until the damage is done.

Measuring AI at the individual level while managing at the system level.

The most common governance mistake in AI coding deployments right now: measuring productivity at the developer level — PRs merged, tasks completed — while delivery stability, time-in-review, and incident rates deteriorate at the system level. The number that looks good in the board deck is not the number that tells you whether AI is working. Design your measurement model around system throughput, not individual velocity.

Adoption without operating model redesign.

McKinsey's 2025 data is stark: 79% of organisations report using generative AI for development. Only 5.5% report real ROI. The gap is not model quality. It is workflow. Deploying an AI coding agent into an unchanged development process adds a fast-moving input to a system not designed to absorb it. Governance, review gates, security checks, and measurement must be redesigned around AI, not bolted onto the workflow that existed before it.

What I Take Away From All of This

The binary question — does AI replace engineers or not — is the wrong frame. The data does not support full replacement. It also does not support the status quo.

What it supports is a structural reorganisation of how software engineering value is created: senior engineering judgment becomes the scarce resource, governance becomes an architectural constraint not a policy layer, and the operating model is the competitive differentiator — not the tooling choice.

The organisations that will create real value from AI in software engineering are not the ones with the fastest adoption. They are the ones that redesigned around it before scaling it.

The needle is not the benchmark. It is the operating model you build around it.

Harnessing machine learning can be transformational, but for it to be successful, enterprises need leadership from the top. This means understanding that when machine learning changes one part of the business, then other parts must also change. Erik Brynjolfsson, Stanford Digital Economy Lab