LLM Coding Agents Fail Under Production Constraints

A May 2026 study identifies a specific failure pattern in AI coding agents — one that correlates reliably with the depth of the backend being built, not with the underlying model's general capability.

A Benchmark Built to Isolate the Constraint Effect

Researchers Francesco Dente, Dario Satriani, and Paolo Papotti published a study introducing the term "constraint decay" to describe a pattern they observed when evaluating LLM coding agents against real backend generation tasks. The core observation: as structural, non-functional requirements accumulate — specific software architecture patterns, designated databases, ORM layers, API contracts across multiple files — agent performance drops sharply, even for agents that perform well on simpler tasks.

To measure this cleanly, the team built a Behavior-Driven Development evaluation framework that separates functional specifications from structural constraints. Each task runs in a fresh Docker environment with an ephemeral PostgreSQL instance, so no carryover state contaminates results. The benchmark covers 100 tasks in total — 80 greenfield backend generation tasks and 20 feature-implementation tasks — and measures agent output using assertion pass rate (A%), which tracks whether the generated code satisfies behavioral correctness requirements, not just whether it compiles.

This methodology matters because it isolates why agents fail, not just that they fail. The three research questions the paper addresses — what the performance drop looks like at scale, which coding frameworks amplify or dampen it, and what failure modes drive it — each produce findings that have direct implications for teams using AI agents in software delivery pipelines. The three key figures from the study are summarized below.

The 30-Point Drop: Shallow Tasks Versus Constrained Production Backends

The most direct finding concerns how fast performance collapses as constraints accumulate. Capable agent configurations start at reasonably high assertion-pass rates on baseline tasks — where the objective is loosely defined and the implementation space is open. As the task specification adds structural requirements (a mandated architecture pattern, a specific ORM, multi-file dependencies, a persistent database layer), those same agents average a 30-point decline in A%.

The drop is not uniform. Weaker agent configurations, which may still perform acceptably on prototyping or frontend tasks, approach near-zero assertion-pass rates under fully constrained backend specifications. This suggests constraint decay is not simply a matter of the task being harder in general — it is a compounding effect where each additional structural requirement narrows the agent's ability to generate code that satisfies all requirements simultaneously.

This helps explain a pattern many engineering teams have noticed informally: AI-assisted code generation works well for scaffolding, for UI components, for isolated utility functions, and for tasks where the agent's output can be easily reviewed and corrected. It tends to degrade precisely where the system is complex enough that incorrect output is also harder to catch — in multi-file backend services where a silent ORM misconfiguration may pass code review but fail at runtime. The chart below shows the assertion-pass rate divergence between capable and weaker agent tiers across baseline and fully-constrained tasks.

Framework Choice Shapes How Quickly Constraints Cause Failures

The second research question — whether the choice of backend framework changes how constraint decay behaves — produces one of the study's more actionable findings. Under identical API contracts, agents perform substantially better in lightweight, explicit frameworks like Python's Flask than in convention-heavy environments like FastAPI.

The mechanism is interpretable: FastAPI's design favors implicit conventions, automatic validation layers, and dependency injection patterns that an agent must satisfy correctly across multiple files simultaneously. Flask, by contrast, makes routing and request handling explicit and local. An agent generating Flask code can satisfy structural constraints one function at a time. In FastAPI, a misunderstanding of the dependency injection chain or a missing Pydantic schema field produces a runtime failure that the agent may not anticipate from the immediate context of any single file.

This is a design tradeoff that human developers navigate consciously — FastAPI's conventions speed up experienced engineers precisely because they reduce boilerplate, but they encode assumptions that are only visible when something goes wrong. For LLM agents, those assumptions appear to be a consistent source of constraint-induced failure. Teams evaluating AI agents for backend automation should account for framework choice as a variable in expected reliability, not just model selection. The directional comparison below reflects the study's relative findings.

Database Queries and ORM Rules Drive Nearly Half of All Agent Failures

The third line of inquiry asked what, specifically, causes constraint decay at the code level. Analysis of agent trajectories and error logs showed that data-layer defects — incorrect database query composition and runtime violations of ORM rules — account for approximately 45% of agent failures in constrained backend tasks.

That concentration is significant. It suggests constraint decay is not evenly distributed across the surface area of backend code. Instead, failure clusters at the interface between application logic and persistence. ORM rules are particularly difficult for agents because they are not explicit in the immediate context of a single file: the rule that a query must be structured a particular way to satisfy a model relationship, or that a transaction must be wrapped correctly to avoid a runtime integrity violation, is an implicit constraint encoded in the ORM framework's behavior, not in the API specification the agent was given.

This points to a specific weakness in how current LLM coding agents handle long-range constraint propagation. An agent can generate a correct API endpoint and a syntactically valid ORM query in isolation; the failure emerges when those two things must be consistent with each other and with a database schema defined elsewhere in the project. The implication for teams building on top of AI coding agents — including those integrating local multi-integration agent frameworks — is that data-layer verification cannot be delegated to the agent. It requires explicit human review or automated test coverage with real database state. Viewed against the compounding costs of AI reliability failures, the failure cluster the study identifies has a direct operational cost implication for teams shipping production backends with significant agent involvement. The breakdown below shows the study's reported data-layer failure share against the remaining, uncharacterized failure modes.