Nolan Lawson benchmarks AI coding costs for JS frameworks

Nolan Lawson, a senior engineer at Couchbase, published a detailed benchmark on his personal blog that quantifies the financial and temporal costs of using AI coding agents to build JavaScript web applications. The study challenges the prevailing narrative in the frontend community that framework selection is driven by developer experience (DX) or abstract performance claims, arguing instead that in the "agentic era," the cost of building with an LLM is a critical, measurable metric that is often ignored. According to the post, the benchmark uses Claude Opus 5.5 to build a simple Pokémon list application across various frameworks, revealing that some stacks are significantly more expensive and slower for AI agents to implement than others.

The methodology and setup

Lawson designed the experiment to be as neutral as possible, avoiding specific architectural mandates like Server-Side Rendering (SSR) vs. Client-Side Rendering (CSR). The agent was given a specification for a content-based site with search and filtering capabilities, along with a conformance test suite based on the accessibility tree to ensure visual consistency without mandating specific HTML structures. To ensure fairness, Lawson allowed the agents to use official MCP (Model Context Protocol) servers or skills for each framework, though he noted that usage was inconsistent. For instance, the Nuxt agent never used its MCP, while the TanStack Start agent only discovered its skill files in two out of five runs. The benchmark excluded CSS frameworks like Tailwind or Bootstrap to focus strictly on the JavaScript framework layer.

The frameworks tested included React Router, Next.js, SvelteKit, Nuxt, Angular, Hono+htmx, Gracile+Lit, TanStack Start, and a "Vanilla" setup using Vite. Lawson ran each configuration five times (or three times for Angular, which failed twice) and reported the median results. He emphasized the stochastic nature of LLMs, noting that token usage and wall-clock time varied significantly between runs due to the random nature of model outputs.

Cost and performance results

The primary finding is that framework choice has a direct impact on the cost of AI-generated code. Lawson reports that SvelteKit, Gracile+Lit, and Angular were the most expensive to build. He attributes SvelteKit’s higher cost to the agent frequently reading the v3 migration guide, likely because the framework is too new for Opus 5.5’s training data. Similarly, the Gracile+Lit agent often cloned the repository or read compiled source code from node_modules due to a lack of knowledge about the framework. Angular’s high cost was linked to the total code size generated.

In contrast, Hono+htmx and React Router emerged as more cost-efficient options, though with overlapping price ranges. The "Vanilla" JavaScript setup using Vite surprisingly underperformed several frameworks in terms of cost and time, suggesting that having a structured framework can be more efficient for an LLM than working with raw infrastructure. Lawson notes that the total time and token usage tracked closely with the financial cost, indicating that more expensive builds also took longer to complete.

Quality and maintainability

Beyond cost, Lawson evaluated the output using Lighthouse for Performance, Accessibility, and Cumulative Layout Shift (CLS). He found that these scores were "nearly saturated" across all frameworks, meaning that the AI agents produced high-quality, accessible, and performant code regardless of the stack. This suggests that the primary differentiator in the agentic era is not the quality of the output, but the efficiency of the process. The variance in Lighthouse scores was minimal, with the main differences appearing in more granular metrics rather than the headline scores.

Implications for developers

This benchmark shifts the conversation from "which framework is best for humans?" to "which framework is best for agents?" For developers using AI coding tools, the choice of framework can no longer be based solely on personal preference or legacy reasons. If a framework requires the agent to read extensive documentation or struggle with unfamiliar patterns, it directly increases the cost and time of development. Lawson argues that this evidence-based approach is necessary to counter the "vibes-based" decision-making that has plagued the frontend community for years. The study suggests that teams should consider the "AI tax" of their stack, favoring frameworks that are well-represented in LLM training data or have robust MCP support.

What to watch

• MCP adoption: The inconsistent use of MCP servers by agents highlights a gap in tooling integration. As MCP becomes more standard, framework-specific MCPs may reduce the "AI tax" for less common stacks.
• Framework maturity: The high cost of SvelteKit and Gracile suggests that newer frameworks may be more expensive for AI to build until they are better represented in model training data.
• Standardization of benchmarks: Lawson’s work may prompt other developers to create similar benchmarks for different application types, such as complex SPAs or real-time applications, to provide a more comprehensive picture of AI coding costs.