Coding agents have strong brand biases, but a 'simulated human' in the loop changes the winners

A new research initiative from Armature has released a detailed analysis of how AI coding agents select third-party services and libraries when building real-world applications. By analyzing thousands of public GitHub repositories to create a statistically representative panel of 75 fake codebases across 10 languages, the team conducted 16,893 experimental runs to observe agent behavior in controlled, sandboxed environments.

The core finding is that coding agents exhibit significant brand bias. When left to their own devices, agents consistently default to a narrow set of well-known providers and cloud-native solutions, often ignoring alternatives that might be more cost-effective or technically superior for the specific task. For instance, in object storage experiments, agents overwhelmingly favored Amazon S3 over competitors like Cloudflare R2, regardless of the context.

However, the study highlights a critical nuance: the presence of a "simulated human" in the loop dramatically shifts these outcomes. By using an LLM (Gemini 3.7 Flash) to act as an orchestrator that asks the agent to analyze the codebase and recommend solutions before implementation, the researchers found that the dominance of default providers decreased. This suggests that agents are not merely executing hardcoded preferences but are responsive to conversational context and permission structures. When allowed to ask for authorization to pick a specific third-party solution, agents were more likely to explore and select non-default options.

The methodology involved rotating between three different sandbox providers (E2B, Blaxel, and Daytona) to ensure environment neutrality. The researchers also tested the impact of prompt specificity, finding that mentioning costs or usage volumes in 20-25% of cases influenced the final output. The full dataset, including 5,292 valid sessions and their traces, has been made public for further analysis by the developer community.

This research provides a valuable baseline for understanding the "default behavior" of current coding agents. It suggests that while agents are powerful, their decision-making is heavily influenced by the interaction pattern. For developers, this implies that simply asking an agent to "implement X" may yield suboptimal or biased results compared to engaging it in a consultative workflow where it is prompted to evaluate options before acting.