GPT-5.6 Luna vs GPT-6 Astra: Code Review Cost and Accuracy

Entelligence published a detailed comparison of OpenAI's GPT-5.6 Luna and GPT-6 Astra models specifically for automated code review tasks. The analysis, available on their blog, tests the hypothesis that a significantly cheaper model can handle the majority of pull request reviews without a prohibitive drop in quality. The headline finding is that GPT-5.6 Luna, priced at $0.20 per million input tokens, identified 69 verified bugs across 50 benchmark pull requests, while the premium GPT-6 Astra, priced at $10 per million input tokens, identified 92 verified bugs. The cost difference is stark: the entire Luna run cost $0.20, whereas the Astra run cost $5.66, a 28x difference in total expenditure for the same volume of work.

The Details

The benchmark utilized 50 public pull requests from the AI-Code-Review-Evals organization, drawn from five major open-source projects: Cal.com, Sentry, Discourse, Keycloak, and Grafana. Each PR introduced specific defects against a clean base branch. Both models received identical prompts requesting findings on correctness, security, concurrency, resource management, and error handling, while explicitly excluding style, naming, and documentation suggestions.

Verification of findings was conducted by two independent judges, GPT-6 Astra and GPT-5.6 Sol, who evaluated an anonymized list of findings from all models. An issue was only counted as verified if both judges agreed it was a real bug. This rigorous process resulted in 143 distinct verified bugs across the entire pool. Luna’s precision was 74% (69 verified out of 93 raised), while Astra’s precision was 96% (92 verified out of 96 raised). Luna was faster, averaging 23 seconds per review compared to Astra’s 36 seconds, and generated 3.1x more output tokens per review, yet remained far cheaper due to its lower per-token pricing.

Context

This comparison follows a previous Entelligence post that compared GPT-6 Astra with GPT-5.6 Sol. The current study specifically isolates the trade-off between cost and accuracy when using the cheapest available model for every task. The data highlights a significant divergence in performance depending on the type of code being reviewed. In repositories like Sentry, Discourse, and Grafana, Luna came within two verified bugs of Astra. However, in Keycloak, an identity and access management server, the gap widened significantly: Luna found 6 verified bugs to Astra’s 14, with only 50% of Luna’s findings holding up against 93% for Astra. This suggests that while Luna is competitive for general application logic, it struggles with complex security and permission models.

What it means for developers

For teams implementing AI code review, the data suggests a tiered approach is necessary. Luna is "good enough" for everyday correctness bugs and general logic errors, offering a massive cost saving for high-volume, low-risk repositories. However, Entelligence explicitly advises against using Luna alone for authentication or permission code. The precision gap is the primary concern: approximately one in four Luna comments was wrong, compared to Astra’s error rate of 4 in 96. For developers, this means that if a team relies solely on Luna, they will encounter a higher volume of false positives, leading to "review fatigue" where engineers may start ignoring AI comments entirely. The cost-per-verified-bug metric favors Luna ($0.0030 vs $0.061), but the risk of missing critical security flaws in sensitive codebases like Keycloak makes Astra the safer choice for those specific modules.

What to watch

• Training Data Contamination: Readers noted that the benchmark PRs are from public repositories with fixes potentially in the models' training data. Entelligence confirmed that the defects were artificially added for the benchmark, mitigating direct recall of the bug, but the surrounding code context remains a potential advantage for models trained on recent versions. A true test requires PRs created after the models' training cutoffs.
• Run-to-Run Variance: Repeated tests on a subset of PRs showed that results vary between runs. Luna showed higher variance, with 7 of 15 bugs repeating in both runs, compared to 10 of 15 for Astra. This indicates that single-run benchmarks may overstate consistency.
• False Negatives: The benchmark does not publish a complete list of all bugs, making it impossible to measure the total number of bugs missed by both models. Entelligence estimates that at least 26 verified bugs were missed by both Luna and Astra, caught only by other reviewers. The true number of missed bugs is likely higher, as unflagged bugs never enter the verification pool.