Boundary-Bench measures how much coding agents lose when you lock down the sandbox

Security and capability are usually framed as a trade-off: the tighter the sandbox, the less an AI coding agent can do. Boundary-Bench, a new open-source benchmark from Accomplish AI, is out to quantify exactly that trade-off. It runs coding agents from Terminal-Bench, a well-known benchmark suite, inside progressively hardened Linux sandboxes, then measures how much success rate drops as network, filesystem, and privilege restrictions are applied.

The project is built on a stack of existing tools: Daytona for sandbox provisioning, OpenRouter for model access, and Harbor for task verification. It's packaged as an Inspect AI extension, so results come out as standard .eval logs that you can view with inspect view or export to CSV. The benchmark includes 89 tasks from Terminal-Bench 2.1, and you can run the whole suite or a subset.

Policies as a lattice

The interesting part is how policies are defined. They're not monolithic profiles but points in a three-dimensional lattice: Network × Filesystem × Privilege. Each dimension can be hardened independently, and the enforcement uses native Linux controls—nftables for network, read-only bind remounts and Landlock for the filesystem, and setpriv, no_new_privs, and capability drops for privileges. The goal is to make denials indistinguishable from ordinary OS errors like EROFS or EPERM, so the agent can't learn to game a fake policy shim.

The published benchmark levels range from control (no hardening) to increasingly restrictive combinations. You can also compose your own policy, picking from options like no-network, no-write-fs, or readonly-os, and the prerequisites are filled in automatically—for example, freeze-home implies readonly-os and non-root. Egress rules never block the model endpoint, so the agent can always reach the LLM, and inbound traffic isn't restricted because the sandbox exposes no services.

Adapted verifiers and not-applicable tasks

A few tasks needed special handling. Five tasks use an adapted verifier because the stock one checks a root-owned path that a non-root agent couldn't touch. The corrected verifiers are applied under every policy so grading stays consistent across policy arms. Another seven tasks—like build-pmars and mailman—simply can't pass under hardened policies because their instructions require system-wide installs or writes under /etc or /var. These are marked policy_applicability: not_applicable and excluded from success rates, though the benchmark will prompt for confirmation before running them unless you pass --yes.

How to run it

To run the full benchmark with the default control policy, you'd do something like:

boundarybench run --model your-model-alias

You'll need OpenRouter and Daytona API keys. The run provisions a Daytona sandbox, applies the policy, lets the agent work, and then Harbor's official verifier runs outside the hardening boundary. Expect to spend a few dollars and a few minutes per task.

You can also compose a policy interactively:

boundarybench run --interactive-policy

Or non-interactively with your own egress allowlist:

boundarybench run --no-network --readonly-os

The results are stored as Inspect logs, and you can export a CSV summary with boundarybench export-results. The leaderboard and task browser are live at boundarybench.com.

Why it matters

For developers building agent harnesses, this is a useful tool to understand where your agent's capability loss comes from—is it the network restriction, the read-only filesystem, or the privilege drop? That's the kind of insight that can guide where to invest in better tooling or prompt engineering. It also sets a precedent for benchmarking agents under realistic security constraints, which is increasingly relevant as enterprises look to deploy these tools in production environments.

Boundary-Bench is MIT-licensed and open source, so anyone can reproduce the results or extend the policy lattice. The project is early, but the approach is solid: measure the trade-off, don't just assume it.