self-bench lets you build private coding-agent benchmarks from your own repo

Benchmarks like SWE-bench are useful, but they're someone else's code. If you want to know which coding agent actually works well on your team's codebase, you need tasks drawn from your own repositories. That's the idea behind self-bench, a new open-source tool that builds private coding-agent benchmarks from work already completed in your Git history.

self-bench scans your local coding sessions and merged GitHub pull requests, then reconstructs each task from the commit before the change. For every accepted task, it creates hidden tests and a reference solution, proves the task fails without the fix and passes with the original implementation, and exports a native task for Harbor, a runner for coding-agent evaluations. The output is a private .tar.gz benchmark you can run against multiple models.

How it works

The quickstart runs the self-bench API, worker, and Temporal workflow state locally while using Modal for disposable task-generation and validation sandboxes. After installing self-bench and authenticating with Modal and GitHub, you set the model and GitHub credentials used by the local worker, plus a random token that protects the local API:

# Set environment variables for the worker and API

Then start Postgres, Temporal, the self-bench API, and a worker in Docker. The worker sends sandbox work to Modal; SELFBENCH_API_URL tells subsequent CLI commands where to reach the local API.

Candidates are classified by difficulty based on changed implementation lines and paths: easy, medium, and hard require at least 20, 50, and 100 changed lines across 1, 2, and 3 paths, respectively. These counts are generation budgets, not guarantees that every candidate will pass validation.

The repository must be a Git checkout with a GitHub origin. self-bench pins its current HEAD, ignores uncommitted changes, and may take hours to author, validate, review, and export the accepted tasks. The --output flag waits for completion and verifies the downloaded archive with SHA-256.

To run evaluations, install Harbor and extract the generated tasks:

# Extract and run Harbor's Codex agent adapter

Harbor's Codex agent adapter runs once for all three models. It evaluates every extracted task at high reasoning and keeps the model results in one job directory. The evaluated agent receives the base repository and task instruction, but not the hidden tests or reference solution.

Closing the waiting CLI does not cancel a submitted workflow. If the local worker or Docker stack stops, work pauses until the worker is restarted. Stop the local stack with the provided command; named Docker volumes retain Temporal history and generated artifacts.

Why this matters

self-bench fills a gap in the AI coding tools space: most benchmarks are static, public, and quickly become contaminated as models train on them. A private benchmark built from your own codebase is more representative and stays fresh as your code evolves. For teams evaluating coding agents, this could become a standard part of the procurement process — and a way to keep agents honest when they claim to "know" your codebase.

The project is MIT-licensed and open source, with documentation covering backend configuration, credentials, persistence, and complete Temporal Cloud deployment. If you're already using Harbor for agent evaluation, self-bench makes it easy to generate tasks that reflect your actual development workflow.