local_bench: A zero-config TUI to benchmark the LLMs already on your machine

Benchmarking local LLMs has always been a split-personality problem. Tools like llm-council or lm-eval-harness handle quality, while utilities like llama-bench focus on throughput. But nothing tied those two halves together in a way that respects the actual hardware you own — until now.

SupermodularAI has released local_bench, a single-command TUI that discovers models installed in your local runner (Ollama, LM Studio, llama.cpp, vLLM, or any OpenAI-compatible server), runs a curated quality suite, measures tokens/sec, time-to-first-token, memory footprint, and optionally joules per token on your real machine, and renders it all as a live leaderboard. No config files, no API keys, no cloud — just clone, run, and get an at-a-glance answer to "which of my local models is actually good, and how fast is it on this laptop?"

Under the hood, the tool keeps things deliberately small. The default suite runs in about 1–2 minutes on a laptop, with tasks graded deterministically (exact numeric match, multiple-choice letter, substring, regex, valid-JSON). Temperature is set to 0 with a fixed seed for reproducibility. For open-ended tasks, an optional --judge MODEL flag enables LLM-as-a-judge scoring (1–5 against a reference answer), using any local or hosted model — even Claude via the Anthropic provider, though that path disables temperature and caching for determinism.

Power users get deeper control: custom task packs in JSON or YAML, auto-saved run logs, and a diff command to compare models across runs. The tool even ships a browser-based pack viewer that warns you if prompts will overflow the context window — a real problem with default Ollama settings, which load models at 4096 tokens regardless of what the model supports. That's a subtle but critical detail: a prompt that exceeds the context doesn't error, it silently discards the older half, and you end up grading a fragment instead of the model.

Notably, local_bench is not on PyPI — you install it by cloning the repo, which keeps it lean and close to the source. It's a refreshingly honest tool in a space full of over-engineered benchmarks, and it's exactly the kind of local-first utility that makes the "which model should I actually run?" question answerable in minutes. For anyone juggling multiple quantizations or fighting with context windows, this deserves a look.