Lumabri Lets You Run Huge Mixture-of-Experts Models Across a Swarm of Peer Machines

Running a large mixture-of-experts (MoE) model like DeepSeek or GLM typically demands a hefty GPU cluster. A new open-source project, Lumabri, takes a different approach: it turns a swarm of ordinary machines—GPUs or not—into a distributed inference engine. The core idea is that you don't need to download an entire model to use it; instead, the bytes that an inference actually touches are fetched from peers on demand and cached locally.

The project, built on the colibri engine in pure C with no dependencies, lets one machine share a model directory while others chat with it. The first question may be slower as the working set crosses the network, but subsequent queries are served from a local mirror at full speed—even if the original server goes offline.

What makes Lumabri notable is its inclusivity: any machine can join, CPU or SSD first. A swarm with no GPUs at all is a working swarm, and the output is byte-identical whether a GPU accelerates it or not. This stands in contrast to GPU-pooling networks that recruit from the few; Lumabri recruits from everyone.

How It Works

The system has three roles: a tracker that indexes which peers hold which files, a maintainer that serves byte-range reads on the model directory, and an expert node that executes the model's compute. A chat client mounts the model through an LD_PRELOAD shim that intercepts the libc calls the engine makes on a model directory, mirroring files as sparse local files.

A key design choice is that donating compute requires the model on disk, but donating disk requires nothing—you start empty and the swarm fills you. The tracker assigns the least-replicated files first, so every donated gigabyte lands where the swarm is thinnest. This rarest-first strategy maximizes resilience.

NAT traversal is handled simply: maintainers keep an outbound control connection to the tracker, and if a direct peer-to-peer dial fails, bytes are relayed through the tracker. This means peers behind home NATs can serve with zero router configuration. Private swarms are supported via a shared token (LUMABRI_TOKEN).

Getting Started

The README walks through a minimal setup: on the machine with a model, you run lumabri serve; on the chatter's machine, you run lumabri chat (or --tracker <ip>:7300). The chat asks for the swarm's address and the operator's public key once, then remembers them in ~/.lumabri/config. A synthetic tiny model is included for testing.

Once connected, /swarm shows live network stats (model held, GB, bytes served), and /model lets you switch between models, even across different architectures (GLM to OLMoE to DeepSeek) because the engine binary is chosen per model.

The Real Test: No Starvation

The project includes a concurrency_test.sh that runs the same generation from N chatters simultaneously. On a 6-core box with everything sharing those cores, the spread between fastest and slowest stays flat while absolute time grows—indicating CPU contention but not a lock convoy. That's a signal that the architecture scales without bottlenecking on shared locks.

Lumabri is a compelling experiment in democratizing large-model inference. It won't replace dedicated GPU clusters for the biggest models, but it offers a graceful way to pool whatever resources you have—and a fascinating look at what distributed systems can do with minimal dependencies.