ExpertCache lets a 120B Mixture-of-Experts model run on a 16GB Mac — slowly, but surprisingly intact

Running a 120-billion-parameter model on a laptop with only 16GB of unified memory sounds like the kind of benchmark sleight-of-hand that usually ends with a downsampled model or a remote API call in the fine print. But Amos Labs' new research release, ExpertCache, claims to do exactly that — with the full GPT-OSS 120B MXFP4 checkpoint — by exploiting the sparse, layer-wise structure of mixture-of-experts (MoE) models and a page-aware Metal runtime for Apple Silicon.

The repository is frank about what it is and isn't. It's an experimental research harness, not a production inference runtime. The headline result: on a physical 16 GiB M1 Pro, ExpertCache ran the 63.4GB checkpoint as a local, fully on-device inference path, scoring 14/16 on a functional evaluation. That score includes a documented correction for a "Sign-ups versus signup" discrepancy; without it, the raw score drops to 11/16. Either way, the takeaway isn't that the model is a frontier champion — it's that an oversized checkpoint can still be useful even when it doesn't fit in memory by conventional means.

The key trick is that ExpertCache doesn't load all 120B parameters into the active Metal working set. Instead, it uses a page-aware approach that schedules expert tensor loads on demand, avoiding the need to bind or copy the full expert tensor into memory. In practice, that means the runtime can execute the model with far less resident memory than the checkpoint size would suggest — at the cost of speed. Decode rates hovered around 3 tokens per second, which the authors themselves describe as "far too slow for interactive production use."

The release is carefully scoped. The 16 GiB path disables automatic fit, explicitly selects all GPU layers, and disables whole-file mmap prefetch. It's a deliberately configured result, not a stock-runtime claim. A second clean-boot repeat is still needed before the team will call the 16 GiB result reproducible. On a 64 GiB M1 Max, the same checkpoint runs more comfortably, but even there the authors stop short of claiming production readiness or publication-grade performance parity.

What makes this more than a novelty is the rigor in the repo. Every claim is tied to a specific experiment matrix, a pinned model revision, and a qualification protocol that includes negative results and protected one-token gates. There are runbooks for repeating the 16 GiB experiment, explicit "no-go" zones (like automatic fit), and a clear boundary between what's decision-grade and what's still a diagnostic.

ExpertCache also includes a 20B portability gate that just passed all four correctness arms, with canonical-response equivalence on three paths and a noted difference in stock behavior. That's a useful sign that the technique generalizes beyond a single model size.

For developers, the most interesting angle isn't the speed — it's the memory mechanics. If page-aware scheduling can keep a 120B MoE useful on 16GB of unified memory, it hints at how future on-device inference might sidestep the traditional memory ceiling for sparse architectures. It's not a production tool yet, but it's a tangible demonstration that the hardware can do more than conventional runtimes let it.

If you want to try it yourself, the repo requires Node.js 22+, Python 3.11, and a physical Apple Silicon Mac. You'll need to download the pinned GPT-OSS 120B MXFP4 GGUF artifact and follow the reproducibility docs. The authors are clear that a 32 GiB Mac is not a valid stand-in for a 16 GiB hardware claim, so don't skip the runbook.

This is research software with narrow, honest claims — and that's exactly what makes it worth reading. ExpertCache doesn't solve the interactive inference problem, but it does show that "the model is too big for this machine" isn't always the end of the story.