[launches] · · 1 min read
SpeechCompass: Google DeepMind's open-source system adds speaker direction to live captions
A new mobile system uses classical DSP on a multi-microphone phone case to diarize and display who is speaking where in group conversations.
By ByteBulletin Editors · Editorial Team
Google DeepMind has open-sourced SpeechCompass, a system that brings speaker direction and diarization to mobile captioning, addressing one of the most significant unmet needs for users who rely on speech-to-text in group settings. The project, published at CHI 2025, combines custom hardware — a phone case with multiple microphones — with lightweight on-device signal processing that runs on a low-power microcontroller.
Rather than using machine learning approaches that require video or speaker embeddings, SpeechCompass relies on classical DSP algorithms (GCC-PHAT and kernel density estimation). This keeps latency low, preserves privacy by not retaining voice data, and makes the system viable on resource-constrained devices. The DSP components can be built and tested with Bazel independently of the hardware, lowering the barrier for experimentation.
The Android app provides multiple visualization options: colored text, directional arrows, and other cues that overlay on captions to indicate which direction speech is coming from. A large-scale survey of 263 frequent captioning users identified speaker distinction as the top unmet need, and a lab study with eight users confirmed that directional guidance was unanimously valued for group conversations.
SpeechCompass builds on the earlier LiveLocalizer project (UIST 2023) and reuses the same hardware platform. The project is licensed under Apache 2.0 for code and CC-BY for other materials, with all code available on GitHub. The release includes a demo video split into ten chapters covering hardware, UI, live demos, and user studies.
Why this matters for devtools
For developers working on accessibility or real-time audio systems, SpeechCompass demonstrates a practical, efficient approach to spatial audio processing without the overhead of deep learning models. The modular, Bazel-based build system and the ability to test DSP algorithms without hardware make it a useful reference implementation for embedded microphone-array localization.
SHARE
