ClearBuds: Wireless Binaural Earbuds for Learning-Based Speech Enhancement

• Main Authors: Chatterjee, Ishan, Kim, Maruchi, Jayaram, Vivek, Gollakota, Shyamnath, Kemelmacher-Shlizerman, Ira, Patel, Shwetak, Seitz, Steven M
• Format: Journal Article
• Language: English
• Published: 27-06-2022
• Subjects: Computer Science - Sound
• DOI: 10.48550/arxiv.2206.13611

We present ClearBuds, the first hardware and software system that utilizes a
neural network to enhance speech streamed from two wireless earbuds. Real-time
speech enhancement for wireless earbuds requires high-quality sound separation
and background cancellation, operating in real-time and on a mobile phone.
Clear-Buds bridges state-of-the-art deep learning for blind audio source
separation and in-ear mobile systems by making two key technical contributions:
1) a new wireless earbud design capable of operating as a synchronized,
binaural microphone array, and 2) a lightweight dual-channel speech enhancement
neural network that runs on a mobile device. Our neural network has a novel
cascaded architecture that combines a time-domain conventional neural network
with a spectrogram-based frequency masking neural network to reduce the
artifacts in the audio output. Results show that our wireless earbuds achieve a
synchronization error less than 64 microseconds and our network has a runtime
of 21.4 milliseconds on an accompanying mobile phone. In-the-wild evaluation
with eight users in previously unseen indoor and outdoor multipath scenarios
demonstrates that our neural network generalizes to learn both spatial and
acoustic cues to perform noise suppression and background speech removal. In a
user-study with 37 participants who spent over 15.4 hours rating 1041 audio
samples collected in-the-wild, our system achieves improved mean opinion score
and background noise suppression.
Project page with demos: https://clearbuds.cs.washington.edu