Real-time Audio Video Enhancement \\with a Microphone Array and Headphones
Authors:
Jacob Kealey,
Anthony Gosselin,
Étienne Deshaies-Samson,
Francis Cardinal,
Félix Ducharme-Turcotte,
Olivier Bergeron,
Amélie Rioux-Joyal,
Jérémy Bélec,
François Grondin
Abstract:
This paper presents a complete hardware and software pipeline for real-time speech enhancement in noisy and reverberant conditions. The device consists of a microphone array and a camera mounted on eyeglasses, connected to an embedded system that enhances speech and plays back the audio in headphones, with a latency of maximum 120 msec. The proposed approach relies on face detection, tracking and…
▽ More
This paper presents a complete hardware and software pipeline for real-time speech enhancement in noisy and reverberant conditions. The device consists of a microphone array and a camera mounted on eyeglasses, connected to an embedded system that enhances speech and plays back the audio in headphones, with a latency of maximum 120 msec. The proposed approach relies on face detection, tracking and verification to enhance the speech of a target speaker using a beamformer and a postfiltering neural network. Results demonstrate the feasibility of the approach, and opens the door to the exploration and validation of a wide range of beamformer and speech enhancement methods for real-time speech enhancement.
△ Less
Submitted 1 March, 2023;
originally announced March 2023.
