Sonar-enabled glasses read wearers’ silently spoken voice commands

Some people just don’t have the ability to speak, and just speaking voice commands aloud in a noisy environment just doesn’t work. Such people may use her EchoSpeech glasses, which read the user’s softly spoken words.

The experimental eyewear is being developed by a team at Cornell University’s Smart Computer Interfaces for Future Interactions (SciFi) Lab.

Two downward facing miniature speakers are mounted under one lens on the underside of the frame, and two mini microphones are placed under the other lens. The speaker emits inaudible sound waves that are reflected by the wearer’s mouth movements and returned to the microphone.

These echoes are analyzed in real time by deep learning algorithms on wirelessly linked smartphones. Its algorithms are trained to associate specific echoes with specific mouth movements, which are associated with specific softly spoken commands.

Cornell University

EchoSpeech can currently recognize 31 such commands with about 95% accuracy, requiring only a few minutes of training per user. And importantly for those with privacy concerns, this system has no built-in cameras and does not send any information to the internet.

Plus, with no power-hungry cameras, you can run up to 10 hours on a single battery charge. In contrast, researchers claim their experimental camera-based system lasts only about 30 minutes on a single charge.

The university is currently working on commercializing the technology.

“For people who cannot produce sounds, this silent speech technology could be an excellent input to speech synthesizers,” said Ruidong Zhang, a Ph.D. student who leads the study. “It can bring back the patient’s voice.”

SciFi Lab previously developed a somewhat similar system called EarIO. It uses a sonar-equipped ear-worn device to capture the wearer’s facial expressions, but is primarily used to create digital avatars.That said, the University of Buffalo’s EarCommand system To do Quietly spoken words are read out through earphones that detect unique ear canal deformations caused by specific mouth movements.

EchoSpeech is demonstrated in the following video.

Source: Cornell University