It may sound like Ruidong Zhang is talking to himself, but in reality, a computer science PhD student unlocks a nearby smartphone and plays the next song in his playlist. silently says the passcode.
It’s not telepathy. These are the seemingly normal, off-the-shelf glasses he wears called EchoSpeech. EchoSpeech is a silent speech recognition interface that uses acoustic sensing and artificial intelligence to continuously recognize up to 31 unspoken commands based on lip and mouth movements. .
Developed by Cornell University’s Smart Computer Interfaces for Future Interactions (SciFi) Lab, this low-power wearable interface requires just minutes of user training data before it can recognize commands and execute them on a smartphone. .
Zhang is the lead author of “EchoSpeech: Continuous Silent Speech Recognition on Minimally-obtrusive Eyewear Powered by Acoustic Sensing,” presented at the Computing Machinery Conference (CHI) on Human Factors in Computing Systems in Hamburg, Germany this month. . .
“For people who cannot produce sounds, this silent speech technology could be a good input to a speech synthesizer. It might be possible to get the patient’s voice back,” Zhang said. Talk about potential uses for the technology.
EchoSpeech, in its current form, can be used to communicate with other users via smartphones in places where conversation is inconvenient or inappropriate, such as noisy restaurants or quiet libraries. Silent speech interfaces can also be used with design software such as CAD in combination with a stylus. No keyboard and mouse required.
Equipped with a microphone and speaker pair smaller than a pencil eraser, the EchoSpeech glasses become a wearable, AI-powered sonar system that sends and receives sound waves across the face and senses mouth movements. A deep learning algorithm, developed by researchers at SciFi Lab, analyzes these Echohips profiles in real time with an accuracy of about 95%.
Chen Chang, Assistant Professor of Computer Science at Cornell Ann S. Bowers College of Computing and Information Science and Director of SciFi Labs, said:
“We are very excited about this system,” he said. Small, low-power, and privacy-friendly, all of which are important features for deploying new wearable technologies into the real world. “
SciFi Lab has developed several wearable devices that use machine learning and tiny wearable video cameras to track body, hand and face movements. Recently, labs have moved away from cameras and into acoustic sensing for facial and body movement tracking, citing improved battery life. Enhanced security and privacy. Smaller, more compact hardware. EchoSpeech is built from a similar acoustic-sensing device in the lab called EarIO, a wearable earbud that tracks facial movements.
Most techniques for silent speech recognition are limited to a pre-determined selection set of commands and require the user to face or wear the camera, which is neither practical nor feasible. No, says Cheng Zhang. Wearables His cameras also have significant privacy concerns for both users and those with whom they interact, he said.
Acoustic sensing technologies such as EchoSpeech eliminate the need for wearable video cameras. And because audio data is much smaller than image and video data, it requires less bandwidth to process and can be relayed to smartphones in real time via Bluetooth, says François Guimbretière, co-author and professor of computer science at Cornell Bowers CIS. says Mr.
“And since the data is processed locally on the smartphone rather than being uploaded to the cloud, privacy-sensitive information is never under the control of the user.”
According to Cheng Zhang, battery life will also improve exponentially.
The team is looking to commercialize the technology behind EchoSpeech, thanks in part to the Ignite: Cornell Research Lab to Market Gap Fund.
Researchers at SciFi Lab are studying smart glasses applications for tracking facial, eye, and upper-body movements.
Cheng Zhang said:
Original: AI-powered glasses can read silent conversations
Than: Cornell Ann S. Bowers College of Computing and Information Science
