Researchers at Carnegie Mellon University have developed a camera system that can apparently detect sound vibrations with a level of precision that allows audio to be recreated without inference or a microphone. A team from the Robotics Institute (RI) at CMU’s School of Computing built the system, which includes two cameras and a laser. It can detect “high-velocity, low-amplitude surface vibrations” that the human eye cannot see, the university said in a press release.
The system features regular cameras rather than the high-speed cameras used in previous research, which should reduce the cost. “We’ve made the optical microphone much more convenient and usable,” said Srinivasa Narasimhan, RI professor and director of the Illumination and Imaging Laboratory. “We have improved quality while reducing costs.”
An algorithm compares speckle patterns captured by a rolling shutter and a global shutter. It uses the differences between the patterns to calculate the vibrations and recreate the audio. A speckle pattern (created by the laser in this case) refers to the behavior of spatially coherent light after it reflects off a rough surface. This behavior changes when the surface vibrates. The rolling shutter rapidly scans an image from edge to edge, while a global shutter captures an entire image at once.
“This system pushes the boundaries of what can be done with computer vision,” said Assistant Professor Matthew O’Toole, co-author of on the system. “This is a new mechanism for capturing high-speed and tiny vibrations, and presents a new area of research.”
The researchers say they were able to isolate the audio of guitars being played simultaneously. They claim the system was able to observe a bag of chips and use its vibrations to reconstruct the audio emitted from a nearby speaker with higher fidelity than previous optical microphone approaches.
There are many potential applications for this technology. The researchers suggest, for example, that the system could monitor the vibrations of machinery in a factory to look for signs of trouble. Sound engineers could also isolate the sound of an instrument to improve the mix. Essentially, it could help remove ambient noise from audio recordings.