Research · RSJ 2025 · 2025-09
RSJ 2025: Audio-informed Imitation Learning
A novel method to integrate audio signals for manipulation tasks unsolvable with visual information alone.
Overview This research proposed a new task called "Acoustic Informed Pick and Place," where sound is essential to distinguish between objects that look identical. Key Achievements Developed a framework that represents sound information from a microphone array as a 2D "acoustic map." This map is combined with visual information and fed into an imitation learning model. Experiments in both simulation and the real world showed this method significantly improves success rates on tasks that depend on sound. Currently extending this research to see how incorporating audio spectrograms impacts performance. YouTube Demo
#Imitation Learning #Robotics #Multimodal #Acoustics #Computer Vision