Analyzing Liquid Pouring Sequences via AudioVisual Neural Networks Existing work to estimate the weight of a liquid poured into a target container often require predefined source weights or visual data. We present novel audiobased and audioaugmented techniques, in the form of multimodal convolutional neural networks (CNNs), to estimate poured weight, perform overflow detection, and classify liquid and target container. Our audiobased neural network uses the sound from a pouring sequencexe2x80x93a liquid being poured into a target container. Audio inputs consist of converting raw audio into melscaled spectrograms. Our audioaugmented network fuses this audio with its corresponding visual data based on video images. Only a microphone and camera are required, which can be found in any modern smartphone or Microsoft Kinect. Our approach improves classification accuracy for different environments, containers, and contents of the robot pouring task. Our Pouring Sequence Neural Networks (PSNN) are trained and tested using the Rethink Robotics Baxter Research Robot. To the best of our knowledge, this is the first use of audiovisual neural networks to analyze liquid pouring sequences by classifying their weight, liquid, and receiving container.

Making Sense of Audio Vibration for Liquid Height Estimation in Robotic Pouring In this paper, we focus on the challenging perception problem in robotic pouring. Most of the existing approaches either leverage visual or haptic information. However, these techniques may suffer from poor generalization performances on opaque containers or concerning measuring precision. To tackle these drawbacks, we propose to make use of audio vibration sensing and design a deep neural network PouringNet to predict the liquid height from the audio fragment during the robotic pouring task. PouringNet is trained on our collected realworld pouring dataset with multimodal sensing data, which contains more than 3000 recordings of audio, force feedback, video and trajectory data of the human hand that performs the pouring task. Each record represents a complete pouring procedure. We conduct several evaluations on PouringNet with our dataset and robotic hardware. The results demonstrate that our PouringNet generalizes well across different liquid containers, positions of the audio receiver, initial liquid heights and types of liquid, and facilitates a more robust and accurate audiobased perception for robotic pouring.