Transparent Liquid Segmentation for Robotic Pouring
Abstract
Liquid state estimation is important for robotics tasks such as pouring; however, estimating the state of transparent liquids is a challenging problem.
We propose a novel segmentation pipeline that can segment transparent liquids such as water from a static, RGB image without requiring any manual annotations or heating of the liquid for training.
Instead, we use a generative model that is capable of translating unpaired images of colored liquids into synthetically generated transparent liquid images.
Segmentation labels of colored liquids are obtained automatically using background subtraction.
We use paired samples of synthetically generated transparent liquid images and background subtraction for our segmentation pipeline.
Our experiments show that we are able to accurately predict a segmentation mask for transparent liquids without requiring any manual annotations.
We demonstrate the utility of transparent liquid segmentation in a robotic pouring task that controls pouring by perceiving liquid height in a transparent cup.
Accompanying video and supplementary information can be found at https://sites.google.com/view/roboticliquidpouring
Introduction
Robots that could pour liquids would enable us to automate tasks such as cooking, pouring medicines into vials in pharmacies or watering our plants.
However, transparent liquids are difﬁcult to perceive in images; the only visual signals a perfectly transparent liquid can provide are the refraction of light passing through the liquid.
Obtaining depth measurements for liquids is similarly difﬁcult since the liquid will refract the projected infrared light.
Previous works have explored robotic pouring in various settings [Do et al., 2016, Dong et al., 2019, Kennedy et al., 2017, 2019, Matl et al., 2020] but all require signiﬁcant compromises in the environment or method for data collection.
For example, several methods for transparent liquid segmentation require heating up the liquid during training to obtain ground-truth labels when viewed by a thermal camera Schenck and Fox [2016, 2017b a]; however, heating up the liquid for training is a tedious process that limits how much training data can be easily collected.
Other approaches require observing the liquid from multiple viewpoints Do et al. [2016], checkerboard backgrounds Kennedy et al. [2019], weight measurements Kennedy et al. [2019], or liquid motion Yamaguchi and Atkeson [2016], Wilson et al.
[2019], Liang et al.
[2019], Do and Burgard [2018]; these requirements on the environment restrict the applicability of these methods.
In this work, we propose a method for perceiving transparent liquid (such as water) inside transparent containers.
Our method requires signiﬁcantly fewer restrictions on the operational domain than previous methods.
Speciﬁcally, our method operates on individual images (we do not require liquid motion or multiple frames), and requires no manual annotations or heating of liquids during training. [Figure]
To accomplish this, our method uses a generative model that learns to translate images of colored liquid into synthetically generated images of transparent liquid.
Because images of colored liquid are easy to segment, this automatically provides us with ground-truth labels for the synthetically generated images of transparent liquid.
Finally, we demonstrate the utility of our method for transparent liquid segmentation on a robot pouring task.
Related Work 2.0.1 Transparent object perception
Perceiving transparent objects is particularly challenging because transparent objects can refract, reﬂect, and absorb light.
Some previous works focus on perceiving transparent containers; methods have been developed for transparent object segmentation Xie et al. [2020], Liao et al. [2020], depth estimation Sajjan et al. [2020], Zhu et al.
[2021], key-point estimation Liu et al. [2020], and transparent object matting Chen et al. Other methods for segmenting transparent objects use light ﬁeld cameras Xu et al. [2015, 2019] or light polarization Kalra et al.
On the manipulation side, other recent works have been developed to directly grasp transparent objects without ﬁrst estimating their 3D shape Weng et al.
Our approach builds on Xie et al. [2020] for transparent container segmentation; however, our focus is on segmenting the transparent liquid inside the container.
Unlike previous work that uses manual annotations for training Xie et al. [2020], our work does not rely on any manual annotations.
2.0.2 Transparent liquid perception
Perception of liquid is more challenging than perception of object due to the lack of a ﬁxed shape or geometry.
While perception of colored liquid can sometimes be done using background subtraction Kennedy et al. [2017], it does not work for transparent liquid.
One approach to transparent liquid perception is to use heated liquid observed by a thermal camera to obtain ground-truth labels for liquid Schenck and Fox [2016, 2017b,a].
However, the requirement to heat the liquid before recording the ground-truth is a tedious process; our method does not require heated liquid.
To segment liquid while it is being poured, one can use optical ﬂow Yamaguchi and Atkeson [2016] or audio signals Wilson et al. [2019], Liang et al.
Our method can segment static liquid, which is important for liquid state estimation before initiating a pouring task.
Some methods reason about the refraction of the infrared light emitted by a depth sensor Do et al. [2016], Do and Burgard [2018], multiple noisy readings from different viewpoints [Do et al., 2016], or from different time points during pouring Do and Burgard [2018], integrated probabilistically.
In contrast, our method can segment the liquid from just a single RGB image.
Another approach is to use a depth sensor to estimate the height of the liquid surface Dong et al. [2019]; however, such depth readings are inaccurate for transparent liquids.
A different strategy is to pour liquid in front of a checkerboard background or to use weight readings from a scale Kennedy et al.
Our method does not require a checkerboard background or a scale.
Finally, some approaches forgo a separate module for transparent liquid perception and learn an end-to-end policy for pouring transparent liquid Lin [Figure]
et al. However, so far such approaches have only been shown to work in simulation due to the sample complexity of learning a sensori-motor policy.
Method
We describe our method for transparent liquid segmentation when liquids are placed within transparent containers (see Figure 2 for an overview).
First, we collect a dataset of colored liquid and another (unpaired) dataset of transparent liquid.
We then use an image translation method to learn to translate an image of colored liquid into a synthetically generated image of transparent liquid that is identical to the input image, except that the liquid is now transparent.
Next, we use background subtraction to ﬁnd the colored liquid pixels in the colored liquid dataset.
We treat the colored liquid segmentation as a ground-truth label for the synthetically generated transparent liquid.
We then train a network to segment transparent liquid, using paired samples of the synthetically generated transparent liquid and colored liquid ground-truth labels.
3.1 Learning to translate colored liquid to transparent liquid
To train a model for transparent liquid segmentation, we ideally want a dataset of labeled images of transparent liquids.
However, labeling such a dataset is tedious.
Instead, we make use of an image translation method to synthetically generate the desired labels.
We collect one dataset of colored liquids in transparent containers D color and a second dataset of transparent liquids in transparent containers D transparent .
Given these two datasets, we learn an image translation model from colored to transparent liquids.
To do so, we use Contrastive Unpaired
Translation (CUT) Park et al. [2020], which we train to convert an image of a colored liquid into an image of a transparent liquid.
We brieﬂy describe CUT and how we adapt it for our method.
The backbone of CUT is a generator that translates an image of the source domain into an image of the target domain.
To encourage this translation, three loss terms are used.
First, a standard adversarial GAN loss is used to encourage the generator to output images that are visually similar to those in the target domain:
[Equation]
where G, D, denote the generator and the discriminator respectively, and X , Y denote the source domain and the target domain, respectively.
The generator G is divided into an encoder G enc and a decoder G dec, such that the output is ˆ y =
G ( x ) =
G dec ( G enc ( x )) , for an image x from the source domain.
Additionally, CUT uses a patch-wise contrastive loss van den Oord et al. [2019] to encourage corresponding patches between the input and output images to be similar to each other in feature space.
Speciﬁcally, given an image x from the source domain X , the image is translated into an image y of the target domain Y .
The patch-wise contrastive loss, L PatchNCE ( G, H, X ) , maximizes the mutual information between H ( G enc ( x )) and H ( G enc ( y )) , where H is a small multi-layer perceptron (MLP).
The generator is also trained with an identity loss L PatchNCE ( G, H, Y ) to help regularize the encoder and minimize unnecessary modiﬁcations to a source image.
The combined loss is:
[Equation]
We apply CUT directly to our two datasets of raw images, where our source domain X = D color and target domain Y = D transparent .
Thus, we use CUT to convert an image of colored liquid x color ∈
D color to a synthetically generated image of a transparent liquid ˆ y transparent =
G ( x color ) .
Importantly, the patch-wise contrastive loss encourages the object parts in the image x color to be in the same location as the parts in the translated image ˆ y transparent .
For example, in Figure 2, the cup, liquid, surface, and even shadows are in the same locations between the image of colored liquid x color that is input to the generator G , compared to the synthetic image of transparent liquid ˆ y transparent that is output by the generator.
The primary difference between these images is that the colored liquid has changed to become transparent; the liquid is in the same location as in the input.
This property (that the liquid is in the same location in the generator input x color as in the output ˆ y transparent ) is crucial to the success of our proposed segmentation method.
If we assume that the only property that has changed as a result of the translation is the liquid color, then we can directly use the segmentation masks from the colored liquid x color as pseudo-ground truth for the generated transparent liquid ˆ y transparent .
Because it is simple to segment an image of colored liquid using color thresholding or background subtraction (see supplementary materials for details), we can then easily generate segmentation labels for the synthetic transparent images ˆ y transparent without requiring human annotations.
Formally, given an image of colored liquid x ( i ) color with corresponding segmentation mask M ( i ) , we generate a synthetic image of transparent liquid ˆ y ( i ) transparent =
G ( x ( i ) color ) to which we associate the colored image’s
segmentation mask M i as a pseudo-ground truth segmentation label.
This creates a synthetic labeled dataset: { ˆ y ( i ) transparent , M ( i ) } which we use to train our transparent liquid segmentation model, as described below.
3.2 Learning transparent liquid segmentation
We can use the aforementioned synthetic labeled dataset to train a transparent liquid segmentation model S using the Binary Cross Entropy loss between the predicted liquid segmentation mask and the pseudo-ground truth M i (described above).
Architectural, hyper-parameter, and implementation details are described in the supplementary materials.
Our image translation and segmentation modules are trained on one Nvidia 2080 Ti GPU.
3.3 Robot liquid pouring
In this section, we describe the details of the robotic pouring system we designed to demonstrate the utility of our transparent liquid segmentation model in pouring tasks.
While other works have explored sophisticated and ﬂexible robotic pouring methods, we emphasize that our robotic system design is a simple testbed for our perception.
Our system consists of two stages: a visual postprocessing stage that converts a liquid segmentation into an estimate of a scalar ﬁll level in a container ˆ l and a pouring controller which drives the system to reach a target ﬁll level l target .
See Figure 3.
3.3.1 Task Design
For our robotic pouring experiments, we use a Franka-Emika Panda 7-DOF robotic arm, with a custom [Figure]
We place a Microsoft Azure Kinect RGB camera directly in front of the robot’s workspace.
We place the target cup directly in front of the camera at a known location.
We then drive the end-effector to a known location directly above the target cup.
We ensure that the source cup in the robot’s end-effector is not included in the image recorded by the Kinect camera.
We ﬁll the source cup with liquid, which will be poured into the target cup.
3.3.2 Visual postprocessing
To minimize any potential errors caused by the segmentation model, we ﬁrst use an off-the-shelf method TransLab Xie et al. [2020] model to detect the location of the cup in the scene.
Next, we crop the image region around the detected cup location.
Then, we use our segmentation model to segment the transparent liquid pixels from within this crop.
Finally, we perform a ﬁltering step that removes noise as well as small water particles in motion (i.e. during pouring).
The remaining segmented points are from liquid inside the target cup.
It is difﬁcult to design a control system for pouring that operates directly on a current and target segmentation mask.
Instead, we deﬁne a target ﬁll height h target from the camera perspective; we then deﬁne the process variable as the fraction of the cup that is ﬁlled: l target = hh cup , where h is the liquid height and h cup is the cup height (as seen from the camera perspective).
This task was inspired by the task of a robot waiter, which must reﬁll everyone’s cups to a certain fraction of the cup height.
Further, keeping the process variable grounded in 2D visual features avoids complexities of estimating the 3D geometry of the scene.
To estimate the current ﬁll level ˆ h , we compute a vertically-aligned bounding box of the segmented pixels.
We then use the upper edge of the box as the estimated height ˆ h of the liquid.
As mentioned previously, we use previous work to segment the transparent cup in the scene Xie et al. [2020]; we use a similar approach as above to ﬁt a bounding box and ﬁnd the height of the cup h cup .
From these computations, we can estimate the process variable ˆ l = ˆ hh cup .
3.3.3 Control
With the robot arm positioned directly above the cup, we restrict the control space to be one of two states: { NotPouring , Pouring } .
The NotPouring state corresponds to the starting postion, where the source cup is completely vertical (upright).
The Pouring state corresponds to a 60 rotation about the central axis of the cup (parallel to the table).
When the control signal changes between the two states, the end-effector rotates at a constant angular velocity until the desired control state is reached.
The system initially commands Pouring until the estimated height is within a margin (cid:15) of the target; afterwards, the system commands NotPouring .
Because the vision system is a simple neural network with simple post-processing, it is very low latency.
Therefore, we can operate our control loop at roughly 10Hz.
Because of the responsive system, bang-bang control of the pouring into the target cup is effective.
Finally, to compensate for some errors that occur in the perception system when the target cup is empty, we always begin the control with roughly 1s of initial pouring.
Further details about our data capture and background subtraction can be found in the supplementary materials.
3.4 Diverse Backgrounds
Previous work has achieved background generalization for object segmentation by synthetically pasting a foreground segment on top of a random background scene Kisantal et al.
[2019], Dvornik et al.
However, this technique is infeasible for learning to segment transparent liquids inside of transparent containers, since the container and liquid will refract the background, which cannot be easily imitated synthetically.
Instead, to generate a diverse set images with physicallyaccurate refraction characteristics, we set up a large ﬂatscreen LED display behind the pouring scene.
We then played videos containing a diverse set of indoor and outdoor scenes during the data collection procedure.
This allowed us to create datasets with a high degree of background diversity, with natural patterns of light refracting through water.
While these procedures could be included as part of the regular data collection procedure, for simplicity, they were only performed for the results in Section 4.3 (Ablation 3).
Results 4.1 Dataset Description
To train our method, we collected 4 distinct pouring videos each of green-colored water and clear water.
This resulted in datasets with 2231 and 2237 RGB frames, respectively.
We also collected a test set of 133 images of transparent liquids in the same scene.
4.2 Segmentation performance
To evaluate the segmentation performance of the method for transparent liquids, we create a test set of 65 images of transparent containers that have varying amounts of transparent liquids (water) in them, placed at different distances from the camera.
We manually annotate the location of the transparent liquids for the test set using Wada [2016]; however, we do not use such annotations for training.
[Table]
Our results for transparent liquid segmentation can be found in Table 1 as well as in Fig 4 (bottom row).
Our method generally succeeds at segmenting the liquid pixels in the image, achieving high
IoU scores across the test set.
However, our method experiences a small drop in performance on images in which the cup is ﬁlled to a high level (i.e. right-most image in “Ours” in Figure 4). We [Figure]
hypothesize that such examples have regions of liquid that are distant from a liquid-cup or liquid-air boundary; thus these cases are harder to classify when there are no refractive patterns to indicate the presence of liquid.
4.3 Pouring Performance
[Figure]
[Table]
To assess the utility and effectiveness of transparent liquid segmentation in our real robotic pouring system, we conduct a series of pouring trials.
We choose four different initial ﬁll-levels and target ﬁll-levels (see Table 2), and conduct 20 pouring trials for each ﬁll-level.
We measure the level of liquid achieved upon the controller reaching the ﬁnal NotPouring state, and report the average error across each ﬁll-level, as well as across all trials.
We conduct two sets of trials: ﬁrst with transparent water, using our transparent liquid segmentation system; second with green-colored water, using the background-subtraction method described in the supplementary materials.
Given the high accuracy of background subtraction, we consider this second task to represent how the system would perform with near-perfect segmentation.
We opted not to evaluate using segmentation methods with poor performance, due to the high risk of spilling liquid in the workspace.
Results can be found in Table 2.
In the case of pouring transparent liquids, we are able to achieve the desired ratio l target with an average accuracy of 0.94%, which corresponds to a roughly 0.13cm error on average.
Surprisingly, in the case of colored liquid (where background subtraction yields high-ﬁdelity segmentation), accuracy is worse, with an average error of 3.38% or 0.47cm error.
However, when observing the system, we noticed that the bounding box computation is sensitive to segmentations with splashing/sloshing: in these cases, the bounding box overestimates the amount of liquid in the cup, and terminates pouring earlier than it should.
Segmentations from our model pick up less of this transient liquid, and thus outperform the “ground-truth” segmentation.
With a more sophisticated postprocessing step and controller, these effects could potentially be mitigated.
4.4 Visual Translation
We show representative examples of the image translation achieved by our CUT-based model in Fig 5.
We observe that the network is able to translate the input image containing green-colored water pixels into images of clear water while still capturing the same background and refraction patterns as that in the source image.
Most importantly, the transparent liquid in the synthetic images is in the same location as the colored liquid in the original images.
This property allows us to apply the background mask from the colored liquids as the ground-truth label for the synthetic images of transparent liquids.
4.5 Visual Ablations
Ablation 1: What is the beneﬁt of training the segmentation model on synthetically generated transparent (instead of colored) liquid?
To answer this question, we train a segmentation model on colored liquid with color jitter and evaluate it on transparent liquid.
We jitter the brightness, contrast and hue to obtain the input image for training the segmentation model.
This ablation explores whether such color augmentation is sufﬁcient to train a model for transparent liquid segmentation.
Quantitative results of this ablation (“Opaque Dataset”) are shown in Table 1.
Qualitative results are shown in Fig 4.
The model fails to detect the correct liquid height during our evaluation.
This analysis shows that using color jitter on a colored liquid image is not sufﬁcient domain randomization [Figure] to capture the texture and patterns required to segment transparent liquids, supporting the idea that the segmentation model should be trained directly on images of transparent liquid.
Ablation 2: What is the beneﬁt of the translation approach over using a small amount of manually-labeled data?
Since our method uses a standard supervised loss function to train the model on synthetic data, a natural comparison is to evaluate performance when the model is trained on images of transparent images with manual segmentation annotations.
Since annotation is very time-consuming ( ∼ 1min per image), we annotate 10% of images in the training domain, and observe how well the model can learn to segment on two supervised subsets: 10% labeled and 1% labeled.
Additionally, we train our method on same fraction of images from our synthetic dataset.
Results can be found in the second section of Table 1.
We draw two conclusions from these results.
First, both our method and the fully-supervised ablation require substantially less data to reach their peak performance on the test set than is available: “Ours (10%)” shows the same performance as our full method (“Ours”), and “Supervised (10%)” outperforms our method.
This is unsurprising, given that our test images are visually quite similar to the training set.
Second, with only 1% of manually-annotated labels available, the supervised baseline (“Supervised (1%)”) performs substantially worse than our full method.
While our dataset is rather small – we were able to manually label 10% of the data – our data collection method is extremely scalable.
In large datasets it is reasonable to expect one might only have access to manually-annotated labels for 1% of a full dataset.
This provides evidence that our method would scale well to liquid datasets where a small amount of supervised data would not provide sufﬁcient generalization.
Ablation 3: Does the approach work if there are diverse cups and backgrounds?
To evaluate our segmentation model’s capacity to generalize to diverse cups and backgrounds, we conduct two studies, shown in Figure 6.
We ﬁrst train a segmentation model on an image dataset consisting of a single cup in a single scene with varying heights of liquids.
We then evaluate on transparent cups of different shapes with different ﬁll levels of transparent liquid.
As can be seen in Figure 6, the model generalizes reasonably to other cups, indicating that it has learned to detect relevant liquid features invariant a speciﬁc container.
Next, we train a model on a cup in front of a diverse set of backgrounds and various water heights, with a dataset collected as described in Section 3.4.
As can be seen in
Figure 6, the model learns a segmentation function that is invariant to novel unseen backgrounds.
These two experiments demonstrate the potential of our approach to scale to a general set of scenes, provided a sufﬁciently diverse dataset generated by our method.
Conclusion
In this paper, we propose a method to segment transparent liquid placed inside transparent containers using static RGB images.
A generative model is used to translate colored liquids to transparent texture.
We show that an encoder - decoder network can be used to predict the segmentation mask directly from RGB images of transparent liquids without using any additional input modalities.
We use background subtraction on colored liquids to obtain the ground truth for training the segmentation model and we do not require any manual annotations.
Our method shows good results for most test cases of transparent liquid segmentation.
Finally, we demonstrate the utility of transparent liquid segmentation on a real robotic pouring task.
We hope that our method paves the way for more ﬂexible and robust perception of transparent liquids for robot pouring.