U NDERSTANDING E MBODIED R EFERENCE WITH T OUCH -L INE T RANSFORMER
Anonymous authors
Paper under double-blind review
A BSTRACT
We study embodied reference understanding, the task of locating referents using embodied gestural signals and language references.
Human studies have revealed that objects referred to or pointed to do not lie on the elbow-wrist line , a common misconception; instead, they lie on the so-called virtual touch line .
However, existing human pose representations fail to incorporate the virtual touch line.
To tackle this problem, we devise the touch-line transformer:
It takes as input tokenized visual and textual features and simultaneously predicts the referent’s bounding box and a touch-line vector.
Leveraging this touch-line prior, we further devise a geometric consistency loss that encourages the co-linearity between referents and touch lines.
Using the touch-line as gestural information improves model performances significantly.
Experiments on the YouRefIt dataset show our method achieves a +25.0% accuracy improvement under the 0.75 IoU criterion, closing 63.6% of the gap between model and human performances.
Furthermore, we computationally verify prior human studies by showing that computational models more accurately locate referents when using the virtual touch line than when using the elbow-wrist line .
I NTRODUCTION
Understanding human intents is critical to intelligent robots when interacting with humans.
However, most prior work in the modern learning community neglects the multimodal aspect in human-robot communication scenarios.
Let us take an example shown in Fig.
1, wherein a person commands the robot to interact with a chair behind the table.
In response, the robot ought to understand what is being referred to before taking any actions ( e.g ., moving towards it and cleaning it).
Notably, both embodied gesture signals and language reference play significant roles.
Without the pointing gesture, the robot would not be able to distinguish the two chairs using the utterance the chair that is occluded .
Without the language expression, the robot would not be able to distinguish the chair from other objects in that region ( e.g ., bags on the table).
To fill this gap, we study the
Embodied Reference Understanding (ERU) task introduced by Chen et al. (2021).
This task requires an algorithm to detect the referent (referred object) using (i) an image/video with nonverbal communication signals and (ii) a sentence as a verbal communication signal.
The first fundamental issue in addressing ERU is the representation of the human pose.
The de facto human pose representation in modern computer vision is defined by COCO (Lin et al., 2014)—a graph consisting of 17 nodes (keypoints) and 14 edges (keypoint connectivity).
This representation is prevalent; The existing model for ERU (Chen et al., 2021) take for granted to use pre-extracted
COCO-style pose features as the algorithm input.
However, we rethink the limitation of the COCO- style pose graph in the context of ERU and identify a counter-intuitive fact:
The referent does not lie on the elbow-wrist line ( i.e ., the line that links the human elbow and wrist).
As shown in Fig.
2, this line (in red) does not cross the referred microwave, illustrating a common misinterpretation of human pointing (Herbort & Kunde, 2018).
Interestingly, a recent psychology study (O’Madagain et al., 2019) provides strong evidence that supports the above hypothesis from the behavior perspective.
It investigates how human beings mentally develop pointing gestures and argues that it is a virtual form of reaching out to touch .
This new finding challenges conventional psychological theories (McGinn, 1981; Kita, 2003) that the pointing gesture is mentally a behavior of using the limb as an arrow.
Inherited from the notations in O’Madagain et al.
(2019), we coin the red line in Fig.
2 as an EWL and the yellow line in
Fig.
2 ( i.e ., a line that connects the eye and the fingertip) as a VTL.
Inspired by this essential observation that VTLs are more accurate than EWLs in embodied reference, we augment the existing COCO-style pose graph with an edge that connects the eye and the fingertip.
As validated in a series of experiments in Sec. 4, this augmentation brings significant performance boosts on YouRefIt (Chen et al., 2021).
The second fundamental issue in addressing ERU is how to model gestural signals and language references jointly.
Inspired by the success of multimodal transformers (Chen et al., 2020;
Li et al., 2020; Tan & Bansal, 2019;
Lu et al., 2019; Kamath et al., 2021) in multimodal tasks (Hudson &
Manning, 2019; Antol et al., 2015; Zellers et al., 2019), we devise the Touch-Line Transformer, a transformer-based model that takes inputs from both the visual and natural-language modalities.
Our
Touch-Line Transformer jointly models gestural signals and language references by simultaneously predicting the touch-line vector and the referent’s bounding box.
To further help our model utilize gestural signals ( i.e ., the touch-line vector), we integrate a geometric consistency loss to encourage co-linearity between the touch-line and the predicted location of the referent, resulting in significant performance improvements.
Leveraging the above two insights, our proposed method achieves a +25.0% accuracy improvement under the 0.75 IoU criterion compared with state-of-the-art methods on the YouRefIt dataset, closing 63.6% of the gap between model performance and human performance.
This paper makes four contributions.
(i) Inspired by recent behavior studies, we devise a novel human pose representation–Virtual Touch Line (VTL)–in computer vision.
(ii)
We introduce the Touch-Line
Transformer that jointly models nonverbal gestural signals and verbal references.
(iii)
We encourage our model to utilize the touch line by a novel geometric consistency loss that improves the co-linearity between the touch line and the predicted object.
(iv) Our model achieves a new state-of-the-art performance on ERU, attaining a +25.0% performance gain under the 0.75 IoU threshold.
R ELATED W ORK
Misinterpretation of pointing gestures Pointing allows observers and pointers to direct visual attention to establish references in communication.
Recent research reveals that observers surprisingly make systematic errors (Herbort & Kunde, 2016); they found that, while the pointer produces gesture using VTLs, observers interpret pointing gestures using the “arm–finger” line.
The VTL mechanism is founded by O’Madagain et al.
Pointing gestures orient toward their targets as if the pointers were to touch these targets.
In neuroscience, gaze effects occur for tasks that require gaze alignment with finger pointing (Bédard et al., 2008).
Together, the above evidence shows that eye position and gaze direction are significant for understanding pointing.
Critically, Herbort & Kunde (2018) verify that when human observers were instructed to extrapolate the touch-line vector, this systematic misinterpretation during human-human communication is reduced.
Inspired by these discoveries, we integrate the touch-line vector to improve the pointing gesture interpretation performance.
Gaze target detection The gazing gesture itself can refer to some objects in the absence of language expressions.
Gaze target detection is the localization of the objects that people are gazing at.
Recasens et al.
and Chong et al.
(2018) proposed two-state methods that generate saliency maps and estimate gaze directions first before combining them and predicting the gaze target.
Tu et al.
(2022) designed a one-stage end-to-end method to simultaneously locate human heads and gaze targets for all people in an image.
Fang et al.
(2021) designed a three-stage method that predicts 2D and 3D gaze direction, predicts field of views and depth ranges, and then predicts the gaze target.
Zhao et al.
(2020) maintains that humans find gaze targets from salient objects on the sight line.
They designed a model to simulate the hypothesized human way of finding gaze targets in images and extended its ability to video frames.
Li et al.
(2021) extended gaze target detection to 360-degree images.
Chong et al.
and Recasens et al.
(2017) extended gaze target detection into videos, where the gazed object may not be in the same frame as the gazing gesture.
Referring expression comprehension Language expressions can also refer to some objects.
Referring expression comprehension is to locate referents in an image using only language expressions.
Multiple datasets ( e.g ., RefCOCO (Yu et al., 2016), RefCOCO+ (Yu et al., 2016), RefCOCO-g (Mao et al., 2016), and Guesswhat (De Vries et al., 2017)) help benchmark models for this task.
On methods and models, Du et al.
(2021) devise a transformer-based framework and use the language to guide the extraction of “discriminative visual features” when encoding images and sentences.
Rohrbach et al.
(2016) recognize the bi-directional nature between language expressions and objects in images, obviating the need for bounding box annotations.
Yu et al.
(2018) devise a modular network; three modules respectively attend to subjects, their location in images, and their relationships with nearby objects.
While the above works locate referents using language expressions, they rarely use nonverbal gestures produced by humans in the input images.
Language-conditioned imitation learning Imitation helps robots to learn skills (Schaal, 1999), and robots need to identify target objects when completing goal-conditioned tasks (Stepputtis et al., 2020).
Stepputtis et al.
(2020) argue that specifying the target object via vectors and one-hot vectors, while helpful for robots to identify the target object, is not flexible enough to support continued learning in deployed robots.
As a result, Stepputtis et al.
(2020) propose to use the more flexible natural language to refer target objects to robots for goal-conditioned manipulation tasks.
Specifically, their architecture contains a semantic module to help robots understand referents referred to by languages.
Their architecture helps robots to locate and attend to the target object when the natural language expression non-ambiguously refers to an object.
However, their approach does not consider pointing gestures and has difficulties in handling ambiguities when referring involves pointing gestures.
As a result, some recent works in language-conditioned imitation learning (Stepputtis et al., 2020; Lynch & Sermanet, 2021; Mees et al., 2022) will potentially benefit from our proposed VTL by more accurately locating the target object, especially when the natural language alone is ambiguous.
[Figure]
M ETHOD 3. N ETWORK A RCHITECTURE
Our framework, shown in Fig. 3, consists of a multi-modal encoder, a transformer decoder, and prediction heads.
We detail each component below.
Multimodal encoder We generate a visual embedding vector by extracting visual features from input images with a pre-trained ResNet (He et al., 2016) backbone, flattening them, and adding them to a set of position embeddings (Parmar et al., 2018; Bello et al., 2019).
Meanwhile, we generate a textural embedding vector from input texts using a pre-trained BERT (Liu et al., 2019).
After obtaining visual and textural embedding vectors, we concatenate and feed them into a transformer encoder to learn multi-modal representations.
Transformer decoder We feed multi-modal representations generated above to our transformer decoder.
Our transformer decoder, additionally, takes a set of learnable object queries and gestural key point queries as inputs.
With multi-modal representations and input queries, our transformer decoder generates object output embeddings and gestural output embeddings.
We use the pre-trained
MDETR (Kamath et al., 2021) decoder.
Prediction heads Object and gestural output embeddings from the transformer decoder are inputs for our prediction heads (MLPs).
Our prediction heads use them to predict bounding boxes (for referents) and gestural key points.
We keep one object bounding box and one pair of gestural key points with the highest scores as the final prediction.
Specifically, we define the score of a bounding box as 1 minus the non-object class column of the softmax of the predicted object logits.
For a pair of gestural key points, the score is the is-a-VTL/EWL column of the predicted arm logits’ softmax.
3. E XPLICIT L EARNING OF N ONVERBAL G ESTURAL S IGNALS
Measure co-
lineaerity
Ideally, a referent should have a high co-linearity with the VTL.
We measure this co-linearity using cosine similarity:
[Equation]
where ( x f , y f ) , ( x e , y e ) , and ( x o , y o ) are the x and y coordinates of the fingertip, the eye, and the center of referent bounding box, respectively.
Encourage co-lineaerity We encourage our model to predict referent bounding boxes that are highly co-linear with VTLs using a referent alignment loss, which is a geometric consistency loss:
[Equation]
where cos _ sim gt is computed using ground truth referent box and cos _ sim pred is computed using predicted referent box.
In both cos _ sim gt and cos _ sim pred , ground truth VTLs are used.
Of note, the design of the offset cos _ sim gt ensures the predicted object is as co-linear as the ground-truth object to the VTL.
In other words, the loss in Eq.
(2) is minimized to zero when the predicted object box is as co-linear as the ground-truth one to the VTL.
Modify for EWLs In experiments using EWLs instead of VTLs, we replace the fingertip ( x f , y f ) and the eye ( x e , y e ) in Eq.
(1) using the wrist ( x w , y w ) and the elbow ( x l , y l ) , respectively.
3. I MPLICIT L EARNING OF N ONVERBAL G ESTURAL S IGNALS
We eliminate postural signals (by removing humans) from images to investigate whether our model can implicitly learn to utilize useful nonverbal gestural signals without being explicitly asked to learn.
Please refer to Appendix A.1 for details of our procedure.
3. A DDITIONAL L OSSES
The total loss in training is defined as:
[Equation]
where λ i s are the weights of various losses, L box is the weighted sum of the L1 and GIoU losses for predicted referent bounding boxes, and L gesture is the L1 loss for predicted VTLs or EWLs.
The soft token loss L token and the contrastive loss L contrastive follow those in Kamath et al.
; they help our model align visual and textural signals.
E XPERIMENTS 4. D ATASET , E VALUATION M ETRIC , AND I MPLEMENTATION D ETAILS
We use the YouRefIt dataset (Chen et al., 2021), consisting of 2 , 950 training instances and 1 , 245 test instances.
It has two portions: videos and images.
We use the image portion; the inputs to our model are images that do not have temporal information.
Each instance contains an image, a sentence, and the location of the referent.
We provide additional annotations of VTLs and EWLs for the YouRefIt dataset.
To make a fair comparison with prior models, we follow Chen et al.
and report precision under three different IoU thresholds: 0.25, 0.50, and 0.75.
A prediction is correct if its IoU with the ground-truth box is greater than the threshold.
Additionally, we choose our best models by the precision at the GIoU (Rezatofighi et al., 2019) threshold of 0.75; compared with IoU, GIoU is an improved indicator of the model’s ability to locate objects accurately.
We report the results using
GIoU thresholds in Sec. 4.5.
During training, we use the Adam optimizer’s AMSGrad variant (Reddi et al., 2018) and train our models for 200 epochs.
We use the AMSGrad variant because we observe a slow convergence of the standard Adam optimizer in experiments.
We set the learning rate to 5e-5 except for the text encoder, whose learning rate is 1e-4.
We do not perform learning rate drops because we rarely observe demonstrable performance improvements after dropping them.
We use A100 GPUs.
The sum of the batch sizes on all graphic cards is 56.
Augmentations follow those in Kamath et al.
The total number of queries is 20 (15 for objects; 5 for gestural key points).
4. C OMPARISON WITH S TATE - OF - THE - ART M ETHODS
Our approach outperforms prior state-of-the-art methods by 16.4%, 23.0%, and 25.0% under IoU threshold of 0.25, 0.
[Table]
and 0.75, respectively; see Tab. 1.
Specifically, our model performs better than visual grounding methods (Yang et al., 2019; 2020) which do not explicitly utilize nonverbal gestural signals.
Our method also performs better than the method proposed in YouRefIt (Chen et al., 2021)
[Table]
which did not leverage the touch line or the transformer models for multimodal tasks.
4. E XPLICITLY L EARNED N ONVERBAL S IGNALS
[Table]
We report the model performance that explicitly predicts either the VTLs or the EWLs.
Results Overall, the model trained to predict the VTLs performs better than the one trained to predict the EWLs under all three IoU thresholds; see Tab. 2.
The model trained to explicitly predict the EWLs performs even worse than the one that was not trained to explicitly predict any gestural signals under the IoU threshold of 0.75.
Analysis Under the IoU threshold of 0.75, the worse performance of the model with explicit EWL prediction can be partly attributed to the unreliability of the arm’s orientation and the stringent precision requirement of the 0.75 IoU threshold.
As we observed in Fig.
4, the EWLs are not reliable for predicting objects’ locations, simply because they oftentimes do not pass through the referents.
This mismatch prevents the model from correctly determining the referents’ locations using the
EWLs’ orientations.
For instance, in Fig.
4a (right), the EWL fails to pass the glass cup in the yellow
[Figure]
box; it passes a ceramic cup instead.
With contradictory nonverbal (the ceramic cup) and verbal (the glass cup) signals, the model struggles to identify the referent correctly.
[Table]
In contrast, VTLs partly explain the model’s improved performance.
For example in Fig.
4b, the
VTL passes the chair in the yellow box while the
EWL fails.
Similarly, in Fig.
4c, 4e, 4f, 4h and 4l,
VTLs passes the referent while EWLs do not.
Additionally in Fig.
4a, 4d, 4g and 4i to 4k, VTLs are closer to the referents’ box centers than EWL are.
The higher consistency between verbal and nonverbal signals increases the chance of successfully identifying the referents.
Under the IoU thresholds of 0.25 and 0.50, the improved model performance can be partly attributed to the rough orientations provided by the EWLs and the more lenient precision requirements of the lower thresholds.
Specifically, the rough orientations provided by the EWLs might help the model eliminate objects that significantly deviate from this direction.
Hence, the model can choose from a smaller number of objects, leading to a higher probability of correctly locating the referents.
4. I MPLICITLY L EARNED N ONVERBAL S IGNALS
[Table]
We further investigate whether our model can implicitly learn nonverbal gestural signals without being explicitly asked to predict the VTLs or the EWLs.
To investigate this, we conduct two sets of experiments: one using original images and another one using inpainted images.
Specifically, when using original images, we do not provide our annotated gestural key points (eyes, fingertips, elbows, and wrists) to our model and we do not ask our model to predict any of these gestural key points.
When using the inpainted images in which humans are removed, we still do not provide annotated gestural key points or ask our model to predict these key points.
The set of experiment using original images but does not provide or predict gestural key points corresponds to the “Original” column in Tab. 3, the “No Explicit Gestural Key Points” column in
Tab. 1, the “No Exp. Ges.
K. Poi.”
column in Tab. 7, and the “None” column in Tab. 2.
original inpainted
[Figure]
original inpainted
[Figure]
original inpainted
[Figure]
Our model performs much worse when nonverbal gestural signals are absent.
Specifically, removing humans from input images results in 5.8%, 6.1%, and 4.8% performance drop under the IoU threshold of 0.25, 0.50, and 0.75, respectively; see details in Tab. 3.
These performance drops indicate that the implicitly learned model lacks useful information given only the inpainted images.
Specifically, removing humans in the scenes also eliminates useful communicative signals, including pointing, poses, and the gaze direction (Sebanz et al., 2006).
The lack of human context introduces additional ambiguities in identifying the referents, especially when the natural language input alone cannot uniquely refer to an object in the scene.
For example in Fig.
5f, given the natural language input “the books in front of me,” determining which piles of books this person refers to is difficult: one on the table and one in the scene’s lower right corner, thus in need of nonverbal gestural signals, such as pointing.
Similarly in Fig.
5e, we cannot distinguish which picture frame the person is referring to without considering the pointing gesture.
4. A DDITIONAL R ESULTS
Attention weights visualizations We visualize the attention weights of our model trained with
VTL (Fig. 6).
We use yellow to visualize the attention weights of matched gesture keypoint queries and blue for matched object queries.
Visualizations show that the attention of object queries can attend to the regions of the target objects while the attention of keypoint queries primarily focus on humans’ heads and hands.
Taken together, these results indicate that our model successfully learns gestural features that boost the model performance.
Failure cases While the VTL effectively helps the model leverage the nonverbal gestural signals, the performance of our model still could not surpass the 74% human performance.
The gap between model and human performance is partly attributed to the model’s inability to distinguish the subtle differences between similar objects.
In other words, while explicitly predicting the VTLs helps our model utilize gestural signals to determine the orientation of the object, it does not help the model recognize the minor differences between multiple objects in the same orientation.
For example, the VTL passes two bottles in Fig.
7a.
While the VTL indeed helps to narrow down the options to these two bottles, it does not help our model recognize which one of them is body lotion.
[Figure]
Similarly in Fig.
7b, the VTL indicates two bottles—one with a red lid and the other with a yellow lid; yet it does not help the model distinguish the subtle differences between the lid colors.
[Figure]
Additionally, in very rare cases (Fig.
8), the application of our VTL is limited because the human head is not visible from the input image.
Without human heads, a touch line cannot be drawn in the image.
[Figure]
Computed cosine similarities The computed cosine similarity is much higher when using the
VTLs (Tab. 4), indicating the VTLs are more co-linear to the referents.
This result verifies our hypothesis that the VTLs are more reliable in referring to the objects’ locations.
[Table] [Table]
Referent alignment loss An ablation study (Tab. 5) shows that our reference alignment loss plays a significant role in leveraging nonverbal gestural signals.
Effects of object sizes We evaluate the model performance w.r
[Table]
t. the object detection of different object sizes (Tab. 6).
We define small (S), medium (M), and large (L) objects following the size thresholds in Chen et al.
Compared to the relatively consistent human performance, deep learning models’ performances are significantly lower when detecting diminutive objects than when detecting larger objects, especially under the less stringent IoU thresholds of 0.25 and 0.50.
The degraded performance indicates that future artificial intelligence models need to improve the performance of small object detection.
[Table]
Precision computed using GIoU thresholds We provide the performance of our models when evaluated using GIoU instead of IoU thresholds; see Tab.
[Table]
[Table] Corrupted language inputs We investigate the performance of our model when using corrupted language inputs by changing each input sentence to the word “object.”
Our results (Tab. 8) show that corrupting language inputs severely impair the performance of our models. [Table]
C ONCLUSION AND L IMITATIONS
We presented an effective approach, named Touch-Line Transformer, to utilize the simple but effective
VTLs to improve an artificial agent’s ability to locate referents referred to by humans in the wild.
Our approach is inspired by recent findings in psychology studies on the touch-line hypothesis, which further revealed that people frequently misinterpret other people’s referring expressions.
Our proposed architecture, combined with the proposed VTL, significantly reduced the gap between model performance and human performance.
Some limitations exist in our work.
First, resizing operations before and after inpainting might influence model performance.
Next, we primarily study the eyes’ location and the upper limbs’ orientation regarding nonverbal signals, leaving the study of other types of nonverbal signals to future works, such as gazing direction, the direction of the finger, and the orientation of the lower limbs.