Handling spatial references in natural language is a key challenge in tasks like autonomous navigation and robotic manipulation. Recent work has investigated various neural architectures for learning multi-modal representations of spatial conceptsthat generalize well across a variety of observations and text instructions . In this work , we develop accurate models for understanding spatial references in text that are also robust and interpretable  . We design a text-conditioned relation network whose parameters are dynamically computed with a cross-modal attention module to capture fine-grained spatial relations between entities. Our experiments across three different prediction tasks demonstrate the effectiveness of our model compared to existing state-of-the-art systems. Our model is robust to both observational and instructional noise, and lends itself to easy interpretation through visualization of intermediate outputs.\\%DIF > improvement in predicting goal locations and a 15\\% improvement in robustness compared to state-of-the-art systems.
Learning representations of spatial references in natural language is a key challenge in tasks like autonomous navigation and robotic manipulation. Recent work has investigated various neural architectures for learning multi-modal representations for spatial concepts. However, the lack of explicit reasoning over entities makes such approaches vulnerable to noise in input text or state observations . In this paper , we develop effective models for understanding spatial references in text that are robust and interpretable , without sacrificing performance . We design a text-conditioned relation network whose parameters are dynamically computed with a cross-modal attention module to capture fine-grained spatial relations between entities. This design choice provides interpretability of learned intermediate outputs. Experiments across three tasks demonstrate that our model achieves superior performance, with a 17\\%DIF > improvement in predicting goal locations and a 15\\% improvement in robustness compared to state-of-the-art systems.
R
Handling
Learning representations of
R
of spatial conceptsthat generalize well across a variety of observations and text instructions
for spatial concepts. However, the lack of explicit reasoning over entities makes such approaches vulnerable to noise in input text or state observations
R
work
paper
R
accurate
effective
D
also
A
, without sacrificing performance
R
relation network
relation network
R
Our experiments across three different prediction tasks demonstrate the effectiveness of our model compared to existing state-of-the-art systems. Our model is robust to both observational and instructional noise, and lends itself to easy interpretation through visualization of intermediate outputs.
This design choice provides interpretability of learned intermediate outputs. Experiments across three tasks demonstrate that our model achieves superior performance, with a 17