  Visual 2.5D perception involves understanding the semantics and geometry of a
scene through reasoning about object relationships with respect to the viewer
in an environment. However, existing works in visual recognition primarily
focus on the semantics. To bridge this gap, we study 2.5D visual relationship
detection (2.5VRD), in which the goal is to jointly detect objects and predict
their relative depth and occlusion relationships. Unlike general VRD, 2.5VRD is
egocentric, using the camera's viewpoint as a common reference for all 2.5D
relationships. Unlike depth estimation, 2.5VRD is object-centric and not only
focuses on depth. To enable progress on this task, we create a new dataset
consisting of 220k human-annotated 2.5D relationships among 512K objects from
11K images. We analyze this dataset and conduct extensive experiments including
benchmarking multiple state-of-the-art VRD models on this task. Our results
show that existing models largely rely on semantic cues and simple heuristics
to solve 2.5VRD, motivating further research on models for 2.5D perception. The
new dataset is available at https://github.com/google-research-datasets/2.5vrd.
