  We investigate Referring Image Segmentation (RIS), which outputs a
segmentation map corresponding to the given natural language description. To
solve RIS efficiently, we need to understand each word's relationship with
other words, each region in the image to other regions, and cross-modal
alignment between linguistic and visual domains. Recent methods model these
three types of interactions sequentially. We argue that such a modular approach
limits these methods' performance, and joint simultaneous reasoning can help
resolve ambiguities. To this end, we propose a Joint Reasoning (JRM) module and
a novel Cross-Modal Multi-Level Fusion (CMMLF) module for tackling this task.
JRM effectively models the referent's multi-modal context by jointly reasoning
over visual and linguistic modalities (performing word-word, image
region-region, word-region interactions in a single module). CMMLF module
further refines the segmentation masks by exchanging contextual information
across visual hierarchy through linguistic features acting as a bridge. We
present thorough ablation studies and validate our approach's performance on
four benchmark datasets, and show that the proposed method outperforms the
existing state-of-the-art methods on all four datasets by significant margins.
