  This paper presents a novel method, termed Bridge to Answer, to infer correct
answers for questions about a given video by leveraging adequate graph
interactions of heterogeneous crossmodal graphs. To realize this, we learn
question conditioned visual graphs by exploiting the relation between video and
question to enable each visual node using question-to-visual interactions to
encompass both visual and linguistic cues. In addition, we propose bridged
visual-to-visual interactions to incorporate two complementary visual
information on appearance and motion by placing the question graph as an
intermediate bridge. This bridged architecture allows reliable message passing
through compositional semantics of the question to generate an appropriate
answer. As a result, our method can learn the question conditioned visual
representations attributed to appearance and motion that show powerful
capability for video question answering. Extensive experiments prove that the
proposed method provides effective and superior performance than
state-of-the-art methods on several benchmarks.
