  Neural module networks (NMN) have achieved success in image-grounded tasks
such as Visual Question Answering (VQA) on synthetic images. However, very
limited work on NMN has been studied in the video-grounded language tasks.
These tasks extend the complexity of traditional visual tasks with the
additional visual temporal variance. Motivated by recent NMN approaches on
image-grounded tasks, we introduce Video-grounded Neural Module Network (VGNMN)
to model the information retrieval process in video-grounded language tasks as
a pipeline of neural modules. VGNMN first decomposes all language components to
explicitly resolve any entity references and detect corresponding action-based
inputs from the question. The detected entities and actions are used as
parameters to instantiate neural module networks and extract visual cues from
the video. Our experiments show that VGNMN can achieve promising performance on
two video-grounded language tasks: video QA and video-grounded dialogues.
