Video-aided Unsupervised Grammar Induction (2021)

Abstract: We investigate video-aided grammar induction, which learns a constituency parser from both unlabeled text and its corresponding video. Existing methods of multi-modal grammar induction focus on grammar induction from text-image pairs, with promising results showing that the information from static images is useful in induction. However, videos provide even richer information, including not only static objects but also actions and state changes useful for inducing verb phrases. In this paper, we explore rich features (e.g. action, object, scene, audio, face, OCR and speech) from videos, taking the recent Compound PCFG model as the baseline. We further propose a Multi-Modal Compound PCFG model (MMC-PCFG) to effectively aggregate these rich features from different modalities. Our proposed MMC-PCFG is trained end-to-end and outperforms each individual modality and previous state-of-the-art systems on three benchmarks, i.e. DiDeMo, YouCook2 and MSRVTT, confirming the effectiveness of leveraging video information for unsupervised grammar induction.

## introduction constituency parsing is an important task in natural language processing, which aims to capture syntactic information in sentences in the form of constituency parsing trees. many conventional approaches learn constituency parser from humanannotated datasets such as penn treebank @xcite . however, annotating syntactic trees by human language experts is expensive and timeconsuming, while the supervised approaches are limited to several major languages. in addition, the treebanks for training these supervised parsers are small in size and restricted to the newswire domain, thus their performances tend to be worse when applying to other domains @xcite . to address these issues, recent approaches @xcite @xcite design unsupervised constituency parsers and grammar inducers, since they can be trained on large-scale unlabeled data. in particular, there has been growing interests in exploiting visual information for unsupervised grammar induction because visual information can capture important knowledge required for language learning that is ignored by text @xcite @xcite . this task aims to learn a constituency parser from raw unlabeled text aided by its visual context. previous methods @xcite @xcite learn to parse sentences by exploiting object information from images. however, images are static and cannot present the dynamic interactions among visual objects, which usually correspond to verb phrases that carry important information. there-fore, images and their descriptions may not be fullyrepresentative of all linguistic phenomena encountered in learning, especially when action verbs are involved. for example, as shown in figure 1 (a), when parsing a sentence "a squirrel jumps on stump", a single image cannot present the verb phrase "jumps on stump" accurately. moreover, as shown in figure 1 (b), the guitar sound and the moving fingers clearly indicate the speed of music playing, while it is impossible to present only with a static image as well. therefore, it is difficult for previous methods to learn these constituents, as static images they consider lack dynamic visual and audio information. in this paper, we address this problem by leveraging video content to improve an unsupervised grammar induction model. in particular, we exploit the current state-of-the-art techniques in both video and audio understanding, domains of which include object, motion, scene, face, optical character, sound, and speech recognition. we extract features from their corresponding state-of-the-art models and analyze their usefulness with the vc-pcfg model @xcite . since different modalities may correlate with each other, independently modeling each of them may be sub-optimal. we also propose a novel model, multi-modal compound probabilistic context-free grammars (mmc-pcfg), to better model the correlation among these modalities. experiments on three benchmarks show substantial improvements when using each modality of the video content. moreover, our mmc-pcfg model that integrates information from different modalities further improves the overall performance. our code is available at https://github.com/ sy-zhang/mmc-pcfg . the main contributions of this paper are: ## background and motivation our model is motivated by c-pcfg @xcite and its variant of the image-aided unsupervised grammar induction model, vc-pcfg @xcite ). we will first review the evolution of these two frameworks in sections 2.1-2.2, and then discuss their limitations in section 2.3. ## multi-modal compound pcfgs in this section, we introduce the proposed multimodal compound pcfgs (mmc-pcfg). instead of purely relying on object information from images, we generalize vc-pcfg into the video domain, where multi-modal video information is considered. we first introduce the video representation in section 3.1. we then describe the procedure for matching the multi-modal video representation with each span in section 3.2. after that we introduce the training and inference details in section 3.3.