Learning a Grammar Inducer from Massive Uncurated Instructional Videos (2022)

Abstract: Video-aided grammar induction aims to leverage video information for finding more accurate syntactic grammars for accompanying text. While previous work focuses on building systems for inducing grammars on text that are well-aligned with video content, we investigate the scenario, in which text and video are only in loose correspondence. Such data can be found in abundance online, and the weak correspondence is similar to the indeterminacy problem studied in language acquisition. Furthermore, we build a new model that can better learn video-span correlation without manually designed features adopted by previous work. Experiments show that our model trained only on large-scale YouTube data with no text-video alignment reports strong and robust performances across three unseen datasets, despite domain shift and noisy label issues. Furthermore our model yields higher F1 scores than the previous state-of-the-art systems trained on in-domain data.

## introduction grammar induction is a fundamental and longlasting @xcite @xcite problem in computational linguistics, which aims to find hierarchical syntactic structures from plain sentences. unlike supervised methods @xcite @xcite @xcite that require human annotated treebanks, e.g., penn treebank @xcite , grammar inducers do not rely on any human annotations for training. grammar induction is attractive since annotating syntactic trees by human language experts is expensive and time consuming, while the current treebanks are limited to several major languages and domains. recently, deep learning models have achieved remarkable success across nlp tasks, and neural models have been designed @xcite @xcite for grammar induction, which greatly advanced model performance on induction with raw text. recent efforts have started to consider other useful information from multiple modalities, such as images @xcite and videos @xcite . specifically, @xcite show that multi-modal information (e.g. motion, sound and objects) from videos can significantly improve the induction accuracy on verb and noun phrases. such work uses curated multi-modal data publicly available on the web, which all assume that the meaning of a sentence needs to be identical (e.g., being a caption) to the corresponding video or image. this assumption limits usable data to several small-scale benchmarks @xcite @xcite with expensive human annotations on image/video captions. the noisy correspondence between form and meaning is one of the main research questions in language acquisition @xcite @xcite , where different proposals attempt to address this indeterminacy faced by children. there has been computational work incorporating such indeterminacy into their models @xcite . for modeling empirical grammar learning with multi-modal inputs, two important questions still remain open: 1) how can a grammar inducer benefit from large-scale multi-media data (e.g., youtube videos) with noisy text-to-video correspondence? and 2) how can a grammar inducer show robust performances across multiple domains and datasets? by using data with only weak cross-modal correspondence, such as youtube videos and their automatically generated subtitles, we allow the computational models to face a similar indeterminacy problem, and exam-ine how indeterminacy interacts with data size to influence learning behavior and performance of the induction models. in this paper, we conduct the first investigation on both questions. specifically, we collect 2.4 million video clips and the corresponding subtitles from instructional youtube videos (howto100m @xcite ) to train multi-modal grammar inducers, instead of using the training data from a benchmark where text and video are in alignment. we then propose a novel model, named @xcite , that extends previous work @xcite by incorporating a videospan matching loss term into the compound pcfg @xcite model. to better capture the video-span correlation, it leverages clip @xcite , a state-of-the-art model pretrained on video subtitle retrieval, as the encoders for both video and text. compared with previous work @xcite that independently extracts features from each modality before merging them using a simple transformer @xcite encoder, the encoders of our model have been pretrained to merge such multi-modal information, and no human efforts are needed to select useful modalities from the full set. experiments on three benchmarks show that our model, which is trained on noisy youtube video clips and no data from these benchmarks, produces substantial gains over the previous state-of-the-art system @xcite trained on in-domain video clips with human annotated captions. furthermore, our model demonstrates robust performances across all three datasets. we suggest the limitations of our model and future directions for improvements through analysis and discussions. code will be released upon paper acceptance. in summary, the main contributions are: ## conclusion in this paper, we have investigated how massive instructional youtube video and subtitle pairs can improve grammar induction. we have also proposed a new model that leverages the latest advances in multi-modal pre-training to learn better videospan correlation. experiments on three benchmarks demonstrate superior and robust performances of our model over previous systems. we leave exploring other pre-trained video-text matching models and more publicly available data (e.g., youtube videos from other domains and tv shows) in future work.