What is Learned in Visually Grounded Neural Syntax Acquisition (2020)

Abstract: Visual features are a promising signal for learning bootstrap textual models. However, blackbox learning models make it difficult to isolate the specific contribution of visual components. In this analysis, we consider the case study of the Visually Grounded Neural Syntax Learner (Shi et al., 2019), a recent approach for learning syntax from a visual training signal. By constructing simplified versions of the model, we isolate the core factors that yield the model’s strong performance. Contrary to what the model might be capable of learning, we find significantly less expressive versions produce similar predictions and perform just as well, or even better. We also find that a simple lexical signal of noun concreteness plays the main role in the model’s predictions as opposed to more complex syntactic reasoning.

## introduction language analysis within visual contexts has been studied extensively, including for instruction following (e.g., @xcite @xcite @xcite , visual question answering (e.g., @xcite @xcite , and referring expression resolution (e.g., @xcite @xcite . while significant progress has been made on such tasks, the combination of vision and language makes it particularly difficult to identify what information is extracted from the visual context and how it contributes to the language understanding problem. recently, @xcite proposed using alignments between phrases and images as a learning signal for syntax acquisition. this task has been long-studied from a text-only setting, including recently using deep learning based approaches @xcite @xcite @xcite . while the introduction of images provides a rich new signal for the task, it also introduces numerous challenges, such as identifying objects and analyzing scenes. in this paper, we analyze the visually grounded neural syntax learner (vg-nsl) model of @xcite . in contrast to the tasks commonly studied in the intersection of vision and language, the existence of an underlying syntactic formalism allows for careful study of the contribution of the visual signal. we identify the key components of the model and design several alternatives to reduce the expressivity of the model, at times, even replacing them with simple non-parameterized rules. this allows us to create several model variants, compare them with the full vg-nsl model, and visualize the information captured by the model parameters. broadly, while we would expect a parsing model to distinguish between tokens and phrases along multiple dimensions to represent different syntactic roles, we observe that the model likely does not capture such information. our experiments show that significantly less expressive models, which are unable to capture such distinctions, learn a similar model of parsing and perform equally and even better than the original vg-nsl model. our visualizations illustrate that the model is largely focused on acquiring a notion of noun concreteness optimized for the training data, rather than identifying higher-level syntactic roles. our code and experiment logs are available at https://github. com/lil-lab/vgnsl_analysis . ## model variations we consider varying the parameterization of vg-nsl, i.e., φ, combine, and score, while keeping the same inference algorithm and learning procedure. our goal is to constrain model expressivity, while studying its performance and outputs. embedding bottleneck we limit the information capacity of the parsing model by drastically reducing its dimensionality from @xmath0. we reduce dimensionality by wrapping the token embedding function with a bottleneck layer φ b (x) = mlp b (φ(x)), where mlp b is a twolayer feed-forward network mapping to the reduced size. this bottleneck limits the expressiveness of phrase embeddings throughout the parsing algorithm. during training, we compute both original and reduced embeddings. the original embeddings are used to compute the visual matching reward signal, whereas the reduced embeddings are used by score to determine parsing decisions. at test time, only the reduced embeddings are used. in the case of @xmath1, the model is reduced to using a single criteria. the low dimensional embeddings are also easy to visualize, and to characterize the type of information learned. ## experimental setup we train vg-nsl and our model variants using the setup of @xcite , including three training extensions: (a) +hi: adding a head-initial inductive bias to the training objective; (b) +fasttext: the textual representations are partially initialized with pre-trained fasttext @xcite ; and (c) -in: foot_0 disabling the normalization of image features. we follow the @xcite setup. we train all vg-nsl variants on 82,783 images and 413,915 captions from the mscoco @xcite training set. we evaluate unsupervised constituency parsing performance using 5,000 non-overlapping held-out test captions. we use additional 5,000 non-overlapping validation captions for model selection, as well as for our analysis and visualization in section 5. we generate binary gold-trees using benepar @xcite , an off-the-shelf supervised constituency parser. we notate model variations as d, score, combine. for example, 1, s ws , c me refers to dimensionality @xmath2, weighted sum scoring function (s ws ), and mean pooling combine (c me ). we train five models for each variation, and select the best checkpoint for each model by maximizing the parse prediction agreement on the validation captions between five models. the agreement is measured by the self-f 1 agreement score @xcite . this procedure is directly adopted from @xcite . we use the hyper-parameters from the original implementation without further tuning. we evaluate using gold trees by reporting f 1 scores on the ground-truth constituents and recall on several constituent categories. we report mean and standard deviation across the five models. training setting 1, sws, cme 2, sws, cme u basic setting 72.0 77.5 87.5 +hi 78.2 80.3 91.8 +hi+fasttext 80.5 83.1 92.3 +hi+fasttext-in 85.6 86.4 92.8 ## conclusion and related work we studied the vg-nsl model by introducing several significantly less expressive variants, analyzing their outputs, and showing they maintain, and even improve performance. our analysis shows that the visual signal leads vg-nsl to rely mostly on estimates of noun concreteness, in contrast to more complex syntactic reasoning. while our model variants are very similar to the original vg-nsl, they are not completely identical, as reflected by the self-f 1 scores in our work is related to the recent inference procedure analysis of @xcite . while they study what biases a specific inference algorithm introduces to the unsupervised parsing problem, we focus on the representation induced in a grounded version of the task. our empirical analysis is related to @xcite , who methodologically, and successfully replicate the results of @xcite to study their performance. the issues we study generalize beyond the parsing task. the question of what is captured by vision and language models has been studied before, including for visual question answering @xcite @xcite , referring expression resolution @xcite , and visual navigation @xcite . we ask this question in the setting of syntactic parsing, which allows to ground the analysis in the underlying formalism. our conclusions are similar: multi-modal models often rely on simple signals, and do not exhibit the complex reasoning we would like them to acquire.