Grounded PCFG Induction with Images (2020)

Abstract: Recent work in unsupervised parsing has tried to incorporate visual information into learning, but results suggest that these models need linguistic bias to compete against models that only rely on text. This work proposes grammar induction models which use visual information from images for labeled parsing, and achieve state-of-the-art results on grounded grammar induction on several languages. Results indicate that visual information is especially helpful in languages where high frequency words are more broadly distributed. Comparison between models with and without visual information shows that the grounded models are able to use visual information for proposing noun phrases, gathering useful information from images for unknown words, and achieving better performance at prepositional phrase attachment prediction.

## introduction recent grammar induction models are able to produce accurate grammars and labeled parses with raw text only @xcite @xcite , providing evidence against the poverty of the stimulus argument @xcite , and showing that many linguistic distinctions like lexical and phrasal categories can be directly induced from raw text statistics. however, as computational-level models of human syntax acquisition, they lack semantic, pragmatic and environmental information which human learners seem to use @xcite @xcite . this paper proposes novel grounded neuralnetwork-based models of grammar induction which take into account information extracted from images in learning. performance comparisons show figure 1 : examples of disambiguating information provided by images for the prepositional phrase attachment of the sentence mary eats spaghetti with a friend @xcite . ## related work existing unsupervised pcfg inducers exploit naturally-occurring cognitive and developmental constraints, such as punctuation as a proxy to prosody @xcite , human memory constraints @xcite @xcite , and morphology @xcite , to regulate the posterior of grammars which are known to be extremely multimodal @xcite . models in @xcite also match embeddings of word spans to encoded images to induce unlabeled hierarchical structures with a concreteness measure @xcite . additionally, visual information is observed to provide grounding for words describing concrete objects, helping to identify and categorize such words. this hypothesis is termed 'noun bias' in language acquisition @xcite @xcite , through which the early acquisition of nouns is attributed to nouns referring to observable objects. however, the models in @xcite also rely on language-specific branching bias to outperform other text-based models, and images are encoded by pretrained object classifiers trained with large datasets, with no ablation to show the benefit of visual information for unsupervised parsing. visual information has also been used for joint training of prepositional phrase attachment models @xcite suggesting that visual information may contain semantic information to help disambiguate prepositional phrase attachment. ## grounded grammar induction model the full grounded grammar induction model used in these experiments, imagepcfg, consists of two parts: a word-based pcfg induction model and a vision model, as shown in figure 2 . the two parts have their own objective functions. the pcfg induction model, called noimagepcfg when trained by itself, can be trained by maximizing the marginal probability p(σ) of sentences σ. this part functions similarly to previously proposed pcfg induction models @xcite where a pcfg is induced through maximization of the data likelihood of the training corpus marginalized over latent syntactic trees. the image encoder-decoder network in the vision model is trained to reconstruct the original image after passing through an information bottleneck. the latent encoding from the image encoder may be seen as a compressed representation of vi-sual information in the image, some of which is semantic, relating to objects in the image. we hypothesize that semantic information can be helpful in syntax induction, potentially through helping three tasks mentioned above. in contrast to the full model where the encoded visual representations are trained from scratch, the imageprepcfg model uses image embeddings encoded by pretrained image classifiers with parameters fixed during induction training. we hypothesize that pretrained image classifiers may provide useful information about objects in an image, but for grammar induction it is better to allow the inducer to decide which kind of information may help induction. the two parts are connected through a syntacticvisual loss function connecting a syntactic sentence embedding projected from word embeddings and an image embedding. we hypothesize that visual information in the encoded images may help constrain the search space of syntactic embeddings of words with supporting evidence of lexical attributes such as concreteness for nouns or correlating adjectives with properties of objects. foot_2 ## vision model the vision model consists of an image encoderdecoder network and a syntactic-visual projector. the image encoder-decoder network encodes an image into an image embedding and then decodes that back into the original image. this reconstruction constrains the information in the image embedding to be closely representative of the original image. the syntactic-visual projector projects word embeddings used in the calculation of lexical expansion probabilities into the space of image embeddings, building a connection between the space of syntactic information and the space of visual information. ## optimization there are three different kinds of objectives used in the optimization of the full grounded induction model. the first loss is the marginal likelihood loss for the pcfg induction model described in equation 1, which can be calculated with the inside algorithm. the second loss is the syntactic-visual loss. given the encoded image embedding e m and the projected syntactic-visual embedding e σ of a sentence σ, the syntactic-visual loss is the mean squared error of these two embeddings: the third loss is the reconstruction loss of the image. given the original image represented as a vector i m and the reconstructed image îm , the reconstruction objective is the mean squared error of the corresponding pixel values of the two images: models with different sets of input optimize the three losses differently for clean ablation. noim-agepcfg, which learns from text only, optimizes the negative marginal likelihood loss (the negative of equation 1 ) using gradient descent. the model with pretrained image encoders, imageprepcfg, optimizes the negative marginal likelihood and the syntactic-visual loss (equation 13 ) simultaneously. the full grounded grammar induction model im-agepcfg learns from text and images jointly by minimizing all three objectives: negative marginal likelihood, syntactic-visual loss and image reconstruction loss (equation 14 ): experiments described in this paper use the mscoco caption data set @xcite and the multi30k dataset @xcite , which contains pairs of images and descriptions of images written by human annotators. captions in the mscoco data set are in english, whereas captions in the multi30k dataset are in english, german and french. captions are automatically parsed @xcite to generate a version of the reference set with constituency trees. 6 in addition to these datasets with captions generated by human annotators, we automatically translate the english captions into chinese, polish and korean using google translate, 7 and parse the resulting translations into constituency trees, which are then used in experiments to probe the interactions between visual information and grammar induction. results from models proposed in this paper -noimagepcfg, imageprepcfg and imagepcfg -are compared with published results from @xcite , which include prpn @xcite , on-lstm @xcite as well as the grounded vg-nsl models which uses either head final bias (vg-nsl+h) or head final bias and fasttext embeddings (vg-nsl+h+f) as inductive biases from external sources. all of these models only induce unlabeled structures and have been evaluated with unlabeled f1 scores. we additionally report the labeled evaluation score recall-homogeneity @xcite for better comparison between the proposed models. all evaluation is done on viterbi parse trees of the test set from 5 different runs. details about hyper-parameters and results on development data sets can be found in the appendix. however, importantly, the tuned hyperparameters for the grammar induction model are the same across the three proposed models, which facilitates direct comparisons among these models to determine the effect of visual information on induction. ## analysis of advantages of visual information we hypothesize three specific ways that visual information may help grammar induction. first, a strong correlation between words and objects in images can help identification and categorization