Visually Grounded Compound PCFG s (2020)

Abstract: Exploiting visual groundings for language understanding has recently been drawing much attention. In this work, we study visually grounded grammar induction and learn a constituency parser from both unlabeled text and its visual groundings. Existing work on this task (Shi et al., 2019) optimizes a parser via Reinforce and derives the learning signal only from the alignment of images and sentences. While their model is relatively accurate overall, its error distribution is very uneven, with low performance on certain constituents types (e.g., 26.2% recall on verb phrases, VPs) and high on others (e.g., 79.6% recall on noun phrases, NPs). This is not surprising as the learning signal is likely insufficient for deriving all aspects of phrase-structure syntax and gradient estimates are noisy. We show that using an extension of probabilistic context-free grammar model we can do fully-differentiable end-to-end visually grounded learning. Additionally, this enables us to complement the image-text alignment loss with a language modeling objective. On the MSCOCO test captions, our model establishes a new state of the art, outperforming its non-grounded version and, thus, confirming the effectiveness of visual groundings in constituency grammar induction. It also substantially outperforms the previous grounded model, with largest improvements on more ‘abstract’ categories (e.g., +55.1% recall on VPs).

## introduction grammar induction is a task of finding latent hierarchical structure of language. as a fundamental problem in computational linguistics, it has been extensively studied for decades (lari and young, 1990; carroll and charniak, 1992; clark, 2001; 1 our code is available at https://git.io/ju0jj . @xcite . recently, deep learning models have been shown very effective across nlp tasks and have also been applied to grammar induction, greatly advancing the area @xcite @xcite . these neural grammar-induction approaches have been generally limited to relying on text, without considering learning signals from other modalities. in contrast, the crucial aspect of natural language learning is that it is grounded in perceptual experiences @xcite @xcite . we thus anticipate improved language understanding by leveraging grounded learning. promising results from grounded learning have been emerging in areas such as representation learning @xcite @xcite . typically, they use visual images as perceptual groundings of language and aim at improving continuous vector representations of language (e.g., word or sentence embeddings). in this work, we consider a more challenging problem: can visual groundings help us induce syntactic structure? we refer to this problem as visually grounded grammar induction. @xcite propose a visually grounded neural syntax learner (vg-nsl) to tackle the task. specifically, they learn a parser from aligned imagesentence pairs (e.g., image-caption data), where each sentence describes visual content of the corresponding image. the parser is optimized via rein-force, where the reward is computed by scoring the alignment of images and constituents. while straightforward, matching-based rewards can, as we will discuss further in the paper, make the parser focus only on more local and short constituents (e.g., 79.6% recall on nps) and to perform poorly on longer ones (e.g., 26.2% recall on vps) @xcite . while for the former it outperforms the text-only grammar induction methods, for the latter it substantially underachieves. this may not be surprising, as it is not guaranteed that every constituent of a sentence has its visual representation in the aligned image; the reward signals can be noisy and insufficient to capture all aspects of phrase-structure syntax. consequently, @xcite have to rely on language-specific inductive bias to obtain more informative reward signals. another issue with vg-nsl is that the parser does not admit tractable estimation of the partition function and the posterior probabilities for constituent boundaries needed to compute the expected reward in closed form. instead, vg-nsl relies on monte carlo policy gradients, potentially suffering from high variance. to alleviate the first issue, we propose to complement the image-text alignment-based loss with a loss defined on unlabeled text (i.e., its loglikelihood). as re-confirmed with neural models in @xcite and @xcite , text itself can drive induction of rich syntactic knowledge, so additionally optimizing the parser on raw text can be beneficial and complementary to visual grounded learning. to resolve the second issue, we resort to an extension of probabilistic contextfree grammar (pcfg) parsing model, compound pcfg @xcite . it admits tractable estimation of the posteriors, needed in the alignment loss, with dynamical programming and leads to a fully-differentiable end-to-end visually grounded learning. more importantly, the pcfg parser lets us complement the alignment loss with a language modeling objective. our key contributions can be summarized as follows: (1) we propose a fully-differentiable endto-end visually grounded learning framework for grammar induction; (2) we additionally optimize a language modeling objective to complement visually grounded learning; (3) we conduct experiments on mscoco @xcite and observe that our model has a higher recall than vg-nsl for five out of six most frequent constituent labels. for example, it surpasses vg-nsl by 55.1% recall on vps and by 48.7% recall on prepositional phrases (pps). comparing to a model trained purely via visually grounded learning, extending the loss with a language modeling objective improves the overall f1 from 50.5% to 59.4%. ## background and motivation our model relies on compound pcfgs @xcite and generalizes the visually grounded gram-mar learning framework of @xcite . we will describe the relevant aspects of both frameworks in sections 2.1-2.2, and then discuss their limitations (section 2.3). ## visually grounded compound pcfgs we use compound pcfgs @xcite and develop visually-grounded compound pcfgs (vc-pcfgs) within the contrastive learning framework. instead of sampling a tree and computing a point estimate of the image-text matching loss, we can compute the expected image-text matching loss under a tree distribution and use end-to-end contrastive learning (section 3.1). since it is inefficient to compute constituent representations relying on the chart, we will introduce an additional textual representation model to encode constituents (section 3.2). moreover, vc-pcfgs let us additionally optimize a language modeling objective, complementing the visually grounded contrastive learning (section 3.3). ## related work grammar induction has a long history in computational linguistics. following observations that direct optimization of log-likelihood with the expectation maximization algorithm @xcite is not effective at producing effective grammars, a number of approaches have been developed, emboding various inductive biases or assumption about the language structure and its relation to surface realizations @xcite @xcite . the recent advances in the area have been brought by flexible neural models @xcite @xcite . all these methods, with the exception of @xcite , rely solely on text. visually grounded learning is motivated by the observation that natural language is grounded in perceptual experiences @xcite @xcite @xcite . it has been shown effective in word representation learning @xcite @xcite and sentence representation learning @xcite . all this work uses visual images as perceptual experience of language and exploits visual semantics derived from images to improve continuous vector representatios of language. in contrast, we induce structured representations, discrete tree structure of language, by using visual groundings. we propose a model for the task within the contrastive learning framework. learning involves estimating concreteness of spans, which generalizes word-level concreteness @xcite . in the vision and machine learning community, unsupervised induction of structured image representations (aka scene graphs or world models) has been receiving increasing attention @xcite @xcite . however, they typically rely solely on visual signal. an interesting extension of our work would be to consider joint induction of structured representations of images and text while guiding learning by an alignment loss. ## conclusion we have presented visually-grounded compound pcfgs (vc-pcfgs) that use compound pcfgs and generalize the visually grounded grammar learning framework. vc-pcfgs exploit visual groundings via contrastive learning, with learning signals derived from minimizing an image-text alignment loss. to tackle the issues of misleading and insufficient learning signals from purely agreement-based learning, we propose to complement the image-text alignment loss with a loss defined on unlabeled text. we resort to using compound pcfgs which enables us to complement the alignment loss with a language modeling objective, resulting in a fully-differentiable end-to-end visually grounded learning. we empirically show that our vc-pcfgs are superior to models that are trained only through visually grounded learning or only relying on text.