Dependency Induction Through the Lens of Visual Perception (2021)

Abstract: Most previous work on grammar induction focuses on learning phrasal or dependency structure purely from text. However, because the signal provided by text alone is limited, recently introduced visually grounded syntax models make use of multimodal information leading to improved performance in constituency grammar induction. However, as compared to dependency grammars, constituency grammars do not provide a straightforward way to incorporate visual information without enforcing language-specific heuristics. In this paper, we propose an unsupervised grammar induction model that leverages word concreteness and a structural vision-based heuristic to jointly learn constituency-structure and dependency-structure grammars. Our experiments find that concreteness is a strong indicator for learning dependency grammars, improving the direct attachment score (DAS) by over 50% as compared to state-of-the-art models trained on pure text. Next, we propose an extension of our model that leverages both word concreteness and visual semantic role labels in constituency and dependency parsing. Our experiments show that the proposed extension outperforms the current state-of-the-art visually grounded models in constituency parsing even with a smaller grammar size.

## introduction grammar induction aims to discover the underlying grammatical structure of a language from strings of symbols. previous work has focused on two grammar formalisms: constituency and dependency grammars. probabilistic context-free grammars @xcite have been used for modeling probabilistic rules in a constituency grammar, including more recent approaches that combine pcfgs with neural models @xcite . the dependency grammar 1 code is available at https://github.com/ ruisi-su/concrete_dep . and are the paths of the neural l-pcfg which uses pure text and our proposed concrete l-pcfg which uses word concreteness, respectively. neural l-pcfg incorrectly predicts observing as the root of the sentence, while concrete l-pcfg selects the correct root fans. the two formalisms are complementary and unified models have been shown to achieve higher induction accuracy than models that are trained for either constituency or dependency parsing alone. for instance, the lexicalized pcfg @xcite is a supervised model that jointly learns the two formalisms by associating a word and a part-ofspeech tag with each non-terminal in the tree of the pcfg. a more recent variant, the neural l-pcfg @xcite ) is an unsupervised model that uses lexical information to jointly learn both constituency-structure and dependency-structure grammars, achieving state-of-the-art performance in both formalisms. @xcite with standard deviations of the english mscoco captioning data @xcite . we exclude words that are unclassified or lack a dominant tag. besides the use of lexical information, grammar induction also benefits from a coarse understanding of the similarities in the syntactic structures shared by many human languages. recent works showed that biases based on such syntactic universals can be applied to the posterior distribution of the dependency structures @xcite @xcite . such universal linguistic rules can be helpful when part-ofspeech tags are given to the grammar induction models. however, explicitly providing a ruleset can make non-terminals sensitive to their context, which is slightly contradictory to context-free grammar based methods, such as pcfgs. thus, a more abstract method is needed to provide both guidance and flexibility to grammar induction models. nevertheless, inducing grammars entirely from pure text is a tall order, and arguably one not even tackled by humans themselves -humans learn language from the surrounding world @xcite . recent work, such as the visually grounded neural syntax learner @xcite show improved performance on constituency grammar induction by learning a shared embedding space for text and its corresponding image. however, @xcite note that the improvement in performance is not a result of learning complex syntactic rules, and is rather of the model indirectly learning word concreteness, a concept that evaluates the degree to which a word refers to a perceptible entity @xcite . however, concisely describing the effect of word concreteness on the constituency structure of a sentence is non-trivial without resorting to languagespecific heuristics such as the head-initial bias of the vgnsl @xcite . on the other hand, describing the effect of concreteness on dependency structure is more straightforward. in most languages, content words, such as nouns and verbs, tend to be more concrete as they are learned through experience via modalities @xcite . our analysis on the english mscoco dataset, shown in figure 2 , affirms this is true for english. content words are also more likely to be heads than dependents in many dependency specifications @xcite . motivated by these observations, in this paper, we incorporate the concept of concreteness in the unsupervised neural l-pcfg model (zhu et al., 2020) at two levels: the word level and the phrase level. first, at the word level, we incorporate a prior to upweight dependency structures that have heads with higher word concreteness scores, which are human-rated numerical values for common english words @xcite . our experiments on the english mscoco dataset show that, at the word level, the word concreteness priors greatly improve the dependency induction performance over the neural l-pcfg, increasing the directed attachment accuracy (das) by 50% (section 4). additionally, we investigate the effect on predicted roots and find that the concreteness priors serve as a "short-cut" to bias perceptible entities towards becoming heads in the dependency parse. in the example in figure 1 , the concrete l-pcfg selects the entity fans as the root word for the sentence, because it is more perceptible than the action observing. adding word concreteness priors to the root encourages the subsequently learned rules to select the lexical head with higher word concreteness. at the phrase level, we present a vision-based heuristic to exploit concreteness by connecting the word referents of the corresponding perceptible entities. these perceptible entities are extracted from images and aligned using a unified vision-language pre-training model @xcite . our experiments show the vision-based heuristic improves the model's constituency parsing performance. finally, we present a model that combines the concreteness priors at both the word and phrase levels, which further improves the f1 score by 12% over the neural l-pcfg for the constituency parsing task. in summary, this paper demonstrates that concreteness interacts with the neural l-pcfg differently at various levels: word-level concreteness helps dependency induction in differentiating between content and function words, and the phraselevel vision-based heuristic helps pruning grammars that produce incorrect spans. both levels help the neural l-pcfg to conceptualize the relationships between words that are highly associated with visually perceptible entities. ## background this section briefly discusses the neural l-pcfg (zhu et al., 2020), which jointly learns constituency-structure and dependency-structure grammars. we encourage readers to refer to the original paper for a more detailed description. the neural l-pcfg is a neural parameterization of the lexicalized pcfg @xcite and it demonstrates improved performance on unsupervised constituency and dependency parsing. the phrase structures from a context-free grammar (cfg) are defined as a five-tuple @xmath0, n , p, σ, r), including the following set of rules r: because it is intractable to integrate over z in equation 1, the evidence lower bound (elbo) of the log likelihood is estimated by monte-carlo sampling and optimized during training of the neural l-pcfg @xcite using the mean and the variance vectors predicted by an inference network. the neural l-pcfg is trained to maximize the log likelihood of the entire corpus. during inference, the most probable tree is approximated using the mean vector µ = f µ (x) (predicted by the inference network) in a dirac delta distribution δ(z -µ) instead of the real distribution p(z|x), which would be intractable: the probability distribution over sentences x is then defined by: in which z(t , z) is the normalizing factor, and @xmath1. the unnormalized probability is calculated by exponentiating a scoring function: pz (t) ∝ exp g θ (t, z), which assumes each rule is independent of the other rules in the parse tree t. ## dependency induction method we propose three model variants: the word concreteness model, the vision-based heuristic model, and a model that combines both concreteness and the heuristic for inducing dependency structures and constituency structures. ## experiments we run experiments for the three model variants we propose in the previous section: the word concreteness prior, the vision-based heuristic, and the combined method. for word concreteness, our experiments aim to answer the following questions: (1) "does incorporating concreteness in the learning process improve dependency induction?" and (2) "once concreteness has been used in learning, can the model still achieve good performance when concreteness information is no longer available during inference?". in particular, the second question asks whether the model can internalize the concept of concreteness during training and accurately induce dependency structure even when concreteness is not explicitly provided. additionally, we experiment with the vision-based heuristic model to explore (3) whether structured priors at the phraselevel can improve constituency parsing, and (4) whether it is also effective on improving dependency parsing. finally, we combine the priors to investigate the utility of concreteness at both the word and the phrase levels for further improving the performance of the neural l-pcfg. this combination allows more effect of the concreteness priors when training the neural l-pcfg to jointly learn the constituency and the dependency structures. we experiment on the combined model that applies both priors either at the root level (as in concrete l-pcfg), or at the rule-level that produces the parsed tree t (as in coupling) to fully understand the influence of the priors and the effect of where they are incorporated in the neural l-pcfg grammars. ## discussion and conclusion overall, our results show that the substantial increase in dependency induction performance using our method is due to the added word concreteness prior. we demonstrate that concreteness is a good approximation for understanding subcategories of words in a dependency parsing model. however, we recognize that this approximation is insufficient for verb phrases, because an action described by a verb is often coupled with visible entities @xcite . for example, in a sentence many people gather around a building with clocks, our approach incorrectly upweights people to be the root rather than gather, because the concreteness of the former is higher than the latter. similar to @xcite , our analysis in figure 4 shows that mscoco includes a large number of noun phrases (almost double the number of verb phrases). this disproportional ratio might contribute to the lower das for the dmv and text-only neural l-pcfg models. because mscoco contains shorter sentences with a simpler syntactic structure as compared to the penn treebank, neural l-pcfg converges to a local optimum that achieves high constituency accuracy but low dependency accuracy ( our experiments with the proposed vision-based heuristic show that both dependency and constituency accuracy measures can be improved by leveraging the joint learning setting of the neural l-pcfg and the visually derived information. unlike previous work on visually grounded models, our coupling heuristic selects what is visually significant from an image in the language modeling objective. however, certain limitations exist in our simple heuristic, including noise propagation from the semantic role labeling model. because mscoco has a large portion of noun phrases, the action predicted by the semantic role labeling model from the image is often misaligned with a noun term in the caption and vice versa. this could mislead the downstream grammar induction task and hurt the constituency accuracy. for example, the left constituent from the root in (d) of figure 5 is grammatically incorrect. given these observations, we removed the rewards to spans that couple the arguments when predicate is not aligned in the combined model. in future work, we plan to explore more ele-gant ways to parameterize the priors in the neural l-pcfg, instead of using the hyperparameter λ. nonetheless, different from other visually grounded models, which jointly learn the textual representations with the paired image,our approach directly influences the language model by encoding the visual information completely in the text domain. one advantage of our approach is that the visual information is directly applied to optimize a language modeling objective, allowing explainability of which parts of the image are helpful for learning the syntax of the corresponding caption. our work provides a basis for solving higherlevel syntax ambiguities. for example, the sentence the girl will put the orange on the tray in the bowl can be decomposed in two ways: [the girl will put][the orange on the tray][in the bowl] and [the girl will put the orange][on the tray in the bowl] @xcite . although both analyses are grammatically correct, only one is correct when a visual reference (i.e., an image) is provided. future dependency induction models could aim at untangling such ambiguity using more complex visual information. finally, although we show that concreteness is useful for dependency induction, using concreteness is seemingly an over-simplification for modeling semantic dependency relationships. in human language, root assignment can involve a priority mechanism: when a verb is present in a sentence, it immediately becomes the root despite how concrete other words are. our work suggests that further investigation into modeling such priority mechanisms to improve the grammar induction performance even more.