The Return of Lexical Dependencies: Neural Lexicalized PCFG s (2020)

Abstract: In this paper we demonstrate that context free grammar (CFG) based methods for grammar induction benefit from modeling lexical dependencies. This contrasts to the most popular current methods for grammar induction, which focus on discovering either constituents or dependencies. Previous approaches to marry these two disparate syntactic formalisms (e.g., lexicalized PCFGs) have been plagued by sparsity, making them unsuitable for unsupervised grammar induction. However, in this work, we present novel neural models of lexicalized PCFGs that allow us to overcome sparsity problems and effectively induce both constituents and dependencies within a single model. Experiments demonstrate that this unified framework results in stronger results on both representations than achieved when modeling either formalism alone.1

## introduction unsupervised grammar induction aims at building a formal device for discovering syntactic structure from natural language corpora. within the scope of grammar induction, there are two main directions of research: unsupervised constituency parsing, which attempts to discover the underlying structure of phrases, and unsupervised dependency parsing, which attempts to discover the underlying relations between words. early work on induction of syntactic structure focused on learning phrase structure and generally used some variant of probabilistic context-free grammars (pcfgs; @xcite @xcite . in recent years, dependency grammars have gained favor as an alternative syntactic formulation @xcite @xcite . specifically, the dependency model with valence (dmv) @xcite forms the basis for many modern approaches in dependency induction. most recent models for grammar induction, be they for pcfgs, dmvs, or other formulations, have generally coupled these models with some variety of neural model to use embeddings to capture word similarities, improve the flexibility of model parameterization, or both @xcite @xcite . notably, the two different syntactic formalisms capture very different views of syntax. phrase structure takes advantage of an abstracted recursive view of language, while the dependency structure more concretely focuses on the propensity of particular words in a sentence to relate to each other syntactically. however, few attempts at unsupervised grammar induction have been made to marry the two and let both benefit each other. this is precisely the issue we attempt to tackle in this paper. as a specific formalism that allows us to model both formalisms at once, we turn to lexicalized probabilistic context-free grammars (l-pcfgs; @xcite . l-pcfgs borrow the underlying machinery from pcfgs but expand the grammar by allowing rules to include information about the lexical heads of each phrase, an example of which is shown in figure 1 . the head annotation in the l-pcfg provides lexical dependencies that can be informative in estimating the probabilities of generation rules. for example, the probability of vp @@xmath0, because ''chasing'' is a present participle. historically, these grammars have been mostly used for supervised parsing, combined with traditional count-based estimators of rule probabilities @xcite . within this context, lexicalized grammar rules are powerful, figure 1 : lexicalized phrase structure tree for ''the dog is chasing the cat.'' the head word of each constituent is indicated with parentheses. but the counts available are sparse, and thus required extensive smoothing to achieve competitive results @xcite . in this paper, we contend that with recent advances in neural modeling, it is time to return to modeling lexical dependencies, specifically in the context of unsupervised constituent-based grammar induction. we propose neural l-pcfgs as a parameter-sharing method to alleviate the sparsity problem of lexicalized pcfgs. figure 2 illustrates the generation procedure of a neural l-pcfg. different from traditional lexicalized pcfgs, the probabilities of production rules are not independently parameterized, but rather conditioned on the representations of nonterminals, preterminals, and lexical items ( §3). apart from devising lexicalized production rules ( §2.3) and their corresponding scoring function, we also follow @xcite compound pcfg model for (non-lexicalized) constituency parsing with compound variables ( §3.2), enabling modeling of a continuous mixture of grammar rules. 2 we define how to efficiently train ( §4.1) and perform inference ( §4.2) in this model using dynamic programming and variational inference. put together, we expect this to result in a model that both is effective, and simultaneously induces both phrase structure and lexical dependencies, foot_2 whereas previous work has focused on only one. our empirical evaluation examines this hypothesis, asking the following question: in neural grammar induction models, is it possible to jointly and effectively learn both phrase structure and lexical dependencies? is using both in concert better at the respective tasks than specialized methods that model only one at a time? our experiments ( §5.3) answer in the affirmative, with better performance than baselines designed specially for either dependency or constituency parsing under multiple settings. importantly, our detailed ablations show that methods of factorization play an important role in the performance of neural l-pcfgs ( §5.3.2). finally, qualitatively ( §5.4), we find that latent labels induced by our model align with annotated gold non-terminals in ptb. ## motivation and definitions in this section, we will first provide the background of constituency grammars and dependency grammars, and then formally define the general l-pcfg, illustrating how both dependencies and phrase structures can be induced from l-pcfgs. ## conclusion in this paper, we propose neural l-pcfg, a neural parameterization method for lexicalized pcfgs, for both unsupervised dependency parsing and constituency parsing. we also provide a variational inference method to train our model. by modeling both representations together, our approach outperforms methods specially designed for either grammar formalism alone. importantly, our work also adds novel insights for the unsupervised grammar induction literature by probing the role that factorizations and initialization have on model performance. different factorizations of the same probability distribution can lead to dramatically different performance and should be viewed as playing an important role in the inductive bias of learning syntax. additionally, where others have used pretrained word vectors before, we show that they too contain abstract syntactic information which can bias learning. finally, although out of scope for one paper, our results point to several interesting potential roads forward, including the study of the effectiveness of jointly modeling constituency-dependency representations on freer word order languages, and whether other distributed word presentations (e.g., large-scale transformers) might provide even stronger syntactic signals for grammar induction. despite the demonstrated success of lexical dependencies, it should be noted that these are only unilexical dependencies, in contrast to bilexical dependencies, which also consider the dependencies between head and dependent words. modeling these dependencies would require marginalizing over all possible dependents for each span-head pair. in this case, the time complexity of exhaustive dynamic programming over one sentence would become o(l there are several potential methods to side-step this problem, including the use of sampling in lieu of dynamic programming, using heuristic methods to prune the grammar, and designing acceleration methods on gpu @xcite .