An Empirical Study of Compound PCFG s (2021)

Abstract: Compound probabilistic context-free grammars (C-PCFGs) have recently established a new state of the art for phrase-structure grammar induction. However, due to the high time-complexity of chart-based representation and inference, it is difficult to investigate them comprehensively. In this work, we rely on a fast implementation of C-PCFGs to conduct evaluation complementary to that of (CITATION). We highlight three key findings: (1) C-PCFGs are data-efficient, (2) C-PCFGs make the best use of global sentence-level information in preterminal rule probabilities, and (3) the best configurations of C-PCFGs on English do not always generalize to morphology-rich languages.

## introduction probabilistic context-free grammars (pcfgs) have been used for unsupervised constituency grammar learning since decades ago @xcite , while learning pcfgs with the expectation maximization algorithm @xcite has been difficult as being involving non-convex optimization. recently, @xcite propose compound pcfgs, an over-parameterized neural model that extends corpus-level pcfgs by defining a mixture of pcfgs per sentence. c-pcfgs have achieved the state-of-the-art performance on english and chinese treebanks. they are also shown to be effective in a visually-grounded learning setting @xcite . however, because of the high time-complexity of chart-based representation and inference, it is hard to inspect c-pcfgs comprehensively. in this work, we rely on a fast implementation 1 of c-pcfgs to conduct a set of experiments complementary to those of @xcite . our 1 https://github.com/zhaoyanpeng/cpcfg . ## compound pcfgs compound pcfgs provide a novel parameterization of pcfgs. unlike pcfgs, which assign each grammar rule r a non-negative scalar π r such that r:a γ π @xmath0, c-pcfgs relax the strong context-free assumption of pcfgs by assuming that rule probabilities follow a compound distribution: learning c-pcfgs is formalized as maximizing the log likelihood of each observed sentence @xmath1. . . w n : c-pcfgs satisfy the context-free assumption conditioned on z and thus admit tractable inference for each given z. inference with c-pcfgs seeks the most probable parse t * of w: though given z, the maximum a posterior (map) inference over p θ (t|w, z) can be exactly solved by using the cyk algorithm, the integral over z renders inference intractable. the map inference is instead approximated by: similarly to c-pcfgs, neural pcfgs (n-pcfgs) also use neural networks to parameterize pcfgs, but their parameterization does not rely on the sentence-dependent z. in the following discussion, we will refer to z as 'sentence embedding'. ## experimental setup datasets: we investigate the parsing performance of c-pcfgs across ten languages. specifically, we conduct experiments on the wall street journal (wsj) corpus of the penn treebank @xcite for english, the penn chinese treebank 5.1 (ctb) @xcite for chinese, and eight additional treebanks from the spmrl 2014 shared task @xcite for the other eight languages (basque, german, french, hebrew, hungarian, korean, polish, swedish). we use the standard data splits for each treebank. following @xcite , punctuation is removed from all data; the top 10000 frequent tokens in the training data of each treebank are kept as the vocabulary. 3 unless otherwise specified, we train c-pcfgs on sentences no longer than 40 tokens. model hyperparameters and evaluation: we re-implement c-pcfgs relying on torch-struct @xcite and adopt the same hyperparameter settings as in @xcite . we train c-pcfgs for each language separately. on each treebank we run c-pcfgs four times with different random seeds and for 30 epochs. the best model in each run is selected according to the perplexity on the validation data. at test time, trivial spans, such as single-word and sentence-level spans, are ignored. we report average corpus-and sentence-level f1 numbers as well as the unbiased standard deviations. ## conclusion we have presented an in-depth analysis of c-pcfgs from a quantitative perspective. the analysis concerns three aspects of c-pcfgs: data efficiency and length generalization, the role of the latent sentence embedding, and multilingual performance. our experimental results show that c-pcfgs can learn well only from short sentences and maintain good performance at test time. the latent sentence embedding is crucial for the good performance of c-pcfgs. among the three rule types, preterminal rules make the most of it. however, the configurations of the best-performing c-pcfgs on english do not always generalize to morphologyrich languages.