Character-based PCFG Induction for Modeling the Syntactic Acquisition of Morphologically Rich Languages (2021)

Abstract: Unsupervised PCFG induction models, which build syntactic structures from raw text, can be used to evaluate the extent to which syntactic knowledge can be acquired from distributional information alone. However, many state-of-the-art PCFG induction models are word-based, meaning that they cannot directly inspect functional affixes, which may provide crucial information for syntactic acquisition in child learners. This work first introduces a neural PCFG induction model that allows a clean ablation of the influence of subword information in grammar induction. Experiments on child-directed speech demonstrate first that the incorporation of subword information results in more accurate grammars with categories that word-based induction models have difficulty finding, and second that this effect is amplified in morphologically richer languages that rely on functional affixes to express grammatical relations. A subsequent evaluation on multilingual treebanks shows that the model with subword information achieves state-of-the-art results on many languages, further supporting a distributional model of syntactic acquisition.

## introduction unsupervised pcfg induction models @xcite induce grammars from raw text and use those induced grammars to build linguistically meaningful hierarchical structures for sentences. recent work on pcfg induction has adopted bayesian or neural word-based pcfg models @xcite @xcite , which have proven to be accurate at discovering syntactic structures solely from word sequences. these results indicate that a human-like grammar can be learned from distributional data @xcite @xcite , providing some evidence against the poverty of the stimulus argument @xcite in language acquisition. however, despite their high performance, these word-based pcfg induction models are not fully suitable as computational-level models of syntactic acquisition. specifically, they treat words as symbols and do not have direct access to subword information. in contrast, child language learners are known to be sensitive to word-internal functional affixes from a very young age @xcite , which may provide crucial information for syntactic acquisition @xcite . this would presumably make wordbased pcfg induction models less appropriate for modeling the acquisition of morphologically rich languages, in which most information about syntactic units and relations is expressed at the subword level @xcite . in order to address this issue, this work first defines a character-based model and a minimallymanipulated word-based counterpart for neural pcfg induction. 1 this formulation allows a clean ablation of subword information, and therefore allows its influence on grammar induction to be studied. experiments using child-directed speech demonstrate that the incorporation of subword information results in more accurate grammars with coherent syntactic categories that the word-based induction model has difficulty finding. additionally, this effect is found to be amplified in morphologically richer languages that rely on functional affixes to express grammatical relations. finally, an evaluation on multilingual treebanks shows that the model with subword information achieves stateof-the-art induction results on many languages, providing further evidence for a distributional model of syntactic acquisition. ## related work early work in unsupervised pcfg induction from raw text @xcite @xcite was not as successful as models of unsupervised constituency parsing @xcite . however, recent work from unsupervised parsing @xcite @xcite and grammar induction @xcite @xcite @xcite shows much improvement over previous results with grammars learned solely from raw text, indicating that statistical regularities relevant to syntactic acquisition can be found in word collocations. for example, @xcite propose a word-based neural compound pcfg induction model for accurate grammar induction on english. zhu et al. (2020) further extend this compound pcfg induction model to jointly induce lexical dependencies using a lexicalized pcfg. @xcite augment a pcfg induction model to use contextualized word embeddings with subword information to allow morphological cues to influence induction. unfortunately, the elmo embeddings @xcite used in that work are trained with a large number of tokens, which limits the value of this approach for investigating child-like syntactic acquisition. in an attempt to answer a similar question, there has also been recent interest in evaluating whether recurrent neural networks like lstms can learn grammar-like representations from word sequence alone. although some initial results seemed promising @xcite , later studies have shown that some of these results may not necessarily generalize to languages other than english @xcite , yielding an inconsistent picture. moreover, studies in this line of research try to model specific syntactic phenomena such as subject-verb agreement, filler-gap dependencies, and auxiliary inversion, and therefore have a slightly different focus from grammar induction. complementary to this approach are studies that aim to reconstruct syntactic representations from contextualized word representations @xcite . again, however, the use of neural language models assumes access to much more data than the input available to the typical child @xcite . ## models experiments described in this paper use two variants of a neural pcfg induction model, which differ minimally in how terminal expansion is modeled. a chomsky normal form pcfg with c nonterminal categories is first factored into two separate parts: binary-branching nonterminal expansion rule foot_1 probabilities, and unary-branching terminal expansion rule probabilities @xcite . given a tree as a set τ of nodes η undergoing nonterminal expansions c η → c η1 c η2 (where η ∈ {1, 2} * is a gorn address specifying a path of left or right branches from the root), and a set τ of nodes η undergoing terminal expansions c η → w η (where w η is the word at node η), the marginal probability of a sentence σ can be computed as: to allow the terminal expansion model to be separated out from the rest of the grammar induction model and make the character-and wordbased models differ only by the terminal expansion model, we first define a set of bernoulli distributions that distribute probability mass between these two sets of rules: there are two fully connected layers before and after the residual blocks: all w's and b's are model parameters, and s c η are the logits for the final softmax in equation 2. b is set to 2 in all models @xcite . this formulation naturally allows the model to learn from the input data how to allocate preterminal and other nonterminal categories, thereby making it more appropriate for multilingual settings (e.g. agglutinative languages that have many word types may need more preterminal categories in comparison to other nonterminal categories). binary-branching nonterminal expansion rule probabilities for a nonterminal category c η are defined as: the binary-branching nonterminal expansion distribution is defined for all models as: the representation of the word w η is usually symbolic in pcfg induction models. in order to probe the effect of character-based models in incorporating subword information to grammar induction, models in this work differ in how words are represented and in turn how the terminal expansion models are defined. the word-based models (neu-ralword) use symbolic representation of words, whereas the character-based models (neuralchar) use character sequences as observations for the terminal expansion models. they differ minimally by the terminal expansion models and the input they take, which allows a clean manipulation of subword information while holding all other model components fixed. ## experiment 1: evaluation on child-directed speech this work first explores the influence of subword information in grammar induction by comparing the performance of the neuralchar and neuralword models on child-directed speech corpora, which represent authentic linguistic input that child learners are exposed to. ## experiment 2: evaluation on multilingual treebanks subsequently, in order to determine if the evaluated models are competitive with state-of-the-art grammar induction models, we present experiments on multilingual newswire treebanks to test the performance of the neuralchar and neuralword models using both labeled evaluation (rh) and standard unlabeled evaluation (bracketing f1). these are evaluated on newswire to compare against models developed on predominantly newswire data. the results show that the neuralchar model outperforms these models in grammar induction on most languages. in addition, comparison of model performance on different languages hints at the inductive biases embodied by each of these systems. ## conclusion and future work this paper presents a character-based model and a minimally-manipulated word-based counterpart for neural pcfg induction. the character-based model allows subword information to influence grammar induction, which is more consistent with the information that child language learners are able to leverage. experiments and analyses using child-directed speech corpora show that the incorporation of subword information results in more accurate grammars with linguistically homogeneous syntactic categories, with its impact being stronger on morphologically richer languages. additionally, this model achieves state-of-the-art induction results on many languages, providing further support for a distributional model of syntactic acquisition. taken together, these results indicate that the proposed character-based model incorporates more realistic input and captures more successful learning outcomes, thus making it a more plausible cognitive model. while this work addresses a major drawback of word-based pcfg induction models, the proposed model nonetheless relies on symbolic representations of characters, which are abstractions from potentially noisy perceptual input. future work could explore the extent to which syntactic knowledge can be acquired from lower-level (e.g. phonemic or acoustic) input alone by including a word segmentation task @xcite for the model. additionally, recent work in unsupervised grammar induction @xcite has shown that incorporating visual information in the form of images and videos helps learn constituents that denote entities or action. adopting such grounded approaches can help answer questions about the extent to which the visual scene or other contexts contain relevant information, and eventually about the nature of input required for syntactic acquisition.