The Importance of Category Labels in Grammar Induction with Child-directed Utterances (2020)

Abstract: Recent progress in grammar induction has shown that grammar induction is possible without explicit assumptions of language specific knowledge. However, evaluation of induced grammars usually has ignored phrasal labels, an essential part of a grammar. Experiments in this work using a labeled evaluation metric, RH, show that linguistically motivated predictions about grammar sparsity and use of categories can only be revealed through labeled evaluation. Furthermore, depth-bounding as an implementation of human memory constraints in grammar inducers is still effective with labeled evaluation on multilingual transcribed child-directed utterances.

## introduction recent work in probabilistic context-free grammar (pcfg) induction has shown that it is possible to learn accurate grammars from raw text @xcite @xcite , which is significant in addressing the issue of the poverty of the stimulus @xcite in linguistics. although phrasal categories and morphosyntactic features can be induced from raw text @xcite , most unsupervised parsing work has been evaluated using unlabeled parsing accuracy scores @xcite @xcite @xcite . this is potentially distortative because children and adults can distinguish categories of phrases and clauses @xcite @xcite , and much of acquisition modeling research has been directed at simulating the development of abstract linguistic categories in first language acquisition @xcite @xcite @xcite . recent work proposed a labeled parsing accuracy evaluation metric called recall-v-measure (rvm) as a method for evaluating unsupervised grammar inducers @xcite , but this metric counts categories as incorrect if they are finergrained than reference categories or if they represent binarizations of n-ary branches in reference trees, which may be linguistically acceptable. we therefore further modify it to recall-homogeneity (rh) calculated as the homogeneity @xcite of the labels of matching constituents of the induced and gold trees, weighted by unlabeled recall. this work uses transcribed child-directed utterances from multiple languages as input to a grammar inducer with hyperparameters tuned using either unlabeled f1 or labeled rh. results show that: (1) the induced grammars capture the preference of sparse concentrations in human grammars only when using labeled evaluation; (2) grammar accuracy increases as the number of labels grows only when using labeled evaluation; ## model all experiments described in this paper use a bayesian dirichlet-multinomial model @xcite to induce pcfgs without assuming any language specific knowledge. this model defines a chomsky normal form (cnf) pcfg with c nonterminal categories as a matrix g of binary rule probabilities which is first drawn from the dirichlet prior with a concentration parameter β: trees for sentences 1..n in a corpus are then drawn from a pcfg parameterized by g: inference in this model uses gibbs sampling to produce samples of grammars and trees with the most probable parses obtained with the viterbi algorithm. ## data and hyperparameters experiments here use transcribed child-directed utterances from the childes corpus @xcite in three languages with more than 15,000 sentences each. english hand-annotated constituency trees are taken from the adam and eve portions of the brown corpus @xcite . mandarin @xcite and german @xcite data are collected from childes with reference trees automatically generated using the stateof-the-art @xcite parser. disfluencies are removed, and only sentences spoken by caregivers are kept in the data. models are run 10 0.01 0.1 0.2 0.5 1.0 5.0 β 0.2 0.4 0.6 unlabeled f1 times with 700 iterations with random seeds following previous work @xcite . the last sampled grammar is used to generate viterbi parses for all sentences. all punctuation is retained during induction and then removed in evaluation. significance testing uses permutation tests on concatenations of viterbi trees from all test runs. we use adam for exploratory experiments and the other three sets for confirmatory experiments. 1 2 3 ∞ depth 0.3 0.4 0.5 rh figure 4: depth-bounding on adam ## conclusion unlabeled evaluation has been used in grammar induction, but experiments presented in this paper show that unlabeled evaluation can reveal unexpected bias in the data which may lead to unhelpful conclusions compared to labeled evaluation. results show that trends of preference of sparsity and use of categories that are consistent with linguistic annotation can only be discovered with labeled evaluation. furthermore, human memory constraints are still effective in grammar induction when labeled evaluation is used throughout all stages of development.