Data-driven Parsing Evaluation for Child-Parent Interactions (2023)

Abstract: We present a syntactic dependency treebank for naturalistic child and child-directed spoken English. Our annotations largely follow the guidelines of the Universal Dependencies project (UD [Zeman et al., 2022]), with detailed extensions to lexical and syntactic structures unique to spontaneous spoken language, as opposed to written texts or prepared speech. Compared to existing UD-style spoken treebanks and other dependency corpora of child-parent interactions specifically, our dataset is much larger (44,744 utterances; 233,907 words) and contains data from 10 children covering a wide age range (18–66 months). We conduct thorough dependency parser evaluations using both graph-based and transition-based parsers, trained on three different types of out-of-domain written texts: news, tweets, and learner data. Out-of-domain parsers demonstrate reasonable performance for both child and parent data. In addition, parser performance for child data increases along children’s developmental paths, especially between 18 and 48 months, and gradually approaches the performance for parent data. These results are further validated with in-domain training.

## introduction research on syntactic dependency parsing has experienced tremendous progress with the continuous development of the universal dependencies project (ud) @xcite . that said, of the 228 treebanks in the latest version of ud (v2.10), only 12 consist of fully spoken data (see also @xcite , while the rest focus on different genres within the written domain. this means most (if not all) state-of-the-art off-the-shelf dependency parsers are oriented towards written texts rather than tailored specifically to spontaneous spoken language. therefore a natural question arises: how well will parsing systems developed for written data perform on spontaneous spoken language? over the past decade, there have been efforts devoted to dependency parsing for the spoken domain, especially for (a subset of) the switchboard corpus @xcite , which contains transcripts of telephone conversations in english. while some focused on parsing the full subset @xcite @xcite , others attended to specific phenomena common in spoken data, such as speech repairment @xcite . the switchboard corpus was manually annotated only for constituency parses; the prior work relied on dependency parses automatically converted from constituency parses without manual verification. in addition to english, dependency treebanks have been developed for the spoken domain of other languages, including french @xcite , czech @xcite , russian @xcite , japanese @xcite , and mandarin chinese @xcite . these treebanks, including switchboard, however, were not always built using ud guidelines. in addition, the customized annotations and trained parsers are not always publicly available, making it less straightforward to carry out evaluation, especially since most dependency parsers are designed for ud-formatted treebanks. this paper presents a wide-coverage dataset of spontaneous child-parent interactions @xcite annotated with syntactic dependencies largely following the ud standards. our thorough annotation guidelines will be useful in the development of any spoken language dependency treebank. our work goes beyond prior work in several respects. first, compared to most of the other spoken dependency treebanks, which contain conversations between adults @xcite @xcite or user-generated content @xcite , our annotation guidelines attend to child and child-directed spoken language. second, in contrast to other spoken treebanks in the ud project (see @xcite ), our dataset as a whole is of considerable size. in our corpus, there are 26,098 child utterances (116,428 words), and 18,646 parent utterances (117,479 words). third, while there are some dependency corpora of child-parent interactions in english @xcite , japanese @xcite , and hebrew @xcite , they include data from only one or two children, and detailed annotation guidelines pertaining to (child) spoken language are sometimes lacking. in this study, we provide annotations for utterances of 10 children across a much wider age range, therefore covering in more detail lexical and syntactic phenomena that are more common in child language (e.g., repetition, speech disfluency) and spoken language more broadly. with this dataset, we ask two additional questions: (1) how do state-of-the-art dependency parsers trained on out-of-domain data perform on naturalistic spoken language of different interlocutors? to address this question, we evaluate parsers trained on three genres within the written domain: news texts, tweets, and learner data, all in english. ( 2 ) what is the relationship, if any, between parser performance and the developmental stage of the child? one might expect a positive correlation between the two, with the expectation that as the child continues to develop their language skills, they will utilize more and more cohesive syntactic structures, instead of, for example, producing grammatical errors, unintelligible speech, or word omissions. on the other hand, it is possible that parser performance might increase as the child reaches a certain developmental stage, then start decreasing, since the child might start producing sentences with more complex or expressive syntactic structures, which are potentially harder to analyze. ## why a dependency treebank? we chose ud-style dependency annotations because of their general ease of adaptability to different domains or languages, as well as the continuous active development of different dependency parsing toolkits. dependency parsing has recently attracted more attention than constituency parsing. compared to constituency structures, dependency structures are ''simpler'', with easier adaptations to ''non-standard'' domains or typologically diverse languages. the popularity of dependency parsing can be largely attributed to the research community's considerable efforts into developing the ud project and the conll shared tasks @xcite . accordingly, tools for building dependency parsers @xcite @xcite are well-maintained; as new treebanks become available and are added to ud, new implementations will continue to become accessible. developers can evaluate their parser models using our dataset to probe their parsers' capabilities. second, from the perspective of studying child language, previous literature has shown that linguistic features derived from dependency parses can be leveraged to replace traditional metrics such as the index of productive syntax @xcite and measure children's syntactic development in an automatic fashion @xcite @xcite . others have utilized dependency trees to study how children produce syntactic alternations along their developmental trajectory @xcite . these prior experiments, however, used parsers trained either on data from one or two children, or on fully written data; therefore the generalizability of the prior findings remains to be verified. providing a dependency treebank (even with differences in annotation details) for data from more children and a wider age range will allow researchers in child language development to train parsers of wider coverage with good performance. the parsers can then be applied to unannotated child-parent interactions. third, while this work focuses on english, there are some corpora of child-parent interactions in other languages that also provide sufficient longitudinal data, though not on the scale of the english sections (e.g., the hebrew berman longitudinal corpus @xcite ). we hope that our work can serve as a useful reference. we note two studies that carried out ud-style dependency annotations for child and/or childdirected spoken language. liu and prud'hommeaux (2021) took a semi-automatic approach to convert a subset of the existing dependency parses from the eve corpus @xcite to ud standards, with a focus on 18-to 27-month-old children. concurrent work by @xcite annotated dependency parses for two languages: english (from the adam corpus @xcite ) and hebrew (the hagar corpus @xcite ), although they only looked at child-directed utterances. the annotations from @xcite cover some phenomena prevalent in child spoken language, such as non-standard vocabulary and ambiguous fragments (with which our annotation standards align), but they do not include guidelines for annotating possible lexical omission or speech restart and repair, which we address. when considering the annotations of the dozen spoken dependency treebanks in ud more broadly, they appear to mostly focus on fillers, discourse particles, and disfluency @xcite ; in some cases, detailed information is lacking largely due to the fact that the treebanks are relatively small @xcite . given that our dataset is on a much larger scale, we are able to carefully note different speech-related phenomena along with providing clear annotation guidelines. while fillers and discourse markers are usually treated consistently across existing ud treebanks, there remain significant inconsistencies in head-attachment and the (over-)application of treebank-specific dependency relations. as pointed out in @xcite , while several treebanks rely on the reparandum dependency relation defined in ud to annotate disfluency, they do not all follow the ud guidelines which suggest applying reparandum right-to-left (i.e., treating the repairment as the syntactic head of the prior disfluent words) @xcite . other work introduces new treebank-specific dependency relation subtypes that do not abide by ud standards for clause discourse markers (e.g., using parataxis:discourse instead of discourse when clause-level discourse markers are automatically identifiable) or speech restart (e.g., using parataxis:restart instead of an augmentation to reparandum) @xcite . here, our dataset adheres to the ud guidelines as much as possible. when introducing new dependency subtypes for speech repairment, restart, and repetition, we do so by extending reparandum. these subtypes also make it straightforward for automatic extraction of different structures as well as converting our annotations when necessary to make them better aligned with the customized annotations of other (spoken) treebanks. ## meet the data the dataset consists of transcripts of english naturalistic parent-child interactions from the childes (macwhinney, 2000) database, accessed through the childes-db interface @xcite . as we are interested in how parser accuracy changes at different developmental stages, we used age as a proxy for developmental stage and set 6-month intervals as bins. for each individual child, we calculated the total number of words produced by the child and by the parent(s) within each age bin of the child. then, from each age bin, for both child and parent data, we randomly sampled a number of utterances that amounted to approximately 2,000 words; the criteria were relaxed in order to include data across a wide range of age bins. this resulted in spoken data of ten children from 6 corpora ( ## annotation our annotation guidelines largely followed those of ud @xcite . annotator a, with advanced training in dependency syntax, initially annotated data of age 18-24 months and 24-30 months from abe and sarah, as a way to take note of any domain-specific or challenging phenomena. these guidelines were discussed with annotator b and modified as needed. then, given each age bin of every child, the two annotators annotated 10% of the data from both child and parent utterances. we calculated agreement scores using cohen's kappa @xcite . the overall agreement score taking into account all syntactic head and dependency relation annotations is 0.97; the average agreement score across each dependency parse is 0.96. (agreement scores for each child were around 0.97.) final annotations were performed and verified by annotator a. here we describe in detail our approach to transcription orthography, tokenization, and dependency annotations for syntactic constructions that are unique to or more common in child production and spoken data more broadly. ## experimental design we now turn to evaluating dependency parsing with our dataset. we aim to address: (1) how well do competitive parsing architectures trained on out-of-domain data perform for spontaneous child-parent interactions? (2) can in-domain data help with parsing performance for child speech, and if so, to what extent? (3) are there any generalizable relationships between parser performance and child production at different developmental stages? to that end, we explored different parser architectures, several dependency treebanks from a range of domains distinct from one another and from child-parent interactions, and different training schemes for in-domain experiments. 2 parsers we used two graph-based parsers, diaparser @xcite , machamp (van der @xcite , and one transition-based parser, uuparser (de @xcite . we picked these parsers based on the level of detail in their documentation and implementation, as well as their availability. the goal of exploring different architectures here is to see whether qualitative observations (see section 7) of parsing results hold regardless of the particular parser applied. for each of two graph-based parsers, we created different variants via combining with different language models (lms) as encoders (bert-base-cased [bert] @xcite , roberta-base [roberta] @xcite , and twitter-roberta-base [twitter] @xcite ). here we mostly focus on results derived from machamp, which was able to achieve the best results on the out-of-domain data ( ## out-of-domain experiments we first performed automatic part-of-speech tagging for all child and parent data using the open source nlp library stanza @xcite . we then applied each of the out-of-domain parsers to child and parent utterances within each 6-month age range of the child; parser performance was again indexed by las across 3 random seeds. we foresee two possible directions regarding the parsing results. on one hand, ewt has significantly more data than tweebank and esl, which might lead to overall better performance. on the other hand, among the three out-of-domain datasets, the domains of tweebank and esl are possibly more relevant or more similar to child-parent interactions, in the sense that they are less ''formal''. additionally, the written texts of esl may contain errors, which may bear similarity to child data, as children are also language learners. this potentially means that parsers trained from tweebank or esl might outperform those based on ewt. we note that the evaluation of parsers trained on out-of-domain data did not consider new dependency relation subtypes that were introduced in our annotations, which are reparandum:restart and reparandum:repetition. ## in-domain experiments in this section, we perform parser training with in-domain data of child-parent interactions. as seen in the results from section 7, on average, parsers trained with machamp using the twitter lm seemed to achieve the most stable performance; we therefore adopted that same parser setup here. our training scheme is as follows. suppose that we wanted to evaluate parser performance for adam @xcite . we first trained parsers using all the data from the other nine child-parent pairs; we then measured parser performance for both adam's and his parents' data at each of adam's available age ranges using las averaged across 3 random seeds. overall the trends of the las scores from in-domain training (figure 9 ) are similar to those obtained from out-of-domain evaluation (figure 8 ): as the children develop, the las scores increase. in contrast, the results for parent data appear to be much more stable across the age span of the children. (note, however, that the actual numerical values of in-domain evaluation are not directly comparable to those of out-of-domain evaluation, given that the latter did not take into account new dependency subtypes that we introduced for speech disfluency.) for child data, performance ranges from 83.30 to 96.25, while for parent utterances, performance spans from 93.23 to 97.64. in early developmental stages (24-36 months), the parsers for child data seemed to be confused the most by utterances of 2-4 words, which potentially involve word omission (e.g., are five), discourse markers, or vocative elements. in these cases, the parsers tended to mis-analyze the root, preferring to treat shorter utterances as compound noun phrases and to assign the last words to be the root. in addition, for child data specifically, we also explored another training scheme, where we trained parsers on the combined data produced by all 10 parents, then applied the parsers to the data of each individual child. observations derived from this training scheme are qualitatively similar to those derived from the training scheme described above. to add statistical rigor when comparing results from the two different in-domain training settings, we combined data from all children, then measured the correlation figure 9 : evaluation of in-domain parsers trained with twitter lm, using child age as the index of developmental stage; to evaluate parser performance for a given child, the parsers were trained using the data from all the other nine child-parent interactions. on average across the data of all children, speech disfluency accounted for very small proportions of all dependency relations (repairment: 0.30%, restart: 0.48%, repetition: 0.08%), and the proportions are comparable when considering the data of all 10 children (repairment: 0.35%, restart: 0.78%, repetition: 0.10%). between the three relation subtypes of disfluency, the parsers performed the best for repetition, though they tended to analyze the disfluent words or phrases as the syntactic heads of the speech repairment and restart. ## notes on developmental index our analysis thus far has largely relied on child age as the index of developmental stage. in this figure 10 : evaluation of in-domain parsers trained with twitter lm, using utterance length as the index of developmental stage; in this case to evaluate parser performance for the data of a given child, the parsers were trained using the data from all the other nine child-parent interactions. this observation, nevertheless, seems somewhat in opposition to the earlier patterns where parser performance appears to increase as children age, given that age and utterance length maintain a positive correlation with each other @xcite . this positive relationship is also evident in the children's data in our dataset; when fitting a linear regression model predicting utterance length as a function of age, the model yielded a significantly positive coefficient value for age (β = 0.82, @xmath0.001). to tease apart the respective effect of age and utterance length, we resorted to mixed-effect linear regression. for each utterance produced by children in our dataset, we calculated the following metrics: sentence length (mcdonald and @xcite , dependency length @xcite , and las score. in the mixed-effect model (after step-wise backward regression), we included las score as the outcome variable, meanwhile adding age, sentence length, dependency length, and the training data size for the parsers; we included interactions between each pair of the aforementioned four fixed effects, as well as children as random effects. based on the mixed-effect regression results, age appears to have a significantly positive influence on las score (β = 0.03 (0.02, 0.04)), meaning that parser performance improves as age increases. on theotherhand, utterance length seems to have a significantly negative effect on parser performance (β = -0.007 (-0.09, -0.005)), which aligns with observations from figure 10 . perhaps a bit surprisingly, dependency length turns out to also play a positive role in parser performance, albeit only weakly (β = 0.002 (0.001, 0.002)). comparing the coefficient values of the three factors, the role of age is much more pronounced than that of utterance length or dependency length. in particular, we attribute the relatively much weaker effect of utterance length to the fact that around 95.22% utterances of children in our data contain fewer than 10 tokens. (note that this is a result from our data-driven approach to building the dataset originally rather than cherry-picking utterances from childes.) we take these observations as indications that there does exist a relationship between children's developmental stage and parser performance, with the latter mostly affected by children's age while modulated by their utterance lengths. ## discussion and conclusion we present a wide-coverage dataset of child-parent interactions annotated with syntactic dependencies, along with detailed annotation guidelines extending the universal dependencies project. evaluations from graph-based and transitionbased dependency parsers with varying hyperparameters demonstrate that parsers trained using a relatively small amount of english tweets (tweebank) are able to yield comparable or even superior performance to parsers trained from much larger dependency treebanks. in addition, we observed the general trend that on average, parser performance increases as the children reach older ages, indicating that as children progress along their syntactic developmental trajectory, they start producing structures that are more cohesive but not too complex for the parsers to handle. the relationship between parser performance and children's age, as shown from our mixed-effect modeling, is modulated by utterance length, which has a significant yet quite weak and negative effect on las scores. it is our hope that this dataset and the trained parsers will be useful for researchers studying both child language development and linguistic characteristics of spoken data more broadly. while in this work we adopted the ud annotation styles, prior work has also given guidelines for annotating spontaneous speech using constituency trees. the most notable example is the switchboard corpus @xcite , which also provides annotations for speech restart, repair, word fragments, and so on. we do not wish to claim that our annotation guidelines are in any way better than those for switchboard; rather, given that the childes database contains child-parent interactions in different languages, we hope that our work here will motivate future relevant research that develops syntactic treebanks for child speech in languages other than english (see also @xcite , potentially with wider coverage of children's age span. we started with english in this study because of data availability; with the ever growing size of childes, others can utilize or adapt our annotation standards to different languages in order to facilitate research on crosslinguistic child language development.