Dependency Parsing Evaluation for Low-resource Spontaneous Speech (2021)

Abstract: How well can a state-of-the-art parsing system, developed for the written domain, perform when applied to spontaneous speech data involving different interlocutors? This study addresses this question in a low-resource setting using child-parent conversations from the CHILDES databse. Specifically, we focus on dependency parsing evaluation for utterances of one specific child (18 - 27 months) and her parents. We first present a semi-automatic adaption of the dependency annotation scheme in CHILDES to that of the Universal Dependencies project, an annotation style that is more commonly applied in dependency parsing. Our evaluation demonstrates that an outof-domain biaffine parser trained only on written texts performs well with parent speech. There is, however, much room for improvement on child utterances, particularly at 18 and 21 months, due to cases of omission and repetition that are prevalent in child speech. By contrast, parsers trained or fine-tuned with in-domain spoken data on a much smaller scale can achieve comparable results for parent speech and improve the weak parsing performance for child speech at these earlier ages

## introduction while the task of dependency parsing has been studied extensively, in both monolingual @xcite and crosslinguistic contexts @xcite , much of the effort thus far has been devoted to parsing data from the written domain. by contrast, dependency parsing of spontaneous speech in general has received little attention, despite the notable and sometimes quantifiable differences of syntactic variations between written and spoken registers @xcite . one obstacle to the development of dependency parsing for speech is that, in comparison to the written domain, which offers numerous corpora with gold-standard (morpho)syntactic annotations in a large variety of languages @xcite , data sets annotated with comparable levels of structural details in the spoken genre are relatively rare, regardless of the particular annotation frameworks adopted. one of the most widely used and largest spoken corpora is the switchboard corpus @xcite , which provides constituency structures for around one million tokens from transcriptions of conversational speech in english. other available spoken corpora are of much smaller scale. for instance, @xcite developed a dependency treebank for spoken slovenian also using transcripts of spontaneous speech, which contains 29,488 tokens in total. @xcite built a dependency treebank for hupa, an indigenous language residing in northern california; the treebank used mainly transcriptions of story telling from the only master speaker of the hupa language and contained around 6,561 mannualy annotated tokens. a few treebanks in the universal dependencies project v2.7 @xcite (hereafter ud) include transcriptions of spoken data, but most such data is mixed in with written texts and the register is not specified for individual sentences. the limited availability of spoken corpora leaves open the questions of how well the current stateof-the-art dependency parser, developed mainly for written texts, performs on low-resource naturalistic speech and to what extent the performance of the parser differs depending on the specific role of the speaker. it is possible that recent advances in the parsing of written texts @xcite @xcite will extend to speech and that errors will be restricted to cases that are unique in the spoken genre, such as omissions of frequent or short words @xcite . alternatively, it is also reasonable to speculate that such parsers would not align well with speech given that they are being applied to a domain that is different from their original training data. nevertheless, the potential differences between the dependency structure of an utterance assigned by a parser and one assigned manually might not necessarily indicate that there are errors but rather that there is more than one way of interpreting the utterance when the discourse context is not considered. understanding whether and where parsers for written texts fall short in low-resource spontaneous speech has the potential to contribute to developing parsing systems for the spoken domain, as well as expanding the availability of spoken data. this study makes a step towards the aforementioned directions. in order to investigate the parsing performance for spoken data in a low-resource setting @xcite , especially spontaneous speech that involves different interlocutors, we focus on child-parent conversational interactions as our targeted evaluation data. specifically, the evaluation sets contain utterance samples of one child, "eve", and her parents from the brown corpus @xcite in the childes database @xcite . the remainder of the paper is structured as follows: we review related work in section 2; section 3 points out some general issues with parsing speech; section 4 describes our procedures of adapting the current dependency annotation scheme in childes to that of ud; section 5 presents our experiments foot_0 ; and we conclude with section 6. ## related work although most work in the dependency parsing and annotation literature has examined the written domain, there have been a number of notable exceptions. looking at the domain of human-computer interactions, @xcite adapted the ud annotation scheme to a data set of 882 sentences that are mostly transcripts generated from automatic speech recognition systems. their work showed that a parser ensemble approach (two parsers in this case) yielded the best results. @xcite also investigated dependency parsing with transcripts from dialogue systems; using a data set of 1,500 sentences annotated with ud labels, they found that the best parsing accuracy was achieved when an out-of-domain parser was fine-tuned with more in-domain data. other work such as @xcite and @xcite has focused on dependency parsing of tweets, which can be considered a domain that lies between the spoken genre and other written contexts. the data set from @xcite contains a total of 3,550 tweets, four times the number used in @xcite . their annotation standards followed the ud style while allowing multiple roots in tweets that have more than one sentences indicated mainly by punctuation. they reported that an ensemble approach with 20 different parsers was the most effective. ## issues with speech orthography one major challenge with parsing speech lies in finding a standard way of transcribing and formatting spoken language. in particular, syntactic parses should be developed for what the speakers have actually said, without further added facilitation during the transcription process. this general issue has not been addressed consistently, at least in the current format of childes. as a demonstration, consider the following annotated examples (adapted from their initial forms in childes). the problems with including punctuation and tagmarkers for syntactic analyses could potentially be relaxed to some extent via automatically removing the added items from the dependency parse following certain heuristics. for example, a rule of thumb could be to change all tokens before the tagmarker to be a dependent of the root of the utterance, with the dependency relation discourse, then discard the tagmarker itself. however, this would not entirely alleviate the problem, since the parser sometimes assign dependents to punctuation or tagmarker. in our manual check of about 250 utterances of the english data from childes that were not labeled as gold-standard (see section 5), this was indeed the case. while in some sentences the root can be assigned as the head of the dependents of the punctuation or tagmarker assigned by an automatic parser, in other cases, the correct syntactic heads are not easily identifiable without actually looking at the utterance. utterances with omitted words are even more challenging than those with punctuation or tagmarkers. ideally, to syntactically analyze spoken data, the focus should be on the actual utterances rather than what the annotators think the utterance should be, especially since the inclusion of additional words in transcribed speech assumes that there were words that were omitted to begin with. this is not to deny that all manual annotations are subjective in some respects, and a less arguable dependency parse could be assigned to an utterance with potential word omissions when the preceding or following contexts are taken into account. for instance foot_1 , in example (4) celery is treated as a nominal modifier of piece; and since the token of, which did not occur in the child utterance, has no further dependents, it could then be removed from the parse without causing any ambiguous issues. to address these issues in a consistent way, in the parsing evaluation experiments here, we did not include any punctuation, tagmarkers, or omitted words in the utterances. ## annotation adaptation to ud the dependency annotations in childes, which now have 37 different dependency relations, were initially developed by @xcite (see also @xcite ). the annotation scheme was then slightly modified by @xcite , with adjustments to cases such as locative phrases and verb particles. given that most of the available dependency parsers for written texts in english abide by the style of ud (zeman et al., 2020), we first adapted the childes dependency relations so that they would be comparable to those in ud. ## conclusion in our low-resource evaluation settings, we found that an out-of-domain parser trained only with written texts performed well with parent speech, but not necessarily so with child speech. on the other hand, similar or better performance was achieved when the parser was fine-tuned with a very limited amount of in-domain data. given the relatively small size of our evaluation data and the narrow age range of the child, it is certain that there are other common phenomenon unique to child-speech that have not been covered, such as disfluency. for future work, we would like to conduct a more thorough and larger-scale parsing evaluation with speech across wider age ranges of different children. in addition, we plan to extend our study with further experimentation of different domain adaptation methods to other spoken domains such as medical conversations as well as languages. besides english, childes also provides dependency annotations in childparent interactions in other languages, such as spanish @xcite , japanese @xcite , dutch @xcite and hebrew @xcite . since annotation standards in in these corpora largely abide by those of english @xcite , our semi-automatic ud conversion scheme described here could be applied to these languages, along with modifications of language-specific morphosyntactic structures.