Chunk Different Kind of Spoken Discourse: Challenges for Machine Learning (2020)

Abstract: This paper describes the development of a chunker for spoken data by supervised machine learning using the CRFs, based on a small reference corpus composed of two kinds of discourse: prepared monologue vs. spontaneous talk in interaction. The methodology considers the specific character of the spoken data. The machine learning uses the results of several available taggers, without correcting the results manually. Experiments show that the discourse type (monologue vs. free talk), the speech nature (spontaneous vs. prepared) and the corpus size can influence the results of the machine learning process and must be considered while interpreting the results.

## introduction the notion of sentence is generally considered as irrelevant to the analysis and treatment of spoken data @xcite . researchers suggested different segmentation units for spoken data as part of projects such as rhapsody or orfeo. the project segcor foot_0 aims to segment the data of talk-in-interaction at different levels. its first level segmentation concerns minimal syntactic units, which are called chunks. a "chunk" is a non-recursive constituent of linguistic units @xcite . chunking (or shallow parsing) identifies the surface syntactic structure of a sentence and it can be done automatically. the purpose of a chunking is to identify constituents of the sentence without specifying their internal structure and their syntactic function, which is based on previous morphosyntactic labeling. chunking is a good way to perform syntactic analysis automatically on spoken data, for which it is not always feasible to provide a full syntactic parsing. chunkers are well adapted for transcribed oral data in which "sentences" are not always syntactically fulfilled. the chunk is supposed to be a relevant unit for spontaneous speech. @xcite has shown that it is in the chunks where the reparation markers often occur in spoken data. some software tools provide this type of analysis but their performance is usually low on oral data. there are several strategies to develop a chunker. there have been attempts to build a chunker that is particularly adapted to french data by using symbolic methods @xcite @xcite . the method consists of iteratively applying finite-state transducers, together with lexical and syntactic resources. the supervised machine learning seems to be particularly effective in this task as shown by @xcite @xcite . the present research continues the work of @xcite and uses the method of supervised learning. oral data are characterized by discursive variety: situational variety (private conversation, public debate ...), language tasks (explain, narrate, describe ...), genres (travel stories, interviews ...) or register @xcite . the nature of the data influences and guides the learning process. in @xcite , the labeled reference corpus consisted of sociolinguistic interviews; in this work, we were interested in two other communicative situations: a university conference and a spontaneous discussion between friends during a dinner. our objective is to develop a chunker for spoken data by using conditional random fields (crfs). we want to find out how the discourse type (prepared monologue vs. spontaneous talk in interaction) can influence the results of the experiments and which features are most relevant to each communicative situation. ## reference corpus constitution we dispose of two corpora for spoken french: eslo2 foot_1 and clapi foot_2 . we selected two types of speech: a conference, i.e. a prepared monologue (10 minutes, 2120 tokens) from the corpus eslo2 (m) and a discussion between three people, a spontaneous interaction taking place in a private context (10 minutes, 2461 tokens) from the corpus clapi (d). ## machine learning the machine learning process aims to indicate the borders of each chunk and to determine its type. the reference corpus has a small size, hence we choose the crf models (conditional random fields) @xcite that has already shown a good performance for this task @xcite @xcite . we applied the chunking to the pos labeled corpus. @xcite showed that it is possible to train a chunker for spoken corpus with non-corrected pos tags and with a small size of reference corpus. the authors obtained a micro-average of 88%. we continue the same approach but with a methodology that we redefined according to the specificities of oral data : (1) our data is more heterogeneous because it includes two types of oral discourse; we tested four taggers: treetagger @xcite ; sem @xcite exploited by @xcite and using morphosyntactic labels of @xcite ; syntactic dependencies parser @xcite developed by researchers of orfeo project ; perceo @xcite , pos tagger for oral data using fno label which is also present in our chunks typology. we performed the experiments on three corpora: eslo2 (m), clapi (d), eslo2 and clapi (m + d). the aim is to check if speech type (monologue / discussion between three people), speech nature (spontaneous / prepared) and corpus size can influence the machine learning results. we tested many configurations by combining and varying templates [token + pos] foot_4 . the features of the crf were based on the pos of three tools treetagger, perceo and orfeo. for orfeo parser, we tried two additional combinations (1) pos label for current token, (2) pos label for current token and its head. first, for each combination, the current line token was tested. then, for each corpus, the three combinations giving the best were selected for the test. we also included the same combinations on token+1 and token-1. after selecting the best result, other columns were added such as lemma, which resulted from treetagger tagging. the figure 2 shows the patterns of the best combinations for each corpus. ## results and evaluation the evaluations on three corpora (m, r, m + r) were made by a 10-fold cross-validation. we evaluate the chunking with the micro-average of the f-measures of the obtained chunks, which is the average of the f-measures of every type of chunk weighted by their frequencies. for the corpus d, the best results are templates composed of treetagger, perceo, orfeo pos and current and preceding tokens (83.2%). the best score for the corpus m was obtained by a similar combination but this time taking into account the pos of the head proposed by syntactic dependencies parser of orfeo (85.8%). the monologue, part of the conference, contains longer utterances and no interaction, this is the reason why dependency links are more present in this corpus. in the case of the corpus m + d, the best results were obtained by using only the pos of treetagger and perceo with current and preceding tokens (85.7%). these results show that the use of labels proposed by two tools learned on oral data: perceo and orfeo parser, in templates is relevant. the corpus size is also an important criterion because we can see in our results that longer corpora don't need a lot of pos labels and the combination of the results token+sem+ttg+orfeo+perceo, token+orfeo, token+orfeo+ttg, token+1 et token-1. of two tools is enough. it is surprising to note that treetagger seems to be more relevant in this case than the orfeo parser, which is developed for spoken data. the corpus m, a prepared monologue, gives the best machine learning results than the corpus d, the spontaneous discussion. m d m + d microaverage 85.8% 83.2% 85.7% pos treetagger x x x pos perceo x x x pos orfeo x x governor orfeo x token token-1 x x x figure 3 : better results of micro-average the evaluation of labels proposed by our chunker shows that the fno label gets worse results (23.52% f-measure). indeed, some tokens are ambiguous, like yes (oui, ouais), no which can be fno or artic (discourse markers). thus, yes (ouais) in response to a question will be considered as autonomous predicate (a sentence like word) and therefore annotated (fno), like here: eli je [vp b] vous [vp i] sers [vp l]? (i serve you) bea ouais [fno u] (yes) on the other hand, the same form can be in the periphery of the predicate. this form will be considered as nonautonomous discursive articulator, as in the example below, where yes (ouais) closes eli's speaking turn: eli non [artic u] mais [artic u] tu [vp b] sais [vp l] (no but you know) tu [vp b] en [vp i] mets [vp l] pas [adp b] beaucoup [adp l] (you don't put a lot) tu [vp b] en [vp i] mets [vp l] un [np b] fond [np l] ouais [artic u] (you put a depth) there are a few other common mistakes. many chunks np are annotated as pp because of the ambiguity between the preposition de followed by a definite article and the partitive article (du, de la, etc.), both have the same form. a quarter of the ap are considered as vp because ap follows often vp. the chunks boundaries (labels b, l, u) are generally better annotated (figure 4 ). ## conclusion the article described the development of a chunker for two different kinds of spoken discourse: a monologue in a conference and a spontaneous discussion between three people during dinner. we trained crfs by using a small reference corpus. we obtained the best results on the corpus d (83.2%), with templates composed of treetagger, perceo, orfeo pos and current and preceding tokens. we obtained the best score for the corpus m (85.8%) with a similar combination, but we also added the governor label proposed by the orfeo syntactic dependency parser. the experiments showed that the discourse type should be considered while training and interpreting the results. thus, the results of dependency parsing are more relevant to integrate in crf model for the monologue in which the utterances are long and dependencies relations are more present. fno label obtained a better score in a discussion because it is very present in this corpus. in the case of the extended corpus m + d, we obtained the best results (85.7%) by using few features: the pos of only two tools: treetagger and perceo with current and preceding tokens. the chunker that we developed is launched with the python language including an interface 7 . it identifies chunks not only from the transcriptions of spoken data without sound but also from the tokens of the transcriptions which are aligned with the sound, by using jtrans @xcite . we obtain the results in elan software @xcite format, which shows the transcription and annotation by tiers (figure 5 ). there are two ways to visualize the results: 1). the label of the chunk type (eg. artic); 2). the label of the chunk type and the bilou (eg. u-artic), which are presented side by side. elan format is convertible to exmaralda, @xcite , a software that is used in the segcor project. the chunker can be successfully applied also on the conll data sets. it is possible to use our chunker for other tasks. for example, we used the results of chunker for automatic segmentation of transcriptions aligned to the sound in macro-syntactic periods @xcite . we have several directions for future work : (1) to add some information from the records as prosody;