Conversational Feedback in Scripted versus Spontaneous Dialogues: A Comparative Analysis (2024)

Abstract: Scripted dialogues such as movie and TV subtitles constitute a widespread source of training data for conversational NLP models. However, there are notable linguistic differences between these dialogues and spontaneous interactions, especially regarding the occurrence of communicative feedback such as backchannels, acknowledgments, or clarification requests. This paper presents a quantitative analysis of such feedback phenomena in both subtitles and spontaneous conversations. Based on conversational data spanning eight languages and multiple genres, we extract lexical statistics, classifications from a dialogue act tagger, expert annotations and labels derived from a fine-tuned Large Language Model (LLM). Our main empirical findings are that (1) communicative feedback is markedly less frequent in subtitles than in spontaneous dialogues and (2) subtitles contain a higher proportion of negative feedback. We also show that dialogues generated by standard LLMs lie much closer to scripted dialogues than spontaneous interactions in terms of communicative feedback.

## introduction while the amount of text data available for training or fine-tuning llms is large and growing steadily, spoken conversational data remains relatively scare. although corpora of spontaneous spoken interactions have been collected for various languages @xcite , those are generally of a modest size and limited to specific topics or tasks. due to this scarcity of available data, a common approach for the development of conversational models is to rely on corpora of authored dialogues extracted from movie scripts @xcite or movie and tv subtitles @xcite . however, those dialogues are markedly different from spontaneous interactions. most importantly, movie scripts and subtitles are explicitly written with the aim of narrating a story. subtitles must also abide to strict length constraints, and thus tend to only transcribe the most salient part of each turn. as a consequence, many conversational phenomena such as disfluencies @xcite , overlapping talk @xcite , and backchannels @xcite are either absent or uncommon in those dialogues, unless their presence happens to contribute to the storyline @xcite . this paper provides a quantitative analysis of how subtitles differ from spontaneous dialogues, focusing more specifically on conversational feedback @xcite and grounding @xcite phenomena. to highlight differences in linguistic properties between subtitles and spontaneous conversation corpora, we first compile a range of lexical statistics and use a dialogue act tagger to estimate the relative frequencies of various feedback signals. to obtain more fine-grained estimates on three core feedback categories, respectively agreement / acceptance, acknowledgement / backchannel and negative feedback, we collect manual annotations on multiple dialogue samples and fine-tune a llm on those annotations to automatically detect the presence of those feedback in our corpora. finally, we apply the fine-tuned llm on synthetic dialogues generated with standard autoregressive llms, and show that those dialogues are comparatively much closer to scripted dialogues than to spontaneous interactions when it comes to the frequency and type of conversational feedback. those experiments are conducted for eight languages (english, chinese, french, german, hungarian, italian, japanese and norwegian) for which corpora of spontaneous dialogues are readily available. the paper is structured as follows. section 2 reviews related work, and section 3 presents the corpora employed in our experiments. section 4 describes the observed lexical distributions of feed-back phenomena and section 5 compares them to estimates derived with a dialogue act tagger. in section 6, we describe the manual annotation of dialogue samples and the fine-tuning of an llm to automate this process. finally, section 7 describes the results of applying this llm-based method to synthetic dialogues, and section 8 concludes. ## corpora we rely on data from both opensubtitles and existing, publicly available corpora of real conversations covering eight different languages (see ## lexical analysis lexical statistics of acknowledgment cues gives us a first picture of the feedback frequency. acknowledgments tend to be produced by the addressee (not the main speaker) and are therefore often short productions uttered in overlap and potentially with a lower voice. out of those three properties (brevity, overlap, lower volume), only the first is practically measurable in our experiments, as the subtitles are by construction text-based. given their relation to acknowledgments, we first analyse "very short utterances" @xcite , defined here as three tokens or less. feedback is also very well represented at initial positions of longer turns/contributions. we therefore targeted two locations: very short utterances (all tokens) and initial positions (one token) of all other utterances. comparing term frequencies between these locations and the overall corpus allowed us to compile language-specific lists of cue words. those lists of cue words (presented in we plot in figure 1 the frequencies of those feedback classes in each corpus, either in terms of absolute frequency (left side) or by looking at the relative proportions of the feedback classes (right side). figure 2 shows the lexical distribution of the most frequent lexical items observed in the utterances of plot (b) for english. we observe that the statistics based on cue words differ substantially between subtitles and spontaneous dialogues. this difference is observed across all languages and sub-genres, (see appendix a for other languages). we sought to identify and reduce other sources of variation between corpora. stac, as a chat corpus, exhibits different patterns than other dialogue corpora, notably due to the presence of emojis. similarly, for english and french, we explored the impact of politeness expression (highly frequent in opensubtitles). those peculiarities did not, however, change the overall picture of our analysis (see figure 13 in appendix a). one key difference between real dialogues and subtitles relates to the overall frequency of feedback cues, which is much higher in spontaneous dialogues (40-50%) than in subtitles (10-20%), as observed in figure1(a). furthermore, as shown in figure 1(b), feedback in spontaneous dialogues consists mostly in positive or neutral (continuers) feedback, while subtitles have few neutral signals but seem to exhibit a much higher proportion of negative feedback and clarification requests. we compared our english cue word lists against the annotations in switchboard. after grouping feedback-related labels into a single feedback category, we find that the cue word lists yield an f 1 score of 0.76. ## dialogue act tagging although lexical statistics do highlight substantial differences in subtitles and spontaneous dialogues, they remain imprecise estimates, as many cue words related to feedback tend to be ambiguous. in this section, we refine our analysis using a dialogue act tagging model trained on the damsl-switchboard corpus. ## further annotations the results from the dialogue tagger do show some clear trends regarding the extent to which communicative feedback is expressed in subtitles compared to spontaneous interactions. however, the use of damsl-switchboard as sole source of training data is a limiting factor in our analysis, in particular when it comes to non-english dialogues, which must be machine-translated prior to labelling. furthermore, the tagger does not provide information about the frequency of negative feedback, although the lexical analysis from section 4 does seem to point towards a higher frequency of those communicative signals in subtitles. we therefore complement the analyses of the two previous sections with a manual annotation effort. to this end, we sample from each corpus a set of 300 utterances to annotate. however, as evidenced by the results of the previous sections, many utterances of our corpora do not seem to contain any communicative feedback. to ensure the annotation process can cover a sufficiently broad variety of feedback signals despite this class imbalance, we do not select the utterances purely at random, but select half among those marked as feedback-relevant by the cue words of section 4, and the other half among those that do not. ## conversational feedback in synthetic dialogues we conclude by investigating the occurrence of communicative feedback in synthetic dialogues generated with autoregressive language models. more precisely, we wish to analyse whether the communicative feedback generated by those models are closer to the patterns found in real interac- figure 5 : frequency of communicative feedback in synthetic dialogues generated using gpt-2 models, either applied without fine-tuning or after fine-tuning on corpora of spontaneous interactions or subtitles. to this end, we use available gpt-2 models @xcite for the eight covered languages foot_3 . the use of gpt-2 models is motivated by practical considerations and the need to obtain pre-trained models for each of the eight languages. for each corpus, we derive a fine-tuned version of its corresponding gpt-2 model by further training the model on the corpus dialogues. to account for the corpus size differences, the number of epochs is adjusted to ensure that the total number of gradient updates is similar across all corpora. the gpt-2 models are then employed to produce synthetic dialogues (100 dialogues of about 50 turns per model. for the fine-tuned models, all turns are automatically generated, while for the base models, the following dialogue start is used as context: hi! -hi, how are you? -fine, and you? to bias the model towards the generation of dialogues. finally, the llm annotator from the previous section is applied on those synthetic dialogues to estimate their frequency of communicative feedback. the results are shown in figure 5 . we observe that the synthetic dialogues generated with the standard gpt-2 models without any further fine-tuning are much closer to the ones derived from subtitles than to those derived from spontaneous interactions when it comes to communicative feedback. this is notably the case for positive and neutral feedback. the occurrence of negative feedback is, however, not as common as in subtitles. although the above results were obtained here using only gpt-2 pretrained models, we expect to find similar patterns for other (and more recent) llms. ## conclusion and future work as evidenced in this paper, movie and tv subtitles exhibit notable linguistic differences to actual spontaneous dialogues in the amount and type of conversational feedback they include. based on a collection of corpora of both spontaneous dialogues and subtitles across eight languages, we provide both lexical statistics and dialogue act estimates derived with a fine-tuned dialogue act tagger. we show that the proportion of conversational feedback is considerably lower in subtitles than in spontaneous dialogues across the corpora included. furthermore, the type of conversational feedback also differs, as negative feedback is proportionally more frequent in subtitles. this is corroborated by manual annotations of 24 dialogue samples from the selected corpora, and the use of a fine-tuned llm trained on those annotations. finally, we also show that dialogues generated from language models are closer to scripted dialogue than real interactions in their use of communicative feedback. beyond their linguistic interest, these results can provide useful insights for the development of conversational models, as those are often trained on scripted dialogues and might therefore struggle both to understand communicative feedback from the user and to produce such feedback themselves. it + ehi, okay, okay, ok, sì, si, vabbè, ecco, perfetto, wow, esatto, certamente, esattamente, assolutamente, sicuramente, decisamente, ovviamente, precisamente, di sicuro, sono d'accordo, concordo, eccellente, grandioso, ottimo, certo, infatti, fantastico, magnifico, naturalmente, giusto, bene, già,lo ben so, ah ah, ah ha, vero, é vero, lo so, lo è, davvero, vero, oh sì, lo è veramente, anch'io, anche io, hai ragione, d'accordo, va bene, benissimo, bello, buono,penso di sì, credo di sì, mi sa di sì, mi pare di sì,anche secondo me, lo penso anch'io, è così, penso che tu abbia ragione, penso tu abbia ragione, credo che tu abbia ragione, credo tu abbia ragione, mi sa che hai ragione, sono d'accordo con te, sono d'accordo con voi, lo era, lo è stato, lo è stata, sono d'accordo con ciò, lo sono, senza dubbio, a posto, ci sto, lo sono stati, lo erano, anche a me oddio, merda, no, non proprio, non molto, non è possibile, cazzo, oh no, macché ? come, davvero, @xmath0, mm-hmm, hmm, mmm, mh, eh, mhmh, eh, m, hm, ah, oh, beh, uh-huh, mmh, eeh ja + そう, はい, ええ, そうか,はあ, どうぞ, 本当, は, あっ, ああ, あ, ね = うん, ふーん,えっ, へえ, うーん, ふん, え,う -ううん, いいえ,いや, いえ, ない, 全くない,ちょっと ? 何 zh + okay, yeah, yes, ok, 对, 哦, 好, 是, 有, 真的, 还行, 然, , 太好了, 耶, 行, 一定, 没错, 那好, 对 了, 真好, 好啊, 好吧, 可以, 太棒了, 太棒了, 好极了, 说得对, 没问题, 我同意, 懂了, 一樣, 我也是, 不错, 是啊, 就是这样, 当然可以 -不, 沒有, 不起, 不是 ? 啊, 是吗, 什麼, 什么, 为什么 = hey, oh, 嘿, 嗯, 呃, 哼, 哈, 嘘, 喔, 呵呵, 噢, 哇, 哦, 哟, 咦