From text to talk: H arnessing conversational corpora for humane and diversity-aware language technology (2022)

Abstract: Informal social interaction is the primordial home of human language. Linguistically diverse conversational corpora are an important and largely untapped resource for computational linguistics and language technology. Through the efforts of a worldwide language documentation movement, such corpora are increasingly becoming available. We show how interactional data from 63 languages (26 families) harbours insights about turn-taking, timing, sequential structure and social action, with implications for language technology, natural language understanding, and the design of conversational interfaces. Harnessing linguistically diverse conversational corpora will provide the empirical foundations for flexible, localizable, humane language technologies of the future.

## keeping unity and diversity in sight while text corpora are sometimes treated as shapeless collections of strings, conversational data is not flat but richly structured @xcite . each turn at talk builds on what came before and shapes the possibility space of what comes next @xcite . conversation analysts call this sequence organization (schegloff, 2007), and cross-linguistic work has uncovered a number of basic sequential positions along with slots for inserts and expansions @xcite . sequences are one of the major tools for organizing social action. studying conversational sequences across diverse languages poses considerable challenges, because it requires access not just to form but also to social action or intent @xcite . while annotated corpora of dialog acts @xcite are available for a small number of well-resourced languages @xcite , they invite an overly categorical view of what is in fact fluid and emergent action ascription. the openendedness of social actions in casual conversation (levinson, 2013) places severe constraints on the utility of slot-filling approaches @xcite , which have their origin in narrow task-oriented interactions @xcite . here we probe conversational sequencing by starting from coarse-grained but robust structural facts about the relative distribution of turns and talk. casual interaction often combine lively spates of equitable exchange with more lopsided moments such as tellings in which one participant secures the floor and the other assumes a recipient role @xcite . some work on english has captured this as chat versus chunk, where a chunk is defined as 'a segment where one speaker takes the floor and is allowed to dominate the conversation for an extended period' @xcite . using a measure of relative skew in contributions in a moving 10 second window, we can identify stretches corresponding to such a distinction, as well as transitions from one state to another, across unrelated languages (figure 5a-c ). knowing about such states and transitions between them is of great relevance to language technology and dialog systems. for instance, the relative predictability of responses differs strongly across states @xcite . our results suggest that it is possible to reliably identify at least some broad activity types across languages, opening up possibilities for investigating the linguistic resources that characterize them, and the ways in which people transition between them. the notions of 'chat' and 'chunk' should not be reified, but the distinction points to a data-driven way to get ana-figure 6 : a. continuers (marked ○) are among the most frequent recipient behaviours in tellings ('chunks') in both english and korean, shown here in two 80 second segments each with a strong skew in contributions. b. however, their relative frequency is about twice as high in korean based on 100 random samples of 80 second segments in both languages: on average, 21% of turns are continuers in korean, against 9% of turns in english. segments were sampled from 53 (55) distinct conversations involving 106 (95) distinct speakers in korean (english). measures are expressed relative to turns instead of time to control for speech rate differences. lytical grip on structural features of activity types in conversation @xcite . work on english has found that tellings can be recognized not just by their skewed division of labour, but also by the use of continuers like mhm @xcite at places where turn transition would be relevant. we find that this is the case in languages in our sample too, so a simple conclusion could be that we have found a way to unearth universal aspects of tellings, or 'chunks', with possible implications for the design of, say, dialog systems sensitive to the interactional achievement of dialog states. however, on closer look, the data also provides reason to take linguistic diversity seriously. figure 6a zooms in on four longer stretches of conversation in english and korean. here, circles highlight the use of the most frequent continuer in the language, which is 'mhm' in english and 응 'eung' in korean. what is already apparent in the four conversations shown in panel a is also borne out in a quantitative analysis of 100 random samples of 80 second stretches of english and korean conversations: while 8% of turns are continuers in english, this is 21% in korean (figure 6b ). this higher frequency also comes with higher susceptibility to overlap: whereas in english, 39% of continuer tokens occurs in full or partial overlap, in korean this is 73%. the difference does not appear to be reducible to transcription conventions; for instance, in both corpora, continuers repeated in quick succession are transcribed as a distinct format (mhm mhm, 응응 eung eung) and excluded from these counts; and in both corpora, the average number of words per turn lies around 6 (korean: 5.7; english: 6.9) and the average number of turns per 10 second window is 5.6 (korean: 5.6; english: 5.6). one implication of this is that continuers are apparently relevant at more points during interaction in korean than in english @xcite , which has consequences for the design of dialog systems, incremental parsers and conversational agents. for instance, a conversational agent in korean might have to issue more displays of recipiency and should be prepared to deal with incoming feedback at a higher pace; in the same context, an agent calibrated to english might need different conversation design. the observed variation is extreme enough to warrant a critical look at the notion of feedback relevance spaces @xcite : perhaps this notion needs to be relativized to cover attested cross-linguistic diversity, as has been suggested in qualitative conversation analytic research @xcite @xcite . we have touched here only on some coarsegrained aspects of sequential structure by way of demonstrating the utility of conversational corpora representing diverse languages. plenty of other phenomena are ripe for similar treatment. ## interactional tools a key finding of linguistics going back to estoup and zipf @xcite is that a small number of items tends to be used for a large amount of work. power law distributions are ubiquitous in linguistic data and well-studied across a range of languages (see piantadosi 2014 for review). most analyses in this line of work tokenize textual data on the basis of the observation that sentences are built out of reusable elements. for such tokenised items (roughly, 'words'), we have come to expect the rank-frequency distribution to look linear on a log/log scale. yet language does not come in stray words, but in turns at talk: communicative moves of varying complexity and conventionality. since communicative turns are rarely studied as holistic units, it is an open question to what extent they may or may not show evidence of linguistic laws. such an organization may seem prima facie unlikely: after all, we know we build complex turns out of simpler elements like words and phrases, and the unlimited expressive power generated by this compositionality is rightly celebrated as one of the hallmarks of human language @xcite . on the other hand, as @xcite noted, "conversation is much more of a roughly prescribed ritual than most people think". indeed a look at conversational data shows that many turns are not one-offs: at least 28% of the utterances in our sample (436 367 out of 1 532 915 across 63 languages) occur more than once, and over 21% (329 548) occur more than 20 times. many of these recurring turn formats are interjections and other pragmatic devices that help manage the flow of interaction and calibrate understanding @xcite @xcite @xcite . the ubiquity and communicative importance of these items opens up the possibility of power law-like distributions at turn level for some subset of turns. here we compare rank-frequency distributions of tokenized items and standalone turn formats in the subset of 22 languages with conversational corpora large enough to feature at least 20 recurring standalone turn formats (figure 7 ). we find that tokenized items, as expected, reproduce some wellknown structural properties of rank-frequency distributions, including their linear nature on a log-log plot and a systematic deviation from this linearity for the highest frequency (lowest rank) words. for standalone turns, distributions trail off sharply towards the lowest frequencies, reflective of the creative and compositional nature of many utterances. however, the considerable subset of recurring turn formats (figure 5 , purple) may also suggest a partial power law distribution: though the data is sparser, a log-log line fitted to the 20% of turns used at least 20 times has a comparable slope in most corpora. the result cannot simply be reduced to the fact that standalone turns are drawn from the larger population of single words. recurrent turn formats tend to have specialized discourse-level functions, and while many are single words like 'm-hm', 'huh?' or 'oh', one out of three are multi-word expressions like english 'but um', japanese あそう なんだ a soo nanda 'oh really' or hungarian nem tudom 'i dunno'. if such recurring formats obey a power law distribution, this provides novel, interactionally motivated evidence in support of the claim that the phrase rather than the word may be a privileged locus for zipf's law of frequency @xcite . in this context it is worth recalling that zipf motivated his observa-tions in terms of tools-for-jobs @xcite . just as the tools of artisans are constructed and arranged in ways that support efficient use, so the tools of language are organized to optimally carry out their jobs. in this sense, we can speak of recurring turn formats as interactional tools. even if interactional tools make up a significant proportion of turns in any exchange (as we saw in §4, continuers alone may account for 10 to 20% of turns at talk), they are easily obscured by premature tokenization or erased by seemingly innocuous procedures like stopword removal. and yet it is precisely these interactional tools that may prove essential to understanding and modelling interactional infrastructure within and across languages. getting at these tools and charting their universality and variability represents a key goal for human language technologies. overlooking interactional tools and the details of their deployment comes with immediate adverse consequences. a recent user study reported that a significant number of participants ran into interactional turbulence and overlap when interacting with a neural conversational agent through an englishbased voice user interface @xcite . the turbulence was traced to the agent making segmentation errors and responding to every single utterance detected. this in turn made it harder for human participants to predict when the agent was done speaking, leading to cascades of overlap and confusion. the study proposed two solutions to deal with this (casting the interactional scuffles as situations to be avoided rather than as the rapid and flexible recalibrations they represent in human interaction). the first is to return the floor to a participant as soon as overlap is detected. this seems to assume that any vocalization by a participant is an attempt to take the floor (rather than, say, a minimal display of understanding-so-far). the second proposal is to "filter out stop words and interjections from the participant" on the grounds that the agent responding to these can confuse participants, "since people often do not even realize they are using stop words or are interjecting" (p. 117). however, people do not produce interjections stochastically, but wield them as interactional tools in the service of calibrating mutual understanding and coordinating joint action @xcite . a continuer like mhm shows understanding, while a repair initiator like huh? requests clarification. indiscriminately filtering out such utterances robs conversational agents of direct access to public displays of understanding and misunderstanding. it also robs people of the very tools they use to coconstruct interdependence and understanding, and therefore of a significant part of their linguistic agency. filtering out interjections to avoid interactional turbulence is like removing all pedestrian crossings to deal with self-driving cars crashing into people. the result may be an incident-free zone, but at significant cost to human flexibility and agency @xcite . more work is needed to explore the distributional properties of recurring turn formats, but at least we can conclude that every corpus in our dataset has a subset of recurrent turn formats with metacommunicative functions whose organization suggests a power-law distribution. their importance in human interaction and by extension humancomputer interfaces can hardly be overstated. to build flexible conversational agents @xcite and localizable conversational interfaces (abushawar and atwell, 2016), we need a solid grip both on possibly universal aspects as well as on the full range of cross-linguistic diversity. ## ways forward recent work has argued that text-based stochastic models may be running into dimishing returns @xcite , has stressed the dearth of relevant conversational data @xcite , and has pointed to formidable challenges in the creation of truly interactive systems @xcite . progress will come from multiple fronts, but careful and mindful data curation must be a fundamental part of it @xcite . this requires a reconceptualization not just of what counts as "nlp work", but also of what counts as data. here we have shown how linguistically diverse corpora of co-present conversation may contribute to such a reconceptualization. now is the time to pivot from text to talk; for few things other than the careful study of interactive language use can bring us closer to an understanding of how language augments human cognition and supports fluid and flexible action coordination. this understanding, in turn, will be critical to make meaningful progress in any domain that involves human language technologies and interactive interfaces. fortunately there are good ways forward. here we summarise three principles to foster a robust and diversity-aware science of human interaction that can underpin engineering solutions, inform language models, and contribute to human-centered applications: the timing, co-construction and sequential positioning of turns is as consequential to their meaning and interpretation as their form. the complex and socially distributed nature of sequence organization exceeds the powers of slot filling approaches and requires renewed attention to interactional tools: the metacommunicative resources people use to construct and calibrate mutual understanding on the fly. ## conclusions cross-linguistically diverse corpora of conversation are increasingly available and can help us to better understand basic interactional patterns and build more flexible, context-sensitive language technologies. for this to work, it is important to keep both linguistic diversity and potential universals in sight @xcite . we cannot assume that a given piece of interactional infrastructure is universal just based on a handful of languages. encouragingly, our results suggest that even relatively small corpora can support robust generalizations about key aspects of interactional infrastructure. one reason this matters is empirical grounding. cross-linguistic and comparative work on human interaction has barely started @xcite . there may be more universals of interaction; equally likely is that there are more patterns of unrecognized diversity. both types of outcomes are important for how they shed light on the structure of human interaction, and both have implications for language technology and human-computer interfaces. more fundamental work in pragmatic typology is needed -and computational approaches to low-resource languages provide a promising starting point. but an equally important reason to consider linguistic diversity in language technology and natural language processing is one of linguistic agency @xcite @xcite . designing interfaces that allow people to flexibly wield their preferred communicative resources lessens the hegemony of any one language and makes technology more inclusive, more humane and more convivial for a larger range of possible users @xcite . localizing user interface elements is only a first step; diversity in how and when basic interactional structures are deployed must ultimately be reflected in the design of conversational user interfaces. in the rush for better language technology we should avoid being driven into the arms of only the best-resourced languages and the easiest-to-get data. we need language models that are representative of the actual ways in which people use language, and conversational interfaces that give people the feeling they do not have to leave their own linguistic identities at the door. comparative and computational work on conversational corpora from a wide range of languages is crucial to develop a strong foundational understanding of universals and diversity in interactional infrastructure, and to ensure we can build the humane and diversity-aware language technologies of the future.