Measuring Lexico-Semantic Alignment in Debates with Contextualized Word Representations (2023)

Abstract: Dialog participants sometimes align their linguistic styles, e.g., they use the same words and syntactic constructions as their interlocutors. We propose to investigate the notion of lexico-semantic alignment: to what extent do speakers convey the same meaning when they use the same words? We design measures of lexico-semantic alignment relying on contextualized word representations. We show that they reflect interesting semantic differences between the two sides of a debate and that they can assist in the task of debate’s winner prediction.

## introduction it is well known that dialog participants often tend to imitate each other. this phenomenon, known as alignment or entrainment, can be of a linguistic nature (lexical @xcite , syntactic @xcite , prosodic @xcite ...) and it has also been observed in nonlinguistic behavior such as posture @xcite or visual attention @xcite . for example, throughout a conversation, speakers may reuse the lexical items used by their partners @xcite , and they tend to use the same referring expressions to refer to the same entities @xcite . this mechanism is said to facilitate language production and comprehension in the interaction @xcite ; and lexical and syntactic repetition have been found to correlate with task success in task-oriented dialog @xcite . one kind of alignment that is less often addressed in the literature is conceptual alignment @xcite . this refers to the extent to which two dialog participants "mean the same things when using the same words" @xcite . the fact that words have pre-established senses does not guarantee conceptual alignment, as speakers may have slightly different mental representations of words (e.g., different associations, figure 1 : we identify words that are used by both sides in a debate (here, meat) and extract contextualized representations from all their instances, which are then compared through our alignment measures. example from the iq2 dataset @xcite . while it has been found that alignment at one level enhances alignment at other levels @xcite , lexical (or surface form) alignment may actually mask conceptual misalignment, which, if undetected, can lead to serious misunderstandings @xcite . 1 nevertheless, conceptual (mis)alignment remains understudied, mainly because it is hard to detect. in this paper, we target a more restricted notion of conceptual alignment: we seek to quantify the divergence or convergence of word meaning that is inferrable from textual information alone, i.e., from the way the same words are used by two speakers throughout a dialog (see figure 1 ). we do not intend to capture conceptual misalignments that are made apparent only through non-linguistic behavior or which involve external referents (e.g., someone performing the wrong action after misunderstanding a command). we refer to this notion as lexico-semantic alignment. we propose, for the first time, a methodology and a set of metrics to explore and quantify lexico-semantic alignment in its definition presented above. our metrics rely on contextualized word representations, which have been found to reflect different aspects of word meaning, including connotation @xcite . we work with a corpus of two-sided debates which constitutes a scenario with interesting dynamics where we can find opinion disparity as well as concessions from either side. the application of an automatic coreference solver additionally allows us to work with different surface forms referring to the same entity. we carry out a qualitative and quantitative analysis of the proposed measures and investigate their usefulness in predicting a debate's outcome. our measures reflect interesting word usage discrepancies between debate sides, and are directly applicable to other kinds of conversations. foot_2 2 related work ## data and preprocessing in this section we explain how we find the common vocabulary between debate sides and how we extract contextualized representations for words and phrases in this shared vocabulary. ## alignment measures we propose measures which reflect different aspects of lexico-semantic alignment and compare them to lexical alignment measures used in previous work. we use a debate entitled "don't eat anything with a face" foot_10 as a running example to show the ranking of words obtained with each measure in several of our measures rely on the averaged pairwise similarities (psim) between the representations of two sets of instances i and j (equation 1 ). sim corresponds to a similarity measure. unless otherwise specified, we use cosine similarity. it can be replaced with a distance measure, such as the euclidean distance, in which case the results need to be interpreted accordingly. with this metric we can examine the words that show the most and the least variation across sides (see ## analysis in section 5.1, we carry out a qualitative analysis of the kinds of phenomena our measures reflect. we do so following our running example and looking at the results for individual words presented in the vocabulary used for the sv metric is v c (d). for all other metrics, we use word lemmas from v t200 (d) provided that at least one instance is available for each time step and side. 12 this consists of 33 lemmas on average. ## toward automatic winner prediction we investigate whether the proposed measures can be used in combination in a supervised classifica- measures sim/dist vocab. accuracy asours cos v 0.57 asours cos v t200+c 0.57 asours eucl v t200+c 0.57 asours eucl v t200 0.57 ours eucl v t200 0.57 asall cos v t200+c 0.54 asdia --0.52 majority class baseline 0.50 length baseline 0.49 features we use three sets of asymmetric measures (calculated for each side): ours (asours), dialign measures (asdia), and all of them combined (asall). additionally, we try using our symmetric and asymmetric measures in combination (ours). we experiment with different parameters when calculating our measures. we use cosine similarity (cos) or euclidean distance (eucl) and different vocabularies: everything (v ) or words that are within the 200 words with highest tf-idf (v t200 ), optionally in combination with v c (v t200+c ). ## conclusion and future work we have introduced and discussed the notion of lexico-semantic alignment. we have proposed a set of measures relying on contextualized word representations which are designed to account for different aspects of alignment, such as temporality and asymmetry. our qualitative analysis shows that our metrics calculated with bert reflect multiple semantic phenomena (e.g., collocations, connotation) that characterize the way each side of a debate uses specific words. we have also shown that the debate-level information provided by these metrics can be helpful for predicting a debate's winner. in future work, we plan to study our measures' behavior on other kinds of conversations where the focus would be on individual speakers, such as task-oriented dialogs or everyday conversations involving multiple topics. we think that they are also potentially useful for detecting cases of misunderstanding due to lexical ambiguity or due to a language proficiency level mismatch between interlocutors. we can also refine our measures with a more fine-grained treatment of temporality and including information of the speaker who introduced each word. finally, an obvious extension would be to experiment with different representations, e.g., from other language models.