A unifying framework for modeling acoustic/prosodic entrainment: definition and evaluation on two large corpora (2020)

Abstract: Acoustic/prosodic (a/p) entrainment has been associated with multiple positive social aspects of human-human conversations. However, research on its effects is still preliminary, first because how to model it is far from standardized, and second because most of the reported findings rely on small corpora or on corpora collected in experimental setups. The present article has a twofold purpose: 1) it proposes a unifying statistical framework for modeling a/p entrainment, and 2) it tests on two large corpora of spontaneous telephone interactions whether three metrics derived from this framework predict positive social aspects of the conversations. The corpora differ in their spoken language, domain, and positive social outcome attached. To our knowledge, this is the first article studying relations between a/p entrainment and positive social outcomes in such large corpora of spontaneous dialog. Our results suggest that our metrics effectively predict, up to some extent, positive social aspects of conversations, which not only validates the methodology, but also provides further insights into the elusive topic of entrainment in human-human conversation.

## introduction a phenomenon that has been repeatedly documented in human-human conversations is the tendency of partners to become more similar to each other in the way they speak. this behavior, known in the literature as entrainment, has been shown to occur along several dimensions during humanhuman interaction (see @xcite @xcite @xcite , being one of these dimensions the behavior of acoustic-prosodic (a/p) features (see, for example, @xcite . a/p entrainment has been associated with multiple social aspects of human-human conversations, such as the degree of success in completing tasks @xcite , the perception of competence and social attractiveness @xcite @xcite @xcite , and the degree of speaker engagement @xcite . nonetheless, empirical evidence also points toward these relations being quite complex. as an example, disentrainment (speakers actively adapting to become more dissimilar to each other) has also been associated with positive social aspects in conversations (see, for example, @xcite @xcite . in spite of these advances, research on the effects of a/p entrainment is still preliminary. in first place, because the way a/p entrainment in conversations is modeled is far from standardised. as an illustrative example, when estimating a/p entrainment metrics, some studies first approximate the evolution of each speaker's a/p features and then use these approximations to calculate a/p entrainment metrics @xcite @xcite ; others study the correspondence between adjacent inter-pausal units (ipus) -defined as speech segments separated by a pause -from different speakers and derive metrics from it @xcite ; and still other studies measure a/p features in different sections of speech for later comparing these values to compute a/p entrainment metrics (see, for example, @xcite . moreover, studies commonly differ in which metrics are analyzed. for this reason, a reliable, simple, general, and flexible framework able to unify the estimation of different types of entrainment metrics is needed. in second place, research is still preliminary because most of the reported findings rely on small corpora, or on corpora collected in experimental setups, making it hard to extrapolate their conclusions to more general contexts. in this way, evidence is still needed on how a/p entrainment relates to social aspects of human-human conversation under different types of natural interactions. the present article has a twofold purpose. first, it proposes a unifying approach for modeling a/p entrainment in conversations. the methodology is simple and flexible enough as to allow calculating adapted versions of several a/p entrainment metrics used in previous studies. second, it evaluates three entrainment metrics derived from the proposed framework on two very different large corpora of spontaneous telephone interactions (the switchboard corpus, in english, and a large collection of call-center conversations, in spanish), testing whether these metrics predict positive social aspects of conversations. to our knowledge, this is the first article testing the relation between a/p entrainment and positive social outcomes in such large corpora of spontaneous natural dialog. overall, our results suggest that metrics derived from the proposed methodology effectively predict, up to some extent, positive social aspects in conversations, which not only validates the methodology, but provides further evidence suggesting that a/p entrainment relates to positive social aspects in human-human conversation under different types of natural settings. additionally, insights on how a/p entrainment metrics relate to social outcomes predictions is provided. the rest of the paper is structured as follows. section 2 presents the proposed framework for modeling a/p entrainment. section 3 details on how we empirically test for relations between metrics obtained using the proposed methodology and positive social aspects of conversations. section 4 presents results from the empirical study. section 5 provides discussion and concludes. ## a unifying framework for modeling a/p entrainment here we present a methodology for modeling a/p entrainment. we divide the process in three steps: 1) extracting a/p features from ipus, 2) estimating the speakers' a/p evolution functions, and 3) calculating a/p entrainment metrics from a/p evolution functions. the following sections describe each step. ## empirical study materials and methods next, we ran a series of machine learning experiments aimed at investigating whether the metrics derived from the proposed methodology have any predictive power over positive social aspects of conversations. this section describes the corpora and the methodology used. ## discussion in this work we proposed a unifying framework for modeling different types of a/p entrainment in natural conversations. we also tested on two very different corpora whether three metrics derived from our framework provide valuable information for predicting positive social outcomes in conversations (perceived naturalness in swbd and self-reported customer satisfaction in ccc). our results suggest that these metrics effectively relate to positive social outcomes. however, several remarks should be made. first, the fact that the achieved auc scores are greater than chance not only validates the proposed metrics, but strongly suggests that a/p entrainment is related to positive social outcomes. importantly, this result was not found on a single corpus, but on two independent ones. both corpora had not only different positive social outcomes attached, but also differed in their domain and even in their language (english and spanish). future research should focus on testing if these results prevail across broader domains and for further social variables. second, even when the obtained results are higher than chance, they are far from being exceptionally high. this suggests that a/p entrainment metrics by themselves -at least the ones testedmay not contain enough predictive information as to achieve competitive results. probably a competitive model should incorporate information regarding the semantic content of conversations and/or, for the case of a corpus like ccc, customer relationship management related information. however, this does not mean that the proposed metrics are of no use. future research should focus on studying whether the information provided by a/p entrainment metrics complements the one provided by other sources. in our experiments we tested this up to some degree. in particular, we observed that a/p entrainment metrics complement the information contained in the external features. this effect is very strong for ccc and less strong for swbd. third, the fact that the best set of features differs across corpora, suggests that which features predict positive social outcomes depends on the outcomes being predicted and the corpus itself. note that a similar pattern was observed in pérez et al. @xcite where, even on the same corpus, the significance of synchrony metrics calculated on different a/p features varied across different social outcomes. fourth, shap dependence plots suggest that the manner in which predictive models make use entrainment metrics is quite complex. first of all, not always are higher entrainment values associated to higher predicted values of a positive social aspect. rather, two more patterns are observed: a negative relation between a/p entrainment and positive outcomes, and an inverted u pattern. additionally, in line with the third remark, it is interesting to note the effects of a given a/p entrainment metric are not the same across corpora, again suggesting heterogeneity across tasks and corpora. an illustrative case are the patterns observed for speechrate-synchrony, for which an inverted u pattern is observed in swbd and a negative relation is observed in ccc. finally, the reason why people do entrain is still unknown (see, for example, @xcite @xcite @xcite . consequently, metrics such as the ones tested in this work, albeit noisy and imperfect, are likely to be capturing part of some more complex phenomenon that we do not fully understand yet. further research on the causes of entrainment in human speech is still needed.