Understanding Computational Models of Semantic Change: New Insights from the Speech Community (2023)

Abstract: We investigate the descriptive relevance of widely used semantic change models in linguistic descriptions of present-day speech communities. We focus on the sociolinguistic issue of contact-induced semantic shifts in Quebec English, and analyze 40 target words using type-level and token-level word embeddings, empirical linguistic properties, and – crucially – acceptability ratings and qualitative remarks by 15 speakers from Montreal. Our results confirm the overall relevance of the computational approaches, but also highlight practical issues and the complementary nature of different semantic change estimates. To our knowledge, this is the first study to substantively engage with the speech community being described using semantic change models.

## introduction research on computational analyses of semantic change has established a range of modeling approaches, standard evaluation practices, and evidence of their practical utility as well as shortcomings in descriptive linguistic applications @xcite . however, most work to date has focused on general-purpose diachronic corpora with little information on the underlying speakers, but the same methods are increasingly used to study finer-grained patterns in present-day speech communities. this raises the vital question of how the output of these models is perceived by the speakers whose linguistic behaviors they purport to describe. we address this issue focusing on the sociolinguistic context of quebec english, which exhibits cross-linguistic semantic change due to contact with french @xcite . we analyze 40 semantic shifts in a regionally stratified corpus of tweets, obtaining semantic change estimates from type-level and token-level word embedding models, * work done at the university of toulouse. as well as a range of empirical linguistic information. we use face-to-face sociolinguistic interviews with 15 speakers from montreal to elicit acceptability ratings and qualitative remarks for the target semantic shifts attested in tweets from our corpus. we provide the following contributions. (1) a quantitative comparison of model-derived semantic change scores, human acceptability ratings, and empirical linguistic properties, confirming the overall relevance of standard approaches and highlighting their shortcomings. (2) a qualitative analysis of the social values linked to semantic shifts, clarifying the complementarity of semantic change estimates. ## related work semantic change over time is mainly analyzed using word embeddings @xcite . typelevel models provide meaning representations that are specific to time periods and can be compared across them @xcite @xcite . token-level models provide contextualized embeddings that can be pooled into time-specific representations or used in clustering @xcite . other deep neural architectures use cross-lingual transfer @xcite or train on temporal data @xcite . similar methods are used to compare word meanings across online communities @xcite ), text types (fišer and ljubešić, 2018), dialect regions @xcite , and languages @xcite . the prevalent evaluation approach uses graded type-level semantic change scores @xcite to assess models on ranking or binary classification. this has enabled improvements in model design, shown by shared tasks on a range of european languages @xcite @xcite . the models have been used in promising linguistic analyses @xcite @xcite , including of quebec english @xcite , but their descriptive value is yet to be fully determined @xcite . further validation can be provided by the targeted speakers, similarly to collecting semantic change ratings from online communities under study @xcite . however, we argue for much more extensive engagement based on variationist sociolinguistic practice @xcite . ## data and method we aim to gain a better understanding of previously proposed computational semantic change measures by examining how they relate to a standard sociolinguistic estimate of the same phenomenon -in this case, contact-induced semantic shifts in quebec english. we begin by presenting the data and methods used to obtain these estimates. ## conclusion we used different embedding-based models to analyze semantic shifts in quebec english, and assessed them based on acceptability ratings and qualitative remarks by speakers from montreal. we found that the models were overall relevant, but that (i) model-derived scores aligned differently, and never decisively, with human ratings; (ii) empirical linguistic properties affected all scores, but in different ways; (iii) semantic shifts with similar quantitative scores were often linked to divergent qualitative assessments. this both demonstrates the complementarity of different semantic change estimates and calls for caution in interpreting them, as they fail to capture the full range of information relevant to speakers from the target community.