Short-term Semantic Shifts and their Relation to Frequency Change (2020)

Abstract: We present ongoing research on the relationship between short-term semantic shifts and frequency change patterns by examining the case of the refugee crisis in Austria from 2015 to 2016. Our experiments are carried out on a diachronic corpus of Austrian German, namely a corpus of newspaper articles. We trace the evolution of the usage of words that represent concepts in the context of the refugee crisis by analyzing cosine similarities of word vectors over time as well as similarities based on the words’ nearest neighbourhood sets. In order to investigate how exactly the contextual meanings have changed, we measure cosine similarity between the following pairs of words: words describing the refugee crisis, on the one hand, and words indicating the process of mediatization and politicization of the refugee crisis in Austria proposed by a domain expert, on the other hand. We evaluate our approach against the expert knowledge. The paper presents the current findings and outlines the directions of the future work.

## introduction words are prone to change their meaning over time. automatic detection and analysis of such change is beneficial for a number of theoretical and computational linguistics tasks, as well as for other research areas, i.e., digital humanities, social and political science. semantic change occurs due to various linguistic and extra-linguistic factors. although it might be rather challenging to distinguish between these two causes of change, the latter offer an opportunity to gain insights into temporal dynamics of social, cultural, political phenomena reflected in texts. the discussion about the relationship between semantic evolution of words and their frequencies has been already introduced in the studies of computational semantic change detection @xcite @xcite @xcite @xcite . although in linguistics, semantic change was shown to be associated with frequency rise @xcite , @xcite emphasized that a change in frequency can be a precursor to semantic change, but does not capture the change itself. therefore, when studying semantic shift it is important to consider the effect of frequency change on the observed shift. we contribute to the field by analysing shortterm semantic shifts and their relation to frequency change. furthermore, we propose an evaluation approach as in the case of short-term change previously proposed strategies cannot be adopted. unlike large-scale semantic shifts that span across decades or centuries, short-term semantic shifts within monthly or yearly time slices reflect changes in usage rather than in meaning as it is understood in lexicography. therefore, our work aims to answer the following question: with respect to frequency, what patterns that represent contextual meaning change emerge when dealing with short time slice data? more specifically, does a significant increase in frequency indicate a semantic shift towards broadening or narrowing of usage contexts? and the other way round, if relative frequency does not fluctuate much, can we expect stability of contextual meaning? we address these questions by investigating the case of the refugee crisis in austria from 2015 to 2016 as covered in the austrian media. we evaluate our approach relying on the evidence from a domain expert. the rest of the paper is organized as follows: in section 2 the related work is discussed. in section 3 the data, on which our analysis is based, is presented. in section 4 we introduce our method-ological approach. details about our experiments are provided in section 5 and the results are discussed in section 6. finally, section 7 contains our conclusions. ## related work recently, there has been seen an increased interest in computational semantic change detection methods. @xcite proposed to distinguish between two types of semantic change: linguistic drift and cultural shift. according to them, comparison of word's vectors in different time periods (global measure) reflects linguistic drift, while comparison of word's nearest neighbors (local measure) is suitable for identifying cultural shifts. furthermore, @xcite pointed out that when detecting short-term semantic shifts, the neighbourhood based model outperforms successful alignment methods of long-term semantic change. however, @xcite mentioned that a global measure is sensitive to slight change in word usage, hence we investigate both measures. with a focus on tracing contextual variability over time, we assume that rising similarity of contextual meaning indicates homogenization of overall word use (narrowing), while a decrease in similarity signifies diversification (broadening) @xcite . in order to bring light on the direction of narrowing of a word's usage, we look at the semantic change of pairs of words which was shown to be reliable in different experiment settings @xcite @xcite @xcite . more specifically, we study temporal dynamics between the words of interest and their most close semantic associates. one of the studies of contextual variability that explore correlation with frequency is by @xcite . they explored the effect of frequency on quantifying synchronic semantics shifts between words in two italian newspaper corpora and found that contextual variability is larger if a word is relatively more frequent in the second corpus. on the other hand, @xcite al. (2017); del tredici et al. (2019); vylomova et al. (2019) showed that significant change in frequency does not necessarily imply semantic context change. @xcite tackled the task of semantic change detection by applying several algo-rithms on english, german, french and chinese data, and conducting evaluation against known diachronic change. according to the law of conformity proposed by them, high frequency words are less likely to undergo semantic change. later research supported @xcite @xcite or denied the law by showing that frequency does not correlate with semantic change @xcite . @xcite proved that due to the training procedure semantic change models capture both the change in meaning and noise. they demonstrated that in controlled conditions, the reported meaning change effects largely disappear or become considerably smaller. therefore, when tracing semantic change there is a risk to over interpret differences in word's meaning representations that actually stem from noise. frequency difference among words possibly accounts for such noise. ## data the austrian media corpus (amc) is a diachronic text corpus that contains austrian newspapers, magazines, press releases, transcribed television interviews, news stories from television, etc. from the last thirty years @xcite . with over 44 million articles, it is one of the largest text corpora for german and the largest for austrian german. the language data is tokenized, part-of-speech tagged and lemmatized. in total, it contains 10.500 billion tokens. although the corpus spans over thirty years, we use the data covering ten years, from 2008 to 2017, and only newspaper articles were taken for our analysis. we split the data into yearly spanned subcorpora, thus obtaining ten corpora for the study with their sizes ranging from 141 to 152 million tokens. next, we extracted lemmas representing common and proper nouns, verbs, adjectives and applied stop words filtering (numerals, names of months and days of the week). a frequency threshold of 100 minimum counts was applied to each subcorpus. ## methods distributional semantic methods adopt the hypothesis that meaning of a word is conveyed in its co-occurrence relationships @xcite . the semantic similarity of two words is then approximated by the cosine similarity (the value ranges from 0 to 1) between their vectors that capture information about its co-occurrence statistics. recent studies mainly make use of dense word representations, usually predictionbased word embeddings models. however, reliability of word2vec based approaches was considered questionable by some studies @xcite @xcite @xcite . furthermore, @xcite evaluated various word's meaning representations and showed that although the best run of the word2vec based model strongly outperforms other methods, its mean performance measured over several runs is comparable with the two count-based representations, namely word vectors from a weighted matrix with positive pointwise mutual information (ppmi) scores and truncated singular value decomposition (svd) of a ppmi matrix foot_0 . in the light of the above and since ppmi vectors were previously demonstrated to be less affected by frequency effects @xcite , we opt for using an average score of 50 runs of the ppmi-based model. as follows, we construct sparse co-occurrence matrices with the window of seven words for each subcorpus and apply ppmi weighting with the parameters suggested by @xcite . a ppmi score with a smoothing parameter α of a word w and its context word c is calculated by the following formula: we employ vector and local neighbourhood similarity measures to get an idea on the overall semantic stability/instability and a time point where the possible change occurs (i.e., allows to discover what and when has changed), while pairwise similarity measure provides information on the direction of context shift (i.e., allows to discover how it has changed). in order to compare to what extent an obtained similarity time series correlates with the frequency time series of a particular word we apply autocorrelation based dissimilarity measure which is the euclidean distance between simple autocorrelation coefficients of the two given time series @xcite . autocorrelation distance is preferred to a simple cor-relation metric due to the fact that autocorrelation implicitly normalizes scores which is helpful when dealing with the time series data of different scale. ## experiments we represent the discourse of the refugee crisis by the following list of words identified by a linguistics specialist: asyl 'asylum', asylwerber 'asylum seeker', ausländer 'foreigner', flüchtling 'refugee', flüchtlingskrise 'refugee crisis', flüchtlingsstrom 'refugee flow', migrant 'migrant', migration 'migration', schlepper 'illegal migrants smuggler' foot_2 , zuwanderung 'immigration'. among other parts of speech, we use only nouns as they are particularly sensitive to cultural shifts in meaning which represent the scope of the current work foot_3 . we explore semantic development of these concepts in the context of the refugee crisis as covered in the media. to enhance the semantic analysis based on these terms and to evaluate the methods we compile two supplementary lists of words which are described below. we expect our approach to show the rise in similarity between between the "seed" concepts and supplementary terms in the years 2015, 2016, 2017 (the refugee crisis period, plus one subsequent year when the flow of refugees declined but the topic of migrants was still widely discussed in the media). our evaluation is based on the statement of an expert in critical discourse analysis that addresses the way societal power relations are established and reinforced through language use @xcite . the statement focuses on the shift to the far-right in the austrian party landscape, mainly the austrian people's party ( övp) and the freedom party of austria (fp ö). during the 2016-2017 election campaign, one of the governing parties ( övp) adopted extreme-right fp ö's policy towards migration and refugees. according to the expert analysis of the political and media discourse, this shift to the right is reflected in language by referring to the migrants as a threat to the welfare of the nation and instigating fear among citizens. moreover, it was proposed to build a border fence and set the maximum limit for asylum applicants. this is pointed to by the increased use of the silent concepts such as angst 'fear', bedrohung, 'threat', grenze 'border', grenzzaun 'border fence', obergrenze 'upper limit', richtwert 'guiding number', terrorismus 'terrorism' in the refugee crisis discourse @xcite . these seven terms constitute our first supplementary list. surprisingly, there was no outrage following the changed attitude towards the refugees even though some of the fp ö claims are clearly against universal human rights. the expert contends that with the purpose of legitimation of unprecedented policy, certain mobilizing and politicizing concepts, i.e., humanität 'humanitarianism', protektionismus 'protectionism', sicherheit 'security', vielvalt 'diversity' were deployed by politicians and the media @xcite . these words form our second supplementary list. ## conclusion short-term semantic shift is a complex phenomenon, and understanding the nature of it implies a lot of challenges. our findings suggest that significant frequency increase is not necessarily followed by significant change in usage, and relatively constant frequencies over time do not imply stability of contextual meaning. we showed that dynamics of word usage is a possible indicator of wider socio-cultural or political shifts. since the paper presents an ongoing study, there is a space for enhancement in many aspects. first, more methods should be compared and comprehensive fine-tuning of the models performed with the careful control for randomness and instability that these models feature. second, one could experiment with the choice of time slices, whether they could be defined empirically or represented in continuous way intersecting one another. third, candidates for semantic shift could be selected in a robust way and fine-grained annotated data would be beneficial for thorough evaluation and scaling up the experiments for providing more evidence of the phenomenon under discussion. fourth, control conditions should be introduced in order to ensure verification of the obtained results.