Sudden Semantic Shifts in S wedish NATO discourse (2023)

Abstract: In this paper, we investigate a type of semantic shift that occurs when a sudden event radically changes public opinion on a topic. Looking at Sweden’s decision to apply for NATO membership in 2022, we use word embeddings to study how the associations users on Twitter have regarding NATO evolve. We identify several changes that we successfully validate against real-world events. However, the low engagement of the public with the issue often made it challenging to distinguish true signals from noise. We thus find that domain knowledge and data selection are of prime importance when using word embeddings to study semantic shifts.

## introduction a well-known adage in natural language processing is that one knows a word by the company it keeps @xcite . yet, this company does not need to be stable and can change in either the long or short term. when this happens, the word undergoes a semantic shift. one common way to study these semantic shifts is by using temporalor diachronic -word embeddings. most semantic shifts are slow and happen over many years or decades. examples are words such as "nice", "broadcast" and "gay" which today have a different meaning than they would have had in the nineteenth century. yet, while such shifts occur over various decennia, other shifts are more rapid. for example, the word "hero" changed its context from "veteran" and " superman" to "frontliner" and "covidwarrior" during the covid-19 pandemic in a matter of months @xcite . the speed of semantic change depends on various factors, such as whether the word has more than one meaning or how common it is in use @xcite . also, sudden semantic change can occur during high-impact events, such as abrupt political, social, or cultural changes. for example, @xcite notes that the meaning of the word "terrorism" changed rapidly after the events of september 11, 2001. this, combined with the knowledge that a change in the meaning of a word also changes the opinions people associate with that word @xcite , makes understanding such sudden shifts relevant if we wish to understand people's changing opinions during real-world events. here, we use word embeddings to focus on an abrupt event in the case of sweden: the country's decision to apply for nato membership in 2022, following the russian invasion of ukraine. this decision was a sudden shift and a marked change in the country's stance on foreign affairs and defense. to study this shift, we focus on the time from september 11, 2021, to september 11, 2022, the day of the 2022 swedish general election. we chose this period as we wished to examine how the language used around nato changed under the assumption that nato would be a major election issue in sweden. to measure the semantic shifts, we use the word embeddings from a word2vec @xcite model to estimate the semantic context of a set of words of interest. we then track these words over time to see if and how they changed by comparing the rank sorting of the most similar words between various periods. from here on, this paper will proceed as follows. first, we will introduce the background to the swedish application for nato membership, and how it can serve as a marked and sudden change. we then introduce our data and the procedure we used for pre-processing. following this, we discuss our methods and the findings that result from them. we end with some brief conclusions and several suggestions for further research. ## background for over two hundred years, sweden followed a self-proclaimed policy of non-alignment ("alliansfrihet") @xcite . as a result, it did not take part in most major wars, nor became part of any military alliance during the cold war. and while it often participated in nato exercises @xcite , full membership was rarely considered. thus, minister for defense peter hultqvist could describe a swedish membership of nato as unthinkable as late as november 2021 @xcite . after the invasion of ukraine in february 2022 though, the government changed its position. this sudden change was possible due to the support of the opposition for membership and the disengagement of most citizens on the issue @xcite . as a result, the government announced its plans to join nato on april 13 and formally applied for nato membership on may 16, 2022. within this timeframe, three events are of note. first, there was the turkish opposition to swedish membership, rooted in that country's opposition to sweden's support for kurdish parties and activists @xcite . second, there was a "no confidence" vote in the swedish house of representatives on the future of minister for justice morgan johansson. while he survived this vote thanks to the support of kurdish-iranian mp amineh kakabaveh, in return the government had to affirm an earlier agreement made in 2021 that stated that "people from those [kurdish] organizations coming to sweden are not terrorists" -a line of reasoning that went straight against turkish demands @xcite . third, there was the nato summit that took place between 28 -30 june, where all nato members (turkey included) extended a formal invitation to both finland and sweden to join nato. a final point of note is that over this period, the application to nato membership was what berglez (2022) calls a "hidden issue". that is, both the government and opposition aimed to -and succeeded -in drawing attention away from it and were thereby followed by most of the media. an illustration of this is that the words "alliansfrihet" and "nato" only occurred respectively 471 and 7936 times in the main swedish media over the period of a year around the application. moreover, the use of both words peaks around may, after which their number drops to almost zero until the elections in september. ## related work we base our decision to use global word embeddings to capture sudden semantic shifts on a wellfounded body of work. not only are they able to capture the semantic similarity and alignment between words, but they are also able to track the shifts in the meaning of political concepts. for example, @xcite show that the meaning of medical words changed before and after the first outbreak of covid-19, while rodman (2020); rheault and cochrane (2020) does the same for parliamentary data, and @xcite successfully use global embeddings to measure sociological concepts such as bias. of note is that all these papers opt to use global word embeddings instead of contextual word embeddings (e.g. elmo @xcite , bert @xcite ). while global word embeddings associate a single embedding vector with a word, contextual word embeddings assign a different vector for the same word depending on the sentence in which it appears. while this has the advantage of being able to take the context of the specific occurrence of a word into account, it does not provide a way to represent the position of a single word in the embedding space. that is, when we care about the global shift of words (as we do here), we need a global and not a contextual embedding. as such, most authors in the social sciences, and we here as well, opt to use global embeddings. ## data to measure our semantic shifts, we rely on swedish-language twitter posts ("tweets") that focus on nato. we do so as twitter's broad user base touches all segments of society, allowing us to get a complete picture of the debate around nato. besides, as tweets have a limit of 280 characters, their length is very similar. this has the advantage that it improves data consistency while reducing computational complexity. within our year-long period, we collected 1, 188, 556 tweets, made by a total of 64, 315 users participating in 507, 359 conversations. of these, 329, 336 are retweets, leaving 859, 220 original tweets. we collected a tweet if it contained any one of a set of search terms relating to nato. to generate these terms, we drew on both theoretical expectations (deductive) as well as first results (inductive). as such, we ended up with seventyfive unique search terms covering nato, alliances, and the war in ukraine (see @xcite for a complete list). many of these words were either compound words that contain "nato" or relate to nato and are specific enough to only occur in that context. thus, we did not include general terms such as "allians" (alliance), unless they were part of the phrase "militär allians" (military alliance) or "allians med turkiet" (alliance with turkey). in the end, we included a tweet when: a) it contained any of the search terms, b) the tweet is a response to another tweet that contained a search term, or c) the tweet has a response containing a search term. based on the background of the nato issue as sketched above, we divide our tweets into four periods. first, there is the pre-invasion period, ranging from september 11, 2021, to 24 february 2022 (the date of the russian military invasion of ukraine). second, there is the post-invasion period running from february 24 to april 13, the date of the joint press conference of the swedish pm andersson and her finnish colleague marin, where both announced the possibility of their countries joining nato. third, there is the pre-application period, running between april 13 and the formal application on may 16. finally, there is the postapplication period, running between may 16 and the elections on september 11. ## pre-processing given that the choice -and order of -preprocessing steps will influence our analysis, we discuss each of these steps in turn @xcite . first, we remove any urls and mentions to other users as well as some minor punctuation. second, we split our tweets into individual tokens. for this, we use the nltk library's nltk.tweettokenizer, as it splits hashtags and emojis better than other tokenizers @xcite . third, we lowercase all tokens, create n-grams (with no limit, so 3-grams can occur), and remove all remaining punctuation. finally, we normalize the spelling of our tokens to address the various spellings of the same word (e.g. "grey" and "gray"). for a more detailed overview of the pre-processing see @xcite . we did not perform the common steps of removing stop words or lemmatizing the tokens, as we found that these steps weakened the relationship between related words. singletons and low-frequency words were filtered out by the gensim library ( řehůřek and sojka, 2010), which was used for the analysis. ## method the model we chose to find our word embeddings is word2vec @xcite . this is a singlelayer neural network that is trained to predict a word from its context -continuous bag-of-words (cbow) -or context from a given word -skipgram (sg). we opted to use both architectures given that they are different in the associations they capture, their computational efficiency, and their sensitivity to less-frequent words @xcite . ## results as both architectures lead to different results, we will look at both the results of the continuous bagof-words (cbow) and the skip-gram (sg) in turn. for each of the two, select four words that we deemed showed interesting patterns. these are "natoansökan" (nato application) and "försvaret" (defense), as well as two unique ones for each -"nato" (nato) and "säkerhet" (security) for cobw and "förskolor" (preschools) and "putin" (putin) for sg. for each word, we give the top four words associated with it based on their cosine similarity. besides these, we will also reflect on several other words that we found showed interesting behavior. ## conclusion our aim with this study was to look at the sudden semantic shift that we expected to occur when sweden decided to apply for nato membership in 2022. looking at various words related to this application process, we find that word embeddings are a powerful tool to capture some of those shifts. moreover, when validating them against real-world events, we find that those shifts are both accurate and meaningful. yet, the sparsity of the dataset often makes it difficult to separate signal from noise when looking at the model results alone. the misalignment between the signals that each of the two model architectures -sg and cbow -manage to capture, as well as the difficulty of validating and interpreting the results exemplifies the challenges in using word embeddings for automatically detecting and measuring semantic shifts. thus, there is a need for extensive human interpretation and validation based on domain knowledge together with a broad range of statistics that can reveal different aspects of the patterns captured by the models. despite this though, word embeddings are still a powerful method that can aid the discovery process. as we showed, they are efficient enough to process large amounts of data and capture several underlying word relationships and sudden semantic shifts.