Investigating the Effects of MWE Identification in Structural Topic Modelling (2023)

Abstract: Multiword expressions (MWEs) are common word combinations which exhibit idiosyncrasies in various linguistic levels. For various downstream natural language processing applications and tasks, the identification and discovery of MWEs has been proven to be potentially practical and useful, but still challenging to codify. In this paper we investigate various, relevant to MWE, resources and tools for Swedish, and, within a specific application scenario, namely ‘vaccine skepticism’, we apply structural topic modelling to investigate whether there are any interpretative advantages of identifying MWEs.

## introduction multiword expressions (mwes) are common word combinations which exhibit idiosyncrasies on a variety of lexical, syntactic, semantic, pragmatic and/or statistical levels. in this paper we investigate the impact of multiword expression (mwe) identification as a text preprocessing step prior to the application of a structural topic modeling (stm) approach @xcite . as a case scenario, for investigating the feasibility of this experimental set-up, we provide an exploratory comparison of the stm analysis on a dataset that contains swedish social medial posts about novel vaccines (mrna, novavax), with and without the identification of mwes. the aim of this work is to answer the following research question: in a mixed-method research design can mwe identification enhance the interpretability and explainability of the generated topics and themes? we start by applying both available mwe lexical resources for swedish (lexicons and processing tools) and standard extraction techniques (e.g., ngram collocations) to preprocess the vaccinerelated narratives by keeping one version of the dataset intact, i.e., without any mwes identified. then, we apply stm in the two versions of the same dataset, to uncover the most prevalent discussion topics. as a methodological step, we utilize an exploratory mixed quantitativequalitative approach @xcite to investigate, compare, discuss, and loosely evaluate these topics with respect to the specific application scenario at hand (cf. section 2). the motivation behind the application is based on the fact that multiword expressions can improve topic coherence, which is positively correlated with human assessment and readability of topics @xcite . mwes can also be used to reduce ambiguity, for example, by recognizing multiword terms/names as opposed to single word tokens could prevent an incorrect interpretation in many domains; for instance, autoimmun reaction 'autoimmune reaction' (instead of autoimmun and reaction separately) or vitamin d-brist 'vitamin d deficiency' (instead of vitamin and d-brist separately) (cf. @xcite . ## application scenario: swedish social media data about novel vaccines we use vaccine skepticism as the application scenario for our case study. vaccine skepticism can be triggered by anxiety about possible side effects and concerns related to novel vaccine technologies, such as the messenger rna (mrna) which can be used as a reason for not receiving (the covid-19) vaccine @xcite . for instance, the university of lund study: "intracellular reverse transcription of pfizer biontech covid-19 mrna vaccine bnt162b2 in vitro in human liver cell line" @xcite , published on february 2022, has been frequently cited since its release, as a confirmation for vaccine skepticism and hesitancy, highlighting a potential misconception that the mrna vaccine alters the human dna. ## related work latent dirichlet allocation (lda) @xcite , is a popular topic modeling method that uses the statistical analysis of textual data to identify themes or topics that occur in a document collection. although topic modelling can identify the topics contained in text, the original bag of words approach used by lda models ignores the order of the words which limits the deeper understanding of the content. therefore, during the last years researchers try to enhance the various flavors of topic models with the addition of n-gram features to improve the results and reduce the complexity of the models. particularly, phrase-based topic modeling has shown significant improvement, especially on short text data @xcite ; and several studies acknowledge the fact lda results can be improved when mwe expressions are included during processing @xcite @xcite . ## structural topic modeling structural topic model (stm) has emerged as an extension to lda allowing the integration of covariates into the prior distributions for document-topic compositions and topic-word proportions. thereby, stm can be used to model how the content of a collection of documents changes as a function of document-level covariates such as day and time, and gain insights and understanding on how topics evolve @xcite . we apply stm to automatically detect latent topics in the dataset which can be used to investigate the nature of these topics reflected in the novel vaccine discussions @xcite . we use a sequential explanatory mixed methods approach which consists of a quantitative phase (collection, cleaning, and natural language processing of the data), followed by a qualitative phase (in-depth analysis of the results from the quantitative phase). this type of design provides greater analysis depth than either singular analysis would @xcite . ## results and discussion we have hypothesized that the identification of multiword expressions can provide us with better and more targeted insights and enhance the interpretability and explainability of the generated topics and themes. the characterization of the multiword expression types which are recognized and applied in this experimental setup follows the order given in section 2.2. we extracted a list of keywords for each topic that have the highest association with that certain topic. for this association, we base ourselves on the frex (frequency and exclusive) value of each word. this value combines the exclusivity of each word (meaning that a word occurs more often in that topic than in others) while also correcting for its overall frequency @xcite . to better understand and label the topics and to label them, we also extracted the top 30 most representative posts for each topic. this qualitative evaluation shows that the proposed method provides only slightly better performance of smt on the sv-socialmedia-mwe dataset. moreover, the frex metric helps us to assign intuitive labels to the subject matter. frex is defined as the ratio of word frequency, and subject to word-topic exclusivity. balancing these two measurements is important as frequent words can often be uninformative, while completely exclusive words can be very rare and not informative. the frex example below, for instance, taken from the sv-socialmedia-mwe version (of topic 2 'women's health issues'), illustrates that three of the tokens, among the top 12, are multiwords: mensrubbning 'menstrual disorder'; polio 'polio'; missfall 'miscarriage'; fertilitet 'fertility'; graviditet 'pregnancy'; stelkramp, 'tetanus'; klimakteriet 'menopause'; rubbning 'disturbance'; ## conclusions and future work the results suggest that the stm models perform only slightly better without mwe than with. one reason could be that the vocabulary is larger without mwes included, so the number of topics capture more words. still, the differences are quite small, and the comparisons of the models based on the two datasets are not 100% fair, since the mwe accounts for about 600 "terms" more and is therefore kind-of another dataset altogether as far as the stm is concerned. still, there is some indication that mwe identification leads to better interpretability of the stm, as calculated by the semantic coherence, which was higher for the dataset with mwes. therefore, we could conclude that the identification of mwes can slightly enhance the explainability of the generated topics and themes, which could lead to a more appropriate labeling of the topic itself during the qualitative interpretation of the generated topics, i.e., incorporating multiword expressions into the models, creates slightly more informative resulting topics. in general, the major differences between the two versions of the topics are also shown in the graphs of appendix b which show the prevalence over time for the topics of the two dataset versions -without stating anything about their quality. major differences for some of the generated topics, that needs further investigation. as a future work it would be also interesting to verify the efficacy of our resources and our method on different domains and types of datasets and explore more resources for multiword recognition for swedish. identification of noncontiguous multiword expressions is another area we need to explore @xcite . there are several opportunities for future research to extend our assessment of the performance and evaluation. as previous research has pointed out, topic modeling algorithms are sensitive to several characteristics such as text length and the text preprocessing applied, for instance no stemming or lemmatization was applied that could have impact on the results @xcite . hence, further investigations including parameters and characteristics are necessary.