What Causes Unemployment? Unsupervised Causality Mining from S wedish Governmental Reports (2023)

Abstract: Extracting statements about causality from text documents is a challenging task in the absence of annotated training data. We create a search system for causal statements about user-specified concepts by combining pattern matching of causal connectives with semantic similarity ranking, using a language model fine-tuned for semantic textual similarity. Preliminary experiments on a small test set from Swedish governmental reports show promising results in comparison to two simple baselines.

## introduction extracting causal relations from natural language text is a popular task that has been tackled using a variety of techniques @xcite . most approaches to causality mining are based on supervised machine learning and presuppose annotated training data, which is lacking for many languages and domains. in this paper, we describe a system for exploring swedish governmental reports, where users can search for statements about potential causes and/or effects related to specific concepts, such as pollution or unemployment. the system should then retrieve sentences that make causal claims involving the specific concepts and rank them by relevance to the original query. in order to delimit the scope of causality mining, we follow the approach of @xcite and focus on causality that is explicitly expressed linguistically, by the use of some causal connective, rather than implicitly expressed causality. a causal connective is any type of linguistic expression that is used to express a causal relation, for instance, verbs like cause, conjunctions like because, nouns like effect, and different types of multi-word expressions like be a result of. while there is plenty of work on the identification of causal sentences or entities, and the extraction of causal relations from text (for an overview, see @xcite , we are not aware of any work focusing on the ranking of extracted causal sentences. it could be viewed as an information retrieval task, for which work is abundant (see, e.g., @xcite . there are also efforts for more specific structured queries, such as taub @xcite . most approaches to causality mining are based on supervised machine learning and presuppose annotated training data, which is available for languages like english @xcite @xcite @xcite and german @xcite . since the single annotated data set available for swedish @xcite is much too small to use as training data, we instead explore a combination of techniques involving keyword matching and pre-trained language models fine-tuned for semantic textual similarity (sts). the idea is to use keywords corresponding to causal connectives -such as the verb cause or the prepositional expression because of -to construct templatic sentences with masked tokens corresponding to the sought cause or effect, for example: "mask causes pollution", "unemployment because of mask". the sts model, in our case contrastive tension @xcite , can then be used to find the sentences in a document collection that are semantically similar to the template sentences and are likely to include instantiations of the sought cause or effect. we believe that this system design can be useful for other tasks where a domain is under-resourced for the target language. we evaluate our approach on a small test set for ranking causal sentences @xcite . preliminary experiments show that the unsupervised method combining keyword matching and semantic similarity search improves over two simple baselines. as far as we are aware, these are the first published results for this task and data set. facing the lack of annotated training data for this task, we instead leverage a pre-trained masked language model tuned for sts. the main idea is to first convert the query to one or more query prompts, that is, templatic sentences with masked tokens corresponding to empty slots, and then search for semantically similar sentences in the document collection. for example, the query [cause: pollution] could be converted to a query prompt such as "pollution causes mask" or "mask is the result of pollution" with the hope that semantically similar sentences make claims about specific phenomena caused by pollution. in the following, we describe the creation of query prompts and the semantic search procedure in more detail. a key component in both is a set of causality keywords, which are used both to create template sentences and to filter sentences in the search procedure. we, therefore, begin by describing the set of causality keywords used. ## data sets we use a previously published test set @xcite , which includes sentences from the swedish government official reports, statens offentliga utredningar (sou) in swedish, from 1994-2020. for more details on the corpus and data set creation, we refer to the original paper. ideally we would evaluate a full list of ranked sentences for each test query. instead, the test set frames an easier annotation task: to classify pairs of sentences for relevance and rank the two sentences internally. for a sentence pair, like the one in figure 1 , the task was to assess their relevance to a given query. the annotation scheme has six cate-gories covering the following cases for sentences 1 and 2: in the example in figure 1 , both sentences are considered relevant, but the second more so, since it explicitly mentioned the term greenhouse effect from the query; hence it is classified as case 3. the test set consists of 800 sentence pairs and their ranked relevance with respect to 43 causal prompts. ## experiments in this section, we compare our system, where we rank causal sentences with ct models, to two baselines: random, which just randomly shuffles the sentences we consider for ranking, and kb-bert @xcite , where sentence embeddings are obtained by mean-pooling the hidden states of kb-bert. for the random baseline we take the average of 10 different random seeds. besides the original ct model, ct-orig, released by @xcite , initialized with kb-bert, and then trained on wikipedia data with the ct objective, we also train two additional in-domain ct models, which we describe next. ## conclusion we describe an initial exploration of causality mining with respect to specific concepts, such as pollution or unemployment, in swedish governmental reports. we present the task in detail and note that there is no available training data. we thus design a search system based on the combination of keyword matching and semantic similarity ranking, which can give reasonable results for a human-inthe-loop scenario. this work can be viewed as a first step towards enabling impact assessment of swedish governmental reports. our system for ranking causal sentences with respect to a theme could potentially feed into more advanced systems for impact assessment, for instance with the goal of exploring trends across sources and over time. although the preliminary results look promising, further evaluation on a larger test set as well as on other document collections will be needed to assess the viability of the approach. it would also be interesting to explore whether syntactic or semantic parsing could be used to improve the model's capacity to distinguish the direction of causality and prevent the confusion of causes and effects. another direction would be to use less aggressive methods than causal keywords for filtering causal sentences. one possibility could be to utilize available data from other languages, to train a cross-lingual model for identifying swedish causal sentences, as proposed in @xcite .