Zero-shot Cross-Lingual Counterfactual Detection via Automatic Extraction and Prediction of Clue Phrases (2022)

Abstract: Counterfactual statements describe events that did not or cannot take place unless some conditions are satisfied. Existing counterfactual detection (CFD) methods assume the availability of manually labelled statements for each language they consider, limiting the broad applicability of CFD. In this paper, we consider the problem of zero-shot cross-lingual transfer learning for CFD. Specifically, we propose a novel loss function based on the clue phrase prediction for generalising a CFD model trained on a source language to multiple target languages, without requiring any human-labelled data. We obtain clue phrases that express various language-specific lexical indicators of counterfactuality in the target language in an unsupervised manner using a neural alignment model. We evaluate our method on the Amazon Multilingual Counterfactual Dataset (AMCD) for English, German, and Japanese languages in the zero-shot cross-lingual transfer setup where no manual annotations are used for the target language during training. The best CFD model fine-tuned on XLM-R improves the macro F1 score by 25% for German and 20% for Japanese target languages compared to a model that is trained only using English source language data.

## introduction a counterfactual statement describes an event that may not, did not, or cannot take place, and the subsequent consequence(s) or alternative(s) did not take place @xcite . counterfactual statements can take the form -if p was true, then q would be true (i.e. assertions whose antecedent (p) and consequent (q) are known or assumed to be false). counterfactual detection (cfd) is an important task in nlp, which has found broad applications such as customer review analysis in e-commerce @xcite , social media analysis @xcite and automatic psychological assessment @xcite . to fur-ther explain the cfd task, consider the following counterfactual statement extracted from a product review: i wish the trouser had ruching so that it could fit me well. this is considered a counterfactual statement because it has the subjunctive mood wished and the author of the review wishes that the trouser had ruching, whereas it does not have in reality. in this particular example, trouser had ruching is the antecedent and it could fit me well is the consequent. ideally, for a user who is searching for trousers with ruching we should not display this particular trouser because it does not have rouching. by accurately detecting counterfactual statements, we can prevent such irrelevant search results. almost all prior work on cfd has been limited to the english language @xcite @xcite @xcite @xcite with the notable exception of @xcite , who looked at german and japanese counterfactuals in addition to english. however, all existing work on cfd require manually labelled languagespecific counterfactual statements for the target language of choice. extending cfd to different target languages has been hindered so far by two main challenges. first, manual annotation of counterfactuality is a time consuming and a costly task, which requires professional linguists as shown by @xcite . moreover, such expert annotators might not be available for all languages we would like to perform cfd. second, counterfactual clues such as wished, would have (in english) or fehlt, wenn es (in german) etc. are highly language-specific, which makes it difficult to transfer a model trained on a source language to a different target language without neither labelled counterfactual examples nor clue phrase lists. to address the above-mentioned challenges, we propose a zero-shot cross-lingual transfer learning method for cfd that learns a cfd model for a tar-get language without using any labelled data for that target language. our proposed method consists of two steps: (a) automatic clue phrase extraction for the target language and (b) learning a cfd classifier for the target language by predicting the clue phrases in the text. we use a neural alignment model @xcite to align machine-translated source language counterfactual sentences to find clue phrases for the target language. we then use those automatically extracted target language clue phrases to induce sequential labels for the sentences in the target language to train a cfd model. for this purpose, we propose a novel training objective that consists of a main task (i.e. predicting whether a given sentence contains a counterfactual statement or not) and an auxiliary task (i.e. predicting whether a given token in a sentence is a clue phrase or not). to the best of our knowledge, we are the first to propose a transfer learning method for cross-lingual cfd, let alone in a zero-shot setting that does not require neither counterfactual clues nor labelled training instances for the target language. using the amazon multilingual counterfactual detection dataset (amcd) @xcite , we evaluate the proposed method for its ability to perform cross-lingual zero-shot transfer. specifically, we use token-embeddings obtained from xlm-r @xcite and mbert foot_0 to train cfd models for german and japanese target languages using counterfact labelled sentences for english source language and automatically extracted clue phrases for each target language. in particular, no human counterfact annotations for the target language are used during training. our proposed method establishes a new state-of-theart for zero-shot cross-lingual transfer with an improvement of 25% in macro-averaged f1 score for german and that of 20% for japanese. the source code implementation for the proposed method will be publicly released upon paper acceptance. ## related work for training and evaluating cfd methods, a dataset was annotated in the semeval-2020 task 5 @xcite covering two subtasks. the first subtask is to classify a given sentence as to whether it expresses a counterfactual statement or not, whereas in the second subtask the participat-ing teams must extract the antecedent and consequent from a given counterfactual statement. our goal in this paper is close to the first subtask, which can be modelled as a sentence-level binary classification problem. most of the high performing methods @xcite @xcite @xcite submitted to semeval-2020 task 5 use state-ofthe-art pretrained language models @xcite @xcite to represent sentences. traditional machine learning methods, such as support vector machines and random forests were also used but with less success @xcite . however, none of these previously proposed methods consider cross-lingual nor zero-shot cfd settings. to achieve the best prediction quality, ensemble strategies are employed. the top performing systems use an ensemble of transformers @xcite @xcite , while others include convolutional neural networks (cnns) with global vectors @xcite embeddings (ojha et al., 2020). various structures are used on top of transformers. for example, lu et al. (2020); ojha et al. (2020) use a cnn as the top layer, while bai and zhou (2020) use a bi-grus and bi-lstms. some other proposed methods use additional modules, such as constituency and dependency parsers in the lower layers of the architectures @xcite . @xcite created the amcd counterfactual dataset by annotating sentences selected from amazon product reviews. unlike the se-meval dataset, which covers only english counterfactuals, amcd covers japanese and german counterfactuals in addition to english. amcd is the only publicly available multilingual cfd dataset. therefore, we use amcd to evaluate the cross-lingual zero-shot cfd models we propose in this paper. @xcite trained cfd models using different approaches such as bagof-words representations of sentences as well as by fine-tuning pre-trained masked language models for each language separately. they also considered a cross-lingual zero-shot setting where they first machine translated the source (english) dataset into each of the target languages (german and japanese), and train cfd models for those target languages using the translated training instances. however, the performance of this approach was significantly worse than that of the inlanguage baselines, which lead to their conclusion "simply applying mt on test data is not an alternative to annotating counterfactual datasets from scratch for a novel target language." this highlights the difficulty of the cross-lingual zero-shot transfer problem setting for cfd, which we consider in this paper. one approach to learn accurate multilingual representations with less supervision in downstream tasks is the few-shot or zero-shot crosslingual transfer learning. here, the goal is to transfer a model trained in the source language into the target language with minimal loss in performance. few-shot transfer learning assumes the availability of a small number of labelled instances in the target language, while the zero-shot setting assumes none. recently, @xcite proposed mad-x, a zero-shot and fewshot language model transfer framework based on the adaptor framework @xcite . moreover, xtreme @xcite , a multilingual benchmark containing many tasks, reported the translate-train performance, where a model is trained on a machine-translated version of the source language dataset into the target languages as a baseline for zero-shot transfer learning. however, to the best of our knowledge, ours is the firstever model proposed for cross-lingual zero-shot transfer for cfd. ## cross-lingual zero-shot cfd let us denote a sentence @xmath0, w 2 , . . . , w |x| consisting of a sequence of |x| tokens w j . cfd is considered as a binary classification task in this paper, where the goal is to predict whether a sentence x contains a counterfactual statement (y(x) = 1) or otherwise (y(x) = 0), indicated by the binary label y(x). in the crosslingual zero-shot cfd setting, we consider the problem of transferring a cfd model trained on a source language s to a different target language t. for this purpose we assume the availability of a counterfactual-labelled dataset, d @xmath1,i , y(x s,i ))} |ds| @xmath2, d @xmath3,i } |dt| @xmath4, for the target language. here, we use the notation x s,i to indicate the i-th sentence in the source (for the target x t,i ) language dataset and its associated counterfactual label y(x s,i ). the source language is assumed to be a language for which it is relatively easier to create a large annotated dataset because it is easier to recruit annotators than for the target language. following prior work on cross-lingual transfer @xcite , we use a machine translation (mt) system to translate the sentences in d s into the target language with the labels unchanged to create a machinetranslated version of d s , denoted by d mt . counterfactual statements are rare in natural language sentences and @xcite report that only 1-2% of sentences contain counterfactual statements in a random collection of sentences. therefore, randomly selecting sentences for annotation purposes results in a waste of annotation resources such as annotator time and cost, and will only result in an imbalanced and low-coverage datasets. to address this issue, prior work @xcite @xcite creating annotated datasets for counterfactuality has used language-specific clue phrases that indicate various expressions frequently used to indicate the presence of a counterfactual to filter candidate sentences for annotation. we use such counterfactual clue phrases as auxiliary training data for cross-lingual transfer. specifically, we require that a cdf model can not only (a) predict whether a given sentence x is a counterfactual or not (main task), but also be able to (b) predict whether a token w in x is a clue phrase or not (auxiliary task). unlike obtaining annotations for counterfactual statements in the target language, it is rela-tively easier to obtain a list of counterfactual clue phrases for the target language. more importantly, as we show later in § 3.1, it is possible to automatically extract an accurate set of target language clue phrases, v t , using d s , d mt and a set of clue phrases for the source language, v s . the auxiliary task is motivated by prior work in semi-supervised learning @xcite and masked language modelling @xcite , where it has been shown that by predicting tokens that are highly related (i.e. clue phrases) to the downstream task (i.e. sentence-level counterfactual detection) we can learn task-specific correlations between tokens. in contrast to the main task, which is modelled as a sentence-level binary classification task, we model the auxiliary task as a token-level sequence labelling task. however, unlike for the main task, where we have at least counterfactual labelled sentences from the source language (i.e. d s ), we do not have any manually annotated training data neither for the source nor for the target languages for the auxiliary task. for this reason, we automatically label training data for the auxiliary task as follows. for the source language, we assign a binary-valued token label y(w j ) for each token w j in each sentence x i in d s , where y(w j ) = 1, if w @xmath5. for example, given a sentence "the bottom fits fine, but i wished there was more ruching like in the photo." we label "wished", corresponding to a clue phrase in english as 1 and other tokens as 0. to generate training data for the target language we can use either sentences in d t or d mt . we empirically compare the different combinations of training data later in §5.1. next, we describe the training objectives associated with the main and auxiliary tasks. let us consider a multilingual masked language model (mlm), h, with pretrained parameters θ that assigns a vector h(w, x; θ) to a word w in a sentence x. we train a feed forward neural network f with parameters ϕ and a sigmoid output unit such that given the embedding x of a sentence x it predicts whether x is counterfactual (i.e. f (x; ϕ) = 1) or otherwise (i.e. f (x; ϕ) = 0). different methods can be used to create sentence embeddings from mlms such as mean or max pooling, attention-based weighting or by simply considering the embedding for the classification (i.e. for the auxiliary task, we train a feed forward neural network, g(h(w, x); ψ), that takes in the contextualised embedding h(w, x) of token w in sentence x and returns 1, if w is a clue phrase or 0 otherwise. we compute the nll loss for the clue phrase prediction task as in (2). finally we add the losses for the main and auxiliary tasks to compute the total loss. our zero-shot transfer model uses d mt on the main task (1) and either of d mt or d t on the auxiliary task (2), i.e. ## experimental settings dataset: we use the amcd dataset, which contains counterfactual statements annotated from amazon product reviews for three languages: english (en), german (de), and japanese (ja). we use the original published training/development/test splits foot_1 in our experiments, for which the number of sentences are shown in to evaluate the level of cross-lingual counterfactual detection (xcfd) performance that can be obtained by directly translating the source language clue phrases to the target language, we create a clue phrase translation (cp translation) baseline. this baseline uses google translate foot_3 to translate individual clue phrases in the source language to the target language without using any contexts for those clue phrases. models and hyperparameters: to obtain token embeddings, we use two multilingual language models in our experiments: on the other hand, xlm-r has been trained on 2.5tb of filtered commoncrawl data containing 100 languages. the initial weights are taken from the bert-base-multilingual-cased and xlm-roberta-large model checkpoints, made available at the huggingface transformers model hub @xcite . we use the adam optimizer (kingma and ba, 2014) with a batch size of 128, an initial learning rate of 0.00001 and train our cfd models for 5 epochs. as the evaluation metric, we report the macro-averaged f1 scores with 95% bootstrap estimated confidence intervals @xcite ). foot_4 5 results ## conclusion we studied zero-shot cross-lingual transfer learning for cfd and proposed a novel training objective that combines (a) token-level clue phrase prediction in target language sentences and (b) sentence-level counterfactuality prediction for source language (and translated to target language) sentences. moreover, we proposed a method to automatically extract clue phrase for a given target language, which obviates the need for manually compiled clue phrases. predicting clue phrases as an auxiliary task improves cross-lingual transfer from english source to german and japanese target languages, obtaining state-of-the-art performances on amcd.