I Wish I Would Have Loved This One, But I Didn’t – A Multilingual Dataset for Counterfactual Detection in Product Review (2021)

Abstract: Counterfactual statements describe events that did not or cannot take place. We consider the problem of counterfactual detection (CFD) in product reviews. For this purpose, we annotate a multilingual CFD dataset from Amazon product reviews covering counterfactual statements written in English, German, and Japanese languages. The dataset is unique as it contains counterfactuals in multiple languages, covers a new application area of e-commerce reviews, and provides high quality professional annotations. We train CFD models using different text representation methods and classifiers. We find that these models are robust against the selectional biases introduced due to cue phrase-based sentence selection. Moreover, our CFD dataset is compatible with prior datasets and can be merged to learn accurate CFD models. Applying machine translation on English counterfactual examples to create multilingual data performs poorly, demonstrating the language-specificity of this problem, which has been ignored so far.

## introduction counterfactual statements are an essential tool of human thinking and are often found in natural languages. counterfactual statements may be identified as statements of the form -if p was true, then q would be true (i.e. assertions whose antecedent (p) and consequent (q) are known or assumed to be false) @xcite . in other words, a counterfactual statement describes an event that may not, did not, or cannot take place, and the subsequent consequence(s) or alternative(s) did not take place. for example, consider the counterfactual statement -i would have been content with purchasing this iphone, if it came with a warranty!. counterfactual statements can be broken into two parts: a statement about the event (if it came with a warranty), also referred to as the antecedent, and ## related work counterfactuals have been studied in various contexts such as for problem solving @xcite , explainable machine learning (byrne, 2019), advertisement placement @xcite and algorithmic fairness @xcite . @xcite proposed an annotation scheme whereby the original data is augmented in a counterfactual manner to overcome spurious associations that a classifier heavily relies upon, thus failing to perform well on test data distributions that are not identical. unlike @xcite and closely related work by @xcite , we are interested in identifying exist-ing counterfacts and filtering these statements to improve search performance. a cfd task was presented in semeval-2020 challenge @xcite . the provided dataset contains counterfactual statements from news articles. however, the dataset does not cover counterfactuals in e-commerce product reviews, which is our focus in this paper. one of the earliest cfd datasets was annotated by @xcite and covers counterfactual statements extracted from social media. both datasets are labelled for binary classification by crowdsourcing and contain only sentences in english. we will compare our dataset to these previous works in § 3.4. to summarise, our dataset is unique as it contains counterfactuals in multiple languages, covers a new application area of e-commerce reviews, and provides high quality annotations. a range of cfd methods was recently proposed in response to the semeval-2020 challenge @xcite . most of the high performing methods @xcite @xcite @xcite use stateof-the-art pretrained language models @xcite @xcite @xcite . traditional ml methods, such as svm and random forests were also used but with less success @xcite . to achieve the best prediction quality, ensemble strategies are employed. the top performing systems use an ensemble of transformers @xcite @xcite , while others include convolutional neural networks (cnns) with global vectors (glove; pennington et al., 2014) embeddings (ojha et al., 2020). various structures are used on top of transformers. for example, lu et al. (2020); ojha et al. (2020) use a cnn as the top layer, while bai and zhou (2020) use a bi-grus and bi-lstms. some other proposed methods use additional modules, such as constituency and dependency parsers, in the lower layers of the architecture @xcite . cfd datasets tend be highly imbalanced because counterfactual statements are less frequent in natural language texts. prior work has used techniques such as pseudo-labelling @xcite and multi sample dropout @xcite to address the data imbalance and overfitting problems. ## dataset curation we adopt the definition of a counterfactual statement proposed by @xcite where they define it as a statement which looks at how a hypothetical change in past experience could have affected the outcome of that experience. their definition is based on linguistic structures of 6 types of counterfactuals as following. conjunctive normal: the antecedent is followed by the consequent. the antecedent consists of a conditional conjunction followed by a past tense subjunctive verb or past modal verb. the consequent contains a past or present tense modal verb. (example: if everyone got along, it would be more enjoyable.) conjunctive converse: the consequent is followed by the antecedent. the consequent consists of a modal verb and past or present tense verb. the antecedent consists of a conditional conjunction followed by a past tense subjunctive verb or past tense modal. (example: i would be stronger, if i had lifted weights.) modal normal: the antecedent is followed by the consequent. the antecedent consists of a modal verb and past participle verb. the consequent consists of a past/present tense modal verb. (example: we should have gone bowling, that would have been better.) wish/should implied: the antecedent is present, the consequent is implied. the antecedent is the independent clause following 'wish' or 'should'. the consequent is implied and can be paraphrased as "would be better off". (examples: i wish i had been richer. i should have revised my rehearsal lines.) ## evaluations we conduct a series of experiments to systematically evaluate several important factors related to counterfactuality such as (a) selection bias due to clue phrases ( § 4.1), (b) effect of merging multiple counterfactual datasets ( § 4.2), (c) use of machine dataset language size cf % son et al. (2017) english 1637 (2137) 10.1 (31.2) yang et al. (2020a) english 20000 11.0 this work english / german / japanese 10000 (5023) / 7000 / 7000 10.0 (18.9) / 69.1 / 9.5 dataset cf definition domain construction annotation son et al. (2017) janocko et al. (2016) twitter keywords filtering mixed: manual (unknown), automatic pattern matching yang et al. (2020a) janocko et al. (2016) news: finance, politics, healthcare keywords filtering, pattern matching manual (crowdsourcing, strong agreement) this work janocko et al. (2016) amazon reviews keywords filtering manual (curated by linguists) for evaluations in (a), (b), and (c), we fine-tune a widely used multilingual transformer model bert (mbert) @xcite to train a cfd model. the model is pretrained for the tasks of masked language modelling and next sentence prediction for 104 languages 4 and is used with the default parameter settings. the model is implemented using the transformer. 5 library we fine-tune a linear layer on top of these pretrained language models for the cfd task using the training process as described next. 6 we use an 80%-20% train-test data split and tune hyperparameters via 5-fold cross-validation. hyperparameters in the already pretrained transformer models are kept fixed. f1, matthew's correlation coefficient (mcc; @xcite , and accuracy are used as evaluation metrics. mcc (∈ [-1, 1]) accounts for class imbalance and incorporates all correlations within the confusion matrix @xcite . accuracy may be misleading in highly imbalanced datasets because a simple classification of all instances to the majority class has a high accuracy. however, for consistency with prior work, we report all three evaluation metrics in this paper. all the reported results are averaged over at least 3 independently trained models initialised with the same hyperparameter values. for tokenisation, unless the tokeniser is prespecified for the model, we use word tokenize 4 https://huggingface.co/ bert-base-multilingual-uncased 5 https://github.com/huggingface/transformers 6 see supplementary for the details on fine-tuning. from nltk.tokenize.punkt 7 for english and german languages; and mecab 8 as the morphological analyser for japanese. ## conclusion we annotated a multilingual counterfactual dataset using amazon product reviews for english, german and japanese languages. experimental results show that our english dataset is compatible with the previously proposed semeval-2020 task 5 dataset. moreover, the cfd models trained using our dataset are relatively robust against selection bias due to clue phrases. simply applying mt on test data results in poor cross-lingual classifica-tion performance, indicating the need for languagespecific cfd datasets.