Pheonix at S em E val-2020 Task 5: Masking the Labels Lubricates Models for Sequence Labeling (2020)

Abstract: This paper presents the deep-learning model that is submitted to the SemEval-2020 Task 5 competition: “Detecting Counterfactuals”. We participated in both Subtask1 and Subtask2. The model proposed in this paper ranked 2nd in Subtask2 “Detecting antecedent and consequence”. Our model approaches the task as a sequence labeling. The architecture is built on top of BERT, and a multi-head attention layer with label masking is used to benefit from the mutual information between nearby labels. Also, for prediction, a multi-stage algorithm is used in which the model finalize some predictions with higher certainty in each step and use them in the following. Our results show that masking the labels not only is an efficient regularization method but also improves the accuracy of the model compared with other alternatives like CRF. Label masking can be used as a regularization method in sequence labeling. Also, it improves the performance of the model by learning the specific patterns in the target variable.

## introduction counterfactual statements describe events that did not actually happen or cannot happen, as well as the possible consequence. for example, in the sentence "if i was tall just like a giraffe, i was eating the leafs of those charming trees.", the first clause is a counterfactual antecedent and the second clause is the consequent. to model counterfactual semantics and reasoning in natural language, semeval-2020 task 5 @xcite aims to provide a benchmark for two basic problems: (1) detecting counterfactual statements as a binary classification task (2) detecting antecedent and consequent as a sequence labeling task. the introduced model in this paper is designed for the 2nd task. the model approaches the antecedent (ant) and consequent (cons) detection as a sequence labeling. sequence labeling is the assignment of a categorical label to each member of a sequence. common examples of a sequence labeling are part-of-speech tagging and named entity recognition. although, sequence labeling can be treated as a set of independent classification tasks, considering the dependency of choices for nearby elements improves the accuracy to choose the globally best set of labels for the whole sequence. it is specially the case for some sequence labeling tasks like squad @xcite , that the task is to find a set of consecutive tokens from a reading passage that respond to the specific question. likewise, in counterfactual detection task, individual tokens in the same noun phrase, clause, or sentence are likely to get the same labels. in order to benefit from the mutual information between the labels of nearby tokens our model is built on top of bert @xcite . bert benefits from two main features: (1) pretraining: different forms of information about the semantic and syntactic of the language is encoded in the model @xcite , and this let fine-tuning the model on a much smaller dataset than it would be required for a model that is built from the ground up. (2) multi-head attention layers: attention mechanism was first introduced in @xcite to focus on the most pertinent parts of the text for language translation and found applications in a wide range of nlp tasks @xcite . then, multi-head figure 1 : consistency check for some sample outputs attention mechanism @xcite initiated a new epoch in nlp by allowing attention heads to focus on different aspects of the language concurrently. some of these heads correspond well to linguistic notions of syntax, like direct objects of verbs, determiners of nouns, objects of prepositions, and coreference mentions @xcite . although bert generates a vector representation for each token as a function of syntactic and semantic features of all tokens in the text segment @xcite , using a typical linear layer with softmax function on top of bert may not be the best fit for some specific sequence labeling like counterfactual detection. to better understand the possible deficiencies of such models consider a simple case where a model has to predict a sequence of two elements where 'ab' and 'ab' are both plausible outcomes but 'ab' and 'ab' outcomes are highly unlikely. then, in case one element is clearly 'a' or 'a' and the other is uncertain, a reasonable sequence classification architecture like bert can ensure that the information flows to the uncertain element to get aligned to the correct value. however, if both elements are very uncertain around 50% probability, then bert will sometimes generates 'ab' or 'ab' in the output. to address this issue, conditional random fields (crf) were used typically to filter out inconsistent labeling patterns. crf is a standard model for prediction where contextual information or state of the neighbors affect the current prediction @xcite . crf found its application in sequence labeling especially as a complement for lstms @xcite . however, the application of crf became limited after the advent of bert and other transformers. for some tasks adding crf on top of bert shows no improvement and in the original bert paper a softmax was used for sequence labeling. and since then, few works observed improvement using crf on top of bert @xcite . our experiment results show very limited improvement for using crf with bert. as an alternative we tried a multi-head attention layer on top of the bert while part of the labels are revealed to the layer during training, a technique we call "label masking". inspired by the cloze task @xcite masking was used as a preliminary task for pretraining the bert model. the mask language model randomly masks some of the tokens from the input, and then the model has to predict the masked tokens @xcite . in this study, however, we mask part of the labels and then the model has to predict the remaining labels based on the text and the revealed labels. we observe a notable improvement in the results while using label masking. our results suggest label masking as an efficient method for sequence labeling task both for regularization and accuracy improvement for sequence labeling. the contribution in the model architecture are as follows: (1) a multi-head attention layer is added on the top of bert. the input of the layer are the sequence of hidden-states from bert and part of labels, then, the model has to predict the missing labels in the output. (2) for prediction, a multi-stage algorithm is used, in which during an iterative process, the model finalize some predictions with higher certainty, and in the next iteration it can use the knowledge from the finalized labels to predict the remaining labels. the code is available at http://github.com/pbabvey/label_masking . in the following, section 2 describes the model architecture and the experiment results are reported in section 3. 2 model description 2.1 model architecture ## experiment results the model specifications are as follows: the bert base module is borrowed from transformers library @xcite in pytorch. for the multi-head attention layer the hidden size is set to @xmath0, and the number of self-attention heads is @xmath1. in the masking layer, between 30% to 100% of all the tokens are selected at random for masking. the learning rate of multi-head attention layer is set to 2 • 10 -3 to expedite its convergence, while the learning rate of other layers is set to 2 • 10 -5 to maintain the accumulated knowledge in bert during training. the train-validation ratio is set to 90%-10%. ## conclusion in this paper we introduced a deep-learning model to solve the counterfactual detection task. the model approaches the task as a sequence labeling. we used label masking with a multi-head attention layer on top of the bert to exploit the mutual information between nearby labels. moreover, we used a multi-stage procedure for label prediction in which the model learns to use the knowledge from predicted labels with high certainty in labeling the remaining tokens. our model achieved considerable results in semeval-2020 task 5 @xcite . our results suggest label masking plus a multi-head attention layer as an efficient method both for regularization and accuracy improvement. the improvement with label masking was higher than the crf, however, further studies is needed to evaluate the advantage of the introduced method in other sequence labeling tasks.