Temporal Relation Classification using B oolean Question Answering (2023)

Abstract: Classifying temporal relations between a pair of events is crucial to natural language understanding and a well-known natural language processing task. Given a document and two event mentions, the task is aimed at finding which one started first. We propose an efficient approach for temporal relation classification (TRC) using a boolean question answering (QA) model which we fine-tune on questions that we carefully design based on the TRC annotation guidelines, thereby mimicking the way human annotators approach the task. Our new QA-based TRC model outperforms previous state-of-the-art results by 2.4%.

## introduction events in stories are not necessarily mentioned in a chronological order. the timeline of events is important for understanding the main narrative of a story as well as the correct order of actions. for example, the timeline may be used directly by clinicians looking for a convenient way to explore the disease course of their patients, or by algorithms to follow instructions in the right order, given as text, such as in cooking recipes. building the timeline is done based on two main subtasks: (1) event extraction, that is, detecting the most important events in a given textual input, and (2) temporal relation classification (trc), also known as temporal relation extraction, which is about putting two events, given as gold spans, in the right chronological order. for example, consider the following text: "before you put the cake in the oven, say a little prayer." in the first subtask, known as event extraction, we would like to detect only the relevant events for our domain of interest. in this case, the words put and say are both verbs representing some relevant actions; therefore, we mark them as events. in the second subtask, trc, we put every two events in a chronological order by classifying them using a closed set of temporal relations. in this case, the two events put and say should be assigned with the label after indicating that put is happening after say in a chronological order. in this study we focus on trc, which is typically handled as a classification problem of two events provided along with the context in which they are mentioned. matres @xcite is one of the dominant datasets for trc comprised of news documents manually annotated with temporal relation labels. the events are deterministically chosen to be all actions (mostly verbs) mentioned in the documents. every pair of events (n, m) are manually labeled with one of four labels: before (n happened before m), after (n happened after m), equal (n and m happened at the same time), and vague (it is impossible to know which event happened before the other). traditional classification approaches have already been demonstrated for trc. in this work, we get inspiration from a relatively new promising approach for solving natural language processing (nlp) tasks, in which the target algorithm is based on a reduction of the task to another problem. in our case, we solve the trc problem using a model that handles the boolean question-answering (qa) task, which is about answering a yes/no question given a passage used as a context. we decide to use boolean qa as our proxy problem due to the way the annotation work for building matres has been done. in the main annotation guidelines of matres @xcite , the annotators are asked to assign a label to a pair of events (n, m) by answering the two following questions: (1) is it possible that the start time of n is before the start time of m? and (2) is it possible that the start time of m is before the start time of n? there are four possible answer combinations, each is mapped to one label: (yes, no) ⇒ before, (no, yes) ⇒ after, (no, no) ⇒ equal, and (yes, yes) ⇒ vague. therefore, we transform an instance of trc, composed of a pair of events and a document, into a pair of yes/no qa instances, one for each of the two questions, and then fine-tune a yes/no qa model to answer them. the final prediction is made based on the combination of the yes/no answers retrieved by the qa model. ## related work trc has received increasing levels of attention in the past decade. there is a relatively long list of related shared tasks @xcite @xcite . modern approaches for trc use some sort of a neural network as a classifier. for example, @xcite showed that a neural network that uses only words as input, performs better than the traditional models that process features which were manually created. a more modern approach for trc is based on large pre-trained language models. @xcite continued to pre-train a language model before fine-tuning it on trc; @xcite incorporated a global inference mechanism to tackle the problem at the document level; @xcite combined a recurrent neural network (rnn) over bert @xcite embedding and a structured support vector machine (ssvm) classifier to make joint predictions; @xcite integrated bert with a temporal commonsense knowledge base, and improved accuracy significantly by 10% over the previously known best result; and @xcite developed a multitask model for the two related subtasks, event extraction and trc. @xcite train a gated relational graph convolution network using rhetorical discourse features and temporal arguments from semantic role labels, in addition to some traditional syntactic features. @xcite use a unified form of the document creation time to improve modeling and classification performance, and @xcite improve the faithfulness of trc extraction model. @xcite built a syntactic graph constructed from one or two continuous sentences and combined it with a pre-trained language model. the best result so far has been reported recently by @xcite , who extract relational syntactic and semantic structures, and encode them using a graph neural network. in another recent work @xcite , the authors introduce a novel method to better model long document-level contexts by detecting and encoding important sentences in the document. none of those studies use qa to address the trc problem. our boolean qa-based approach continues to improve on @xcite work, achieving a new stat-of-the-art result for trc. ## datasets we conduct experiments with two datasets. ma-tres @xcite ) is a composition of three datasets (timebank, aquaint and plat-inum) which were re-annotated following new guidelines. following previous work, we use timebank and aquaint together as a training set and platinum as a testing set. for validation and development we use a different dataset named tcr @xcite , which has been used similarly in other works @xcite . as mentioned above, matres has four labels: before, after, equal, and vague. timebank-dense @xcite , or tb-dense in short, is the second dataset which we use in this work. tb-dense has two additional labels: includes and is-included. following common practices, we evaluate our models using the relaxed micro-average f1 score (i.e., for ma-tres ignoring all mistakes on vague instances during evaluation, and for tb-dense completely removing vague instances from the validation and testing sets). overall, matres contains 12, 736 training instances, 837 testing instances, and 2, 600 validation instances from trc. tb-dense contains 4, 032 training instances, 1, 427 testing instances, and 629 validation instances. the label distributions is summarized under appendix b. ## methodology we design our problem as yes/no question answering problem. therefore, we fine-tune a pre-trained language model (plm) by taking a yes/no qa classification approach for which every instance is composed of a passage (text) and a question, provided along with a yes/no answer. our qa model is designed as a traditional classifier; the input is a concatenation of the passage and the question with a special separator token in between, and the output is a two-way label distribution vector. we use roberta @xcite , which comes in two sizes, base and large; we use both. an instance of trc is composed of a document, two event spans, and a label. in order to use our qa model for trc, we convert each such instance into two or three yes/no qa instances, which we use for fine-tuning and testing. each qa instance input trc instance document: before you put the cake in the oven, say a little prayer. event 1: put event 2: say convert to a yes/no qa instances yes/no qa instance passage: before you @put@ the cake in the oven, @say@ a little prayer. question: is it possible that @put@ started before @say@? yes/no qa instance passage: before you @put@ the cake in the oven, @say@ a little prayer. question: is it possible that @say@ started before @put@? yes/no qa model (w/ or w/o fine-tuning for qa) no yes convert to trc output after trc figure 1: using yes/no qa for trc. the input is a trc instance from matres that we convert into two yes/no qa instances, which we process with the qa model. the results are converted back into a trc label using mapping rules (see the text and each qa instance is processed independently during fine-tuning. at inference time we run the instances through the model and assign a trc label based on the answers. naturally, a document may contain more events than the two relevant ones. therefore, we use markers @xcite in order to mark the two relevant events. specifically, each relevant event is surrounded by the '@' character in both, the passage and the question. figure 1 demonstrates how we process a matres instance. ## experiments and results we fine-tune our models for the duration of five epochs and evaluate them on the validation set every epoch; we use the best checkpoint as the output model. we run every experiment three times using different seeds and report on the averaged accuracy and standard deviation on the testing set. 2 the ma-tres model with the best question variation (qv3) has been further processed with two additional procedures: perturbation and fine-tuning with boolq. perturbation. to achieve better model generalization, we perturb the instances of the training baseline algorithms. to assess the contribution of our yes/no qa design, we define two baseline algorithms. the first baseline is a traditional multiclass qa model, which is given with the same passage as in our original yes/no qa model, paired with only one question that takes one of the labels as an answer. we experiment with two question variations: qv1: what is the chronological order of the two marked events: <event 1> @xmath3: @xmath4, after or at the same time @xmath5 the second baseline is a simple multiclass sentence-classification roberta model, which receives as input for this model comprises only the passage, and the output is one of the labels from the dataset. as seen in ## conclusions we proposed a novel approach for trc using a pretrained language model fine-tuned for a yes/no qa classification task. our model was fine-tuned to answer questions which were originally designed to support decision making during the annotation process. we believe we have demonstrated the potential of this method to leverage the yes/no qa design to break down the prediction process into a set of yes/no questions; our approach outperforms existing methods, achieving a new state-of-the-art result for trc on two datasets. there is a potential practical limitation to this work, which is related to time complexity and speed performance. since every instance is transformed into multiple qa instances, it may take a relatively long time to process a document.