Textual Entailment for Temporal Dependency Graph Parsing (2023)

Abstract: We explore temporal dependency graph (TDG) parsing in the clinical domain. We leverage existing annotations on the THYME dataset to semi-automatically construct a TDG corpus. Then we propose a new natural language inference (NLI) approach to TDG parsing, and evaluate it both on general domain TDGs from wikinews and the newly constructed clinical TDG corpus. We achieve competitive performance on general domain TDGs with a much simpler model than prior work. On the clinical TDGs, our method establishes the first result of TDG parsing on clinical data with 0.79/0.88 micro/macro F1.

## introduction and background temporal information extraction from text is an important part of natural language understanding. many works have framed temporal relation extraction (re) as the task of identifying temporal relations between pairs of events, or an event and a time expression (timex3) @xcite @xcite @xcite . this pairwise framing can make it hard to decide when to annotate a temporal relation, and the resulting timelines are usually fragmented @xcite as not all events or timex3s are linked to each other. heuristics are typically applied to constrain the search space of pairwise relations, both for annotators and machine learning models. for example, many annotation efforts have constrained temporal relations to adjacent sentences: tempeval @xcite @xcite , clinical tempeval, @xcite @xcite , and timebank-dense @xcite . a more principled approach is to model the temporal information in a document as a dependency tree structure @xcite xue, 2018b). this approach was extended by @xcite to temporal dependency graph (tdg) structure for a more comprehensive representation. an example is shown in figure 1 . with this approach, human annotators inspect each entity and find at most 2 reference times. a complete tdg can be constructed from these decisions. the automatic temporal re task then becomes a parsing task: produce a tdg as output given a text as input. tdg datasets have been constructed for news articles @xcite and contracts @xcite . the current state-of-the-art (sota) tdg model @xcite reports 0.77 f1 score in the general domain, and 0.64 f1 on the contract dataset, showing the learnability of the tdg approach in those two domains. in the current work, we make the following contributions: our tdg parser is inspired by works applying nli-based methods to other information extraction tasks, including relation extraction @xcite on the tacred data set @xcite , event argument extraction @xcite on the ace @xcite and wikievents @xcite datasets, and biomedical relation extraction @xcite . ## creating a clinical tdg corpus a temporal dependency graph (tdg) is defined as a 4-tuple (t, e, m, l), where t is a set of timex3s, e is a set of events, m is a set of pre-defined "meta" nodes (e.g. root), and l is a set of edges. the definitions and guidelines 2 of @xcite describe the steps to create a tdg from events and timex3s: (1) for each timex3 t, if t is locatable (i.e., t is not a quan-tifier, duration or set), find its reference time expression (reference timex), otherwise assign no reference timex; (2) for each event e, find its reference timex; and (3) for each event e, find its reference event if there is one. fig. 1 shows examples of such reference decisions, where the reference timex of "friday" is "doctime", the reference timex of "sigmoidoscopy" is "doctime", and the reference event of "sigmoidoscopy" is "sees". we semi-automate this tdg construction process by leveraging the existing annotations over the thyme corpus @xcite . our approach is visualized in fig 2 . first, we take all the events and timex3s in the thyme corpus as the building blocks of the graph. steps, we include as many tlinks (temporal links) from the thyme corpus as possible to maintain the richness and informativeness of the thyme annotations. in some cases, we reverse the tlink label (e.g. <e 1 before e 2 > @xmath0.1). identifying the reference timex for a timex3. tlinks between two timex3s are not annotated in the thyme corpus as the temporal relations between a pair of timex3s can be inferred if their normalized values are available. 3 for a locatable timex3, we use the gold temporal operators annotations from the semantically compositional annotation of time expressions (scate; @xcite to get a timex3-timex3 relation by mapping temporal operators to temporal relations as shown in ## corpus statistics following the clinical tempeval tasks, we use the colon cancer set of the thyme+ corpus and the same data splits as described in @xcite . our statistics on the development (dev) set show that there are on average 155 events per note. since 91.5% of the tlinks occur withinsection, and all the cross-section tlinks are contains-subevent relations, we split each note into sections, drop the cross-section tlinks, and create one tdg per section. with our carefully designed conversion method, we automatically translated most of the pairwise annotations in the thyme+ corpus into tdgs. ## tdg parsing the task of temporal dependency graph parsing is to find the parent node(s) for each child node, where the child node can be an event or a timex3. we cast the tdg parsing task as a textual-entailment (nli) task. given a child node x i and a list of candidate parent nodes {y 1 , y 2 , . . . , y i , . . . , y n }, our model first verbalizes the possible relations @xmath1, y @xmath2.3). for example, the verbalization for the before relation is: x i happened before y i . to get the premise, we concatenate the sentence that contains the child node and the one that contains the parent node. 5 if y i is not the parent of x i , the relation label between x i and y i will be no-edge. we also add descriptions after the premise regarding the distance between two nodes, following previous works @xcite @xcite . 6 then, we run the nli model to obtain the probability the premise entails/contradicts/is neutral to the hypothesis. in the training stage, we finetune a pre-trained nli model on the entailment data generated from the tdg training data. in a tdg, each event can have at most two parents: a reference timex, and a reference event. that is, most of the time, a candidate parent node is not the gold parent for the child, which means the relation between a child node x i and a candidate parent y i is no-edge in most cases. to obtain a relatively balanced training set, we first divide the candidate parent nodes into two sets: a and b, where a contains the gold parent nodes for the current child node, and b contains the rest. then, we randomly sample n e examples from a, and n c examples from b. for each instance sampled from a, we generate an entailment example using the gold label, and randomly sample n c incorrect labels to generate neutral examples. for each instance sampled from b, we generate one contradiction example (see appendix a.3 for an example). in the inference stage, for each candidate parent node y i of a child node x i , we verbalize all possible relations between them and pick the candidate parent with the highest entailment probability as the final parent for x i . ## experimental setup we evaluate our model on the two tdg data sets: the general-tdg and thyme-tdg (clinical-tdg). 7 when sampling the training data, we set n e to 1 and n c to 3. for general-tdg, to generate the reference timex candidates for each timex3 or event, we include all the timex3s in the document; to generate reference event candidates for each event, we include all the events from the beginning of the document to two sentences after the child node. for clinical-tdg, we include candidates in the window of 6 sentences before and 4 sentences after the child node. for both data sets, our candidate parent window setting @xmath3. for the general domain tdg parsing, we finetune the roberta-large-mnli @xcite model via huggingface @xcite for 3 epochs. for the clinical domain tdg parsing, we finetune the pubmedbertmnli-snli-scinli-scitail-mednli-stsb @xcite model for 3 epochs. for model initialization, we experimented with 5 random seeds: {42, 52, 62, 72, 82}. see appendix a.2 for other hyperparameters. we use gold events and timex3s as input for the tdg parsers. @xmath4, relation, @xmath5. on the general-tdg, we report the average f1 scores across all documents (macro-f) following previous practice @xcite @xcite . on the clinical-tdg data set, we report both macro-and micro-f1 scores. 7 the tdg dataset of 100 contracts used in @xcite is not publicly available to the best of our knowledge. dev test nli-based tdg (best) 0.88 (0.79) 0.88 (0.79) nli-based tdg (average) 0.87 (0.79) 0.88 (0.79) ## results and discussion compared to the doctime model @xcite , our nli-based tdg model (best) achieved slightly lower but competitive performance, while being much simpler. the doctime model contains 3 graph neural networks and relies on off-theshelf nlp tools including a co-reference resolution model, a dependency parser, a pre-trained model for sentence embeddings, and a document-level rhetorical structure theory parser. @xcite did not list exact tools or configurations (e.g., what is the model used for coreference resolution?) and the code is not publicly available, so it's very hard to re-implement or apply this model to other data sets currently. we evaluated our nli-based tdg approach on the newly created clinical-tdg data set ( ## conclusion we explore tdg representation and parsing in the clinical domain. we convert the pairwise annotations over the mayo clinic ehr notes in the thyme corpus to tdgs semi-automatically. we then develop a nli-based tdg parser that is much simpler than prior tdg parsers, yet achieves performance competitive with the sota in the general domain. on the clinical tdg data set, our parser also achieves strong performance, which can serve as a baseline for future research on clinical tdg parsing.