Exploring Text Representations for Generative Temporal Relation Extraction (2022)

Abstract: Sequence-to-sequence models are appealing because they allow both encoder and decoder to be shared across many tasks by formulating those tasks as text-to-text problems. Despite recently reported successes of such models, we find that engineering input/output representations for such text-to-text models is challenging. On the Clinical TempEval 2016 relation extraction task, the most natural choice of output representations, where relations are spelled out in simple predicate logic statements, did not lead to good performance. We explore a variety of input/output representations, with the most successful prompting one event at a time, and achieving results competitive with standard pairwise temporal relation extraction systems.

## introduction extracting temporal information from texts is critical in the medical domain for prognostication models, studying disease progression, and understanding longitudinal effects of medications and treatments. the standard route for extracting temporal information is by casting it as a relation task between time expressions and medical events. this relation extraction task is approached by forming relation candidates by pairing potential relation arguments and training a classifier to determine whether a relation exists between them. this pairwise approach is taken by a state-of-the-art temporal relation extraction system @xcite , which uses a pretrained language model such as bert @xcite for representing the training examples. the goal of this paper is to investigate a generative approach to relation extraction as an alternative to the traditional pairwise method. we investigate whether it is possible for a sequence-to-sequence (seq2seq) model such as t5 @xcite , bart @xcite to ingest a chunk of clinical text, often containing multiple sentences, and generate humanreadable output containing all relation instances in the input. this goal proved to be more ambitious than we anticipated, but ultimately we succeeded in designing input/output representations that were competitive with state-of-the-art. using generative models for relation extraction has received little attention and no work exists on using these models for temporal relation extraction. @xcite use natural language to encode sentence-level relations but mapping the output text to the input arguments is not trivial and requires an alignment algorithm. @xcite formulate relation extraction as a template generation problem but their approach requires a complex cross-attention guided copy mechanism. we explore sentence-as well as cross-sentence relations and encode relations in a structured and humanreadable form in which the relation arguments can be easily mapped to the reference entities in the input. in our experiments, we use semeval-2016 task 12: clinical tempeval data @xcite , which annotated time expressions, events, and temporal relations, specifically the contains relation that links times and events to their narrative containers @xcite . for example, in ## results and discussion only one input/output variant was competitive with baseline systems: the 1-container variant ( bart performed worse than the other seq2seq models across all input/output variants although its performance could potentially be improved by a much more extensive hyperparameter search. we leave an exploration into why its "out-of-the-box" performance was inferior for future work. ## conclusion engineering input/output representations for seq2seq models proved difficult as obvious choices of output representations, such as explicit relations encoded as predicate logic statements led to poor performance. by exploring alternative input/output representations, we were able to improve performance. our 1-container input/output variant with a t5 model was competitive with or better than the current state-of-the-art without requiring additional training data. this is likely due to several factors. first, predicting one relation at a time allowed the model to mitigate the limitation on the maximum length of the output and capture longdistance relations, which was more challenging for the other variants. second, it required generating only a single word, which is more like the text generation tasks the seq2seq models were trained on than generating predicate logic expressions like the other variants required. future research may want to explore different pretraining objectives for seq2seq models that would be more appropriate when downstream tasks require generating structured output.