Set-Aligning Framework for Auto-Regressive Event Temporal Graph Generation (2024)

Abstract: Event temporal graphs have been shown as convenient and effective representations of complex temporal relations between events in text. Recent studies, which employ pre-trained language models to auto-regressively generate linearised graphs for constructing event temporal graphs, have shown promising results. However, these methods have often led to suboptimal graph generation as the linearised graphs exhibit set characteristics which are instead treated sequentially by language models. This discrepancy stems from the conventional text generation objectives, leading to erroneous penalisation of correct predictions caused by the misalignment of elements in target sequences. To address these challenges, we reframe the task as a conditional set generation problem, proposing a Set-aligning Framework tailored for the effective utilisation of Large Language Models (LLMs). The framework incorporates data augmentations and set-property regularisations designed to alleviate text generation loss penalties associated with the linearised graph edge sequences, thus encouraging the generation of more relation edges. Experimental results show that our framework surpasses existing baselines for event temporal graph generation. Furthermore, under zero-shot settings, the structural knowledge introduced through our framework notably improves model generalisation, particularly when the training examples available are limited.

## introduction understanding the temporal relation between events mentioned in long documents is crucial to modelling complex text with articulated narratives. one of the widely adopted benchmarks for event temporal relation understanding is the semeval 2013 @xcite , requiring end-to-end generation of event temporal graphs directly from raw text. an event temporal graph is a natural representation of temporal information, with the nodes representing events and the edges the temporal relationships between them, such as "before", "after", or "simultaneous". existing studies typically approach the problem of constructing event temporal graphs through a two-step pipeline, with the first step focusing on detecting events in text, and the second step on classifying the temporal relations between them @xcite . however, such pipeline-based approaches suffer from wellknown limitations, including (i) the need for finegrained annotations at each step; and (ii) the potential for error propagation throughout the pipeline. in the first step, the event extractor aims at locating as many event triggers as possible in the given documents, leading to the inclusion of numerous trivial events that often lack relevance to the narrative and have no relation with other events. as a result, the next step for temporal relational extraction becomes burdened with many noisy events, significantly impacting the overall accuracy and efficiency of the models. to address these limitations, @xcite introduced a reformulation of the task by generating event temporal graphs directly through conditional text generation. this approach allows for the use of pre-trained language models and, more importantly, overcomes the typical limitations associated with the pipeline architecture. while this method involved fine-tuning a text generation model, such as gpt-2, for the generation of linearised event temporal graphs as sequences, it fails to consider an important aspect. specifically, it does not account for the fact that the target sequence (i.e. the list of event temporal relations) is order-invariant, and should therefore be treated as a set rather than as an ordered sequence. for example, the following two sequences represent the same temporal graph: s1: [(cuomo leaving his office, before, speak to reporters), in this scenario, the conventional text generation loss will (mistakenly) yield a high value because most of the tokens in the corresponding positions do not match, even though the event relations are the same. this issue has a detrimental effect on the model performance for several reasons. first, it discourages the language model from generating additional edges. generating more edges implies a greater number of potential permutations in the edge sets, making it less likely to match the target. secondly, if the initially generated edge in the sequence differs in token count from the one in the target, it causes all subsequent edges to misalign with the target, even if they are identical, leading to a high loss value. in this work, we propose a set-aligning framework (saf) that enables efficient employment of llms for auto-regressive event temporal graph generation. saf incorporates a group of novel regularisations, named set property regularisations (spr), along with augmented data, which aims at tackling the problems associated with the use of lm loss in contextualised graph generation by mitigating its penalisation towards the target sequences. for example, the s1 and s2 above are different sequences of the same edge set. even if s1 has the same order as the target edge sequence and thus has a lower lm loss than s2, both of them will be added with the same spr. therefore, the relative difference of their loss values becomes smaller, which avoids overfitting the model towards the specific edge order of s1. moreover, if the model explores generating one more edge after s2 and the edge is correct, the spr value will decrease while the lm loss will probably increase. using the proposed saf, we fine-tune language models from the t5 @xcite family with weak supervision. additionally, we introduce the first human-annotated dataset for contextualised event temporal graph generation built on the new york times, which we combine with existing event relation extraction datasets to evaluate the effectiveness of the saf framework. experiments on the newly annotated new york times corpus foot_1 show that saf significantly increases the number of generated edges, resulting in improved recall. further-more, we assess the performance of our approach on existing sentence-level event temporal relation extraction datasets, namely matres @xcite and tb-dense @xcite , under zero-shot settings, and we find that the structural knowledge introduced through the proposed saf has an even greater impact on model generalisation when the training examples available are limited. our contributions are three-folded: ## set-aligning framework madaan and yang (2021) first explored the possibility of end-to-end event temporal graph generation using neural language modelling. since then, however, this task has remained under-explored, with numerous unresolved issues. to elaborate, the first concern is that @xcite framed graph generation as a conventional sequence generation problem, whereas it is fundamentally a set generation problem. secondly, the dataset they built primarily consists of small-sized graphs, failing to challenge the model in terms of documentlevel understanding. lastly, their investigation mainly centred on gpt-2, while the landscape of llms has evolved with the emergence of models featuring distinct structures (e.g., encoder-decoder) and new paradigms (e.g., in-context learning) in recent years. in this study, we address these three aspects to enhance the understanding of sequenceto-sequence temporal graph generation. although our proposed framework is designed to be model-independent, several factors have led us to choose flan-t5 as the base model for our experiments: (i) based on our preliminary experiments, flan-t5-base hits the sweet spot in terms of performance vs. resource consumption, allowing us to test more variants; (ii) its encoder-decoder structure is well-suited to document-level graph generation, due to its efficiency in processing comprehensive information in lengthy documents. ## conclusion this study proposes a framework for fine-tuning language models to generate event temporal graphs directly from raw documents in an end-to-end manner. the proposed framework includes a data augmentation method and set property regularisations to mitigate the problem caused by conventional generation loss, promoting the generation of more edges by language models and, consequently, leading to improved performance. extensive experiments show the effectiveness of our proposed model on multiple widely used datasets with realworld articles. the thorough analysis demonstrates that our framework can encourage language models to generate more edges for constructing event temporal graphs in various settings.