Discourse Information for Document-Level Temporal Dependency Parsing (2023)

Abstract: In this study, we examine the benefits of incorporating discourse information into document-level temporal dependency parsing. Specifically, we evaluate the effectiveness of integrating both high-level discourse profiling information, which describes the discourse function of sentences, and surface-level sentence position information into temporal dependency graph (TDG) parsing. Unexpectedly, our results suggest that simple sentence position information, particularly when encoded using our novel sentence-position embedding method, performs the best, perhaps because it does not rely on noisy model-generated feature inputs. Our proposed system surpasses the current state-of-the-art TDG parsing systems in performance. Furthermore, we aim to broaden the discussion on the relationship between temporal dependency parsing and discourse analysis, given the substantial similarities shared between the two tasks. We argue that discourse analysis results should not be merely regarded as an additional input feature for temporal dependency parsing. Instead, adopting advanced discourse analysis techniques and research insights can lead to more effective and comprehensive approaches to temporal information extraction tasks.

## introduction temporal information extraction (tie) is the task of modelling the relative and/or absolute temporal relations between all the temporal nodes in an article. a temporal node can be either an event or a time expression (timex). tie is a core component task of text comprehension. despite its importance, tie remains one of the lowest performing natural language understanding tasks. it is a difficult task, and the challenge is further compounded when expanding it to the document level, as the number of temporal relations scales quadratically with the number of temporal nodes, and the requi-site amount of reasoning must incorporate longer spans of text. to address these challenges, @xcite ; @xcite ; @xcite proposed the use of temporal dependency structures to represent the overall temporal relational structure within an article. this approach is based on the phenomenon of temporal anaphora, where the interpretation of the occurring time of one temporal node depends on knowing the occurring time of another temporal node. by modelling these temporal dependency relations, the overall temporal structure of an article can be obtained without the need for exhaustively labelling every pair of temporal nodes. as a result, temporal dependency parsing not only models the temporal relations between events but also captures narrative and discourse structure. there are striking similarities between temporal dependency structures and the constituency discourse tree structures @xcite used for discourse parsing in the context of rhetorical structure theory (rst; @xcite , and not just in their use of trees or graphs. more importantly, temporal dependency relations can be viewed as a specific type of anaphoric relation that discourse analysis models attempt to capture. this observation suggests a potential connection between dependency parsing and discourse analysis, warranting further investigation into their relationship and potential synergies. this connection between document-level temporal structure and discourse structure was corroborated by choubey and huang (2022), who discovered that incorporating discourse profiling (dp) information, specifically the functional role of each sentence, could enhance the overall performance of temporal dependency graph parsing (tdg; @xcite . their evaluation may not have been sufficiently comprehensive, however. tdg parsing encompasses three distinct types of relation parsing: timex to timex (t2t), event to timex (e2t), and event to event (e2e), each requiring a different prediction mechanism. upon a more detailed reexamination of choubey and huang's (2022) findings, dp information in fact does not consistently improve performance across all three relation types; it reliably enhances e2e, but may lead to a decline in performance for the other two. we believe this is caused by two major limitations of choubey and huang's (2022) approach. first, dp is a hard problem in its own right. the state-of-the-art dp system (choubey and huang, 2021) only yields a 59.21% f1 performance. this means tie systems following choubey and huang's (2022) guidance will only have access to noisy and inaccurate dp features. second, sentence function is a relatively high-level, descriptive type of discourse structure. temporal dependency structure, on the other hand, can also benefit from a lot of simple surface-level discourse information, such as precedence @xcite . to address these issues, we have experimented with incorporating surface-level sentence-position information into a tie system, and in two ways: encoding absolute sentence-position by appending the sentence number directly onto the context sentences, following choubey and huang (2022), and proposing a novel sentence position embedding (spe) using a sinusoid. our experiments demonstrate that spe could significantly enhance temporal dependency graph parsing performance across all relation types, with the performance increase being mostly greater or at least comparable to that provided by dp information. the resulting tdg parsing system foot_0 with spe obtains the state-of-the art performance. figure 1 : an example of a tdg from @xcite . in the example text (upper), event triggers are highlighted in green and timexes are highlighted in orange. in the tdg (lower), different types of dependency relations have different edge colours (t2t, e2t and e2e). each arrow points from the parent node to the child node. this pairwise annotation scheme, however, fails to generalize to the document level. the number of temporal node pairs is quadratic in the number of temporal nodes (n), i.e., n 2 ∈ o(n 2 ). @xcite pointed out that this quadratic increase, together with the increase in the complexity and number of vague relation links for annotators to consider will, in practice, inevitably cause errors to annotation. to address this issue, @xcite ; zhang and xue (2018b); @xcite have advocated for using dependency structures to represent document-level temporal relations. @xcite annotated a children's story with temporal dependency trees. each event u only depends on one other event v iff the interpretation of when u occurred requires knowing when v occurred. @xcite temporal dependency tree structure only includes events, but this standard may yield disconnected structures. zhang and xue (2018b) refined temporal dependency tree structure to allow the inclusion of timex vertices as well as two special vertices: a document creation time (dct) vertex and a root vertex. the inclusion of timex vertices allows for capturing the missing events in timex (e2t) temporal dependencies and timex to timex (t2t) temporal dependencies. the addition of the dct and root vertices ensures each document is always parsed into a valid tdt. both @xcite and zhang and xue (2018b) assumed that each event or timex had exactly one reference temporal node (to which the dependency edge points), resulting in a tree structure. @xcite , on the other hand, argued that this assumption is overly stringent, and that it is possible for an event to have both a reference timex and an reference event. they therefore proposed to characterise temporal structure with temporal dependency graphs (tdg), in which each event can have a timex parent, an event parent, or both. as depicted in figure 1 , the event alerted depends on both the timex friday and the event fall. as a result, tdg is more expressive than the earlier tdts. in this work, we used the tdg corpus released by @xcite . ## discourse analysis for tdg parsing based on dijk's (1986) schemata of news content, choubey et al. (2020) proposed the task of discourse profiling (dp). the task is to classify each sentence into one of eight content types (see appendix b). there are two ways of encoding dp information, as proposed by @xcite . the first (dp feature) appends the content type label directly, marked with a special token #. for instance, if the sentence represents a main event, the label #m1# is appended to the sentence. we obtained the same model-generated content type labels from choubey and huang (2022). the second (dp distillation) involves using model distillation. in this approach, the model is equipped with two decoders: one predicts the reference score, while the other performs dp classification. the training of both tasks occurs simultaneously, distilling the dp information into the underlying language model. ## discussion before choubey and huang (2022), the relationship between discourse and tie had not been explored, and indeed our own experiments corroborate the value of their insight to incorporate discourse information into constructing documentlevel temporal structures. merely using the output of a discourse system as an additional input feature for document-level tie may not be the most effective strategy, however. a very superficial, but novel sentence position embedding effectively encodes surface-level sentence-order information, and seems to be more reliable as a proxy for the discourse structure of news articles. incorporating this information leads to state-of-the-art performance in tdg parsing. the success of sentence-position embedding offers a significant opportunity to bridge discourse analysis and document-level temporal dependency parsing. it suggests that we should not naïvely rely on discourse information as a separate, modular input source. instead, the similarities between the two tasks indicate that various techniques and insights can be transferred and applied across both domains, leading to more effective models and a deeper understanding of the relationship between discourse analysis and temporal dependency parsing. owing to resource constraints, our experiments were conducted using only one type of language model, roberta-base. however, other models such as bert @xcite , deberta @xcite , and ernie @xcite have demonstrated impressive performance across various natural language understanding benchmarks. we aim to evaluate these models in future research, and we encourage other researchers to reproduce our work using these alternative models.