Timeline Extraction from Decision Letters Using C hat GPT (2024)

Abstract: Freedom of Information Act (FOIA) legislation grants citizens the right to request information from various levels of the government, and aims to promote the transparency of governmental agencies. However, the processing of these requests is often met with delays, due to the inherent complexity of gathering the required documents. To obtain accurate estimates of the processing times of requests, and to identify bottlenecks in the process, this research proposes a pipeline to automatically extract these timelines from decision letters of Dutch FOIA requests. These decision letters are responses to requests, and contain an overview of the process, including when the request was received, and possible communication between the requester and the relevant agency. The proposed pipeline can extract dates with an accuracy of .94, extract event phrases with a mean ROUGE- L F1 score of .80 and can classify events with a macro F1 score of .79.Out of the 50 decision letters used for testing (each letter containing one timeline), the model correctly classified 10 of the timelines completely correct, with an average of 3.1 mistakes per decision letter.

## introduction timeline extraction is the process of extracting dated events and ordering them along a timeline @xcite . the task can be seen as a variant of event extraction, where the date is the operand of the event, and a type is associated with each date-event pair. our goal is to retrieve all triples of the form (date, event, event class) from a given document using a pipeline consisting of spacy and chatgpt. after the triples have been extracted, we place them along a timeline to create an overview of the decision process, an example of which can be seen in figure 1 . note that each event is grounded in the document, and can be hyperlinked to the exact position in the document, allowing for quick verification. these constructed timelines can have several purposes, such as the graphical summarization of content @xcite , as well as being a part of process mining, where the event classification helps to gain insights in the different parts and their durations in a process. furthermore, timeline extraction over a (dynamic) corpus is a valuable tool in automatic process monitoring. our timelines are machine interpretable overviews of processes, making it easier to control and check them in real-time. thus, timeline extraction also creates valuable metadata about temporal relations and intervals of events and event sequencing @xcite . this study focuses on extracting timelines from decision letters produced by the dutch government in response to a request made under the dutch foia legislation. we used spacy to detect and extract dates from sentences, and chatgpt to extract the event phrases and their classes. out of the 524 triples in the test set, roughly 76% of them were classified correctly, and out of the 50 decision letters, the timelines of 10 of them were extracted perfectly. ## related work the field of timeline extraction has seen quite some interest in recent years, and it was featured as part of the semeval 2010 tempeval and semeval 2015 timeline challenges @xcite , where several aspects, such as the grounding of dates with events as well as the creation of cross-document timelines for entities were addressed. traditionally, the systems used for timeline extraction have consisted of pipeline approaches, with a system containing a part-of-speech tagger, named entity recognition (ner) and coreference resolution modules in succession to extract event phrases @xcite @xcite , and systems such as heideltime @xcite to extract temporal phrases. as the annotation of these timelines can be quite expensive, some works focused on automatically constructing additional data, such as work done by @xcite , who use distant supervision to create timelines for entities and use this additional data to train their pipeline model. a major downside of these pipeline approaches, as discussed by @xcite , is the propagation of errors from individual components in these systems, harming overall performance. the authors propose a method that does not use a pipeline, but instead uses a bert model to extract events by posing the problem as a question answering task and querying the model, which is in some regards similar to our approach using chatgpt. another approach that replaces part of the pipeline with a neural component is work from @xcite , which relies on entity annotations being present, and uses an lstm network to predict the temporal durations of these entities, relative to each other. with the advent of pre-trained large language models (llms) such as @xcite , new event extraction methods have been developed using these models @xcite . these methods are similar to the ones using bert, but instead prompt these large language models to extract (actor, event, event type) triples directly from the input text. although some of the models can be fine-tuned, a pre-trained llm can usually perform quite well on new tasks, especially when using few-show prompting or in-context-learning. this involves providing several examples of the task that has to be performed to the model in the same prompt, helping the model in performing the task. several techniques and bestpractices for in-context learning exist, as surveyed by @xcite . in our paper we experiment with the selection of the in-context examples, and use bm25 to select the top-k most similar datapoints from the trainingset, an approach similar to that used by @xcite . ## discussion although the proposed pipeline achieves good performance on the task of event extraction and classification for decision letters, the fact that it relies on chatgpt, a commercial and closed-source product has certain downsides. although we have tried to mitigate the inherent randomness in the chatgpt model, it is possible that there are minor inconsistencies in performance between runs. a possible direction for future work is the usage of open-source llms such as llama-2 to facilitate the usage of this work in practice, and to alleviate some of the aforementioned problems. the goal of this work was to evaluate a pipeline consisting of spacy and chatgpt, with as little components as possible, to prevent the propagation of errors. although several components could have been implemented by using different models, such as a parsing-based approach for the event extraction, or by using another neural model such as bert, the fact that chatgpt is pre-trained meant that there was very little need for training data, and a small dataset could be used for evaluating the proposed approach. ## conclusion we have shown that a quite accurate timeline extractor for a specific domain can be constructed using a promptable llm like chatgpt, with a very limited number of training examples, in a relatively low-resource language such as dutch, using few-shot prompting and selecting similar examples using bm25. for future work, we could look into fine-tuning an open-source llm such as llama for this specific task, or maybe consider generating training samples for the task using an llm and using these to train another system.