Evaluating of Large Language Models in Relationship Extraction from Unstructured Data: Empirical Study from Holocaust Testimonies (2023)

Abstract: Relationship extraction from unstructured data remains one of the most challenging tasks in the field of Natural Language Processing (NLP). The complexity of relationship extraction arises from the need to comprehend the underlying semantics, syntactic structures, and contextual dependencies within the text. Unstructured data poses challenges with diverse linguistic patterns, implicit relationships, contextual nuances, complicating accurate relationship identification and extraction. The emergence of Large Language Models (LLMs), such as GPT (Generative Pre-trained Transformer), has indeed marked a significant advancement in the field of NLP. In this work, we assess and evaluate the effectiveness of LLMs in relationship extraction in the Holocaust testimonies within the context of the Historical realm. By delving into this domain-specific context, we aim to gain deeper insights into the performance and capabilities of LLMs in accurately capturing and extracting relationships within the Holocaust domain by developing a novel knowledge graph to visualise the relationships of the Holocaust. To the best of our knowledge, there is no existing study which discusses relationship extraction in Holocaust testimonies. The majority of current approaches for Information Extraction (IE) in historic documents are either manual or OCR based. Moreover, in this study, we found that the Subject-Object-Verb extraction using GPT3-based relations produced more meaningful results compared to the Semantic Role labeling-based triple extraction.

## introduction understanding unstructured texts using computational methods is considered a challenging task due to the complexity of the natural language. holocaust testimonies are firsthand ac-counts provided by survivors, witnesses, and others who experienced or observed the atrocities of the holocaust during world war ii @xcite . holocaust testimonies also belong to the category of unstructured texts, which presents unique challenges for computational assessment. these testimonies are often emotionally charged and contain highly-sensitive and personal information which is scattered everywhere in the testimony. moreover, holocaust testimonies contain a range of linguistic complexities, such as archaic language, regional dialects, and highly specialised terminology related to the holocaust, which can be challenging to parse using traditional nlp techniques. extracting relationships from holocaust testimonies is essential for historians as these firsthand accounts provide valuable information about the holocaust. by uncovering hidden connections and associations between entities from the testimonies, historians can gain deeper insights into the historical context, dynamics between individuals and groups, and the broader narrative of the holocaust. this information helps to enhance the understanding of the events, identify patterns, and shed light on the social, political, and cultural aspects of this tragic period in history. however, existing approaches for ie in historic documents are mainly manual (reference), or a few based on advanced digitalised approaches such as ocr @xcite . recently, there have been some efforts made towards ie in historic documents using nlp @xcite . relationship extraction (re) plays a crucial role in discovering meaningful connections and associations between entities from holocaust testimonies to enhance our understanding of the historical context. however, the unstructured nature of these testimonies presents additional challenges when it comes to extracting relationships, making the task even more difficult for humans. dependencies between words and phrases, captured by dependency parsing, provide valuable insights into the syntactic and semantic relationships within the text. there are many downstream applications which are based on extracted relations, such as information retrieval @xcite , question answering (qa) @xcite , and knowledge graph construction @xcite . a knowledge graph is a type of graph database that is designed to systematically organise and present knowledge in a structured format. in the context of unstructured data, knowledge graphs are used to extract and organise the information into a structured format using different cutting-edge nlp techniques. further, graphical representations improve the accuracy and relevance of the search and retrieval results of information from unstructured texts. visualising relationships between entities and events in the holocaust using a graph enables people to identify all of the personal names, places, and locations mentioned in a collection of testimonies. however, with recent advancements in large language models (llms) such as gpt3, there has been significant progress in uncovering hidden relationships within specific content @xcite . these llms, trained on vast amounts of textual data, have demonstrated their capability to learn complex patterns and capture nuanced relationships between entities. llms have introduced a novel paradigm known as in-context learning (icl) @xcite . this paradigm, as exemplified by studies such as @xcite , formulates nlp tasks as language generation problems, allowing the models to make predictions based on demonstrations provided within the context. instead of relying solely on fine-tuning with labelled data, llms leverage the power of language generation to produce outputs such as named entity recognition. the objective of this paper is to examine the performance of llms by employing chatgpt on holocaust testimonies for re. following are the contributions of the proposed paper. the rest of this paper is organised as follows. we critically analyse related work in section 2. we present our methodology in section 3. in section 4, we describe our experiments and report the results and section 5 discusses the next steps of this research. finally, a brief conclusion is provided in section 6. ## related work in this section, we critically analyse existing research in the field of nlp for relationship extraction. we will discuss and establish the context of re within historic documents in particular historical testimonies. previous studies have paid little attention to the computational approaches for information extraction in holocaust testimonies. the valuable information embedded within these testimonies remains largely unexplored, representing a hidden knowledge source within historical data. leeuw, d. et. al shed light on the existing digital infrastructure for holocaust studies and underscored the significant limitations inherent in this domain @xcite . they emphasised the pressing need for computational approaches to effectively address these challenges and overcome the limitations. moreover, some rule-based computational approaches were performed on multi-source holocaust victim reports to extract biographical information @xcite . to date, no study has been conducted specifically focused on identifying the relations and entities present in holocaust testimonies. this research gap highlights the untapped potential for leveraging computational techniques. relationship extraction is a common downstream task that is often performed in conjunction with named entity recognition in various domains, including biomedical @xcite , finance @xcite , and more. the goal of re is to identify and extract meaningful connections or associations between entities mentioned in the text. according to previous studies, several approaches have been considered in identifying relationships. these extracted relationships can then be used to build knowledge graphs @xcite , which serve as representations of the extracted information. the recent advancements in llms have led to their widespread adoption in various nlp tasks, including text classification @xcite . these studies have leveraged gpt models to improve the performance of text classification tasks. furthermore, a recent study @xcite explored the use of llms for relationship extraction. however, their findings indicate that llms reveal lower performance than fully-supervised baselines, such as fine-tuned bert. despite the application of transformer-based models in relationship extraction across various domains, there is currently a lack of annotated datasets specifically tailored for the holocaust domain. this poses a challenge in leveraging the power of these models for extracting relationships from holocaust testimonies and gaining domain-specific insights. addressing this gap by creating annotated datasets tailored to the holocaust domain would greatly contribute to the development of more accurate and contextually relevant relationship extraction models. despite the promising performance of llms in various nlp tasks, the application of in-context learning (icl) for relation extraction (re) still presents challenges. re involves identifying the semantic relationship between two entities mentioned in a sentence, which requires a comprehensive understanding of natural language. recent research by @xcite has explored the application of gpt3 icl for biomedical re and evaluated the complete dataset, suggesting that there is room for improvement in this area for domain-specific contexts. ## methodology in this section, we describe the proposed pipeline employed for creating the knowledge graph. as shown in figure 1 ## results and comparative analysis in this section, we evaluate the results obtained from the methods described above. we adopted the above-described methods to the original transcripts of testimonies and their gptderived relations. after conducting a comparative analysis between gpt3 results and the original testimonies relations, we identified that gpt3 results have less noise and they were properly arranged. furthermore, another finding from this research is that the argument-verb extraction method (method 02) also failed to identify many relations in the context of the holocaust, both in the original data and in the relations generated by the gpt model. this suggests that the argument-verb extraction method may not be suitable for accurately capturing the full range of relations in this specific domain. the primary contribution of this paper is the identification of relations through dependency-based svo extraction, semantic role labelling, and gpt prompts in holocaust testimonial data which describe about the survivor experience. these identified relations are then visualised in a graphical format, providing a clear representation of the relationships within the data. based on our findings, we observed that the relations generated by the gpt3 api and the triplet extraction method based on subject-verb-object were able to provide the most accurate and effective results when identifying relations in holocaust data. currently, our research primarily focuses on identifying relations from individual holocaust testimonies. however, our future plans involve expanding this work to link individual testimonies together by establishing additional relations. this broader network of relations will enable a deeper understanding of the collective experiences, interactions, and events within the holocaust, contributing to a more comprehensive and interconnected understanding of this historical period. moreover, we plan to extend our experiment with name entity recognition combine with re and integrate the results got with the svo extractions as a part of triple integration to construct an nto-n knowledge graph. by integrating the extracted triples from the holocaust testimonies and mapping the predicates to a common schema, we aim to create a comprehensive and interconnected knowledge graph. this graph will capture the relationships, associations, and connections between entities, events, and concepts related to the holocaust. ## conclusion this research, evaluates and compares the performance of traditional rule-based dependency methods for relationship extraction with the recent advancements in llms. through our proposed novel knowledge graph relationships can be visualised better than baseline approaches, hence proving the usefulness of the work specifically for the historians for better synthesis and presentation of the hidden information. this study represents the first-ever investigation into the domainspecific analysis of holocaust text data. it focuses on examining the unique characteristics and challenges presented by this specific domain in the process of relationship extraction.