Fine-tuning Language Models for Triple Extraction with Data Augmentation (2024)

Abstract: Advanced language models with impressive capabilities to process textual information can more effectively extract high-quality triples, which are the building blocks of knowledge graphs. Our work examines language models’ abilities to extract entities and the relationships between them. We use a diverse data augmentation process to fine-tune large language models to extract triples from the text. Fine-tuning is performed using a mix of trainers from HuggingFace and five public datasets, such as different variations of the WebNLG, SKE, DocRed, FewRel, and KELM. Evaluation involves comparing model outputs with test-set triples based on several criteria, such as type, partial, exact, and strict accuracy.The obtained results outperform ChatGPT and even match or exceed the performance of GPT-4.

## introduction knowledge graphs (kgs) represent knowledge in a semantically rich and intuitive way, enabling one to better understand and utilize gathered information. a kg is a data structure representing real-world entities and the relationships between them in the format of a triple, e.g., ⟨head entity, relation, tail entity ⟩ or ⟨ subject, predicate, object ⟩ @xcite . the majority of available knowledge is composed of unstructured textual data. the need to 'convert it' into a structured format via extracting entities and relationships between them drives the construction of kgs. large language models, like chatgpt or gpt-4, have a remarkable capacity for understanding and generating text. it makes them useful tools for automating the process of knowledge extraction from textual sources. they can capture nuances and complexities of language, allowing for a deeper comprehension of the text's meaning. therefore, they can be employed to create kgs that accurately and fully capture complicated semantic relations and the meaning of texts. extracting triples from texts poses several challenges @xcite . finding accurate and comprehensive entities and representative relationships from the text can be difficult, especially with various language usage, implicit references, and context-dependent interpretations. additionally, processing and analyzing enormous quantities of text can be computationally demanding and resource-intensive. therefore, methods for capturing reliable contextual information are paramount for kg's growth and development. advanced context-aware techniques must be developed to identify and separate contextual references, capture relationships, and identify implicit connections. this work aims to tune large language models (llms) to perform triple extraction from text. we have conducted several experiments using various models and datasets of different quality and sizes. the construction of triples adhering to the dbpedia ontology format has been particularly interesting. the webnlg dataset @xcite , predominantly using the dbpedia vocabulary for its entities and properties or emulating its ontological style, serves as the basis for our training data. we have introduced a set of procedures to generate various prompts, instructing models about different processes related to triple extraction and understanding. this has led to the augmentation of the original webnlg data and the creation of various versions of training datasets. eleven models, each with seven billion parameters, have been trained. their efficacy has been evaluated in comparison with gpt-3.5 and gpt-4 on webnlg. additionally, we have preliminary assessed larger models with thirteen, thirty, and thirty-three billion parameters and trained them similarly. the ultimate objective is to propose and illustrate a training methodology capable of elevating domain-specific models to or beyond the proficiency of leading-edge models. the findings of the work that constitute our contributions are: ## related work in the field of triple extraction, lstm is a conventional technique to explore. seq2rdf @xcite employs an lstm-based sequenceto-sequence model to map natural language text to rdf triples in one step, using pre-trained word and knowledge graph embeddings for initialization. however, it is limited to extracting single triples and cannot handle multi-triple extraction. the chatie framework @xcite achieves zeroshot information extraction by promoting chat-gpt, without requiring any labeled data for training. it allows interactively querying the model to extract structured information piece by piece in a multi-turn conversational format. the chatie relies on llm like chatgpt which is not open source. the performance depends heavily on how well the prompts are engineered and provides many details. the head to tail benchmark @xcite ) provides a systematic way to evaluate how knowledgeable llm are about facts in diverse domains(movies, books, academics). the benchmark is still limited in size and diversity compared to the vast world knowledge, 18k qa pairs may not comprehensively cover all entity types, relationships, and knowledge domains. few-shot learning with gpt-3 @xcite achieves state-ofthe-art performance on standard relation extraction datasets, surpassing existing fully supervised models. fine-tuning flan-t5 on explanations generated by gpt-3 further enhances performance. treating relation extraction as a text-generation task provides flexibility in expressing entities and relations. however, gpt-3 is opaque, not open source, and significantly costly. ## problem and experimental setup the paper focuses on extracting information from plain text. it is the task of building triples of the form ⟨subject, predicate, object⟩ based on the content of a sentence. triple extraction is a domainindependent task. two entities of a triple, i.e., subject and object, appear in the text, while a relation between these two entities is often deduced by 'understanding the meaning' of the sentence. all the components of a triple are extracted at the same time. here is a more formal description of the task. given a set of sentences d := {w 1 , w 2 , ..., w n }, we want to obtain a set of facts built from and based on these sentences. let this set be f acts := {f act 1 , f act 2 , ..., f act n }, and each fact is denoted as ⟨s, p, o⟩, @xmath0, @xmath1, @xmath2, where s, p, o are sets of subjects, predicates, and objects respectively. these triples are the basic units of knowledge graphs, resulting from the development of the semantic web concept. the classes (types of entities) and properties (relationships and attributes) used to describe triples' components are defined using ontologies. one of the most well-known ontologies is the one used by dbpedia @xcite . within the dbpedia dataset, triples are generated and represented using the dbpedia ontology as the schema. this ontology consists of 320 classes organized into a subsumption hierarchy and 1650 dis- ## discussion and limitations the obtained results and their analysis have led to a few observations that confirmed known facts about tuning large models and allowed to draw some new ones. we can categorize them into three parts: data size, model selection, and interaction with a model (prompt and data preparation). size and quality of datasets. it is a well-known fact that larger datasets lead to better results. such an obvious statement is also true for the triple extraction process. it is seen in once we collected results for the other two datasets -docred and fewrel -we investigated the content of the training datasets. it has become apparent that the reference triples that were supposed to be constructed from sentences were of poor quality: limited to a few relations, incoherent structure, a limited number of triples (quite often just one) form small paragraphs. ## conclusion the paper aims to investigate different scenarios of a triple extraction task. various models and a few datasets have been used in the experiments. a prime contribution is the development of a procedure/methodology for augmenting the original dataset. the additions included several tasks indirectly related to the triple extraction process: explaining the extraction steps, reconstructing sentences from triples, and determining the correctness of extracted triples. it resulted in enlarged training datasets (3-or 4-fold). as an outcome, the performance of 7b tuned models is comparable to or even better than that of well-known models from the gpt family. the applied procedures concentrated on generating triples containing elements compatible with a specific vocabulary, in our case, dbpedia. while our models suffer from occasional looping and hallucinations, they effectively extract triples following dbpedia ontology from sentences. the results demonstrate that achieving and exceeding gpt performance with fine-tuned models is possible without large datasets.