AREE j: A rabic Relation Extraction with Evidence (2024)

Abstract: Relational entity extraction is key in building knowledge graphs. A relational entity has a source, a tail and a type. In this paper, we consider Arabic text and introduce evidence enrichment which intuitively informs models for better predictions. Relational evidence is an expression in the text that explains how sources and targets relate. This paper augments the existing SRED ^{FM} relational extraction dataset with evidence annotation to its 2.9-million Arabic relations. We leverage the augmented dataset to build AREEj, a relation extraction with evidence model from Arabic documents. The evidence augmentation model we constructed to complete the dataset achieved .82 F1-score (.93 precision, .73 recall). The target AREEj outperformed SOTA mREBEL with .72 F1-score (.78 precision, .66 recall).

## introduction we define relational extraction with evidence (ree) as the task of extracting related entities (source and target), identifying the relation type between them, and providing evidence from text to support the relations. relation extracting (re) is an important task to build knowledge graphs. enriching knowledge graph edges with evidence labels helps in several ways: (i) it supports explainable ai tasks potentially by large language models @xcite , and (ii) it improves the performance of the re models as evidence plays the role of a hint for an existing relation. evidence is important as it helps understanding machine learning (ml) decisions @xcite and provides decision-makers with more confidence in ml model predictions. in turn, detecting evidence boosts the performance of ml models as it is a step in a chain-of-thought to infer the relation and its type @xcite . a relational entity with evidence @xmath0, t, s ne , t ne , rt, e⟩ where s is the source, t is the target, s ne and r ne are named entity types of the source and the target, rt is the relation type, and e is the evidence. the elements s, t, and e are extracted from the text itself, and the rest are predefined categories and classes. one document may contain multiple rees and rees may have elements in common. figure 1 shows a sentence with four relations where the main entity s 1 entity is connected as source with three other entities: t 1 inception date (2010), t 2 headquarters location (doha), and t 3 country (qatar). doha is also a source s 2 = t 2 connected to t 3 indicating it is in qatar. the evidence shows between brackets explaining why the the model reported these relations and their types. researchers developed systems capable of extracting relations from arabic text including a morphology-based and regular expression based approach @xcite , and a crosslanguage learning approach @xcite that leveraged knowledge transfer for the lack of large arabic relational datasets. research leveraged wikipedia data to build a dataset with 19 relation types @xcite . multilingual data from wikipedia was also utilized to construct a larger dataset, across 14 languages and with 36 relation types @xcite , where at most one relation was assigned per text. bert @xcite was used to train two models: the claim is that multilingual models achieve better results than single language models. the arabic subset of the data contains 9,000 rows with 9 unique relation types. later work presents sred fm with 40-million relations in 18 different languages to be the largest re dataset up to our knowledge. its arabic subset covers 393 out of a total of 400 relation types with 2.9-million relations. sred fm is extracted from wikipedia using the crocodile tool @xcite . the mrebel model was trained on sred fm to perform relational extraction @xcite . this paper presents work that augments the arabic subset of sred fm with relational evidence, and then to use the resulting dataset to train areej, a relational extraction with evidence model. we perform the augmentation via finetuning an open-source large language model to extract evidence from the arabic text given a relational sequence. the evidence augmentation task performed well with .82 f1-score, .93 precision and .73 recall areej extracted relation with evidence with .72 f1-score, .78 precision and .66 recall. we make the annotated dataset available online for the research community. the mrebel tool @xcite ) is trained and tested on the (sred fm ) and (red fm ) multilingual re datasets extracted from wikipedia data. each relation is specified as a tuple wit source, target, entity types for source and target, and relation type. the training dataset was automatically generated, while the testing dataset was manually revised. mbart @xcite , which is typically used for translation, was used to map and normalize the relational types across all included 18 languages. a rule-based model extracts the source, target, and relation spans depending on part-of-speech tags @xcite . the stanford arabic parser @xcite and wordnet @xcite were leveraged to extract triplets, which works well for sentences with simple structures. in @xcite , an approach was introduced to extract relations from documents and use attention to extract evidence. documents are multiple sentences and the evidence is several sentences that support a detected relation. this is different from our target evidence as we target finegrained relations at the sentence level. we also extract the relation and the evidence simultaneously. the work in @xcite improves on memory efficiency and lack of annotations. our work solves the annotations problem by using chatgpt-4 to annotate the data with evidence, fine-tune an open-source llm using the generated data, and then use the fine-tuned llm to annotate 2.9 million relations with evidence. to do that we leverage the chain of thought (cot) technique in our prompts @xcite where requiring intermediate steps in inference improves the quality of the final result and avoids inaccuracies and hallucinations. we constructed arsred via augmenting the 2.9-million relations arabic subset of sred fm with evidence as illustrated in figure 2 . we did this in several steps. ## training arbartev and areej arbartev requires relations to produce evidence. arbartev takes @xmath1, r⟩ composed of text t and relation r and produces @xmath2, e⟩ where e is the evidence. this produced dataset arsred as discussed in section 3. the intuition behind including r in both input and output is that transformers tend to work better when asked to perform tasks in steps and with hints. we trained areej based on the mbart seed model and arsred to take an arabic text t and produce a relation r with evidence e. p(@xmath3, e|t) = len(y) we target only arabic text so we limited the tokenizer to arabic and kept the named and relation entity types as special tokens as specified in rebel and mrebel. model fine-tuning was executed on 1x nvidia rtx a6000 48 gb vram gpu. the hyperparameters used are .00005 learning rate, 5000 warm-up steps, 16 batch size, and 156140 max steps. the rest are the default hyperparameters of mbart. ## results and evaluation for evidence equivalence for arbartev, we computed the number of word-based intersections between its evidence results and those produced by chatgpt-4 on red fm . for entries with no intersection, we count that as a miss. for entries with intersections, we performed a manual check and a human expert deemed the extracted evidence as true or not. arbartev achieved an f1-score of .82 on evidence extraction (p=.93,r=.73). for areej, we performed a beam search at the output level and obtained six predictions. note that these predictions may have relations that share relational elements. we noticed that some extracted relations were correct as evaluated by a human expert, yet the testing set missed them. areej correctly identifies 72 relations that mrebel misses, while mrebel identifies 47 relations areej misses. areej achieves .72 f1score (p=.78, r=.66) with correct evidence .86% of the time as detailed in the extarction row of fourteen relation types are detected by areej but not by mrebel. examples include 'employer', 'use', 'member of', 'industry', 'main subject', and 'date of death'. four relation types are detected by mrebel and not by areej including 'characters', and 'member of sports team'. lower rows of ## conclusion adding evidence annotations resulted in extracting more relations and relation types. recall increased significantly and precision suffered slightly. in future work, we plan to use the extracted evidence to improve precision via relating to relation and entity types and reducing the "none" samples to improve recall.