Event-Arguments Extraction Corpus and Modeling using BERT for A rabic (2024)

Abstract: Event-argument extraction is a challenging task, particularly in Arabic due to sparse linguistic resources. To fill this gap, we introduce the corpus (550k tokens) as an extension of Wojood, enriched with event-argument annotations. We used three types of event arguments: agent , location , and date , which we annotated as relation types. Our inter-annotator agreement evaluation resulted in 82.23% Kappa score and 87.2% F_1 -score. Additionally, we propose a novel method for event relation extraction using BERT, in which we treat the task as text entailment. This method achieves an F_1 -score of 94.01%.To further evaluate the generalization of our proposed method, we collected and annotated another out-of-domain corpus (about 80k tokens) called and used it as a second test set, on which our approach achieved promising results (83.59% F_1 -score). Last but not least, we propose an end-to-end system for event-arguments extraction. This system is implemented as part of SinaTools, and both corpora are publicly available at https://sina.birzeit.edu/wojood

## introduction understanding and extracting events from text is crucial in natural language understanding @xcite for applications like disaster monitoring @xcite , emergency response @xcite , insurance decision support, and fostering community resilience @xcite . events, a type of named entity mentions @xcite , are connected to other entities through their arguments. this connection forms the foundation of the event-argument extraction task, which is closely related to relation extraction. by identifying events and linking them with arguments like agents, locations, and dates, we establish meaningful relationships that enhance applications such the economic forum meeting chaired by professor schwab was held on 2023 in davos as information retrieval systems @xcite , word sense disambiguation @xcite , and knowledge graph construction @xcite . figure 1 shows how the event "the economic forum meeting" is connected with its three arguments. despite the significance of event-argument extraction, there is a notable gap in the availability of comprehensive, annotated corpora for this purpose, especially for under-resourced languages like arabic @xcite . to address this gap, we have developed an event-argument extraction corpus specifically for arabic. we extended the wojood corpus @xcite , which is the largest and most recent ner corpus for arabic @xcite @xcite . we annotated arguments of event entities in wojood corpus by manually linking each event entity with its corresponding argument entities. as shown in figure 2 , three types of arguments are introduced (agent, location, date) for each event. our wojood extension (i.e., the event-arguments annotations) is called wojood hadath . furthermore, we introduce a novel method for event-argument extraction using bert and achieved 94.01% f 1 -score. based on wojood hadath , we generated a dataset of premisehypotheses sentence pairs (we call it hadath nli ). we used this dataset to fine-tune bert, as a natural language inference (nli) task. to test the generalization of our approach, we have constructed an additional out-of-domain dataset (about 80k tokens) called wojood outofdomain and used it as a second test set. our model achieves, again, promising results (83.59% f 1 -score). finally, to streamline the event-argument extraction, we propose an end-to-end system specifically designed for the arabic event-argument extraction task. in summary, the contributions of this paper are: all datasets are available online foot_0 , and the endto-end system is implemented and can be used as part of the sinatools @xcite this paper is organized as follows: section 2 reviews previous research. section 3 discusses corpus annotations. section 4 presents the interannotator agreement analysis. section 5 outlines the methodology. section 6 details the dataset construction, while section 7 covers the experimental setup. section 8 introduces the out-of-domain dataset. section 9 elaborates on the end-to-end system, and section 10 explores ablation studies. finally, section 11 concludes the paper and discusses future directions. ## related works events are occurrences or actions that happen over time, involving specific participants and locations. they have temporal components and rely on physical entities to take place @xcite . extensive research has focused on event extraction and event argument extraction (eae), especially in english. however, eae in arabic is limited, leaving a gap in the literature @xcite . automated methods, utilizing either statistical algorithms or nlp techniques, are employed to identify relationships between words, using large corpora to detect related terms through their cooccurrence patterns @xcite . for instance, @xcite proposed a model to extract event entities from arabic news articles using the gate tool, employing a five-stage entity identification process that establishes links between events and their corresponding arguments. however, their dataset is small as it consists of only 1, 650 sentences. in a similar context, al-smadi and qawasmeh (2016) proposed to use an unsupervised rule-based technique to extract events and the relationship with their associated entities from 1, 000 arabic tweets covering time, agent, location, target, trigger, and product. they linked extracted events and entities to a knowledge base, achieving 75.9% accuracy. this accuracy pertains to the textual representation within the tweet corresponding to the event expression, event type identification (97.7% accuracy), and event time extraction (87.5% accuracy). however, despite these achievements, there remains a need for larger and more diverse datasets to further validate these findings. multilinguality has increasingly attracted attention across various fields due to its potential to enhance the understanding and processing of diverse linguistic data @xcite @xcite . this direction has notably influenced relation extraction tasks, as seen with smiler @xcite , which aims to improve entity and relation extraction, including arabic. @xcite fine-tuned herberta on the arabic subset of smiler, achieving high performance in identifying relations and entities despite its smaller dataset of 9k sentences. this suggests that the unique linguistic features of arabic may contribute to the model's robustness. furthermore, @xcite introduced valuable resources, namely, sred fm and red fm , designed for multilingual relation extraction. sred fm encompasses over 40 million triplet instances across 18 languages, featuring 400 relation types and 13 entity types. their mrebel model, pre-trained on sred fm , exhibited a remarkable improvement of 15 points in micro-f 1 compared to herberta. however, their qualitative error analysis excludes arabic and chinese because the authors are not proficient in these languages. joint extraction models have become increasingly important in nlp. addressing the need for improved event extraction in arabic text, el khbir et al. (2022) introduced a joint model for event extraction in arabic text using the ace 2005 dataset. they used a graph-based representation to extract entities, relationships, and event triggers, along with their arguments. this approach led to improved arabic ner accuracy through different experiments and tokenization schemes. in the realm of biomedicine, xu et al. (2022) introduced the nbr (nli improved biomedical relation extraction) method, which verbalizes relations in natural language hypotheses, allowing the model to utilize semantic information effectively in prediction, even with limited data. in contrast, @xcite addresses cross-lingual eae challenges through their innovative language-oriented prefix-tuning network (lapin) approach. lapin utilizes a languageoriented prefix generator module to handle language variations and a language-agnostic template constructor module to create adaptable templates. experimental results demonstrate lapin's outperforming, achieving an average f 1 improvement of 4.8% and 2.3% on two multilingual eae datasets compared to the previous state-of-the-art models. these efforts collectively contribute to advancing the field of eae in various languages, including arabic. ## inter-annotator agreement to evaluate the quality of our annotations, we randomly selected 5% of the annotations and asked our two annotators to annotate in parallel. we computed the inter-annotator agreement (iaa) using cohen's kappa and f 1 -score. the results in ## even-argument extraction (eae) in this section, we provide an in-depth explanation of our approach to addressing the eae task through the nli task. figure 5 illustrates our framing of the eae problem. ## eae modeling experiments the hadath nli is used to train an eae model. ## additional wojood outofdomain dataset to evaluate the model's generalization and its robustness to contexts beyond the wojood hadath do-mains, we constructed a new out-of-domain corpus (referred to as the wojood outofdomain corpus). we then extracted an nli dataset and used it for model testing. ## ablation studeis best template: to choose the best template to implement in the eae system for each relation (figure 5 ), we evaluated them using the test set. large language models (llms) can further enhance our work by extracting information, including named entities and relationships, from text to populate knowledge graphs and improve other knowledge graph tasks like embedding and completion @xcite . in future work, we will explore the capabilities of llms to enhance event-argument extraction. integrating llms into our framework could potentially improve the accuracy and scalability of our event extraction system.