A rab IE : Joint Entity, Relation and Event Extraction for A rabic (2022)

Abstract: Previous work on Arabic information extraction has mainly focused on named entity recognition and very little work has been done on Arabic relation extraction and event recognition. Moreover, modeling Arabic data for such tasks is not straightforward because of the morphological richness and idiosyncrasies of the Arabic language. We propose in this article the first neural joint information extraction system for the Arabic language.

## introduction information extraction (ie) is the task of identifying and classifying information of interest in a textual document. ie is an important area of research in nlp since it has many practical applications. in this article, we are interested in joint modeling of three ie tasks: named entity recognition (ner), relation extraction (re), and event recognition (er). a joint multi-tasking system, in comparison to a pipeline system, has the advantage of avoiding the propagation of errors among tasks. this area of research is well explored in many languages such as english, chinese, and spanish. @xcite proposed a model that jointly extracts entity mentions, event triggers and event arguments using shared hidden representations in a deep learning framework. @xcite provided a framework for extracting entities, relations, and triggers using bert embeddings and graph propagation to capture context relevant for these tasks. @xcite proposed a joint neural framework that extracts entities, relations and events from an input sentence as a globally optimal graph. for arabic, however, most proposed models are restricted to ner @xcite . limited efforts have been dedicated to re and er @xcite , and no previous work has addressed them jointly. we attempt to fill this gap in the present work. similar to @xcite , the model we propose in §2 extracts a graph from an input sequence in two steps: (a) two crfs @xcite with bio-based tags are used to identify spans (subsequences of tokens) corresponding to entities and event triggers (graph nodes); then (b) greedy decoding is used to obtain the output graph. since arabic is morphologically rich (habash, 2010), entities are not limited to sequences of words like english for instance. some entities correspond to affixes and some words carry multiple entities. therefore, modeling on the subword level is necessary. to address this issue, we compare two approaches which we describe in detail in §3. in the first approach, we resort to word tokenization as a preprocessing step. we aim to split morphologically complex words into tokens, each of which corresponds to (or is a part of) one entity at most. an entity can thus be modeled as a sequence of tokens using the standard bio tags. in the second approach, we augment the bio tags to encode multiple entities per word, eliminating the need for prior tokenization. our contribution in this article is twofold: ## multitask joint extraction model given a text document as input, we aim at extracting, from each sentence, entities and binary relations between them, event triggers, and their arguments. formally, for an input sequence x of length l, the information extraction task is the operation that yields, as an output, a graph @xmath0, e) whose nodes v are spans of tokens of the input sequence representing identified entities and triggers, and whose edges e represent relations between two entities or event roles (relations between event triggers and their arguments entities). each node and edge in the graph has a type. similar to @xcite , our model performs end-to-end ie in four stages. token encoding several combinations of representations from bert's layers are inspected for encoding the input sequence, as done by @xcite for english data. ultimately, the input sequence is encoded using the concatenation of bert's last and third last layers to obtain an embedding for each token, as using these layers improves the performance on most subtasks. @xcite showed that bert last layers contain semantic information about the text, which is beneficial for the processing of arabic texts. input sequences are optionally tokenized in a preprocessing step ( §3). identification token embeddings are passed to a network composed of a feed-forward network (ffn) layer followed by a conditional random field (crf) layer. the network labels the sequence using the bio scheme to identify spans of tokens that correspond to entities or event triggers. we use separate crf taggers for entities and triggers so that each one specializes in one task. the sequence of labels produced by the crf encodes a segmentation of the input sequence so that identified entities cannot overlap, the same applies for triggers. on the other hand, entities can overlap with triggers in some cases. the verb (awqft; she arrested) is for example a trigger of type justice and the pronoun (t) is an entity of type per. classification at this stage, entities and triggers are identified, but their types are not yet assigned. a fixed-size representation for each span is computed as the average of its first and last token's bert embeddings. the output is passed to an ffn to obtain a score for each possible type. again, we use separate ffns for entities and triggers. scoring relations and event roles is performed in a similar manner. an edge between two spans is represented by concatenating their vectors. a relation edge links two entities while a role edge links a trigger to an entity. representations of edges are passed to an ffn to compute a score for each relation or role type. a special none label to indicate the absence thereof. we also use a separate ffn is used for relations and roles. decoding we use unconstrained greedy decoding to obtain the output graph: for each node and edge of the graph, we select the highest-scoring type. in our experiments, we tried adding to the graph score a penalty on invalid graph configurations and decode with beam search similar to @xcite but didn't get any improvements. training the parameters of all networks are jointly trained end-to-end to minimize the sum of individual task losses. we use the negative loglikelihood of gold bio paths as a loss function for the crfs and of the gold label for the ffn classifiers. ## subword entities as discussed earlier in ( §1), a word in arabic can hold two or more entities anchored on its root or affixes. for example, the word (mrasltna; our reporter) comprises two entities: word tokenization subword entities typically correspond to morphemes. we, therefore, use a morphological analyzer to tokenize words in context. the probability that each resulting token corresponds to multiple entities decreases dramatically. in practice, we use the analyzer provided by cameltools @xcite and refer to this tokenization scheme by tok_morph. the word in the above example is tokenized into three morphemes (mrasl +p +na), the first two tokens correspond to the entity per, the third one to org. to obtain supervised training data for tokenized sequences, we align each word with its tokens (morphemes) at the character level and use the alignment to project gold entities onto the tokens. an entity is projected onto a token if the majority of its characters align with the token. if multiple entities are projected onto one token, only one of them is randomly selected. to validate our hypotheses that morphemes are the right level for modeling entities, we compare the morphological analyzer to word pieces @xcite , a statistical tokenizer which does not necessarily produce valid affixes. this tokenizer produces (mrasl tna) for the example word where the second token is not a morphologically valid suffix and does not exactly match the gold entity (na; our). we refer to this tokenization scheme by tok_wp. projection of entities onto tokens is not always perfect either because an entity doesn't correspond to a morpheme in gold data; the tokenizer doesn't produce a valid morpheme; or both. this results in some data loss that we later quantify and take into account during the evaluation phase. an example of the data loss in tok_wp is that of the sentence (@xmath1; label concatenation instead of tokenization then projection, we concatenate labels of subword entities into one complex entity. the example word is thus labeled per-org. this approach is appealing because of its simplicity, but it results in a much larger label set, as some words contain up to four entities. in practice, we restrict the label set to the labels seen in training data. we refer to this tokenization scheme by concat. ## error analysis error analysis is important to understand the model's weaknesses and to attempt to fix them in future work. thus, we examined a sample of 32 sentences where we found 110 remaining errors from experiments with tok_morph tokenization and type labels. entity errors about 23% are errors related to pronominal entities; these errors either come from entities predicted by the model and not annotated in the gold data or vice-versa or from correctly identified entities but incorrectly classified. for example, in the word (sadrtha; confiscated it), the pronoun (t) is annotated in gold data as a per entity that the model does not predict. these errors are most likely due to the lack of labeling of a considerable number of pronominal entities of the gold data. as example, for the word (alm-slhyn; armed), the model predicts the pronoun (yn) as a per entity but it's not annotated in the gold data, although this pronoun was annotated based and machine learning approaches. (benajiba et al., 2008b) proposed an svm-based model with a combination of language-dependent and language-dependent features, showing the relevance of morphological features for rich languages like arabic. (benajiba et al., 2010) built a system augmented by deeper lexical, syntactic, and morphological features that were extracted from noisy data obtained via projection from an arabic-english parallel corpus. (helwe et al., 2020 in addition, they remove all annotations that they consider not oriented to the entity detection and recognition tasks, such as the nominal and pronominal entities, and only keep the named ones, which leads them to a total number of entities in the training and test corpora of 10218. this makes their performance incomparable to ours because we evaluate the model with almost 32000 entities for all our proposed approaches. other work @xcite @xcite ) same preprocessing of @xcite . @xcite tested their model performance on date, time, price, measurement, and percent entities of ace05, while we test our model on the principal entity types (per, loc, org, fac, veh...). relation extraction arabic re works include @xcite , who proposed a distant supervised learning model with specific features that characterize arabic relations. ## conclusion we presented the first joint ie model for arabic and showed a comparable performance to other languages. we also proposed two approaches to address subword entities, a situation specific to morphologically rich languages including arabic, and showed that morphological information is important to their recognition. our hope is that our work will provide a strong baseline for further research and increase interest in ie tasks which remain understudied by the arabic nlp community.