SERAG : Semantic Entity Retrieval from A rabic Knowledge Graphs (2021)

Abstract: Knowledge graphs (KGs) are widely used to store and access information about entities and their relationships. Given a query, the task of entity retrieval from a KG aims at presenting a ranked list of entities relevant to the query. Lately, an increasing number of models for entity retrieval have shown a significant improvement over traditional methods. These models, however, were developed for English KGs. In this work, we build on one such system, named KEWER, to propose SERAG (Semantic Entity Retrieval from Arabic knowledge Graphs). Like KEWER, SERAG uses random walks to generate entity embeddings. DBpedia-Entity v2 is considered the standard test collection for entity retrieval. We discuss the challenges of using it for non-English languages in general and Arabic in particular. We provide an Arabic version of this standard collection, and use it to evaluate SERAG. SERAG is shown to significantly outperform the popular BM25 model thanks to its multi-hop reasoning.

## introduction recent years have witnessed a rapid growth in the use of kgs. both large-scale graphs that connect millions of entities through billions of relationships and smaller domain-specific kgs have become widely available. one way to access the information stored in kgs is through a dedicated query language, such as sparql. this, however, requires proficiency with the syntax of the language and knowledge of the exact schema of the database. given a query in natural language, the task of entity retrieval aims at presenting a ranked list of entities relevant to the query. consider, for example, the query "museums in arab capitals". figure 1 shows how the subgraph around the entity e3, labelled "egyptian museum", supports its relevance to the query. this complex, indirect relationship is not represented textually in any single document. the current standard test collection for entity search is dbpedia-entity v2 (dev2). it is provided with the performance results of a dozen baseline methods @xcite . lately, @xcite presented kewer, a system that significantly outperformed previous methods on dev2. kewer, nevertheless, was developed for and tested on english kgs. in this work, we build on it and propose serag, 1 a method for semantic entity retreival from arabic knowledge graphs. to evaluate serag, we create and share a modern standard arabic (msa) version of dev2. to our knowledge, this is the first attempt to offer dev2 in a non-english language. serag is shown to perform well on arabic dev2 and to significantly improve on the classic bm25 retriever. ## background and related work there are several attempts in literature to define kgs @xcite . 1 (sirāj) is arabic for a lantern. 2 the translated queries and usage directions can be found at https://doi.org/10.5281/zenodo.4560653 . in this work, we follow @xcite and view a kg as a finite set of rdf triples in the form of subject-predicate-object (s, p, o). graph embedding graph embeddings transform nodes and edges into a low dimension vector space while preserving structural information, an idea that was adopted from language modeling. @xcite survey various graph embedding methods. these methods, however, focus on the relational structure of the graph and do not leverage the information present in the surface forms of entities and predicates. consider again the graph in figure 1 . p1 links e3 and e2, but there is additional information in the labels attached to those. entity retrieval since the introduction of dev2, several works have reported significant progress on entity retrieval. naseri et al. (2018 ) explored enriching entity representations using information from related entities. kadilierakis et al. (2020) adapted elasticsearch foot_0 for supporting keyword search over rdf datasets. gerritse et al. (2020) studied utilizing wikipedia2vec (yamada et al., 2020) for entity reranking. nikolaev and kotov (2020) proposed kewer, a system that employs joint word and entity embeddings to rank entities. kewer was evaluated on dev2 but can be applied to any kg and does not require a large textual corpus. question answering (qa) qa systems are concerned with automatically providing answers to natural language questions. such systems have been integrated in search engines and virtual assistants. answers may be constructed by querying a structured database or pulled from an unstructured collection of documents. inspired by drqa @xcite , @xcite al. (2019 ) presented so-qal, a system for open domain factual arabic qa using wikipedia. soqal was evaluated on two novel datasets: arabic reading comprehension dataset and a machine translated version of stanford qa dataset. samy et al. (2019) overview the work done in arabic qa and recommend tools and linguistic resources for future systems. ## method serag, our proposed framework for arabic entity retrieval, is inspired by kewer. as illustrated in figure 3 , for each entity, dbpedia is used to generate (1) a document formed of directly linked textual information (recipe provided with dev2), and (2) a set of random walks starting at that entity. given a query, bm25 scores entities based on the relevance of their documents to the query. it does not, however, utilize information about the graph structure or entities indirectly connected. to enhance the ranking, a word2vec @xcite model is built using the random walks. each walk consists of entities and predicates along a path in the arabic dbpedia graph. entities appear in their entity id form (e.g., e3) with a probability p and replaced with their surface form (e.g., egyptian museum foot_7 ) otherwise. each walk is considered a sentence and all sentences are concatenated into one corpus, used to train the word2vec model. hence, the vocabulary of the model includes both words and entity ids. figure 4 illustrates how random walks and sentences are formed. we use a standard, two-stage ranking approach: given a query q, we first select 1000 entities using bm25, then we rerank them using embeddings. let e be an entity retrieved by bm25, v e its embedding, and v q the embedding of q. the final score of e with respect to q is based on a mix of its bm25 score and the embeddings' cosine similarity: in our implementation, we modified kewer's code to allow for arabic kgs. to keep everything in python, we reimplemented the document generator and used gensim @xcite for bm25. while kewer also uses bm25f @xcite , a fielded version of bm25, in this work we only consider bm25. kewer tunes the parameters of bm25 and β, the mixture weight, using a 5-fold split. our data, however, is prohibitively small. therefore, serag is tested in an unsupervised setup using the default values of gensim, a fixed β = 0.9, chosen to give random walks a higher impact, and the default values kewer uses for the number of walks (100), their length (10), and p (0.1). we are confident that serag will benefit from parameter tuning and hope to include it when a larger dataset becomes available. ## experiments and discussion the evaluation metric of dev2 is normalized discounted cumulative gain (ndcg) at ranks 10 and 100. in section 3 we noted that dev2 in general, and arabic dev2 in particular, do not offer judgements for all relevant entities. @xcite introduced bpref, a retrieval measure designed specifically for incomplete judgment sets. it only considers the relative ranks of relevant and nonrelevant judged entities. we applied bpref to the first 10 and 100 retrieved entities. in both cases, serag outperformed bm25 (0.17 vs. 0.142 for top 10 and 0.336 vs. 0.314 for top 100). to the best of our knowledge, bpref has not been used to evaluate english dev2 systems. ## conclusions there has been significant progress in offering tools and resources for arabic nlp applications. in this work, we addressed the task of entity retrieval from an arabic kg. our contribution is twofold. we introduced an arabic version of dev2, the standard test collection for entity retrieval and then proposed serag, a system for end-to-end entity retrieval from arabic kgs. we believe arabic dev2 can already be used as a benchmark for kg entity retrieval. it is, however, only the first step. we hope to collaborate with the community to expand both the queries and relevant entities. a fresher dump of wikipedia arabic should be considered in order to increase coverage. we also plan to translate the collection to other languages and create a multi-lingual set. while serag was shown to improve over bm25, we observed that arabic entity retrieval faced unique challenges. to address those, we plan to employ the recently introduced arabic nlp toolkit @xcite and utilize pretrained language models such as arabicbert @xcite and @xcite ). 11 the word , for instance, can take over a dozen combinations of diacritics, each with multiple meanings. 12 13