P ars S imple QA : The P ersian Simple Question Answering Dataset and System over Knowledge Graph (2022)

Abstract: The simple question answering over the knowledge graph concerns answering single-relation questions by querying the facts in the knowledge graph. This task has drawn significant attention in recent years. However, there is a demand for a simple question dataset in the Persian language to study open-domain simple question answering. In this paper, we present the first Persian single-relation question answering dataset and a model that uses a knowledge graph as a source of knowledge to answer questions. We create the ParsSimpleQA dataset semi-automatically in two steps. First, we build single-relation question templates. Next, we automatically create simple questions and answers using templates, entities, and relations from Farsbase. To present the reliability of the presented dataset, we proposed a simple question-answering system that receives questions and uses deep learning and information retrieval techniques for answering questions. The experimental results presented in this paper show that the ParsSimpleQA dataset is very promising for the Persian simple question-answering task.

## introduction a knowledge graph (kg) represents a network of real-world entities, with a massive semantic net that integrates various, inconsistent and heterogeneous information resources to represent knowledge about different domains @xcite . some kgs contain information from multiple domains to permit machine learning applications to operate on various tasks such as question answering (qa), recommender systems, and search systems by allowing them to retrieve and reuse comprehensive answers for a given query over kg. many studies used well-structured kgs as external resources to support open-domain qa, whereas the knowledge graph-based question answering * these authors contributed equally to this work. (kgqa) system aims to answer natural language questions (nlqs) automatically. kg as a data structure underpins digital information systems, assists users in finding and retrieving resources, and serves navigation and visualization purposes. in the humanities, knowledge graphs are usually rooted in knowledge organization systems that have a centuries-old tradition and have undergone a digital transformation with the advent of web-connected data @xcite . this work addresses the persian language, which in general is underrepresented in nlp and also within digital humanities. considering open-domain qa in low resources languages such as persian, this work certainly benefits also the research on digital humanities. generally, knowledge graphs and applications could be vehicles for formalizing and connecting findings and insights derived from the analysis of possibly large-scale corpora in the digital humanities domain. where with help of such applications we can digitize archive collections for librarians or social science research. in the english language, there are more valuable research works, but there is limited work that has been carried out for the persian kgqa. the farsbase @xcite is the first persian kgs that uses hybrid techniques to extract knowledge from various sources, such as wikipedia, web tables, and unstructured texts. the farsbase was published in 2018; since then, only a few research works have been published to incorporate the farsbase in nlp tasks due to the unavailability of datasets that refer to the kg. this also limited research on digitalizing datasets for opendomain qa tasks, since, human readers often rely on a certain amount of broad background knowledge obtained from sources outside of the text. it is perhaps not surprising then, that machine readers also require knowledge external to the text itself to perform well on qa tasks where kgs are the best source of such information. to overcome these concerns, @xcite were the first researchers who proposed the pecoq dataset as the first dataset for persian complex qa over kg. although the existing dataset is very well developed for complex qa, many general-purpose kgqa systems in other languages (e.g english) are designed to deal with complex qa by decomposing complex questions into simple questions. the knowledge graph simple question answering (kgsqa) is a key building block for complex qa, and its performance depends on kgsqa @xcite . so putting more emphasis on kgsqa is necessary for kgqa. to facilitate research on persian kgqa, the first simple qa dataset and system have been introduced in this work. to the best of our knowledge, this is the first step toward persian kgsqa. in this work, we first proposed the parssimpleqa dataset, a simple qa dataset in persian. next, we proposed a simple qa framework for the persian kgsqa. the rest of the article is organized as follows. we first describe the problem statements and definitions in section 2. the kgqa studies are studied in section 3. section 4 presents the parssimpleqa dataset. the first persian simple qa model is presented in section 5. section 6 describes experimental setups, and results are covered in section 7. finally, in section 8 we conclude the article. ## problem statement and definitions our study aims to design the first simple qa dataset and system for the persian language that can map a simple nlq q to a matching query q consisting of the subject and relation to be executed in the kg g to retrieve answers. kg g comprises triples in the form of (s, r, o) where s, r, and o denote the head entity, predicate/relation, and the tail entity, respectively. in this work, g is farsbase kg, the first persian multi-source kg. a simple question is a question that contains a single relation that can be queried through g to extract facts. for example, the question "who is the director of alone in berlin?" contains a director relation which can be answered using g fact that "vincent perez" is the director. the kgsqa task is defined as @xcite : given a kg @xmath0, r i , o i )} that represents a set of triples, and a nlq @xmath1, w 2 , . . . , w t }, where w @xmath2, the simple qa task is to find a triple (s ′ , r ′ , o ′ ) ∈ g, such that o ′ is the answer to the question. ## related works the kgqa has attracted a considerable body of research in recent years, and increasingly, researchers are building end-to-end neural network models for this task. a straightforward decomposition of the kgqa pipeline is entity recognition, relation prediction, entity linking, and evidence integration. this work explored relationship prediction, entity linking, and evidence integration to transform natural language into queries to extract simple factoid question answers from kgs. ## parssimpleqa dataset to construct a kgqa system in the practical environment, we should solve the following tasks: entity detection, el, rp, and ei where each task can be addressed in a supervised or unsupervised learning fashion using an appropriate dataset. the proposed dataset is suited for training and evaluation of models for suitable and optimal queries to extract answers from kg g. for automated qa dataset generation, we had to deal with two challenges: creating high-quality templates and creating logical/correct qas which the parssimpleqa datasets consider these challenges properly. figure 1 depicts the process of creating the dataset. in the following, we discussed the details for parssim-pleqa creation. next, the rejected template was removed from the final list to obtain 149 templates. then, we asked annotators to analyze templates to see whether there are relationships that share the same information or not. thereby, we relationship selection entity selection automatic qa generation generating templates templates evaluation and cleaning post-processing figure 1: the parssimpleqa dataset generation procedure combined multiple relationships to obtain 32 relationships in total. on average, 4.65 templates per relationship were constructed for the parssimpleqa dataset. 4. entity selection: two ranges of data sources have been used to build the farsbase knowledge base. the first source involves extracting rule-based information from dumped wikipedia data, and the other source includes other knowledge bases such as yago and wikidata. rule-based information extracted from the wikipedia dumps naturally contains a lot of noise, including meaningless words, as well as information from other knowledge bases, including entities from different languages such as chinese and korean. none of them are appropriate for this task. therefore, we considered the following three conditions in the entity selection for dataset creation: 1) the entity must have a uri field (a unique id in kg), 2) ignoring entities from dbpedia, yago-knowledge.org, ecowlim.tfri.gov.tw, data.linkedmbd.org, data.linkedopendata.it, and 3) considering only entities with persian letters. we employed 60% train, 20% validation, and 20% test split rates while creating datasets. ## method our methodology for the persian kgsqa uses the parssimpleqa dataset and comprises the following components: relation prediction (rp), entity linking (el), and evidence integration (ei). figure 2 depicts a proposed framework. in the proposed sets # of samples # of templates # of question entities train 29,360 78 10,945 validation 2,261 34 1,898 test 4,501 37 3,929 overall 36,122 149 16,772 ## experimental setups metrics: commonly used performance measures include accuracy, precision, recall, and f-measure @xcite . since for each question there is an expected set of correct answers, these are called the gold standard answers. we can define the metrics as followings (n is a number of samples): where precision indicates how many of the answers are correct, recall indicates how many of the returned answers are in the gold standard. f-measure is the weighted average between the (macro) precision and recall. training setups: we have used the parssimpleqa dataset for training and evaluating the first persian kgsqa model. we imported farsbase kg into the neo4j database. next, after tuning several hyperparameter models, the final model was trained. training and hyperparameter tuning was done using the nvidia tesla v100 gpu machine. ## conclusion this paper introduced parssimpleqa, the first kgsqa dataset for the persian language that contains questions with corresponding entities, entity links, relation types, and answers. the machinegenerated questions using human-generated templates are preprocessed and divided into train, validation, and test sets for training and analyzing machine learning techniques. next, we introduced the first persian kgsqa to perform el, rp, and ei. our experimental results on the parssimpleqa dataset show that our proposed framework is robust and generalized well. this framework will play a baseline model that opens many works in persian kgsqa. moreover, our generated parssimpleqa dataset is available to the research community at the github foot_0 repository.