Developing PUGG for P olish: A Modern Approach to KBQA , MRC , and IR Dataset Construction (2024)

Abstract: Advancements in AI and natural language processing have revolutionized machine-human language interactions, with question answering (QA) systems playing a pivotal role. The knowledge base question answering (KBQA) task, utilizing structured knowledge graphs (KG), allows for handling extensive knowledge-intensive questions. However, a significant gap exists in KBQA datasets, especially for low-resource languages. Many existing construction pipelines for these datasets are outdated and inefficient in human labor, and modern assisting tools like Large Language Models (LLM) are not utilized to reduce the workload. To address this, we have designed and implemented a modern, semi-automated approach for creating datasets, encompassing tasks such as KBQA, Machine Reading Comprehension (MRC), and Information Retrieval (IR), tailored explicitly for low-resource environments. We executed this pipeline and introduced the PUGG dataset, the first Polish KBQA dataset, and novel datasets for MRC and IR. Additionally, we provide a comprehensive implementation, insightful findings, detailed statistics, and evaluation of baseline models.

## introduction question answering (qa) systems serve as a sophisticated interface between humans and computers. to further enhance their utility, we need qa systems capable of answering questions based on extensive knowledge @xcite . the knowledge base question answering (kbqa) task addresses this need by using structured knowledge graphs (kg) to provide accurate and relevant answers @xcite . kbqa leverages these graphs, which are rich with interconnected entities and relationships, to decode complex queries and deliver precise answers. importantly, systems that reason over kgs are more resistant to the phenomenon of hallucinations, common in large lan-guage models (llm) @xcite . additionally, the inherent flexibility of kgs facilitates easy modification and updating, ensuring the use of only the most current and accurate facts. however, a significant gap exists in kbqa datasets. many are schematic and not natural in their language, or they rely on discontinued knowledge graphs @xcite @xcite . by natural we refer to naturally occurring questions @xcite . while a broader range of kbqa datasets is available for english, most low-resource languages, including polish, lack such resources @xcite . this scarcity is part of a broader issue prevalent in the field of nlp concerning low-resource languages @xcite . recognizing this gap, we set out to create a kbqa dataset for polish. we faced several challenges during extensive studies of existing works to find the most efficient methods for dataset creation. many datasets were built on simpler predecessors @xcite , and also many construction pipelines are inefficient regarding human labor, as they do not utilize modern tools that could reduce human work, such as assisting large language models (llm). llms have opened new opportunities for assisting human annotators, especially in low-resource languages where the range of pre-trained models is limited @xcite . consequently, we decided to design, implement, and execute a modern approach to creating kbqa datasets tailored explicitly for the low-resource environment. we selected wikidata as kg due to its extensive, multilingual coverage and dynamic, open, and free nature @xcite . notably, we did not use any translation, ensuring that the output data sounded natural. moreover, an advantageous byproduct of this pipeline was the concurrent development of machine read-ing comprehension (mrc) and information retrieval (ir) datasets, requiring no extra human annotation. mrc is essential for ai to understand and analyze texts like a human reader @xcite , while ir is crucial for efficiently extracting relevant information from vast databases @xcite . we summarize our contributions as follows: the study by @xcite ) outlines the various question generation techniques used in existing kbqa datasets. for generating natural questions in our research, we adopted a question generation technique based on query suggestion, initially introduced by @xcite . this technique is effective for acquiring natural factoid questions likely to be posed to a qa system, similar to the approaches used in datasets like nq @xcite and wikiqa @xcite , which were built from questions asked to search engines. for template-based questions, our approach involved creating questions from predefined reasoning templates, a standard method in many kbqa datasets @xcite @xcite . several kbqa datasets used crowdsourced paraphrasing for question diversification @xcite @xcite . in contrast, our approach only automates this process by incorporating humans during final verification. ir many valuable resources for information retrieval in the polish language were recently created. the beir-pl @xcite benchmark was proposed as an automatic machine translation of the beir @xcite benchmark. this popular zero-shot retrieval benchmark was originally only for the english language. the mqupqa @xcite dataset is a composition of multiple already existing polish and multilingual datasets, like czywiesz @xcite , mkqa @xcite . additionally, the mqupqa dataset incorporates other automatic methods for question and answer generation, such as utilizing the generative capabilities of the gpt-3 model @xcite or employing templates inspired by the structure of wikipedia. the poleval (łukasz @xcite competition featured a passage retrieval task. it comprised three datasets from various domains: wikipedia-based, e-commerce shop faq, and legal questions. currently, a polish information retrieval benchmark (pirb) @xcite provides a platform to evaluate models across various datasets. the models referred to in this benchmark represent the current state-of-the-art in polish ir. mrc qa datasets often have a close relationship with ir datasets. the czywiesz dataset is based on the did you know? section of wikipedia, with provided answers and also relevant articles. another example is the polqa @xcite dataset, which is comprised of general questions from quiz shows annotated with relevant passages from wikipedia. the poquad @xcite dataset is structured around questions manually annotated to correspond with the best articles on wikipedia, mirroring the methodology of the squad @xcite dataset. contrastively, our dataset consists of naturally occurring questions, which are afterward annotated to relevant articles. ## definitions a common element in the tasks of kbqa, mrc, and ir is the textual question q. we denote the set of questions as q. despite query being common in the field of ir, we use question and query interchangeably, as our dataset's queries take the form of questions. kbqa we denote kg as a multi-relational heterogeneous graph @xmath0, r, t ), composed of three elements: a set of entities e, a set of relation predicates r, and a set of triples (facts) t . each triplet (h, r, t) ∈ t indicates a relation predicate r between two entities, a head entity h and a tail entity t, where h, @xmath1. in the kbqa task, a textual question q and associated topic entities e q ⊂ e are given. the objective is to retrieve answer entities a q ⊂ e that satisfy the question based on facts in the g. therefore, we denote kbqa dataset as mrc mrc aims to answer a textual question q based on a given text passage p q . we denote mrc dataset as d m @xmath2, p q , a q )}, where a q is the answer extracted from p q . ir the ir task focuses on finding a passage p from a large corpus relevant to a query q. the corpus c is defined as a set of passages, i.e., @xmath3, p 2 , ..., p n }. the ir dataset is denoted as d @xmath4, p q )}, where p @xmath5. ## construction pipeline this section introduces the construction pipeline for the pugg dataset, specifically designed to create a dataset with natural and factoid questions in a semi-automated manner. this approach significantly reduces the workload of human annotators. we outline the pipeline's fundamental design, presented in figure 1 , emphasizing its adaptability to various environments. while this part focuses on the general framework, specific implementation details, such as the models and algorithms used, will be discussed in section 5. question formulation the initial step of our pipeline involves acquiring a variety of natural factoid questions. we initiated our process using existing datasets to minimize the need for manual annotation. from previously existing qa datasets, we extract question prefixes ranging from basic ('who...', 'when...') to more specific ('which canadian athlete...', 'which theater co-created...'). then, the gathered prefixes are completed to formulate a set of questions. we can employ various methods, including rule-based approaches and language models @xcite , and for natural questions, we can also integrate external services. at this stage, we have a collection of question candidates q ′ , as some of which may be incorrect. these inaccuracies are not a concern at this point, as they will be filtered out during the human verification process, detailed in section 4. ## pipeline execution this section delves into the specific implementation of the construction pipeline for the pugg dataset, as previously outlined in a general framework in section 4. our implementation was adapted for polish nlp resources, which face challenges like limited task-specific pre-trained models and lower performance than english. ## experimental setup in this section, we outline the evaluation methodology used to assess the performance of baseline models on the pugg dataset. kbqa for the kbqa baseline, we evaluated the performance of kaping @xcite , a zero-shot framework that leverages an llm for retrieving answer entities. we slightly modified the knowledge retriever module by incorporating a step that retrieves a subgraph of the kg by traversing n edges, regardless of their direction, from the topic entities. our preliminary experiments demonstrated enhanced performance of the modification, showcasing improvements in both accuracy and processing speed. subsequently, we follow the original procedure, which involves retrieving k triples based on their textual embeddings. for embedding purposes, we utilized the mmlw-retrievalroberta-large retrieval model @xcite . we employed gpt-3-turbo as the llm, prompted with tailored queries as detailed in appendix f. the hyperparameters were selected empirically, setting @xmath6. as a metric, we employed accuracy, which measures the proportion of answers included in the llm's response for each question. it is calculated as follows: while @xcite also used accuracy, we refined it by calculating the correct answer proportion per example and excluding entities' aliases, providing a more realistic measure of kbqa efficacy. mrc for the mrc task, we selected models commonly used for the extractive question answering task. we trained and evaluated herbert @xcite models in an extractive fashion alongside a generative approach using the plt5 @xcite models. models were trained for 10 epochs and evaluated with squad metrics @xcite . exact match measures the percentage of predictions that exactly match the gold answer. the f1 metric measures the average token overlap between the prediction and ground truth answer, where both the prediction and answer are treated as a bag of tokens. ir recently, ir has gained significant interest within the polish research community, and many models have been developed and are open to the research community. these models have already been pre-trained on large datasets, which is why we did not fine-tune them to our dataset. the silver retriever @xcite model was trained on the maupqa dataset. we also evaluated e5 @xcite multilingual embedding models, which were trained on contrastive objectives on large weakly-labeled text pairs and afterward fine-tuned on existing datasets and are performing very well on polish texts. the mmlw retrieval models @xcite were trained on a parallel corpus with polish-english text pairs with a bge-large-en @xcite teacher model and are currently on the top of the pirb leaderboard. we also provide results of well-established bm25 @xcite baseline with morfologik foot_3 plugin in elasticsearch. additionally, we evaluated reranker models, focusing on those developed in the beir-pl benchmark and recent models that have appeared on the pirb leaderboard. those models were trained by dadas and grębowiec, 2024 with knowledge distillation from mt5-13b model introduced in the mmarco publication @xcite . for reranking, we employed the bm25 retrieval algorithm to select the top 100 passages for subsequent analysis. finally, we provide a score of the combination of the best retriever and reranker, namely multilingual-e5-large retriever and polishreranker-large-ranknet reranker, to evaluate currently the best ir pipeline available. we calculated the well-established metrics for the ir task: mrr@k, ndcg@k, recall@k @xcite . ## conclusion to address the significant resource gap for lowresource languages, our work introduces the pugg dataset, the first polish kbqa dataset, which also encompasses mrc and ir tasks. it consists of natural and template-based factoid questions. the dataset is the outcome of our proposed semiautomated construction pipeline, designed for lowresource environments. leveraging modern tools like llms as annotation assistants have significantly reduced the need for human labor. additionally, we developed few utility methods, such as entity linking, which are useful in various contexts. the pugg dataset and our pipeline's comprehensive implementation, findings, and detailed statistics from the pugg dataset construction provide valuable insights for future research. further-