A Question Answering Benchmark Database for H ungarian (2023)

Abstract: Within the research presented in this article, we created a new question answering benchmark database for Hungarian called MILQA. When creating the dataset, we basically followed the principles of the English SQuAD 2.0, however, like in some more recent English question answering datasets, we introduced a number of innovations beyond SQuAD: e.g., yes/no-questions, list-like answers consisting of several text spans, long answers, questions requiring calculation and other question types where you cannot simply copy the answer from the text. For all these non-extractive question types, the pragmatically adequate form of the answer was also added to make the training of generative models possible. We implemented and evaluated a set of baseline retrieval and answer span extraction models on the dataset. BM25 performed better than any vector-based solution for retrieval. Cross-lingual transfer from English significantly improved span extraction models.

## introduction in this research, our goal was to create a hungarian question answering dataset that enables the training of hungarian question answering systems and the automatic evaluation of their performance. in the paper we first review existing systems and resources, then describe the annotation procedure we followed and features of the dataset, closed by the presentation and evaluation of baseline retrieval and extractive answer span extraction models trained and tested on the dataset. 1 the dataset and trained models can be found on github and the hugging face model hub searching for the term milqa. ## background early question answering databases were either very small in size or did not contain questions in the form of grammatical interrogative sentences, but they consisted of so-called cloze-type "questions": these are declarative sentences, a part of which is masked and this part must be filled in based on the text. the latter resources were machine-generated, so they were easy to create, but the sentences containing the masked part do not resemble real questions at all. one of the most important milestones in the series of databases used for training question answering systems was the english squad database (stanford question answering dataset) @xcite created at stanford university. this is a much larger database than the previous ones, containing more than 108,000 question-answer pairs in its first version, which was later further supplemented with questions that could not be answered based on the given text passage (151,000 questions, @xcite ) in the second version. the publicly available training and tuning set contains 143,000 (93,000 answerable and 50,000 unanswerable) questions. in addition to its size, this resource can be considered a breakthrough because, on the one hand, unlike previous resources containing cloze-type questions (e.g. cnn/daily mail @xcite ), it actually contained well-formed questions and on the other hand, it was not built of multiple-choice questions (e.g. mctest @xcite or wikiqa @xcite ). furthermore, it gave a huge boost to the development of question answering systems. among question answering datasets and systems, we can distinguish extractive and generative approaches. in the case of the former, the answer is simply a highlighted part of the text (as if we were working with a text highlighter, this is what squad is like), and in the case of the latter, the answer is actually formulated in well-formed human language (e.g. ms marco @xcite , narrativeqa @xcite ). in addition, some of the qa databases contain questions that require the execution of multi-step inference chains to arrive at an answer (multi-hop/multi-step qa tasks). this not only means a greater complexity of the underlying logical derivation, but this type of task can also go beyond the level of individual documents or text fragments, if the given question can only be answered by combining the information contained in several different documents or text fragments (e.g. hotpotqa @xcite , narrativeqa @xcite ). in the case of the multi-step question answering tasks and squad, it was the task of the annotators to formulate questions based on given texts. companies operating large search engines, however, created resources in which relevant documents were collected based on frequent questions entered into the search engine, and the annotators selected or formulated the answers using these results. natural questions (nq, @xcite ) based on questions entered into the google search engine belongs to the former extractive type. in nq, the documents used as context were wikipedia articles, similar to squad. the ms marco qna dataset based on microsoft bing queries belongs to the latter abstractive/generative type @xcite . resources based on existing quiz and literacy question sets were also created using similar web query techniques (e.g. triviaqa @xcite ). perhaps one of the sources of squad's popularity was that it assumes a relatively simplistic model, according to which a single coherent span of text can be selected as an answer for each answerable question, which greatly simplifies the implementation of squad-based systems. this restriction can be implemented well if the annotators are instructed to ask only questions that can be answered in this manner. however, in the case of a non-negligible part of real-life questions, the answer is some kind of list, the elements of which do not necessarily occupy a single contiguous span of the text. in such cases, a single span including all relevant answers may contain a significant amount of text that is irrelevant to the answer. for example, in the natural questions dataset based on real questions, the an-swer is not a single span for 6.9% of the questions. in the case of squad, the context of the questions (the part of the text in which the answer to the question must be found) has a relatively limited length: between 150 and 4000 characters, with an average of 740 characters, which also limits the complexity of the task. yes-no questions naturally occur in datasets similar to nq (natural questions: 2.5%) that originate from actually asked questions. typically, the answer to these questions is not a selected part of the text, but a (usually probable, not clear) yes/no answer follows from a relevant part of the text. there are also datasets specifically containing only yesno questions (e.g. boolq @xcite , also based on wikipedia, amazonyesno @xcite , based on texts related to amazon product reviews, or the biomedical pubmedqa based on article abstracts @xcite ). at the same time, boolq and amazonyesno show significant overlap with the yes-no questions in naturalquestions and amazonqa @xcite databases (in the case of amazon resources, there is essentially a subset relationship). in biomedical question sets of "natural origin", similarly to nq, the proportion of "non-squad-compatible" questions is often much higher than previously mentioned in relation to the nq database. for example, in the case of the clinical questions collection (cqc) data set @xcite @xcite containing questions formulated by actual practicing doctors during their daily professional activities and the pubmed query log dataset @xcite composed of questions formulated by pubmed users in a single day, the proportion of yesno questions is 28.1%, and that of list-type answers is 21.9% @xcite . in addition to the lack of list-type answers and the scarcity of yes-no questions, another problem with extractive datasets arises from the fact that questions about a given text often do not use the same words that appeared in the original context. during the compilation of squad, annotators were encouraged to paraphrase the part of the question anchored to the context when formulating the questions, and not simply copy it. this in itself is not necessarily a serious problem for neural models based on current pre-trained language models, since these usually have sufficiently abstract internal semantic representations to often avoid that paraphrases confuse them. however, if we use a verb in the question different from the one in the original context, then this often involves a different argument frame, which means that the given expression should often appear in the answer in a form different from that in the original text. in the case of squad, the solution to this problem was that prepositions were not made part of the answer, but only the minimal lexical content (annotators were instructed to do so, see fig. 1 .). in the case of english, an essentially isolating language, this solves the above problem in most cases, but at the price that the answer of the system is often not formulated in a pragmatically appropriate form (the latter would include the preposition). in the case of languages, where case is marked morphologically, this solution obviously does not work. in such cases an extractive qa system will definitely give an inadequate answer, because it returns the answer with the original case appearing in the text. at the same time, this does not represent a real problem if the answer is presented as highlighted text in context, since in this case the user does not feel that the machine "answered in a strange manner", but rather that it "highlighted the answer correctly in the text". if, however, the answer is presented as an answer, then it is definitely necessary to move on and use a generative model. we illustrate the problem with an example in hungarian. in the context of péternek az idegeire ment a zaj. 'the noise got on peter's nerves.' (here péter 'peter' is in the dative case), the adequate short answer to the question kit idegesített a zaj? 'who was annoyed by the noise?' would be pétert (in accusative), but this cannot be extracted in this form from the original context. here, the complete sentence would be an adequate (but not minimal) answer to the question. however, this is often not the case, especially when the original context contains the answer in a derived form. in the context a duna európa második leghosszabb folyama az oroszországi volga után. 'the danube is the second-longest river in europe, after the volga in russia.', the adequate answer to the question melyik országban található európa leghosszabb folyama? 'in which country is the longest river in europe?' would be oroszországban 'in russia' (inessive of oroszország). the word form oroszországi in the original context is an adjective derived from the name of the country (and as such, it is decapitalized). here, the original sentence would not be an adequate answer, either. there are some additional question types: question-answer pairs that require counting, the execution of some arithmetic operations, or comparison (how many, how much, which is the most... etc.), which are not a problem even for people with minimal education, but the models must be specially prepared to perform such tasks in order to prevent the machine from failing miserably. the drop @xcite ) question-answer database primarily focuses on such questions. some resources approach the problem of answering questions in the context of a dialogue. the questions are often ambiguous or incomplete, and additional information is needed to answer them. data sets such as sharc @xcite aim at modeling such situations. training the groundbreaking chatgpt model of openai required extensive dialog modeling resources as well as further human-in-the-loop annotation for reinforcement learning. ## a new hungarian question answering benchmark dataset within the research presented in this paper, we created the first publicly available extractive question answering benchmark dataset in hungarian. when creating the database, we largely followed the principles of squad 2.0, however, similar to some of the more recent english q&a databases (natural questions, ms marco, drop) mentioned in section 2, we introduced a number of new question-answer types, which contain more difficult but more realistic tasks. similarly to squad 2.0, the corpus is characterized by the following: a) high-quality wikipedia articles serve as context for the questions, b) factual (not opinion-type) questions are included, c) also contains questions that are not answered in the given text, d) in the original text, we marked the shortest possible answer to the given question (if any), e) when formulating the questions, we paraphrased the original text, so in most cases the answer cannot be found using a lexical search, f) the questions can be interpreted not only in the context of the given text, but also as independent questions (e.g. they do not contain unanchored pronouns). compared to squad, we introduced the follow-ing innovations (special question types are explicitly marked in the database): a) there may be more than one short answer to the given question in the given text (list type answer, approx. 8.5% of the answered questions). b) in addition to the short answer, we also gave a long answer, which includes all the relevant information necessary to answer the question (min. 1 clause, often several sentences). ## models and performance we created and evaluated a number of document retrieval and reader (answer span extraction) models using the dataset. for document retrieval, we evaluated both traditional lexical and various vectorbased retrieval models. for span extraction, we finetuned both a monolingual hungarian model and multilingual models. we also tested to what extent cross-lingual transfer from english can be applied to this specific task. ## conclusions we presented a new qa benchmark database in hungarian, that in several aspects, goes beyond squad-type datasets: it is not limited to single contiguous short extractive answer spans, contains yes/no questions, non-contiguous multispan short answers, long answers, questions requiring arith-