W iki O mnia: filtration and evaluation of the generated QA corpus on the whole R ussian W ikipedia (2022)

Abstract: The General QA field has been developing the methodology referencing the Stanford Question answering dataset (SQuAD) as the significant benchmark. Compiling factual questions datasets requires manual annotations, limiting the training data’s potential size. We present the WikiOmnia dataset, a new publicly available set of QA pairs and corresponding Russian Wikipedia article summary sections, composed with a fully automated generation and filtration pipeline. To ensure high quality of generated QA pairs, diverse manual and automated evaluation techniques were applied. The WikiOmnia pipeline is available open-source and is also tested for creating SQuAD-formatted QA on other domains, like news texts, fiction, and social media. The resulting dataset includes two parts: raw data on the whole Russian Wikipedia (7,930,873 QA pairs with paragraphs for ruGPT-3 XL and 7,991,040 QA pairs with paragraphs for ruT5-large) and cleaned data with strict automatic verification (over 160,000 QA pairs with paragraphs for ruGPT-3 XL and over 3,400,000 QA pairs with paragraphs for ruT5-large).

## introduction generative abilities of large and high-performing pre-trained language models (lms) are widely investigated now, and special interest is aroused around generating datasets in a fully unsupervised way @xcite ). question answering (qa) datasets can be easily adjusted to the generation pipeline formats and become a source for training generative reading comprehension systems @xcite , dialogue systems @xcite , various tasks in the field of information retrieval for various languages @xcite . in this work, we present wikiomnia -the largest qa dataset for russian, obtained in a fullyautomated way. the dataset contains qa pairs for every article of russian wikipedia foot_0 , based on the summary sections. wikiomnia consists of 2 parts: apart from the data, we present a fully-automated pipeline for squad-like data generation for russian, based on generative part represented by the rugpt-3 xl 2 and rut5-large 3 models, and filtering part that includes russian bert 4 baseline and rich heuristic approach. all stated models were fine-tuned on sberquad @xcite ) that is based on the methodology of the original english squad @xcite . the whole automated and unsupervised generation and filtration pipeline was also tested for creating squadformatted qa on other domains: news texts, customer reviews, fiction, and social media. qa datasets generated with rugpt3xl and rut5 will be available on huggingface. after some related work overview in section 2, qa generation and filtration details are demonstrated in sections 3 and 4 respectively, followed by the corpus statistics in section 5. evaluation details are described in sections 6 and 7. ## related work the proposed work is based upon the recent architectures in transformer language modelling -gpt-3 @xcite and t5 @xcite , and solves a standard squad format problem, resulting in triplets "text paragraph -question based on paragraph -answer from the paragraph", see the following example: the following subsections of this section will break down previous work on the topic of qa datasets and their generation. datasets. for english, squad 1.1 @xcite consists of 107,785 question-answer pairs. squad 2.0, combines squad 1.1 questions with over 50,000 unanswerable questions (questions that cannot be answered based on the corresponding paragraph) @xcite . the following datasets for english were of comparable size or bigger. trivia qa @xcite includes 95 thousand qa pairs. natural questions (nq) @xcite contains questions from google search queries and corresponding spans from wikipedia articles as answers: 307,373 training examples, 7,830 development and 7,842 test examples. with the development of deep learning models, over 80 new datasets on qa and reading comprehension appeared in the past two years @xcite . several multilingual qa datasets contain russian examples: mkqa @xcite , tydi qa @xcite , a dataset for 7 languages @xcite . @xcite conducted experiments on the cross-lingual question answering dataset (xquad) benchmark that consists of a subset from squad v1.1 and its translations into 10 languages. wikipedia is commonly used as a relevant source for new datasets: for example, @xcite presented wikiqa dataset of qa pairs. it contains 3,047 questions from bing query logs, where each one is associated with a wikipedia page. manual annotation was used to check if a sentence from a page summary paragraph is the correct answer to the question. @xcite automatically generated 65m qa pairs from wikipedia paragraphs, using four steps with separate models: passage selection, possible answer extraction (with bert), question generation (with bart), and filtering. for russian, sberquad @xcite foot_3 is the main resource for the qa system development and evaluation. the dataset was created following the methodology of the original english squad, it contains about 50 thousand qa pairs. no bigger qa datasets for russian were created yet, and synthetic qa generation approaches were not applied to russian yet. although, pretrained language models, which are suitable for generative tasks, might help create better qa systems: rugpt-3 models (rugpt3xl, rugpt3large, rugpt3medium, rugpt3small) and rut5 models (rut5-base, rut5-large) exist for russian and can be implemented for the task. question-answer generation.. classical qa pair generation pipeline lets firstly choose among text points that should be asked, then ask a question based on these points, and after that find the most likely candidate from the answer spans in text @xcite @xcite . joint models, for question and answer generation can be also used @xcite -i.e. based on bart. @xcite proposed bertbased model which generates questions heuristically from summaries. some filtering steps can be done after creating qa too @xcite @xcite . @xcite proposed likelihood of the generated question-answers as a measure for it. in the recent years, pre-trained language models as unsupervised open-domain qa systems, that incorporate factual knowledge, were studied @xcite @xcite and criticized @xcite . other pre-trained language models were also examined for the task: @xcite fine-tuned bart to answer closedbook questions, and wang et al. (2020) studied gpt-2-based models performance for constructing knowledge graphs. to the best of our knowledge, the only approach to gpt-based qa generation and filtration was suggested in @xcite , who used a qa generation pipeline to generate diverse question-answer pairs from unlabeled text corpus. for question generation, gpt-2 small model, fine-tuned on squad 1.1 training dataset, was used. to filter out lowquality generated data, fine-tuned bert-based qa model utilizing the squad 1.1 dataset was used: examples were kept if f1 similarity score between the answer span and the answer span predicted by bert-based qa was above 0.9. the performance of question generation was evaluated by bleu, rouge-l, meteor metrics. however, the approach was examined only for english. ## implementation details we used the biggest freely available russian gpt3 model: rugpt-3 xl. the model was trained using deepspeed and megatron code and had sparse attention blocks. maximal sequence length for generation was 2048 tokens. we fine-tuned the model on sberquad dataset with the following parameters: batch-@xmath0, sequence @xmath1, learning @xmath2.000015. the model fine-tuning required 10 gpus per worker, and it took 135,000 iterations. after that we ran parallel qa generation with the parameters: maximal @xmath3, beam search with 7 as a number of beams, all 3grams can only occur once, repetition @xmath4. we also fine-tuned rut5-large model for russian on sberquad dataset with such parameters: number of @xmath5, maximal @xmath6, batch @xmath7, number of @xmath8. for rugpt-3 xl, we turned each example into a line starting with a special text beginning token (<[text]>), a text, then a special question beginning token (<[question]>), a question, a special answer beginning token (<[answer]>), and, finally, an answer, followed with the end-ofsequence special token. for rut5-large, we presented each example in the same way, but special text beginning, question beginning and answer beginning tokens were in russian. both models were fine-tuned to generate 3 qa pairs for a text. for qa generation we crawled all wikipedia for the russian language (up to march 2021) -2,682,680 articles in general. we took only text from summary sections in every wikipedia article. based on page categories, we excluded disambiguation articles from the data. then we kept wikipedia article categories for each summary, for filtration and analysis purposes. for processing purposes, we splitted all wikipedia data into 20 batches. both for rugpt-3 xl and for rut5-large, we generated 3 qa pairs per summary. so the dataset contains summaries, qa pairs for them, and additional information, such as page title and corresponding wikipedia categories. all qa pairs for a summary are included in one batch, and each summary appeared in the wikipedia summaries dataset only once. the dataset is presented in 20 batches, it lets use any 18 batches as train set, and the remaining two batches as development set and test set, if needed. both rugpt-3 xl and rut5-large fine-tuning, generation, filtration and evaluation tasks were performed on 4 tesla v100 gpu (32gb ram) server and in google colab. ## filtration of generated data inspired by @xcite , we applied a set of hand-crafted heuristics to filter out generated qa pairs of poor quality in the following steps, based on manual evaluation (see subsection 6.1.). 1. first of all, we dropped qa pairs with more than one interrogative pronoun in a question. after that we left strings with exact match between lemmatized generated answer and bert model answer, with intersection of lemmas between two answers over 70%. this threshold was chosen manually based on the analysis of one data sample -batch 2 (random 50,000 examples from 90,927 summaries). several additional options were implemented and can be used too, but they were not included into the final heuristics version for this specific task after the manual analysis (see subsection 6.1.): 1) rouge @xcite , meteor @xcite and bleu @xcite metrics in the second step. for each pair among 'gold' answer -generated answer, question -generated answer, textgenerated answer, text -question, three metrics for lemmatized strings were calculated. the mean result for each pair was counted, and qa pairs where values were less than the corresponding thresholds 60%, 50%, 40%, were removed. 2) matching persons and locations separately instead of the third step. 3) checking if the 'gold' bert model score is over 0.99, filtering out complicated examples. 4) calculating if word mover's distance between generated answer and 'gold' answer is between 1.1 and 1.5, using the fasttext model for russian 9 . the overall pipeline is presented in figure 1 . ## corpus statistics we describe the main characteristics of the resulting corpus. for synthetic data, it is especially important to control their diversity and frequency of words. basic statistics. for rugpt-3 and rut5 generated data, generation and filtration results in a detailed way are presented in tab. 1 for wikipedia batches 1-5. we see that quality of rut5 generated qa pairs is much better; however, both models require a filtration step for a 'clean' dataset version. in general, the raw dataset version for rugpt-3 contained 7,930,873 examples, and filtered version had more than 160,000 examples. for rut5, the raw dataset version consisted of 7,991,040 examples; filtered version included over 3,400,000 examples. the most frequent words in questions and answers for all 4 setups do not differ: they are about years, names, places, numbers etc. for instant, in questions the most frequent lemmas are 'god' (year), 'nazyvat'sja' (to be named), 'rodit'sja' (to be born), 'skol'ko' (how many), 'gorod' (town), and in answers the most frequent lemmas are 'god' (year), 'rajon' (district), 'chelovek' (human, person), 'gorod' (town), 'rossijskij' (russian). 10 average length for rut5 before and after filtration is about 52 characters (7 tokens) for questions and about 24 characters (4 tokens) for answers. for rugpt-3, average length before filtration is 47 characters (7 tokens) for questions and 19 characters (3 tokens) for answers; after filtration its is slightly shorter: 46 characters (7 tokens) for questions and 12 characters (2 tokens) for answers. in sberquad train set, questions (64.4 characters, 8.7 tokens) and answers (25.9 characters, 3.7 tokens) are longer @xcite . self-bleu for questions diversity. we computed self-bleu as a metric of diversity for generated questions, as they are more specific for a model than answers that depend on questions. we followed @xcite approach that is based on @xcite . it yields how one sentence (a question) resembles other generated questions in the collection: for each question as a hypothesis and all other questions as references, the bleu score is calculated. due to computational reasons, we took random samples of 5,000 examples from raw rugpt-3 data (batch 2), raw rut5 data (batch 2), filtered rugpt-3 data (including batch 2), and filtered rut5 data (including batch 2). to compare, we measured self-bleu for a random sample of 5,000 questions from the original sberquad too. for each text, there was only one corresponding question in the data. questions were lemmatized before calculation. median self-bleu scores are presented in tab. 2, where lower self-bleu scores represent higher diversity. sberquad data demonstrated the highest diversity. while rut5 generated questions imply higher diversity after filtration, for rugpt-3 the most relevant questions that remain after filtration are less diverse. wh-questions ratio. we also use wh-questions during the final stage, we also checked manually, in addition, several random samples of 10,000 qa pairs for specific evaluation tasks. wikipedia topics before and after filtration. to estimate if filtration ratio varies for different topics, we checked the ratio of examples that remained after filtration for various wikipedia categories groups (on the example of batches 1-5): history events, famous persons biographies, plants, technical descriptions, geography, mathematics, sports, actors, and movies. categories for the selected topics were grouped using heuristics rules, based on saved wikipedia category names for each example (one example could have multiple categories). both rugpt-3 and rut5 generated qa pairs showed the best results for articles about sports, perhaps due to simple and well-structured summaries. error analysis showed that rugpt-3 also performed rather well on history and plants topics, but answers to the correct questions, also correct in meaning, did not match the 'gold' answers well. in addition to sports, rut5 qa pairs for technical, history and geography articles also yielded higher quality, they did not contain additional information not from the corresponding summaries, unlike rugpt-3. in general, for technical topics (i.e. computer science), generated qa pairs yielded worse quality before filtration than for other topics. example of an erroneous qa pair generation with rut5, detected by filtering: the filtered dataset may still contain two types of errors that were not detected by the filters: 1) questions about information that was not presented in a summary (0.008% for rugpt-3, based on a random example of 10,000 qa pairs); 2) erroneous answers with numbers (if not years). ## pipeline evaluation on other domains for evaluation purposes, we also tested the full pipeline on data samples of four other text genres in russian: news stories, social media posts, product reviews, and fiction texts. each sample has 2,000 examples taken randomly from the following datasets: 1) news from the newspaper gazeta, foot_10 @xcite ) with text lengths up to 3,500 characters; 2) social media texts from the taiga corpus foot_11 @xcite , with texts lengths up to 3,000 characters; 3) reviews from the dataset foot_12 on product reviews about clothes from an e-commerce website @xcite , with text lengths over 500 characters and up to 1,007 characters, as these texts are rather short; 4) fiction texts from the collection of russian classical literature texts foot_13 : fragments from texts, with text lengths up to 3,000 characters. for every text, three qa pairs were generated. filtration steps were the same as for qa pairs based on wikipedia summaries. after filtration, we got the following results for rugpt3xl: 497 pairs remained for fiction texts, and 559 pairs were left for news texts. for reviews, the pipeline performed in the best way: 1379 pairs were left. the worst performance was for social media: only 154 pairs remained. rut5-large also yielded good performance on reviews: 1,542 pairs were left after filtration. the explanation might be that review texts as a genre usually have definite patterns and structure. the worst rut5-large results were also for social media: only 945 pairs remained. social media texts looked mostly like opinionated pieces where it would be hard to create qa pairs manually too. both pipelines, for rugpt3 xl and rut5-large, can be generalized comparatively well to other genres. although rut5-large performed generally better on all four genres, the results mostly differed on news texts: 3,204 texts remained after filtering. other filtration techniques should be investigated, handling the remaining errors, i.e. how to check quality of numerical answers (especially by rugpt-3), or how to check question and answer similarity to the corresponding summary, considering paraphrases. reasons of the results of the automated evaluation on sberquad development and test sets should be also explored further. the dataset implementation for various and diverse tasks and its evaluation on them remains a separate point for further research. ## conclusions we propose wikiomnia, the new largest questionanswering dataset for russian: it contains qa pairs and corresponding russian wikipedia article summaries. it can be used to improve the quality of monolingual and multilingual information re-trieval systems, open domain question answering, etc. quality of generated qa pairs in the filtered part of the dataset is ensured by diverse automated filtration techniques, manual and automated evaluation. we also present the automated generation and filtration pipeline that can be applied to various sources of text data, including expanding the applicability of qa systems to news data, fiction, reviews. we welcome researchers in the fields of information retrieval and language technology to use both the dataset to train the models, and the pipeline to expand the capabilities and robustness of the existing qa systems. we invite the community to reproduce the work on materials of other languages, using multilingual models and existing baselines.