A Data-centric Framework for Improving Domain-specific Machine Reading Comprehension Datasets (2023)

Abstract: Low-quality data can cause downstream problems in high-stakes applications. Data-centric approach emphasizes on improving dataset quality to enhance model performance. High-quality datasets are needed for general-purpose Large Language Models (LLMs) training, as well as for domain-specific models, which are usually small in size as it is costly to engage a large number of domain experts for their creation. Thus, it is vital to ensure high-quality domain-specific training data. In this paper, we propose a framework for enhancing the data quality of original datasets. (Code and dataset are available at https://github.com/IvaBojic/framework). We applied the proposed framework to four biomedical datasets and showed relative improvement of up to 33%/40% for fine-tuning of retrieval/reader models on the BioASQ dataset when using back translation to enhance the original dataset quality.

## introduction data-centric approach emphasizes the collection of high-quality data as a centrally important step in the model development @xcite . while model-centric approaches were more prominent in the past, recently data-centric approaches are also gaining importance @xcite . this trend was especially emphasized since 2021 when andrew ng launched his campaign for a more data-centric approach to ai by starting the data-centric competition foot_1 , which encouraged participants to increase accuracy by solely improving the datasets while keeping the model fixed. large language models (llms), such as generative pre-trained transformer 3 (gpt-3) @xcite , generate text that is grammatically correct, fluent, and informative. however, there is little to no control over the data that were used for model training. consequently, llms are prone to hallucinating and providing untruthful outputs @xcite . ironically, this reflects llms' ability to be better at learning the training distribution and consequently follow inverse scaling law @xcite . and while some of the recent research efforts are focused on providing explanations of where the llm's outputs came from @xcite , such research is in its infancy. in this work, we focus on language models with a transformer encoder architecture such as bert @xcite , that extract relevant outputs from a domain-specific evidence-based text corpus. deep neural networks trained on domain-specific datasets, including those used in natural language processing (nlp), are most heavily dependent on the quality of the training dataset, which is usually small in size @xcite as it is costly to engage a large number of domain experts for annotation. it is thus important to create high-quality training data for language models to perform better. in this paper, we propose a data-centric framework for machine reading comprehension (mrc) datasets that increases the original dataset quality by both (i) keeping the size of the original dataset fixed, and (ii) augmenting the original dataset by adding new training samples. mrc is a natural language understanding (nlu) task. its goal is to answer questions based on the information provided in a passage @xcite . training datasets for mrc models come in the form of triplets: passage (i.e., positive context), question, and answer. typically, the mrc pipeline works in two phases, where a passage retriever is followed by a passage reader @xcite . for a given question, the retriever first extracts a set of relevant passages from a knowledge base (i.e., text corpus), and then the reader selects an answer (e.g., text span) from one of the retrieved passages @xcite . ## related work data-centric approaches can be divided into (i) data quality enhancement methods that keep the original size of the dataset fixed (e.g., data filtering or label consistency checking), and (ii) data augmentation methods that increase the original dataset size (i.e., adding more training samples). results from the literature on using data-centric approaches to improve model performance in mrc are inconclusive. several studies have reported that data filtering can lead to significant model improvements @xcite @xcite . however, this might not hold if data are filtered in a random way @xcite . additionally, while increasing labelling consistency and excluding or cleaning noisy data points were shown to improve model performance on the bioasq dataset @xcite , shortening answers in aasdqa led to a decrease of f1-score by 4% @xcite . adaptation of data augmentation is still comparatively less explored in nlp @xcite , with a body of work presenting positive results @xcite @xcite as well as papers showing little or no improvements for the given task @xcite @xcite . to the best of our knowledge, this paper is the first that proposes framework for data quality enhancement for improving domain-specific mrc datasets by (i) keeping the original dataset size of data the same and (ii) increasing the original dataset size using augmentation methods. our framework includes methods for (i) a better selection of negative passages for retriever training, and (ii) reformulating questions using paraphrasing, word substitution, and back translation. paraphrasing, word substitution, and back translation were previously used as data augmentation methods in various nlp tasks @xcite @xcite . however, those papers did not look at how each of these methods enhanced the original dataset without increasing its size. keeping the size of the dataset fixed is necessary in resources-constrained scenarios, as the resources (e.g., time) needed for finetuning are proportional to the size of training sets. moreover, previous studies did not present a costbenefit analysis of the resources needed to generate extended training sets and perform fine-tuning processes in comparison with the performance increase. ## a data-centric framework for mrc in our framework, we first generate new training sets using four data quality enhancement methods. we then fine-tune retrieval and reader models on each new training set individually. secondly, we fine-tune retrieval/reader models continually starting from the best individual checkpoint using enhanced training sets that showed improvements in the first step. finally, we create new augmented datasets by concatenating all training sets if they show fine-tuning improvements in the first step. labels in mrc datasets are triplets which include a passage, a question, and an answer. in mrc datasets, an answer is part of a passage which is also called a positive context. to fine-tune a retrieval model as proposed in @xcite , it is necessary to not only provide a positive context of passages that contains the answer to a given question, but also negative contexts. some previous work employed a method of randomly selecting negative contexts from a text corpus @xcite . here we propose a method to improve the random selection of negative contexts. one of the problems with manually collecting labels for mrc datasets is that questions are too similar to their answers @xcite . to solve this, we investigate the use of three different methods applied to the original set of questions: (i) paraphrasing -we use two different language models fine-tuned for paraphrasing; (ii) word substitution -we use two libraries: one to extract a keyword from a given question and another to obtain a list of synonyms for the chosen keyword; and (iii) back translation -we use 25 different machine translation language models to translate a source text into another language, and back into the original language. ## datasets to test our framework, we made adjustments (see appendix a) to four biomedical datasets: bioasq @xcite , covid-qa @xcite , cpgqa @xcite and sleepqa @xcite . we refer the reader to original bioasq dataset contained over 3k manually-annotated biomedical labels. questions in these datasets came in four different flavours: factoid, list, yes/no, and summary. we extracted only factoid questions for which the exact answer can be found in the positive context. original covid-qa dataset was annotated by biomedical experts and contained 2k labels on covid-19 pandemic-related topics. original cpgqa dataset contained 1k manually annotated labels in the domain of clinical practice guidelines. original sleepqa was a manually annotated dataset in the sleep domain with 5k labels. ## evaluation we evaluated our framework by performing finetuning of retrieval and reader models using bi-olinkbert @xcite and biobert bioasq foot_7 respectively. we used biolinkbert for retrieval model fine-tuning as it was recently shown to achieve state-of-the-art performance on lowresource bio-mrc tasks @xcite . biobert bioasq was used for fine-tuning of reader model as proposed in @xcite . intrinsic evaluation of fine-tuned models was done using automatic metrics on test sets: recall@1 for retrieval and exact match (em) for reader models.