B io MRC : A Dataset for Biomedical Machine Reading Comprehension (2020)

Abstract: We introduceBIOMRC, a large-scale cloze-style biomedical MRC dataset. Care was taken to reduce noise, compared to the previous BIOREAD dataset of Pappas et al. (2018). Experiments show that simple heuristics do not perform well on the new dataset and that two neural MRC models that had been tested on BIOREAD perform much better on BIOMRC, indicating that the new dataset is indeed less noisy or at least that its task is more feasible. Non-expert human performance is also higher on the new dataset compared to BIOREAD, and biomedical experts perform even better. We also introduce a new BERT-based MRC model, the best version of which substantially outperforms all other methods tested, reaching or surpassing the accuracy of biomedical experts in some experiments. We make the new dataset available in three different sizes, also releasing our code, and providing a leaderboard.

## introduction creating large corpora with human annotations is a demanding process in both time and resources. research teams often turn to distantly supervised or unsupervised methods to extract training examples from textual data. in machine reading comprehension (mrc) @xcite , a training instance can be automatically constructed by taking an unlabeled passage of multiple sentences, along with another smaller part of text, also unlabeled, usually the next sentence. then a named entity of the smaller text is replaced by a placeholder. in this setting, mrc systems are trained (and evaluated for their ability) to read the passage and the smaller text, and guess the named entity that was replaced by the placeholder, which is typically one of the named entities of the passage. this kind of question answering (qa) is also known as cloze-type questions @xcite . several datasets have been created following this approach either using books @xcite or news articles @xcite . datasets of this kind are noisier than mrc datasets containing human-authored questions and manually annotated passage spans that answer them @xcite @xcite . they require no human annotations, however, which is particularly important in biomedical question answering, where employing annotators with appropriate expertise is costly. for example, the bioasq qa dataset @xcite currently contains approximately 3k questions, much fewer than the 100k questions of a squad @xcite , exactly because it relies on expert annotators. to bypass the need for expert annotators and produce a biomedical mrc dataset large enough to train (or pre-train) deep learning models, @xcite adopted the cloze-style questions approach. they used the full text of unlabeled biomedical articles from pubmed central, foot_0 and metamap @xcite to annotate the biomedical entities of the articles. they extracted sequences of 21 sentences from the articles. the first 20 sentences were used as a passage and the last sentence as a cloze-style question. a biomedical entity of the 'question' was replaced by a placeholder, and systems have to guess which biomedical entity of the passage can best fill the placeholder. this allowed @xcite to produce a dataset, called bioread, of approximately 16.4 million questions. as the same authors reported, however, the mean accuracy of three humans on a sample of 30 questions from bioread was only 68%. although this low score may be due to the fact that the three subjects were not biomedical experts, it is easy to see, by examining samples of bioread, that many examples of the dataset do in this paper, we introduce biomrc, a new dataset for biomedical mrc that can be viewed as an improved version of bioread. to avoid crossing sections, extracting text from references, captions, tables etc., we use abstracts and titles of biomedical articles as passages and questions, respectively, which are clearly marked up in pubmed data, instead of using the full text of the articles. using titles and abstracts is a decision that favors precision over recall. titles are likely to be related to their abstracts, which reduces the noise-tosignal ratio significantly and makes it less likely to generate irrelevant questions for a passage. we replace a biomedical entity in each title with a placeholder, and we require systems to guess the hidden entity by considering the entities of the abstract as candidate answers. unlike bioread, we use pubtator @xcite , a repository that provides approximately 25 million abstracts and their corresponding titles from pubmed, with multiple annotations. 2 we use dnorm's biomedical entity annotations, which are more accurate than metamap's @xcite . we also perform several checks, discussed below, to discard passage-question instances that are too easy, and we show that the accuracy of experts and nonexpert humans reaches 85% and 82%, respectively, on a sample of 30 instances for each annotator type, which is an indication that the new dataset is indeed less noisy, or at least that the task is more feasible for humans. following @xcite , we release two versions of biomrc, large and lite, containing 812k and 100k instances respectively, for researchers with more or fewer resources, along with the 60 instances (tiny) humans answered. random samples from biomrc large where selected to create lite and tiny. biomrc tiny is used only as a test set; it has no training and validation subsets. in all versions of biomrc (large, lite, tiny), the entity identifiers of pubtator are replaced by pseudo-identifiers of the form @entityn (fig. 1 ), as in the cnn and daily mail datasets @xcite . we provide all biomrc versions in two forms, corresponding to what pappas et al. (2018) call settings a and b in bioread. 6 in setting a, each pseudo-identifier has a global scope, meaning that each biomedical entity has a unique pseudo-identifier in the whole dataset. this allows a system to learn information about the entity represented by a pseudo-identifier from all the occurrences of the pseudo-identifier in the training set. for example after seeing the same pseudo-identifier multiple times a model may learn that it stands for a drug, or that a particular pseudo-identifier tends to neighbor with specific words. then, much like a language model, a system may guess the pseudoidentifier that should fill in the placeholder even without the passage, or at least it may infer a prior probability for each candidate answer. in contrast, setting b uses a local scope, i.e., it restarts the numbering of the pseudo-identifiers (from @en-tity0) anew in each passage-question instance. this forces the models to rely only on information about the entities that can be inferred from the particular passage and question. this corresponds to a nonexpert answering the question, who does not have any prior knowledge of the biomedical entities. ## experiments and results we experimented only on biomrc lite and tiny, since we did not have the computational resources to train the neural models we considered on the large version of bioread. @xcite also reported experimental results only on a lite version of their bioread dataset. we hope that others may be able to experiment on biomrc large, and we make our code available, as already noted. ## related work several biomedical mrc datasets exist, but have orders of magnitude fewer questions than biomrc @xcite or are not suitable for a cloze-style mrc task @xcite @xcite . the closest dataset to ours is clicr @xcite , a biomedical mrc dataset with cloze-type questions created using full-text articles from bmj case reports. 13 clicr contains 100k passage-question instances, the same number as biomrc lite, but much fewer than the 812.7k instances of biomrc large. šuster et al. used clamp @xcite to detect biomedical entities and link them to concepts of the umls metathesaurus @xcite . cloze-style questions were created from the 'learning points' (summaries of important information) of the reports, by replacing biomedical entities with placeholders. šuster et al. experimented with the stanford reader @xcite and the gated-attention reader @xcite , which perform worse than aoa-reader @xcite . the qa dataset of bioasq @xcite contains questions written by biomedical experts. the gold answers comprise multiple relevant documents per question, relevant snippets from the documents, exact answers in the form of entities, as well as reference summaries, written by the experts. creating data of this kind, however, requires significant expertise and time. in the eight years of bioasq, only 3,243 questions and gold answers have been created. it would be particularly interesting to explore if larger automatically generated datasets like biomrc and clicr could be used to pre-train models, which could then be fine-tuned for human-generated qa or mrc datasets. outside the biomedical domain, several clozestyle open-domain mrc datasets have been created automatically @xcite @xcite , but have been criticized of containing questions that can be answered by simple heuristics like our basic baselines @xcite . there are also several large open-domain mrc datasets annotated by humans @xcite @xcite @xcite . to our knowledge the biggest human annotated corpus is google's natural questions dataset @xcite , with approximately 300k human annotated examples. datasets of this kind require extensive annotation effort, which for open-domain datasets is usually crowd-sourced. crowd-sourcing, however, is much more difficult for biomedical datasets, because of the required expertise of the annotators. ## conclusions and future work we introduced biomrc, a large-scale cloze-style biomedical mrc dataset. care was taken to reduce noise, compared to the previous bioread dataset of @xcite . experiments showed that biomrc's questions cannot be answered well by simple heuristics, and that two neural mrc models that had been tested on bioread perform much better on biomrc, indicating that the new dataset is indeed less noisy or at least that its task is more feasible. human performance was also higher on a sample of biomrc compared to bioread, and biomedical experts performed even better. we also developed a new bert-based model, the best version of which outperformed all other meth-ods tested, reaching or surpassing the accuracy of biomedical experts in some experiments. we make biomrc available in three different sizes, also releasing our code, and providing a leaderboard. we plan to tune more extensively the bertbased model to further improve its efficiency, and to investigate if some of its techniques (mostly its max-aggregation, but also using sub-tokens) can also benefit the other neural models we considered. we also plan to experiment with other mrc models that recently performed particularly well on opendomain mrc datasets @xcite . finally, we aim to explore if pre-training neural models on bioread is beneficial in human-generated biomedical datasets @xcite .