CODA-19 : Using a Non-Expert Crowd to Annotate Research Aspects on 10,000+ Abstracts in the COVID-19 Open Research Dataset (2020)

Abstract: This paper introduces CODA-19, a human-annotated dataset that codes the Background, Purpose, Method, Finding/Contribution, and Other sections of 10,966 English abstracts in the COVID-19 Open Research Dataset. CODA-19 was created by 248 crowd workers from Amazon Mechanical Turk within 10 days, and achieved labeling quality comparable to that of experts. Each abstract was annotated by nine different workers, and the final labels were acquired by majority vote. The inter-annotator agreement (Cohen’s kappa) between the crowd and the biomedical expert (0.741) is comparable to inter-expert agreement (0.788). CODA-19’s labels have an accuracy of 82.2% when compared to the biomedical expert’s labels, while the accuracy between experts was 85.0%. Reliable human annotations help scientists access and integrate the rapidly accelerating coronavirus literature, and also serve as the battery of AI/NLP research, but obtaining expert annotations can be slow. We demonstrated that a non-expert crowd can be rapidly employed at scale to join the fight against COVID-19.

## introduction as covid-19 spreads across the globe, it's hard for scientists to keep up with the rapid acceleration in coronavirus research. researchers have thus teamed up with the white house to release the covid-19 open research dataset (cord-19) @xcite , containing 130,000+ related scholarly articles (as of august 13, 2020). the open research dataset challenge has also been launched on kaggle to encourage researchers to use cutting-edge techniques to gain new insights from these papers (ai, 2020). to comprehend 1 covid-19 research aspect dataset (coda-19): http://coda-19.org figure 1 : an example of the final crowd annotation for the abstract of @xcite . the core arguments and contributions of a scientific paper effectively, it is essential to understand that most papers follow a specific structure @xcite , where aspects or components of research are presented in a particular order. a paper typically begins with the background information, such as the motivation to study and known facts relevant to the problem, followed by the methods that the authors used to study the problem, and eventually presents the results and discusses their implications @xcite . prior work, as shown in for both humans and machines to comprehend and process the potentially life-saving information in these 13,000+ papers. to reach good performance levels, modern automated language-understanding approaches often require large-scale human annotations as training data. researchers traditionally relied on experts to annotate the structures of scientific papers, as shown in researchers have used non-expert crowds to annotate text, for example, for machine translation @xcite @xcite @xcite , natural language inference @xcite , and medical report analysis @xcite @xcite . while domain experts are still valuable in creating high-quality labels @xcite , and there are concerns about the use of mturk @xcite , employing non-expert crowds has been shown to be an effective and scalable approach to creating datasets. however, annotating papers is still often viewed as an expert task. the majority of the datasets only used experts ( this paper introduces coda-19, the covid-19 research aspect dataset and presents the first outcome of our exploration in using non-expert crowds for large-scale scholarly article annotation. coda-19 contains 10,966 abstracts randomly selected from cord-19. each abstract was segmented into sentences, which were further divided into one or more shorter text fragments. all 168,286 text fragments in coda-19 were labeled with a "research aspect," i.e., background, purpose, method, finding/contribution, or other. this annotation scheme was adapted from sol-vent @xcite , with minor changes. figure 1 shows an example annotated abstract. in our project, 248 crowd workers from mturk were recruited and annotated the whole coda-19 within ten days. 2 nine different workers annotated each abstract. we aggregated the crowd labels for each text segment using majority voting. the resulting crowd labels had a label accuracy of 82% when compared against the expert's labels of 129 abstracts. the inter-annotator agreement (cohen's kappa) was 0.741 between the crowd labels and the expert labels, and 0.788 between two experts. we also established several classification baselines, showing the feasibility of automating such annotation tasks. ## related work a significant body of prior work has explored revealing or parsing the structures of scientific articles, including composing structured abstracts @xcite , identifying argumentative zones @xcite @xcite , analyzing scientific discourse (de @xcite @xcite , supporting paper writing @xcite , and representing papers to reduce information overload @xcite . in this section, we review the datasets that were created to study the structures of scientific articles. we sorted all the datasets that denoted the structures of scientific papers into two categories: (i) datasets that used human labor to manually annotate the sentences in scientific articles ( in the first category, researchers recruited a group of annotators-often experts, such as medical doctors, biologists, or computer scientists-to manually label the sentences in papers with their research aspects (e.g., background, method, findings). coda-19 belongs to the first category. in the second category, researchers used the section titles that came with structured abstracts in scientific databases (e.g., pubmed) to label sentences. a structured abstract is an abstract with distinctly labeled sections (e.g., introduction, methods, results) @xcite . different journals have different guidelines for section titles. to form a coherent and standardized dataset, researchers often mapped these different titles into a smaller set of labels. the sizes of the datasets in the second category ( ## coda-19 dataset construction coda-19 has 10,966 abstracts that contain a total of 2,703,174 tokens and 103,978 sentences, which were divided into 168,286 segments. the data is released as an 80/10/10 training/validation/test split. ## data quality assessment we worked with a biomedical expert and a computer scientist to assess label quality; both experts are co-authors of this paper. the biomedical expert (the "bio" expert in ## classification baselines we further examined the capacity of machines for annotating research aspects automatically. seven baseline models were implemented: linear svm, random forest, multinomial naive bayes (mnb), cnn, lstm, bert, and scibert. ## discussion annotating scientific papers was often viewed as an expert task that was difficult or impossible for non-expert annotators to do. many datasets that labeled scientific papers were thus produced by small groups of experts. for example, two researchers manually created the acl rd-tec 2.0, a dataset that contains 300 scientific abstracts @xcite . a group of annotators "with rich experience in biomedical content curation" created medmentions, a corpus containing 4,000 abstracts @xcite . many datasets used in biomedical nlp shared tasks were manually created by the organizers and/or their students, such as scienceie in semeval'17 @xcite and relation extraction in semeval'18 @xcite . our work suggests that non-expert crowds, such as crowd workers or volunteers, can be used for these types of data-labeling tasks. meanwhile, some prior work used crowd workers to annotate pieces of lower-level information on papers or medical documents, such as images @xcite , named entities (e.g.,, medical terms @xcite , disease @xcite , and medicine @xcite .) our work shows that to a certain extent, crowd workers can comprehend the high-level structures and discourses in papers, and therefore can be assigned more complex, higher-level tasks. ## conclusion and future work this paper introduces coda-19, a humanannotated dataset that codes the background, purpose, method, finding/contribution, and other sections of 10,966 english abstracts in the covid-19 open research dataset. coda-19 was created by a group of mturk workers, who achieved labeling quality comparable to that of experts. we demonstrated that a non-expert crowd can be rapidly employed at scale to join the fight against covid-19. one future direction is to improve classification performance. we evaluated the automatic labels against the biomedical expert's labels, and the scib-ert model achieved an accuracy of 0.774 and a cohen's kappa of 0.667, indicating potential for further improvement. furthermore, one motivation to automatically annotate research aspects is to help search and information extraction @xcite . we have teamed up with the group who created covidseer foot_2 to explore the possible uses of coda-19 in such systems.