Good Data, Large Data, or No Data? Comparing Three Approaches in Developing Research Aspect Classifiers for Biomedical Papers (2023)

Abstract: The rapid growth of scientific publications, particularly during the COVID-19 pandemic, emphasizes the need for tools to help researchers efficiently comprehend the latest advancements. One essential part of understanding scientific literature is research aspect classification, which categorizes sentences in abstracts to Background, Purpose, Method, and Finding. In this study, we investigate the impact of different datasets on model performance for the crowd-annotated CODA-19 research aspect classification task. Specifically, we explore the potential benefits of using the large, automatically curated PubMed 200K RCT dataset and evaluate the effectiveness of large language models (LLMs), such as LLaMA, GPT-3, ChatGPT, and GPT-4. Our results indicate that using the PubMed 200K RCT dataset does not improve performance for the CODA-19 task. We also observe that while GPT-4 performs well, it does not outperform the SciBERT model fine-tuned on the CODA-19 dataset, emphasizing the importance of a dedicated and task-aligned datasets dataset for the target task.

## introduction the rapid growth of scientific publications, particularly during the covid-19 pandemic, has made it increasingly challenging to keep up with the latest research advancements. to address this issue, researchers have developed various systems, such as search engines @xcite , visualization tools @xcite , claim verification systems @xcite , questionanswering systems @xcite , and summarization techniques @xcite . these tools help efficiently comprehend publications by organizing large amounts of information. one critical aspect of understanding scientific literature is the classification of sentences within abstracts according to their research aspects, such as background, purpose, method, and finding. this is particularly crucial in the biomedical domain, where abstracts tend to be longer and more complex. with the annotated aspects, readers can quickly grasp the key aspects of a scientific paper. for example, facetsum @xcite summarizes papers in four aspects to quickly convey the information. several datasets have been proposed for research aspect classification, including pubmed 200k rct @xcite , pubmed-pico-detection @xcite , @xcite , and more. in this paper, we focus on the research aspect classification task using crowd-annotated coda-19 @xcite and explore the impact of different datasets on model performance. specifically, we investigate whether the automatically curated large dataset (pubmed 200k rct @xcite ) can help the target task despite its shifted data distribution from the target task. additionally, we examine whether large language models (llms), trained on a massive general textual corpus, can solve the task with limited or no task-specific data provided. in particular, we evaluate six llms, including three open-sourced models (llama-65b @xcite , @xcite , and dolly-12b @xcite ) and three closed models (gpt-3 @xcite , chatgpt (openai, 2022), and gpt-4 (openai, 2023)). our study suggests that using pubmed 200k does not improve the performance of the coda-19 research aspect classification task, regardless of the approach used. we experimented with (i) training purely on pubmed data, (ii) simplymixing pubmed and coda-19 data, (iii) upsampling coda-19 data and mixing it with pubmed data to address the data imbalance issue, and (iv) using a two-staged training approach where the model is trained on pubmed and coda-19 sequentially. however, none of these approaches could improve the performance of the target task. we hypothesize this is due to the use of the scibert @xcite , which has pre-trained on papers from scientific domains and thus reduces the advantage of incorporating pubmed 200k. our results also showed that although gpt-4 performed well in both zero-shot and few-shot settings, it was not able to outperform the scibert model fine-tuned on the coda-19 dataset. this finding suggests that having a dedicated dataset that aligns well with the target task is still important. ## methodology in this section, we describe models trained to explore the impact of different datasets and training strategies on the coda-19 research aspect classification task. for detailed model training details such as hyperparameters, please refer to appendix b. ## experiments and results we conducted three experiments to ( i ## conclusion in this paper, we investigate the impact of different datasets and llms on the coda-19 research aspect classification task. our findings show that using a huge but slightly different dataset, pubmed 200k, does not help improve performance. llms, trained with massive web corpus, are also unable to outperform the scibert trained on the target dataset, emphasizing the importance of taskaligned datasets. in the future, we will explore methods for the model to consider the context and predict all sentences within a single abstract at once.