Dataset Debt in Biomedical Language Modeling (2022)

Abstract: Large-scale language modeling and natural language prompting have demonstrated exciting capabilities for few and zero shot learning in NLP. However, translating these successes to specialized domains such as biomedicine remains challenging, due in part to biomedical NLP’s significant dataset debt – the technical costs associated with data that are not consistently documented or easily incorporated into popular machine learning frameworks at scale. To assess this debt, we crowdsourced curation of datasheets for 167 biomedical datasets. We find that only 13% of datasets are available via programmatic access and 30% lack any documentation on licensing and permitted reuse. Our dataset catalog is available at: https://tinyurl.com/bigbio22 .

## introduction natural language prompting has recently demonstrated significant benefits for language model pretraining, including unifying task inputs for largescale multi-task supervision @xcite and improving zero-shot classification via explicit, multi-task prompted training data @xcite . with performance gains reported when scaling to thousands of prompted training tasks @xcite , tools that enable largescale integration of expert-labeled datasets hold great promise for improving zero-shot learning. however, translating these successes to specialized domains such as biomedicine face strong headwinds due in part to the current state of dataset accessibility in biomedical nlp. recently data cascades was proposed as a term-of-art for the costs of undervaluing data in machine learning @xcite . we propose a similar term, dataset debt, to capture the technical costs @xcite of using datasets which are largely open and findable, but inconsistently documented, structured, and otherwise inaccessible via a consistent, programmatic interface. this type of debt creates significant practical challenges when integrating complex domain-specific corpora into popular machine learning frameworks. we claim that biomedical nlp suffers from significant dataset debt. for example, while hug-gingface's popular datasets library @xcite contains over 3,000 datasets, biomedical data are underrepresented and favor tasks with general domain appeal such as question answering or semantic similarity (pubmedqa, scitail, biosses). to assess the state of biomedical dataset debt, we built, to our knowledge, the largest catalog of metadata for publicly available biomedical datasets. we document provenance, licensing, and other key attributes per @xcite to help guide future efforts for improving dataset access and machine learning reproducibility. our effort found low overall support for programmatic access, with only 13% (22/167) of our datasets present in the datasets hub. despite a proliferation of schemas designed to standardize dataset loading and harmonize task semantics. there remains no consistent, api interface for easily incorporating biomedical data into language model training at scale. ## data-centric machine learning deep learning models are increasingly moving to commodified architectures. data-centric machine learning (vs. model-centric) is inspired by the observation that the performance gains provided by novel architectures are often smaller than gains obtained using better training data. we outline some key challenges and opportunities in data-centric language modeling. these are broadly applicable to nlp, but have strong relevance to biomedicine and the current state of dataset debt.