Biomedical Language Models are Robust to Sub-optimal Tokenization (2023)

Abstract: As opposed to general English, many concepts in biomedical terminology have been designed in recent history by biomedical professionals with the goal of being precise and concise. This is often achieved by concatenating meaningful biomedical morphemes to create new semantic units. Nevertheless, most modern biomedical language models (LMs) are pre-trained using standard domain-specific tokenizers derived from large scale biomedical corpus statistics without explicitly leveraging the agglutinating nature of biomedical language. In this work, we first find that standard open-domain and biomedical tokenizers are largely unable to segment biomedical terms into meaningful components. Therefore, we hypothesize that using a tokenizer which segments biomedical terminology more accurately would enable biomedical LMs to improve their performance on downstream biomedical NLP tasks, especially ones which involve biomedical terms directly such as named entity recognition (NER) and entity linking. Surprisingly, we find that pre-training a biomedical LM using a more accurate biomedical tokenizer does not improve the entity representation quality of a language model as measured by several intrinsic and extrinsic measures such as masked language modeling prediction (MLM) accuracy as well as NER and entity linking performance. These quantitative findings, along with a case study which explores entity representation quality more directly, suggest that the biomedical pre-training process is quite robust to instances of sub-optimal tokenization.

## introduction in order to communicate complex concepts precisely and efficiently, biomedical terminology has been designed by researchers and medical professionals by combining existing meaningful mor- ## supervised morpheme segmentation recent work evaluating morphological segmentation at scale such as the sigmorphon 2022 shared task @xcite demonstrates the impressive performance of supervised methods compared to unsupervised methods like bpe @xcite or @xcite , even for languages with more limited annotated data than english. in the sigmorphon 2022 shared task, the organizers compile a large quantity of segmented morpheme data, over half a million english words obtained from wiktionary and other sources using both hand-crafted and automated methods @xcite . ## conclusion in this work, we first note that current biomedical tokenization methods are not well aligned with human judgments and highlight that the agglutinating nature of biomedical terminology could be affected by this sub-optimal segmentation. to understand whether this limited segmentation performance has an effect on downstream applications, we first build a biomedical tokenizer which is better aligned with human judgments using supervised morpheme segmentation and biomedical domain-knowledge. we then leverage this tokenizer to pre-train our biovo-cabbert model and compare it with a replicated pubmedbert model on the ner and entity linking tasks. surprisingly, we find that these models achieve almost exact parity in all datasets evaluated, suggesting that pubmedbert's domain specific tokenization and pre-training process was already quite robust to sub-optimal tokenization. we further verify this idea with a case-study which qualitatively confirms our observations. we hope that our work can give researchers and practitioners some insight into how instances of sub-optimal segmentation, which are often jarring to human experts, could have little effect on a model's downstream applicability.