The Effectiveness of Uncased Tokeniziaion for Clinical Notes (2025)

Abstract: The impact of case-sensitive tokenization on clinical notes is not well understood. While clinical notes share similarities with biomedical text in terminology, they often lack the proper casing found in polished publications. Language models, unlike humans, require a fixed vocabulary and case sensitivity is a trade-off that must be considered carefully. Improper casing can lead to sub-optimal tokenization and increased sequence length, degrading downstream performance and increasing computational costs. While most recent open-domain encoder language models use uncased tokenization for all tasks, there is no clear trend in biomedical and clinical models. In this work we (1) show that uncased models exceed the performance of cased models on clinical notes, even on traditionally case-sensitive tasks such as named entity recognition and (2) introduce independent case encoding to better balance model performance on case-sensitive and improperly-cased tasks.

## introduction casing is one of many ways to increase the readability of written text. however, more often than not it is not a necessity -after all we communicate verbally without specifying, "capital a" in conversation. language models (lms) have fixed vocabularies. for them, case sensitivity is a trade-off between retaining case information to resolve casesensitive ambiguities and improving efficiency as well as robustness by introducing case-invariance. case-sensitive tokenization is often seen as useful for resolving ambiguties, e.g., in named entitiy recognition (ner). however, for some types of text -such as clinical notes -the observed case is merely an interpretation of spoken language during transcription, which should be re-constructable from lower-case text with sufficient context and knowledge. this is distinct from applications where preserving case is absolutely necessary, such as in programming languages, where case is semantically significant. moreover, when case-sensitive tokenization results in truncated sequences, then the task becomes more challenging due to missing tokens. for this application, it should be obvious that missing case information will always be easier to infer than truncated tokens. prior work in the biomedical domain has shown mixed results on the impact of casing on downstream tasks, with some work showing that cased models perform better on ner tasks @xcite , and others showing that uncased models perform better across the board @xcite . to the best of our knowledge, no work has been done to investigate the impact of casing on clinical notes. clinical notes share some similarities with biomedical text such as pubmed articles with regards to their domain and in the sense that both are littered with case-sensitive acronyms. pubmed articles are much more polished and typically have proper casing, whereas clinical notes are often semi-structured and lack proper casing. in this work we (1) show that uncased models match or exceed the performance of cased models on clinical notes within 1%, even on traditionally case-sensitive tasks such as ner, and (2) introduce independent case encoding to better balance the trade-off of potentially case-sensitive or improperly-cased tasks. note that we are only concerned with encoder models, not decoder models, which typically use cased tokenization for the output @xcite @xcite due to other motivating factors (e.g., the fact that humans prefer text with proper casing). lee et al. (2020) introduce biobert-which is initialized from the bert cased model and trained on pubmed abstracts. while @xcite report that a cased vocabulary performs slightly better on downstream tasks, they do not provide any further analyses or ablations. @xcite introduce scibert, which uses the same architecture as bert, but was pretrained from scratch on scientific text with a new tokenizer. they use cased models for ner and uncased for all other tasks following the original bert methodology, but note that the uncased models sometimes performed slightly better in their initial testing than cased ones on ner. @xcite introduce pubmedbert, which is trained from scratch on pubmed articles and diverges from previous work by using uncased tokenization. they do not provide ablations to support this decision, but note that the cased and uncased versions had similar performance in prelimenary experiments. ## independent case encoding we introduce a novel model extension by adding independent case encoding to the existing uncased bert model. this is implemented similar to the existing positional and token type embedding in that is an additive embedding with 4 possible values: uppercase, lowercase, capitlized, or mixed case. this case information is found during tokenization. while training from scratch may provide better results, in our experiments we simply add this embedding to the existing bert uncased models. for this reason, we only provide independent casing results on models that continue pre-traininig on mimic-iii. the motivation for independent case encoding is to retain some useful case information without being overly burdened when the case is not important. this gives the model the ability to easily ignore case information, unlike the cased tokenization. ## experiments 4.1 dataset statistics to analyze the prevalence of improper casing in clinical data, we count the total number of tokens and the number of sequences with greater than 128 tokens for mimic-iii and the downstream datasets. we also include these metrics for the c4 common crawl dataset @xcite to serve as a baseline of tokenizing a different open-domain corpus. ## results note that the model with independent case encoding uses the uncased tokenizer and vocabulary. ## analysis: robustness of the independent case encoding to demonstrate the robustness of the independent case encoding, we construct 2 evaluation scenarios by transforming the evaluation dataset either to all upper-case or all lower-case. these transformations naturally do not affect the uncased model, but have a profound impact on the cased model. the upper part of however, the results on the upper-cased evaluation data in the bottom part of ## conclusion our paper provides an analysis of the impact of different casing strategies on clinical notes. we find that uncased models outperform cased models on downstream tasks while requiring fewer tokens. our findings highlight the trade-off of casesensitive tokenization, especially as it pertains to clinical data and other text that may not have proper casing. finally, we introduce independent case encoding to better balance performance on casesensitive and improperly-cased tasks.