Don’t stop pretraining! Efficiently building specialised language models in resource-constrained settings. (2025)

Abstract: Developing specialised language models for low-resource domains typically involves a trade-off between two specialisation strategies: adapting a general-purpose model through continued pretraining or retraining a model from scratch. While adapting preserves the model’s linguistic knowledge, retraining benefits from the flexibility of an in-domain tokeniser – a potentially significant advantage when handling rare languages. This study investigates the impact of tokenisation, specialisation strategy, and pretraining data availability using classical scholarship – a multilingual, code-switching and highly domain-specific field – as a case study. Through extensive experiments, we assess whether domain-specific tokenisation improves model performance, whether character-based models provide a viable alternative to subword-based models, and which specialisation strategy is optimal given the constraints of limited pretraining data. Contrary to prior findings, our results show that in-domain tokenisation does not necessarily enhance performance. Most notably, adaptation consistently outperforms retraining, even with limited data, confirming its efficiency as the preferred strategy for resource-constrained domains. These insights provide valuable guidelines for developing specialised models in fields with limited textual resources.

## introduction transformer-based language models have achieved remarkable success through transfer learning, where models pretrained on large general-purpose corpora are fine-tuned for downstream tasks. though relatively straightforward, this approach proves more challenging for tasks involving highly domain-specific fields or rare languages. in such settings, it might be beneficial -if not essentialto develop specialised language models (e.g. @xcite @xcite . however, there is no consensus on the optimal specialisation strategy -whether to pretrain a model from scratch or to adapt an existing one. adapting involves further pretraining a generic model on domain-specific data. the approach has been shown to increase downstream performance (e.g. @xcite , while preserving the broad linguistic knowledge acquired during the initial pretraining phase. however, this strategy does not grant infinite flexibility. a key obstacle to specialisation often lies in the model's predefined vocabulary. commonly used subword tokenisation methods (e.g. @xcite tie the model to a fixed vocabulary, which can be suboptimal or utterly inappropriate for certain languages. thus, several tokenisation-free models have been developed to address this issue. canine @xcite and charformer @xcite use a broad set of unicode points as a vocabulary. however, while they circumvent tokenisation issues, these models often perform below their subword-based counterparts and significantly limit the maximum input sequence length. while domain-specific tokenisation may improve model performance, modifying the model's vocabulary and embeddings requires retraining from scratch. this creates a critical trade-off between optimising tokenisation at the cost of pretrained knowledge or leveraging existing models despite suboptimal tokenisation. as the decision is constrained by the availability of pretraining data, the dilemma is particularly crucial in lowresource settings. studies reporting superior results from retraining domain-specific models (e.g. @xcite often have access to extensive training resources. while some studies claim better results with smaller pretraining datasets @xcite , many advocate for adapting over retraining @xcite , but few systematically control for tokenisation, leaving its precise role in model performance an open question. this study examines these specialisation strategies in the context of classical scholarship, a field characterised by intense multilingualism, frequent code-switching, and highly domain-specific vocabulary. these factors, along with the extensive use of rare characters and diacritics, pose significant tokenisation challenges, particularly for texts with a high proportion of latin and ancient greek, which are often absent from generic multilingual models, making the field an ideal case study for tokenisation and specialisation strategies. while characterlevel models may offer greater adaptability, subword tokenisers often struggle with historical texts due to transcription errors, spelling variations, and morphological inconsistencies. retraining also remains impractical given the scarcity of clean textual data. although this study assembles the largest classics-related corpus to date, it remains constrained to 1.4b tokens for six languages, less than half the 3.3b tokens used by the first monolingual bert @xcite . this study assesses the impact of tokenisation, specialisation strategy, and the availability of domain-specific data. it asks three research questions: what are the benefits of in-domain tokenisation? do characterbased models provide a viable and more adaptable alternative to subword models? finally, which specialisation strategy is most effective given the constraints of available data? ## general discussion the importance of tokenisation the conclusions of these experiments are multifaceted. first, experiments confirm previous findings on the critical role of tokenisation @xcite . our experiments compare the performance of two xlm-r models pretrained on the same data but using two distinct tokenisers of equal size: a generic and a domain-specific tokeniser. as shown in second, unlike previous studies, these experiments illustrate that in-domain tokenisation does not necessarily lead to better performance. analyses indicate that intrinsic tokenisation evaluation methods relying on fertility do not correlate with downstream results. on the contrary, in a resource-limited environment, lower (i.e. better) fertility also leads to a lower average token frequency and less performant models. we argue that in low-resource settings, tokenisation should balance input sequence length and token-type density. while high-fertility, small-sized tokenisers produce longer input sequences by breaking words into smaller subwords, they also enable more frequent representation of each token within the corpus, which correlates with model performance. this study therefore advocates the adoption of token density as a novel intrinsic evaluation metric. although smaller in-domain tokenisers consistently yield better results than their larger indomain counterparts, they still do not surpass the performance of a larger, generic tokeniser. this improved performance may be attributable to the substantially larger size training corpus, which could favour a more robust and efficient vocabulary. this hypothesis is left for future research. the potential of character-based models the second series of experiments consistently shows that canine-c is outperformed by xlm-r across all downstream tasks. however, the limits of this comparison must be highlighted. notably, the two models are pretrained on different corpora with distinct training objectives and exhibit slight architectural differences, with canine-c incorporating downsampling and upsampling convolutional layers around its central transformer blocks. thus, the findings merely indicate that even in a domain where tokenisation is suboptimal, xlm-r achieves better performance than canine-c. second, the results demonstrate canine-c's strong adaptability to the domain, with the adapted model yielding an average improvement of 5% in f1 score over the generic model. this improvement is substantial when compared to xlm-r's average improvement of 1.5% in f1 score. researchers working with highly specific or underrepresented languages not covered by large multilingual models may therefore find character-based models like canine-c advantageous when adapted to their domain. the superiority of adapted models finally, the third series of experiments shows an undeniable superiority of adapted models over retrained and generic models, regardless of the amount of available pretraining data. this result aligns with the principle that adaptation preserves the extensive linguistic knowledge embedded in the generic base-model, a solid foundation difficult to replicate when training from scratch on limited resources. although retraining has shown success in domain-specific fields with ample pretraining corpora, such as biomedical or legal domains, resource-constrained fields appear to benefit most by leveraging the power of scale utilised by the model during its original pretraining. the findings also reveal that an adaptation corpus of 300m tokens already achieves 75% of the overall performance gains, indicating adaptation as an efficient and resource-effective specialisation strategy. as the superiority of adaptation over retraining is especially evident in data-ablation scenarios, it suggests that researchers and practitioners working with limited pretraining data should prioritise this approach. moreover, adaptation may offer the only viable pathway to specialising large language models, an approach also left for future work. ## conclusion this study provides a comprehensive analysis of the impact of tokenisation and specialisation strategies on the performance of language models in the field of classical scholarship. our results show that in-domain tokenisation does not necessarily lead to better model performance in a resourceconstrained environment, and that token density is a more reliable predictor of extrinsic performance. our experiments also show that character-based models can offer a viable alternative to subword models, especially when adapted. finally, we show that adaptation is the most effective specialisation strategy in a resource-constrained environment, and that even relatively small adaptation corpora can yield significant performance gains. these findings provide valuable insights for researchers working in resource-limited environments and highlight the importance of tokenisation and specialisation strategies in the development of large language models. we leave the investigation of our findings in other historical domains for future work, make our models available to the research community foot_9 .