One Model is All You Need: B y T 5- S anskrit, a Unified Model for S anskrit NLP Tasks (2024)

Abstract: Morphologically rich languages are notoriously challenging to process for downstream NLP applications. This paper presents a new pretrained language model, ByT5-Sanskrit, designed for NLP applications involving the morphologically rich language Sanskrit. We evaluate ByT5-Sanskrit on established Sanskrit word segmentation tasks, where it outperforms previous data-driven approaches by a considerable margin and matches the performance of the current best lexicon-based model. It is easier to deploy and more robust to data not covered by external linguistic resources. It also achieves new state-of-the-art results in Vedic Sanskrit dependency parsing and OCR post-correction tasks. Additionally, based on the Digital Corpus of Sanskrit, we introduce a novel multitask dataset for the joint training of Sanskrit word segmentation, lemmatization, and morphosyntactic tagging tasks. We fine-tune ByT5-Sanskrit on this dataset, creating a versatile multitask model for various downstream Sanskrit applications. We have used this model in Sanskrit linguistic annotation projects, in information retrieval setups, and as a preprocessing step in a Sanskrit machine translation pipeline. We also show that our approach yields new best scores for lemmatization and dependency parsing of other morphologically rich languages. We thus demonstrate that byte-level pretrained language models can achieve excellent performance for morphologically rich languages, outperforming tokenizer-based models and presenting an important vector of exploration when constructing NLP pipelines for such languages.

## introduction it is generally acknowledged that morphologically rich languages (mrl) are challenging for nlp @xcite . while language modeling has addressed this challenge, e.g. by integrating subword information (see e.g. @xcite , there is surprisingly little systematic research on how efficient models for low-level tasks such as tokenization, lemmatization, morphosyntactic analysis, and dependency parsing can be designed for mrls. access to this low-level information is relevant for downstream tasks such as information retrieval and question answering, as well as for linguistic and literary studies. in this paper, we introduce a unified model that jointly performs these tasks for sanskrit, an ancient south-asian mrl, which has been continuously attested since 1,300 bce. vedic, its archaic level, primarily focusses on the description of the soma and the fire sacrifice. starting around 300 bce, the majority of sanskrit literature was composed in classical sanskrit, encompassing a vast array of domains from religious hymns to scientific and narrative texts (see we propose a framework in which we pretrain a character-level sanskrit language model based on byt5 on a large body of sanskrit data before jointly fine-tuning it on a number of downstream nlp tasks, which we reformulate as sequence-generation tasks. this paradigm brings large performance gains, leading to new sota results on established sanskrit nlp benchmarks. we emphasize creating a system that is as simple as possible to train and deploy, without depending on complex pre-or postprocessing steps and retaining high performance on data that shows challenges such as ocr mistakes or the use of non-standard language not sufficiently covered by available linguistic resources. in concrete terms, we achieve a gain of 8.8 points on perfect sentence matching score (pm) for the hackathon sws benchmark compared to the current state-of-the-art, while we come close by 0.13 in performance on the sighum dataset to the currently best performing lexicon-driven model. we achieve 4.88 points improvement on the sws dcs 2018 benchmark. on vedic dependency parsing, we achieve 2.18 points improvement on uas and 2.60 points on las compared to the current state-of-the-art. on ocr post-correction, we outperform the currently best approach by 0.29 lower cer and 3.16 lower wer. we also show that our approach yields the best performance on lemmatization and dependency parsing for three other mlr languages. we also present a novel dataset for the training and evaluation of three central sanskrit nlp tasks based on the digital corpus of sanskrit (dcs): word segmentation, lemmatization, and morphosyntactic tagging. we show that our pretrained model outperforms other baselines on these new tasks. we also demonstrate that jointly training on the tasks of sws, lemmatization, and morphosyntactic tagging on top of the pretrained language model leads to the best performance. this enables the deployment of one single model without dependence on external linguistic resources to handle all relevant nlp tasks for annotated sanskrit corpus building with the best performance. we show that training and evaluating this model on pseudo-paragraph-level, where multiple sentences are predicted at once, gives a distinct performance advantage due to the available contextual information. in section 2, we give an overview of the relevant research literature. in section 3, we discuss the pretraining and fine-tuning datasets used in this paper. section 4 introduces the layout of our proposed multitask framework. in section 5, we first evaluate the model on established sanskrit word segmentation, vedic sanskrit dependency parsing, ocr post-correction tasks, as well as on other mlr languages, and then present the performance of the unified model trained on the new dataset. we also perform a detailed manual analysis of the error patterns of the multitask model. we make the code, all relevant datasets, the pretrained base model as well as the fine-tuned multitask model available under the apache license 2.0 at https: //github.com/sebastian-nehrdich/ byt5-sanskrit-analyzers. we make an interactive live version of the multitask model accessible at https://dharmamitra.org . ## related research the pretrain-fine-tune paradigm, where a pretrained language model (plm) trained on a large corpus of unlabeled data is subsequently fine-tuned on a smaller dataset of task-specific labeled data, is the de-facto standard approach for nlp tasks such as part-of-speech and morphosyntactic tagging, sentence classification, and many more since the publication of the encoder-only approaches bert @xcite and elmo @xcite in 2018. when it comes to morphologically rich languages, the good performance of this paradigm is demonstrated for turkish in özc ¸ift et al. (2021), while @xcite and @xcite show the superior performance of bert on linguistic annotation tasks for the morphologically rich classical language latin. t5 @xcite introduced a new pretraining paradigm where both encoder and decoder are trained. this encoder-decoder architecture enables the fine-tuning of the same base model on diverse tasks such as translation, question answering, and text classification with the same hyperparameters and loss function. @xcite further show how the t5 paradigm can be used efficiently in a multitask setup with large variation between the different tasks. for morphologically rich languages, language models that make use of character-level information show superior performance to those operating on word-level alone @xcite . while a number of openly available pretrained language models exist, only @xcite followed a tokenizer-free byte-level approach, resulting in strong performance on linguistic tasks and achieving the best performance on the morphological inflection task. most approaches to sanskrit nlp tasks such as sanskrit word segmentation (sws) can be broadly separated into two groups: lexicon-based and datadriven. for a recent, comprehensive overview of the relevant literature, see @xcite . lexicon-driven approaches rely on external linguistic resources, while data-driven approaches learn from data alone and are therefore less complex to train and deploy. the main drawback of data-driven approaches is that they cannot access latent knowledge contained in lexical resources. @xcite combine lexicon-based and data-driven aspects, formulating sws as a character-level sequence labeling task that uses lexical information whenever available. @xcite presents a lexicon-based multitask model that handles sws, morphological parsing, dependency parsing, syntactic linearization, and prosodic linearization. to our knowledge, this is the only other published multitask approach to central sanskrit nlp tasks. pretrained language models supporting sanskrit are available, but they are not yet widely used for sanskrit linguistic tasks. @xcite included sanskrit data in its pretraining setup. @xcite trained and evaluated encoder-only plms for the task of vedic sanskrit dependency parsing, coming to the conclusion that they do not offer clear advantages in performance yet due to the comparatively small amount of training data used. ## data for pretraining, we use the sanskrit data of the sangraha dataset @xcite as a basis, which mostly consists of data gained by a comprehensive ocr effort of the sanskrit-related literature available at the internet archive foot_0 . we only use the language-verified split of this dataset and none of the synthetic data. we decided to use this noisy ocr-based dataset following the observation made in bamman and burns (2020), where a plm for latin trained on a noisy corpus consisting of largely ocr'd data achieved new state of the art results on latin pos tagging tasks. we augment this data with high-quality human input sanskrit data from the gretil collection foot_1 and the digital sanskrit buddhist canon. 3 the statistics of the dataset are shown in ## proposed method we propose the combination of the following paradigms in order to generate an efficient, highperforming end-to-end framework for various sanskrit nlp tasks: we first pretrain a byte-level sanskrit plm based on the byt5 architecture, which is distributed under the apache license 2.0, overcoming the limitation of lack of access to latent information for data-driven approaches. then, we reformulate the central sanskrit nlp tasks of word ## experiments models were trained on gpu nodes of 8 nvidia a6000 48gb gpus. the time needed for pretraining was one week, while the fine-tuning runs varied between 2 and 8 hours. the joint multitask model took 32 hours to finetune. we leveraged the deepspeed library https://www. deepspeed.ai/ for training in half precision bf16, and for making efficient use of the multi-gpu setup. for the foundation model, we further pretrain a byt5 model @xcite in the "base" configuration with 582m parameters on the entire dataset for 100,000 steps with a batch size of 512 and a sequence length of 512. the resulting model is called byt5-sanskrit in this paper. according to the scaling laws presented in @xcite , the optimal number of parameters for our training dataset size of 6.5b tokens is about 325m parameters. this matches the byt5 "small" configuration with 300m parameters. we decided to train a model one category larger than that to ensure we get optimal performance. we decided against including open-source and commercial general purpose decoder-only llms in our evaluation setup since their tokenizers are not permitting precise handling of sanskrit on the morpheme-level. ## conclusion and future work we have demonstrated that by pretraining a bytelevel language model on a large collection of mostly noisy data, new state-of-the-art results for sanskrit word segmentation, vedic dependency parsing, and ocr post-correction are achieved, closing the performance gap between lexiconbased and data-driven sanskrit nlp approaches. we further demonstrated that this pretrained language model can be used as a basis for a multitask model that handles word segmentation, lemmatization, and morphosyntactic tagging jointly with high accuracy. we further demonstrated that this multitask model benefits greatly from training and inference on pseudo-paragraph-level. for the joint fine-tuning on these tasks, we presented a novel dataset. the resulting unified model, being independent of external linguistic resources, is simple to deploy and is already used for sanskrit corpus annotation projects as well as in information retrieval and machine translation setups. we also showed that our approach generalizes to other morphologically rich languages, where the application of a byte-level plm yields best results for two languages on lemmatization and for three languages on dependency parsing. we thus establish that bytelevel plms are a crucial vector of exploration when builiding nlp pipelines for mlr languages.