Heidelberg-Boston @ SIGTYP 2024 Shared Task: Enhancing Low-Resource Language Analysis With Character-Aware Hierarchical Transformers (2024)

Abstract: Historical languages present unique challenges to the NLP community, with one prominent hurdle being the limited resources available in their closed corpora. This work describes our submission to the constrained subtask of the SIGTYP 2024 shared task, focusing on PoS tagging, morphological tagging, and lemmatization for 13 historical languages. For PoS and morphological tagging we adapt a hierarchical tokenization method from Sun et al. (2023) and combine it with the advantages of the DeBERTa-V3 architecture, enabling our models to efficiently learn from every character in the training data. We also demonstrate the effectiveness of characterlevel T5 models on the lemmatization task. Pre-trained from scratch with limited data, our models achieved first place in the constrained subtask, nearly reaching the performance levels of the unconstrained task’s winner. Our code is available at https://github.com/bowphs/ SIGTYP-2024-hierarchical-transformers

## introduction unlike modern languages, historical languages come with a notable challenge: their corpora are closed, meaning they cannot grow any further. this situation often puts researchers of historical languages in a low-resource setting, requiring tailored strategies to handle language processing and analysis effectively @xcite . in this paper, we focus on identifying the most efficient methods for extracting information from small corpora. in such a scenario, the main hurdle is not computational capacity, but learning to extract the maximal amount of information from our existing data. to evaluate this, the sigtyp 2024 shared task offers a targeted platform centering on the evaluation of embeddings and systems for historical languages. this task provides a systematic testbed for * equal contribution. for the constrained subtask, participants received annotated datasets for 13 historical languages sourced from universal dependencies @xcite , along with data for old hungarian that adheres to similar annotation standards @xcite . these languages represent four distinct language families and employ six different scripts, which ensures a high level of diversity. the rules imposed in this subtask strictly forbid the use of pre-trained models and limit training exclusively to the data of the specified language. this restriction not only ensures full comparability of the applied methods, it also inhibits any cross-lingual transfer effects. we demonstrate that, even in these resourcelimited settings, it is feasible to achieve high performance using monolingual models. our models are exclusively pre-trained on very small corpora, leveraging recent advances in pre-training language models. our submission was recognized as the winner in the constrained task. notably, it also delivered competitive results in comparison to the submissions in the unconstrained task, where the use of additional data was permitted. this highlights the strength of our approach, even within a more restricted data environment. ## pre-trained language models for ancient and historical languages much of the previous work on pre-trained language models (plms) for ancient and historical languages has focused on cross-lingual transfer learning techniques @xcite @xcite or languages with relatively large corpora compared to most historical languages, such as ancient greek and latin @xcite . in this work, we are interested in maximizing performance in more resource-limited environments while training exclusively on monolingual data. ## results our results are computed using the sigtyp 2024 official evaluation script. 1 the script computes pos tagging scores as the unweighted average of the accuracy and the f 1 score. for morphological tagging, it computes the averaged accuracy across each token, with deductions for any feature categories predicted by the model but absent in the label. the lemmatization scores are the unweighted average of the accuracy@1 and the accuracy@3. we report our results in results in lemmatization display greater diversity, likely due to the differing architectures in participants' approaches. our model achieves 99.18% in classical chinese (lzh), a language where distinct lemmata do not really exist, usually turning the task into mere form replication. this score, though precise, is somewhat lower than the nearperfect range of 99.81 to 99.96% achieved by the other methods in the shared task. ## negative results multi-task learning. we hypothesized that a model simultaneously doing pos tagging, morphological tagging and dependency parsing could benefit from the training signals of related tasks. foot_1 however, this approach did not significantly improve morphological analysis and resulted in longer training times due to slower convergence. on the other hand, jointly performing morphological and pos tagging in a multi-task learning setup yielded minor improvements in pos tagging. we believe that including pos information offers little extra insight to the model for morphological tagging and simultaneously pressures it to form representations apt for pos tagging. conversely, enriching the coarser pos tagging task with morphological labels provides the model with useful additional insights. furthermore, our dependency parsing technique differs from the more direct classification approach used in pos and morphological tagging, potentially leading to instabilities during training. we mark the winner of each subtask in bold and underline the overall winner. see appendix c for language identifiers. tall models. @xcite found that transformers with a narrower and deeper architecture might surpass the performance of similarly sized models in masked language modeling tasks. inspired by this finding, we experimented with doubling the number of layers to 24 while reducing the hidden size from 768 to 512 and the number of attention heads from 12 to 8. however, although this adjustment seemed to yield a marginal improvement in pre-training with mlm, it did not result in any performance changes when training with rtd. ## conclusion we present our approach for the sigtyp 2024 shared task on historical language analysis. our method employs a hierarchical transformer that first focuses on a word's characters, applying self-attention to generate initial word embeddings. these embeddings are then further developed by integrating the contextual information from surrounding words. we pre-train hlm-deberta-v3 and t5 models with small datasets of historical texts. the character-based methodology of our architecture yielded promising results, effectively leveraging the available data. contrary to our expectations, the implementation of multi-task learning had only a negligible effect on enhancing our models' performance.