Multilingual Sequence-to-Sequence Models for H ebrew NLP (2023)

Abstract: Recent work attributes progress in NLP to large language models (LMs) with increased model size and large quantities of pretraining data. Despite this, current state-of-the-art LMs for Hebrew are both under-parameterized and under-trained compared to LMs in other languages. Additionally, previous work on pretrained Hebrew LMs focused on encoder-only models. While the encoder-only architecture is beneficial for classification tasks, it does not cater well for sub-word prediction tasks, such as Named Entity Recognition, when considering the morphologically rich nature of Hebrew. In this paper we argue that sequence-to-sequence generative architectures are more suitable for large LMs in morphologically rich languages (MRLs) such as Hebrew. We demonstrate this by casting tasks in the Hebrew NLP pipeline as text-to-text tasks, for which we can leverage powerful multilingual, pretrained sequence-to-sequence models as mT5, eliminating the need for a separate, specialized, morpheme-based, decoder. Using this approach, our experiments show substantial improvements over previously published results on all existing Hebrew NLP benchmarks. These results suggest that multilingual sequence-to-sequence models present a promising building block for NLP for MRLs.

## introduction in recent years, large pretrained language models showed impressive results in a variety of nlp tasks @xcite , domains @xcite , and languages @xcite . these models were trained in a self-supervised fashion on large corpora with language modeling objectives. by doing this, models are taking advantage of information available in the training data without any access to explicit labels @xcite . recent works @xcite argue that language models capabilities scale as a power-law with model size, dataset size, and the amount of compute used for training. following the success such models achieved on english benchmarks, analogous language-specific models were developed to improve benchmark results in a variety of languages such as arabic and french to name a few @xcite . hebrew nlp was no different, with a number of encoder-based bert variations proposed, specifically hebert @xcite , alephbert @xcite , and recently alephbertgimmel (abg), a variation of alephbert with a much larger vocabulary @xcite . despite the scaling laws proposed in @xcite , all of the hebrew variations of bert are trained with a relatively small set of pretraining data, and are under-parameterized. in terms of training data, hebert and alephbert (and similarly abg) were trained on 10.5gb and 16gb of hebrew pretraining data respectively. mt5 @xcite in comparison, was trained on mc4, where its public replica @xcite is a 27tb collection of natural text in 101 languages drawn from the public common crawl, 66gb of which is hebrew (4.125x more hebrew training data compared to alephbert). in terms of parameterization and model size, abg, the largest hebrew lm, has 60x and 945x fewer parameters compared to english t5 xxl @xcite and gpt3 @xcite , respectively, language models that were released two years earlier. crucially for downstream use, although english t5 is a much larger model than available hebrew language models, it can still be fine-tuned on common gpus. see appendix a for more details. in addition to scale differences, all previous hebrew lms use an encoder-only architecture, even though the morphological complexity of hebrew and other morphologically rich languages (mrls) foot_0 pose challenges for the efficacy of this model. consider, for instance, the task of pos tagging. assigning pos tags for the phrase "babayit halavan" foot_1 requires to initially segment the phrase to its morphemes and only then assign each morpheme its matching pos tag. since the number of input tokens does not match the number of output tags (2 input words and 5 output tags, one for each morpheme), a one-to-one token-to-tag classification head, as commonly employed in encoder-only models, is not feasible. the same problem appears in semantic tasks like question-answering (qa) and named entity recognition (ner). for example, the named entity for "babayit halavan" is "habayit halavan". this goes beyond what encoder-only models can do by requiring the model to label a string that is not part of the input text. to overcome this architectural obstacle in encoder-only models, the authors of alephbert and abg @xcite used @xcite 's three-step segmentation and tagging approach: contextualize the input, pass resulting embeddings to an lstm decoder, which then generates the segmentation separated by a space symbol, in a char-by-char fashion. then they pass the whitespace representation to a classification head. while effective for morpho-syntactic tasks, these additional components do not enable full generative capabilities, and are not pretrained, therefore the representation of morphemes cannot enjoy the pretrained lms ad-vantages. the departure point of this work is that, in contrast to pre-trained encoders, sequence-to-sequence models can simply take the raw text as input and for any sequence labeling task, generate the morphemes and tags in a sequence. in pos tagging, for example, the generated output can be: this work thus identifies the challenge of current hebrew llms as a three-faceted problem: underparameterization, limited training data, and the use of a suboptimal pre-training architecture. 3 to address these three challenges at once, we propose using mt5 @xcite , a large multilingual sequence-to-sequence model that was pretrained on a vast amount of multilingual data including a significant amount of hebrew data. 4 to adapt classification, span prediction, and token/morpheme classification tasks to mt5's text-to-text paradigm, we propose the text-only formulations illustrated in figure 1 . subsequently, we report here that this paradigm change produces empirical improvements on all tasks evaluated compared to previous state-of-the-art, some of which are dramatic, as a 27.9 f1 increase in hebrew question-answering. ## modeling we use mt5 @xcite , a multilingual generative text-to-text version of t5 @xcite , trained simultaneously on 101 languages. we evaluate mt5 on all its available sizes -small, base, large, xl and xxl -ranging from 300m to 13b parameters. subsequently, we propose casting all hebrew nlp tasks for which evaluation benchmarks exist as text-to-text tasks: the input text is fed into the model, and targets are produced in a generative manner. in contrast to text-to-text formulations of classification and span prediction, token classification is not as common in the literature, and specifically when the tokens consist of multiple morphemes, as is the case in mrls. for example, in pos tagging for mrls, each morpheme is assigned a pos tag, therefore multiple tags are assigned per word. as a result, a generative model cannot simply generate tag predictions one after the other, but it requires to first segment the text and only then label it accordingly. e.g., an unsatisfactory generation for "habayit" is det, noun as we cannot recover which morpheme belongs to which tag. an acceptable model output, on the other hand, is ha-det, lavan-adj as we can recover that ha was tagged with a det and lavan with a adj. throughout our experiments we tested a number of different textto-text formulations. the best formulations for the tasks at hand are depicted in fig. 1 . ## results tables 1,2 summarize our empirical findings. our results demonstrate a marked improvement over previously published results on existing hebrew benchmarks. mt5 produces the biggest performance boost for the qa task of parashoot, with mt5-base already surpassing baseline models and mt5-xxl outperforming alephbert by 27.9 f1 points. for ner, mt5 produces better results than evaluated baselines on both of the dataset annotation levels. the largest performance boost comes in nemo's morpheme-level version where mt5 learns to segment and label entities in an end-to-end fashion. for sentiment analysis, mt5 outperforms the baseline models by a small fraction, however, manual error analysis we performed shows that 34% of its errors are annotation errors and for further 30% our annotators were not able to decide on the correct label. we conclude that work towards a cleaner, more challenging sentiment analysis dataset in hebrew is needed. for segmentation and pos tagging we report error reduction of 30.3% and 32.8% compared to previous state-of-the-art. for the lemmatization task we report an increase of 16.93 mset f1 points compared to yap. all of these are an important step towards closing the gap in morphosyntactic tasks compared with other languages. ## related work hebert @xcite is the first pretrained transformer-based language model trained on hebrew wikipedia and oscar @xcite for the task of user-generated sentiment analysis. alephbert @xcite was pretrained on the same copora in addition to a very large number of hebrew tweets. @xcite tackled the extreme data sparseness in mrls lexica @xcite by pretraining with roughly 2.5x of alephbert vocabulary size, leading to performance improvements. orthogonally, @xcite proposed using char-level lms to mitigate the same sparseness problem, however results were inconclusive. @xcite showed that mt5 outperforms baseline models on a number of multilingual datasets but did not directly evaluate on hebrew. alternatively, monolingual hebrew lm papers only compared against mbert @xcite as the sole multilingual baseline. while larger mt5 models perform better than available lms, they require more powerful hardware accelerators and take longer to train and infer. however, this is a reasonable trade-off from pretraining designated monolingual models from scratch, a more expensive task by itself. additionally, the inclusion of data from 101 languages in the training of mt5 may have negatively impacted its performance on hebrew, as some of the data may not have been relevant or beneficial to this particular language. future work will need to address this issue by training a monolingual hebrew lm in order to further improve performance for hebrew. an inherent risk in sequence-to-sequence models is that they can generate inconsistent text with respect to the input text @xcite . while potentially sensitive in different applications, a number of evaluation frameworks have been suggested to reduce the number of such "hallucinations" @xcite ). another limitation of our evaluation framework is that, for lack of available datasets, we did not evaluate mt5 on purely generative tasks such as summarization and paraphrasing. ## conclusions all of the hebrew lms to date are encoder-only models, which could not directly generate morpheme sequences, and thus necessitate a specialized monolingual decoder. in this work we propose to take advantage of mt5, a publicly available multilingual large language model that was trained on a considerable amount of multilingual and hebrew data. additionally the generative approach of text-to-text modeling is more aligned with the morphological challenges inherent in hebrew and by that dispense with the need for specially-tuned decoders. we fine-tuned and evaluated mt5 on a set of hebrew downstream tasks and report dramatic improvements. subsequently, we propose that multilingual sequence-to-sequence models provide a more suitable pretraining alternative for mrls, compared with the smaller, monolingual, encoderonly models.