A leph BERT : Language Model Pre-training and Evaluation from Sub-Word to Sentence Level (2022)

Abstract: Large Pre-trained Language Models (PLMs) have become ubiquitous in the development of language understanding technology and lie at the heart of many artificial intelligence advances. While advances reported for English using PLMs are unprecedented, reported advances using PLMs for Hebrew are few and far between. The problem is twofold. First, so far, Hebrew resources for training large language models are not of the same magnitude as their English counterparts. Second, most benchmarks available to evaluate progress in Hebrew NLP require morphological boundaries which are not available in the output of standard PLMs. In this work we remedy both aspects. We present AlephBERT, a large PLM for Modern Hebrew, trained on larger vocabulary and a larger dataset than any Hebrew PLM before. Moreover, we introduce a novel neural architecture that recovers the morphological segments encoded in contextualized embedding vectors. Based on this new morphological component we offer an evaluation suite consisting of multiple tasks and benchmarks that cover sentence-level, word-level and sub-word level analyses. On all tasks, AlephBERT obtains state-of-the-art results beyond contemporary Hebrew baselines. We make our AlephBERT model, the morphological extraction model, and the Hebrew evaluation suite publicly available, for evaluating future Hebrew PLMs.

## introduction contextualized word representations provided by models such as bert @xcite , roberta @xcite , gpt3 @xcite , t5 @xcite and more, were shown in recent years to be a critical component for obtaining state-of-the-art performance on a wide range of natural language processing (nlp) tasks, from surface syntactic tasks as tagging and parsing, to downstream semantic tasks as question answering, information extraction and text summarization. while advances reported for english using such models are unprecedented, previously reported results using plms in modern hebrew are far from satisfactory. specifically, the bert-based hebrew section of multilingual-bert @xcite (henceforth, mbert), did not provide a similar boost in performance as observed by the english section of mbert. in fact, for several reported tasks, the results of the mbert model are on a par with pre-neural models or neural models based on non-contextual embeddings @xcite ). an additional hebrew bert-based model, @xcite , has been recently released, yet without empirical evidence of performance improvements on key components of the hebrew nlp pipeline. the challenge of developing plms for morphologically-rich and medium-resourced languages such as modern hebrew is twofold. first, contextualized word representations are obtained by pre-training a large language model on massive quantities of unlabeled texts. in hebrew, the size of published texts available for training is relatively small. to wit, hebrew wikipedia (300k articles) used for training mbert is orders of magnitude smaller compared to english wikipedia (6m articles). second, commonly accepted benchmarks for evaluating hebrew models, via morpho-syntactic tagging and parsing @xcite , or named entity recognition @xcite require decomposition of words into morphemes, foot_0 which are distinct of the sub-words (a.k.a. wordpieces) provided by standard plms. such morphemes are as of yet not readily available in the plms' output embeddings. evaluating bert-based models on morphemelevel tasks is thus non-trivial due to the mismatch between the sub-word tokens used as sub-word figure 1 : plm morphological extraction pipeline. the two-word phrase " ‫הלב‬ ‫,"לבית‬ transliterated as "lbit hlbn", mapped to word-pieces which are consumed by a plm to generate contextualized vectors and extract the sub-word morphological units. in this example the wordpiece tokenizer splits the first word, "lbit", into two pieces while leaving the second word, "hlbn", intact. consequently, alephbert generates 3 embedded vectors -the vectors associated with the split word pieces are averaged to form a single contextualized vector. finally, the resulting two word vectors are used by the morphological extraction model that generates the disambiguated morphological segments. in this paper we introduce alephbert, a hebrew plm trained on more data and a larger vocabulary than any hebrew plm before. foot_2 moreover, we propose a novel architecture that extracts the morphological sub-word units implicitly encoded in the contextualized vectors outputted by plms. using alephbert and the proposed morphological extraction model we enable evaluation on all existing hebrew benchmarks. we thus present a processing and evaluation pipeline tailored to fit morphologically rich languages (mrls), i.e., covering sentence-level, word-level and most importantly sub-word morphological-level tasks (segmentation, part-of-speech tagging, full morphological tagging, dependency parsing, named entity recognition (ner) and sentiment analysis), and present new and improved sota for modern hebrew on all of these tasks. ## previous work contextualized word embedding vectors are a major driver for improved performance of deep learning models on many natural language understanding (nlu) tasks. initially, elmo @xcite and ulmfit (howard and ruder, 2018) introduced contextualized word embedding frameworks by training lstm-based models on massive amounts of texts. the linguistic quality encoded in these models was demonstrated over 6 tasks: question answering, textual entailment, semantic role labeling, coreference resolution, name entity extraction, and sentiment analysis. the next big leap was obtained with the introduction of the gpt-1 framework by @xcite . instead of using lstm layers, gpt is based on 12 layers of transformer decoders with each decoder layer composed of a 768-dimensional feed-forward layer and 12 self-attention heads. devlin et al. (2019) followed along the same lines and implemented bidirectional encoder representations from transformers, or bert in short. bert attends to the input tokens in both forward and backward directions while optimizing a masked language model and a next sentence prediction objective objectives. bert benchmarks an integral part involved in developing various plms is providing nlu multitask benchmarks used to demonstrate the linguistic abilities of new models and approaches. english bert models are evaluated on 3 standard major benchmarks. the stanford question answering dataset (squad) @xcite is used for testing paragraph-level reading comprehension abilities. @xcite selected a diverse and relatively hard set of sentence and sentence-pair tasks which comprise the general language understanding evaluation (glue) benchmark. the swag (situations with adversarial generations) dataset @xcite presents models with partial description of grounded situations to see if they can consistently predict subsequent scenarios, thus indicating abilities of commonsense reasoning. when evaluating hebrew plms, one of the key pitfalls is that there are no hebrew versions for these benchmarks. furthermore, none of the suggested benchmarks account for examining the capacity of plms for encoding the word-internal morphological structures which are inherent in mrls. in this work we enable a generic morphological-level evaluation pipeline that is suited for plms of mrls. multilingual vs. monolingual @xcite produced 2 bert models, for english and chinese. to support other languages, they trained a multilingual bert (mbert) model combining texts covering over 100 languages, in the hoped to benefit low-resource languages with the linguistic information obtained from languages with larger datasets. in reality, however, mbert performance on specific languages has not been as successful as english. consequently, several research efforts focused on building monolingual bert models as well as providing languagespecific evaluation benchmarks. @xcite trained camembert, a french bert model evaluated on syntactic and semantic tasks in addition to natural language inference tasks. @xcite trained herbert, a bert plm for polish. they evaluated it on a diverse set of existing nlu benchmarks as well as a new dataset for sentiment analysis for the e-commerce domain. @xcite created alberto, a bert model for italian, using a massive tweet collection. they tested it on several nlu tasks -subjectivity, polarity (sentiment) and irony detection in tweets. in order to obtain a large enough training corpus in low-resources languages, such as finnish @xcite and persian @xcite , a great deal of effort went into filtering and cleaning text samples obtained from web crawls. bert for mrls languages with rich morphology introduce another challenge involving the identification and extraction of sub-word morphological information. in many mrls words are composed of sub-word morphological units, with each unit acting as a single syntactic unit bearing as single pos tag (mimicking 'words' in english). @xcite addressed this for arabic, a semitic mrls, by pre-processing the training data using a morphological segmenter, producing morphological segments to be used for training arabert instead of the actual words. by doing so, they were able to produce output vectors that correspond to morphological segments rather than the original space-delimited word-tokens. however, this approach requires the application of the same segmenter at inference time as well, and like any pipeline approach, this setup is susceptible to error propagation. this risk is magnified as words in mrls may be morphologically ambiguous, and the predicted segments might not represent the correct interpretation of the words. as a result, the quality of the plm depends on the accuracy achieved by the segmenting component. a particular novelty of this work is not making any changes to the input, letting the plm encode morphological information associated with complete hebrew tokens. instead, transforming the resulting contextualized word vectors into morphological-level segments via a novel neural architecture which we discuss shortly. evaluating plms for mrls across all of the above-mentioned language-specific plms, evaluation was performed on the word-,sentence-or paragraph-level. non examined the capacity of plms to encode sub-word morphological-level information which we focus on in this work. şahin et al. (2019) probed various information types encoded in embedded word vectors. similarly to us, they focused on languages with rich morphology where linguistic signals are encoded at the morphological, subword level. their work is more about explainability -showing high positive correlation of probing tasks to the downstream tasks, especially for morphologically rich languages. unlike us, they assume a single pos tag and set of features per word in their probing tasks. in hebrew, arabic and other mrls, tokens may carry multiple pos per word, and are required to be segmented for further processing. we provide a framework that extracts subword morphological units given contextualized word vectors, that enables to evaluate plms on morphologically-aware datasets where words can have multiple pos tags and feature-bundles. corpus file size sentences words oscar (deduped) 9.8gb 20.9m 1,043m twitter 6.9gb 71.5m 774m wikipedia 1.1gb 6.3m 127m total 17.9gb 98.7m 1.9b ## alephbert pre-training data the plm termed alephbert that we provide herein is trained on a larger dataset and a larger vocabulary than any hebrew bert instantiation before. the data we train on is listed in the hebrew portions of oscar and wikipedia provide us with a training-set size orders-ofmagnitude smaller compared with resource-savvy languages, as shown in operation to optimize gpu utilization and decrease training time we split the dataset into 4 chunks based on the number of tokens in a sentence and consequently we are able to increase batch sizes and dramatically shorten training time. we trained for 5 epochs with learning rate 1e-4 followed by an additional 5 epochs with learning rate at 5e-5 for a total of 10 epochs. we trained alephbert base over the entire dataset on an nvidia dgx server with 8 v100 gpus which took 8 days. alephbert small was trained over the oscar portion only, using 4 gtx 2080ti gpus taking 5 days in total. ## the morphological extraction model modern hebrew is a semitic language with rich morphology and complex orthography. as a result, the basic processing units in the language are typically smaller than raw space-delimited tokens. subsequently, most standard evaluation tasks require knowledge of the internal morphological boundaries within the raw tokens. to accommodate this granularity requirement we developed a neural model designed to produce the disambiguated morphological segments for each token in context. these linguistic segmentations are distinct of the word-pieces employed by the plm. in the morphological extraction neural model, each input token is represented by (one or more) contextualized word-vectors produced by the plm. each word-piece token is associated with a vector, and for each space-delimited token, we average the word-piece vectors. we feed the resulting vector into a seq2seq model and encode the surface token as a sequence of characters using a bilstm, followed by a decoder that generates an output sequence of characters, using space as a special symbol signaling morphological boundaries. for tasks involving both segments and labels (part-of-speech tagging, morphological-features tagging, named-entity recognition) we expand this network in a multi-task learning setup; when generating an end-of-segment (space) symbol, the model also predicts task label, and we combine the segment-label losses. the complete morphological extraction architecture is illustrated in figure 2 . ## experimental setup goal in order to empirically gauge the effect of model size and data quantity on the quality of the language model, we compare the performance of alephbert (both small and base) with all existing hebrew bert instantiations. in this section, we detail the tasks and evaluation metrics. in the next section, we present and analyze the results. ## conclusion modern hebrew, a morphologically-rich and medium-resourced language, has for long suffered from a gap in the resources available for nlp applications, and lower level of empirical results than observed in other, resource-rich languages. this work provides the first step in remedying the situation, by making available a large hebrew plm, named alephbert, with larger vocabulary and larger training set than any hebrew plm before, and with clear evidence as to its empirical advantages. crucially, we augment the plm with a morphological disambiguation component that matches the input granularity of the downstream tasks. our system does not presuppose hebrewspecific linguistic-rules, and can be transparently applied to any language for which 2-level segmentation data (i.e., the standard ud benchmarks) exists. alephbert base obtains state-of-the-art results on morphological segmentation, pos tagging, morphological feature extraction, dependency parsing, named-entity recognition, and sentiment analysis, outperforming all existing hebrew plms. our proposed morphologically-driven pipeline foot_10 serves as a solid foundation for future evaluation of hebrew plms and of mrls in general.