Do Pretrained Contextual Language Models Distinguish between H ebrew Homograph Analyses? (2023)

Abstract: Semitic morphologically-rich languages (MRLs) are characterized by extreme word ambiguity. Because most vowels are omitted in standard texts, many of the words are homographs with multiple possible analyses, each with a different pronunciation and different morphosyntactic properties. This ambiguity goes beyond word-sense disambiguation (WSD), and may include token segmentation into multiple word units. Previous research on MRLs claimed that standardly trained pre-trained language models (PLMs) based on word-pieces may not sufficiently capture the internal structure of such tokens in order to distinguish between these analyses.Taking Hebrew as a case study, we investigate the extent to which Hebrew homographs can be disambiguated and analyzed using PLMs. We evaluate all existing models for contextualized Hebrew embeddings on a novel Hebrew homograph challenge sets that we deliver. Our empirical results demonstrate that contemporary Hebrew contextualized embeddings outperform non-contextualized embeddings; and that they are most effective for disambiguating segmentation and morphosyntactic features, less so regarding pure word-sense disambiguation. We show that these embeddings are more effective when the number of word-piece splits is limited, and they are more effective for 2-way and 3-way ambiguities than for 4-way ambiguity. We show that the embeddings are equally effective for homographs of both balanced and skewed distributions, whether calculated as masked or unmasked tokens. Finally, we show that these embeddings are as effective for homograph disambiguation with extensive supervised training as with a few-shot setup.

## introduction semitic morphologically-rich languages (mrls) such as arabic, hebrew, and aramaic are characterized by extreme ambiguity at the word level (wintner, 2014; tsarfaty et al., 2020). in a standard text, many (and often most) of the words will be homographs with multiple possible analyses. the high ambiguity derives from several factors. first, prepositions, conjunctions, accusative pronouns, and possessive pronouns are often seamlessly affixed to words. next, vowels are generally omitted in written texts. finally, proper nouns are not differentiated from common nouns (no capital letters). the task of distinguishing between hebrew homograph analyses is related to the general task of word sense disambiguation (wsd) (agirre and edmonds, 2006; navigli, 2009), yet it is more challenging. in the standard case of wsd, a single orthographic form is associated with a single word that can be analyzed in terms of two or more senses; also, the analyses are generally pronounced identically, and often have the same morphosyntactic properties (e.g bank of a river vs. savings bank). in contrast, in semitic languages, the need for disambiguation often goes beyond a determination of sense. hebrew word ambiguities can be divided into three primary categories ( pretrained contextualized language models with standard word-piece tokenization mechanisms have been shown to excel at wsd in english and other indo-european languages (yaghoobzadeh et al., 2019). however, for hebrew and other semitic languages it has been argued that such models would not sufficiently capture the structure of mrls in order to distinguish between internally-complex homograph analyses (klein and tsarfaty, 2020; tsarfaty et al., 2020). in this work, we take modern hebrew, a semitic language with rich and highly ambiguous morphology, as a case study, and we investigate the extent to which homographs can be disambiguated by contextualized embeddings, regarding all three levels of ambiguity. regarding arabic -a sister language to hebrew -a wide survey of wsd methods is presented by abderrahim and abderrahim (2022). they raise the possibility of utilizing pretrained contextualized embeddings, yet leave its evaluation to future work. @xcite hebrew is a particularly challenging language on which to perform a homograph disambiguation due to the limited available corpora. first of all, currently existing hebrew treebanks are severely limited in size, such that most of the words in the language are not amply represented. furthermore, even regarding common hebrew words, this corpus is problematic, because the nature of language is such that many homographs are skewed in their distribution; thus, even if the primary analysis is sufficiently represented within a tagged corpus, the secondary analysis will often be hopelessly underrepresented. for instance, one common hebrew homograph is ‫מה‬ (mhm), which can be analyzed as a preposition with pronominal suffix, or as an interrogative. the ratio of these two analyses in naturally-occurring hebrew text is over 50:1; thus, occurrences of the secondary analysis within existing tagged corpora are insufficient. in theory, these homograph ambiguities could be addressed using pos tagging systems. for instance, habash and rambow (2005) consider the 1 additional studies in arabic wsd include merhben et al. (2010), merhbene et al. (2013), and shah et al. (2010). for analogous cases of skewed distribution in other languages, researchers have proposed the creation of dedicated challenge sets, containing hardto-classify sentences not easily found in naturallyoccurring text (gardner et al., 2020; elkahky et al., 2018). in the aforementioned previous study, we produced 22 such challenge sets for hebrew homographs, and demonstrated that a bi-lstm of noncontextualized embeddings can obtain high accuracy on this task, establishing the current sota for hebrew homograph disambiguation (shmidman et al., 2020). in this paper, we extend the investigation by considering whether contextualized embeddings from pretrained language models (plms) can provide a more optimal solution. we consider all existing contextualized hebrew plms: multilingual bert ("mbert") (devlin et al., 2019); hebert (chriqui and yahav, 2021); and aleph-bert (seker et al., 2021) ( our experiments demonstrate that contextualized plms pre-trained on sufficiently large unlabeled data and vocabulary size are excellent at disambiguating the word-internal structures of homographs, yet face some challenge with pure sense disambiguation. we show the efficacy of these models in cases of homographs with skewed distribution, and in a few-shot setup. finally, we establish new state-of-the-art results on the challenging task of homograph disambiguation for a morphologicallyrich language printed without vowels, along with a novel benchmark for assessing the morphological reach of future plms in hebrew. ## the data ## experimental setup to evaluate the ability of pre-trained language models (plms) to disambiguate the in-context analyses of morphologically rich and highly ambiguous homographs in hebrew, we adopt a "word expert" approach, producing dedicated classifiers for each individual homograph (zhao et al., 2020). we use two types of plms, contextualized and non-contextualized. for the non-contextualized case, we replicate our previous method detailed in shmidman et al. (2020). for each training example, we use a bilstm on top of the word2vec embeddings of all of the words in the sentence (other than the homograph itself) to produce an encoding for disambiguation. @xcite an mlp is trained to predict the correct homograph analysis based on this encoding. @xcite for the contextualized case, we run the sentence through a pretrained contextualized language model and retrieve the 768-dimension embedding representing the homograph in question. an mlp is trained to predict the correct analysis based on the homographs embeddings alone. in the standard "unmasked" scenario, the sentence is fed into the model as is, including the homograph in question. in the "masked" scenario, the homograph is replaced with a [mask] token. we evaluate the performance of each given method on each given challenge set using 10-fold cross-validation. we calculate an f1 score for each homograph analysis, based upon the precision and recall scores micro-averaged across all folds. we then calculate the macro-average of the f1 scores for all possible analyses for a given homograph, and this is the score reported in the charts herein. ## results and analysis standard (unmasked) scenario figure 1 presents the cumulative f1 score obtained by the models for all challenge sets. our results show that hebert and alephbert far outperform mbert, with alephbert achieving the higher score. the poor performance of mbert is likely due to its smaller pre-training data size and exceedingly lean hebrew vocabulary (cf. figure 2 demonstrates alephbert's performance on different ambiguity types. alephbert performs equally well on cases of segmentation ambiguity and morphosyntactic ambiguity. in contrast, when it comes to ambiguities that are purely semantic, the scores are noticeably lower. this is in line with the findings of ettinger (2020), who shows that bert is stronger with syntax than semantics; goldberg (2019) also notes bert's strong syntactic abilities. interestingly, the same gap exists with the w2v-based method. thus, both contextualized and non-contextualized embeddings struggle to differentiate between senses which are morphologically equivalent. although such cases are only of minimal import when it comes to sentence parsing, they are critical for downstream tasks such as coreference resolution and relation extraction. it thus remains a desideratum to improve disambiguation of purely semantic hebrew homographs. the results in figure 3 demonstrate that aleph-bert performs equally well on cases of binary homographs as on cases of three-way homograph classification. however, when faced with cases of 4-way or 5-way classification, accuracy declines. the effect of word-pieces previous studies have hypothesized that word-pieces are not adequate for capturing complex morphosyntactic structures due to arbitrary (non-linguistic) word-splits. to probe into this we investigate the question, do such splits affect performance. our 75 homographs are all treated as single tokens in hebert and alephbert. however, many of the homographs are broken up into word pieces in mbert, due to its meager hebrew vocabulary. we thus compare mbert's results on words treated as single tokens versus those that are broken up into two or three pieces, which are aggregated using first, sum, or average of the vectors. with regard to cases of split words, we train models using three separate methods: providing the mlp with only the embedding of the first word piece; with an average of the word piece embeddings; or with the sum of the embeddings. as shown in figure 4 , the splitting of a homograph into three word-pieces appears to have a negative impact on the ability of the resulting embedding to differentiate between homograph analyses, for all aggregation methods. ## conclusion in this study we have utilized a wide-ranging collection of hebrew homograph challenge sets in order to evaluate the extent to which raw contextualized embeddings can be leveraged to disambiguate hebrew homographs. we found that contextualized embeddings can effectively disambiguate analyses of homographs, much more so than non-contextualized ones, regarding multiple types of ambiguity: segmentation, morphosyntactic and sense. yet, efficacy on pure sense ambiguity is lower than on the other two types. additionally, an increasing number of splits, or an increasing number of different possible analyses of a token, each lower efficacy. finally, we found that contextualized embeddings can function effectively for this purpose on a few-shot basis, with as little as 5 examples of each analysis. this indicates that with relatively modest effort, highly ambiguous homographs may be effectively treated.