Different Tokenization Schemes Lead to Comparable Performance in S panish Number Agreement (2024)

Abstract: The relationship between language model tokenization and performance is an open area of research. Here, we investigate how different tokenization schemes impact number agreement in Spanish plurals. We find that morphologically-aligned tokenization performs similarly to other tokenization schemes, even when induced artificially for words that would not be tokenized that way during training. We then present exploratory analyses demonstrating that language model embeddings for different plural tokenizations have similar distributions along the embedding space axis that maximally distinguishes singular and plural nouns. Our results suggest that morphologically-aligned tokenization is a viable tokenization approach, and existing models already generalize some morphological patterns to new items. However, our results indicate that morphological tokenization is not strictly required for performance.

## introduction in natural language processing (nlp) pipelines, tokenizers segment unstructured text into smaller, discrete constituents ("tokens") for further processing. importantly, different tokenizers can incur performance and efficiency trade-offs. assigning a unique token to each word in a corpus may lead to high-precision semantic representations, but the resulting models might be less robust to unseen words and require more computational resources. most existing tokenizers allow words to be decomposed into subword tokens @xcite . they can do so along morphological boundaries (e.g. books to @xcite ), but this behavior is not guaranteed. segmenting words into their lemmas and morphemes might simultaneously allow models to more robustly learn morphosyntactic patterns, more efficiently represent such patterns, and better *equal contribution. in the current work, we ask whether and how the tokenization strategy employed facilitates successful language model predictions. we evaluate the effect of three types of plural noun tokenization in spanish-single-token plurals, morphemically-tokenized plurals, and nonmorphemically-tokenized plurals-in the context of a masked article prediction task ( §4). 1 we focus on tokenization schemes for plural forms in spanish, as it offers relatively simple and frequent examples of morphologically complex words. spanish leverages two primary plural marking strategies, which are highly predictable for any given lemma. we specifically focus on cases where the plural form is composed of the singular form with the addition of '-s' or '-es'. we find that tokenization schemes are differentially successful, although the effect is small, and article agreement accuracy is high across all tokenization types. artificial tokenization schemes, where we coerce an initially single-token or nonmorphemically-tokenized plural into a morphemic representation, leads to successful task performance, but does not improve performance beyond the original tokenization scheme. in an exploratory analysis, we compare singular and plural form embeddings across all tokenization schemes. we find axes with high overlap between all plural forms (regardless of tokenization scheme) and high discriminability between plural and singular forms, but other axes can still separate different plural tokenization schemes. this work contributes to a growing literature examining the impact of tokenization on the language modeling objective. code and data are available: https://github.com/catherinearnett/ spanish-plural-agreement . ## related work several studies have investigated morpho-syntactic agreement in bert-style models across multiple languages @xcite @xcite , finding generally high agreement accuracy. in a subject-verb agreement task, however, beto incurs a relatively high rate of agreement errors for certain spanish nouns (despite the ability to extend number agreement to novel words; @xcite . it is unclear to what extent degraded performance is attributable to tokenization scheme, but the word "comanas"-listed as an example of a frequently mis-numbered word-is tokenized nonmorphemically into @xcite . indeed, recent work has demonstrated that morphologically-aware tokenization improves nlp model performance on a variety of downstream benchmarks @xcite @xcite @xcite . most relevantly, @xcite devise a tool to classify english words in terms of whether they are stored as single tokens ("vocab"), as multiple morphemic tokens ("morph"), or as multiple non-morphemic tokens ("alien"). the authors find that how multi-morphemic english words are tokenized is correlated with the language model's downstream performance on several tasks. following @xcite , our work investigates how the tokenization of spanish nouns affects language model predictions involving a specific morphosyntactic rule, providing insight into how morphologically-aware tokenization affects nlp model performance. ## model and data all experiments use beto, a spanish pre-trained bert model @xcite with 110m parameters trained on approximately 3b words. beto uses a sentencepiece tokenizer @xcite with a 32k vocab size. ## study: article-noun agreement our primary research question concerned the impact of the original tokenization (tokenization scheme) on an article agreement task, similar to that implemented by @xcite . in spanish, articles must agree with the number of the noun (e.g., la mujer vs. las mujeres); learned representations for the target noun should thus be conducive to predicting article number. we asked: ## linear discriminant analysis (lda) to identify potential causes for the observed agreement patterns across noun types (singular vs. different plural tokenizations), we considered the embeddings of those nouns in the language model representation space. we took each noun's mean embedding across the last four (out of twelve) beto transformer layers, averaging over all tokens in the noun. to minimize confounds from averaging embeddings over different numbers of tokens, we considered only two-token plurals in all multitoken scenarios for embedding analyses. we first identified the linear axis that maximally separated single-token singular from plural nouns. to do this, we ran linear discriminant analysis (lda) with two classes of embeddings: singular nouns (all single-token) and single-token plural nouns. 2 we then projected all noun representations linearly onto this axis, essentially projecting each embedding into a single value. as expected, we found that singular nouns clustered separately from plural nouns (figure 2 , left). notably, all types of plurals (single-token, artificially tokenized, twotoken morphemic, and two-token non-morphemic) patterned together and were not linearly discriminable along this axis. this suggests that the model could rely on similar number agreement mechanisms for different types of plurals, but future work would need to demonstrate causal impacts of this singular-plural axis on number agreement predictions (e.g. as in @xcite . while the singular-plural lda axis mapped different plural types to similar values, other axes could separate embeddings for the different plural types. we used lda to identify the three linear axes that maximally separated the four types of plurals. as shown in figure 2 (right), singletoken plurals and two-token non-morphemic plu-rals were separable from one another and from all other plural types. the artificial and default morphemic plurals had distinct clusters, but they were not entirely separable from one another. this indicates that even though the artificial tokenization was never seen by the model during training, the representations were still quite similar (e.g. due to the presence of the '##s' or '##es' token). the slight separation between these clusters may be driven either by frequency effects or by veridical differences in how the models represent number in the two plural types. ## discussion and conclusion we assessed whether distinct tokenization schemes impacted the ability of beto (a spanish language model) to predict appropriate articles for spanish plural nouns. single-token representations facilitated slightly better predictions overall. however, the model did show evidence of generalization consistent with having learned morpheme-like "rules": artificially re-tokenizing plural nouns along morpheme boundaries produced representations amenable to article prediction-despite the language model never having previously observed that sequence of tokens (see figure 1 )-though this approach was slightly less accurate than relying on the original tokenization scheme. this provides further insight into work on language models generalizing morphological patterns @xcite ; however, this does not work equally well for all languages or models @xcite . notably, the similar agreement performance across single-token, morphological, nonmorphological, and artificially-tokenized plurals could indicate multiple different agreement mechanisms in the model. at least on this task, tokenization along morpheme boundaries was not correlated with improved agreement performance; this is in contrast to other work suggesting that morphologically aware tokenization improves performance, e.g., in machine translation @xcite or similarity judgments @xcite . future work might apply causal interventions on different embedding axes (as found in §5), to determine the extent to which the same model subnetworks are involved in number agreement for different types of plural tokenizations, shedding light on the impacts of tokenization on language model processing.