Functional Lexicon in Subword Tokenization (2025)

Abstract: The distinction between function and content units of the lexicon has been somewhat neglected in recent NLP work, but it could still be useful when working with low-resource languages, and, in particular, to improve cross-lingual transfer. In this paper, we investigate to what extent BPE subword tokenization can be used to identify units of the functional lexicon in a language without any annotated data. We analyze subword tokens in terms of their productivity and attempt to find thresholds that best distinguish function from content tokens. On a sample of seven diverse languages, we find that the best results are obtained with 50 BPE merges. We also show that this subword tokenization setting can be beneficial for the interlinear glossing task.

## introduction the distinction between function and content units of the lexicon used to be important in nlp before neural models were introduced. a common preprocessing step in many tasks was the removal of "stop words" from input text before any feature extraction. lists of stop words typically included elements of the functional lexicon such as articles (e.g. a, the), pronouns (e.g. it, they, you), prepositions (e.g. at, in) and similar words. at the subword level, stemming was used to remove suffixes (e.g. -s in plays, toys). these steps were performed to reduce data sparsity by separating the form from the content so that the content could be modeled better, which is the main goal in most tasks. continuous representations in neural models removed the need for the function vs. content distinction, especially when large data sets are used for training. when working with the smaller data sets of low-resource languages, however, we expect this distinction still to be useful, especially in the context of cross-lingual transfer. function units are @xcite . the representation of function units is difficult through the lens of distributional semantics because they do not appear in fixed contexts @xcite . rather, function units tend to cooccur widely with all other units in a language. it has been shown that language models struggle to use function units' embeddings to abstract knowledge about syntax. instead, they tend to rely on previously seen, surfacelevel evidence when processing information about ideas such as tense or negation @xcite . already identifying a target language's function units as a preprocessing step could reduce confusion and improve the mapping between token vocabularies across languages. however, such pre-processing would require access to a target language's functional lexicon, i.e., the set of all of its function words and subwords. with the long-term goal of eventually introducing a function-content distinction as part of preprocessing text data for nlp systems, we address a prerequisite question in the present study: to what extent can subword tokenizers be used to extract a language's functional lexicon from raw text data? we assess the degree to which units such as those marked in figure 1 can be identified while pre-processing the text so that they can be given a special treatment depending on the application (e.g., they can be mapped to a special token or removed). more specifically, our goal is to assess the extent to which small bpe vocabularies contain function units in diverse languages and whether a specific tokenization can be used for identifying function units without any labeled data. we show that bpe models trained for 50 merging steps result in subword vocabularies that contain the most prominent function units in diverse languages. we examine the interaction between this tokenization and nlp models on the task of interlinear glossing -which implies distinguishing between lexical and functional elements -and show the benefits of its use for some models. ## related work the arbitrariness of subword tokenization is increasingly recognized as an important problem in nlp motivating efforts towards removing the need for tokenization as a pre-processing step from the nlp pipeline @xcite . the other direction is developing methods for aligning popular subword tokenizers like byte-pair encoding (bpe) @xcite , sentencepiece unigram model @xcite or wordpiece @xcite with linguistically meaningful analyses. sometimes linguistically motivated tokenization improves processing @xcite @xcite , but there is often no clear impact @xcite @xcite @xcite @xcite . the level of text compression achieved by tokenizers has been shown to impact the performance of nlp models, especially on the tasks that involve text generation @xcite @xcite . the degree of compression has also been associated with the morphological types of languages @xcite . paying attention to how the text is tokenized seems to be especially important in languages with richer morphology @xcite @xcite . more generally, the type of script and other inherent properties of languages have been shown to influence the tokenization and cause biases against some types of languages @xcite . language-specific tokenization gives better performance of nlp mod-els across languages @xcite , but this is hard to achieve in practice for multilingual pretrained models. some efforts have been directed towards a more balanced multilingual tokenization via regularization @xcite or sampling (the nllb team et al. ( 2022 )). our focus on function units is intended to contribute to a more uniform and balanced tokenization across languages, exploiting a high-level binary distinction highly relevant to describing languages with glossed examples foot_0 but also to linguistic theory @xcite . ## extracting true labels from interlinear glossing our goal is to identify function units using modern subword tokenizers so that the same method can be applied to any language for which there are written texts. to evaluate the proposed methods, we need samples of segmented texts annotated with true labels showing whether each subword unit belongs to the functional lexicon or not. glossed linguistic examples contain most of the information that we need for our experiments. glosses are traditionally used when describing languages to show how sentences are structured. they represent sentences segmented into subword units. each subword unit is annotated to show its meaning (if the unit is lexical) or its form (if the unit is an element of the functional lexicon). glosses for function units are special codes, usually written in capital letters, signaling grammatical categories. for example, the short sentence in gitksan foot_1 in figure 1 is glossed in the following way: morphemes ii n gya'a -hl xadaa glosses conj 1.i see -cn moose function yes yes no yes no text ii na gya'ahl xadaa translation and i saw the moose to obtain the annotation of function units, we add another line of annotations (function) to the data format that is provided in the glossed sentence. the mapping from the glosses to the binary labels (function unit or not) is created semi-automatically. several rules and heuristics were written into a python script to identify glosses that were representative of function units. first, we seek out glosses that were entirely written in capital letters, digits and periods or any combination of these three types of symbols (e.g. "neg", "past", "2s.sub"). this was not enough because some function units that would be included in a list of stop words are sometimes glossed with english words instead of special codes. for instance, this can happen with auxiliary verbs, pronouns, prepositions, interjections, and conjunctions. we identified such cases with a hardcoded list of function words adapted from tang (2020) and classified them as function units. the output for each language was manually reviewed by a human annotator. ## productivity analysis of bpe subword units to isolate a language's function units, we rely on a productivity analysis of subword units created by a tokenizer. in short, we test the hypothesis that the most productive subword units are function units and that we can define a productivity threshold ζ that can identify function units in any language. though we use annotated data here to evaluate various productivity thresholds, we aim to find a value of ζ that is applicable cross-linguistically so that a language's function units can be identified with just unlabeled text data. our approach is inspired by previous work showing that, cross-linguistically, texts are maximally compressed by applying a bpe model trained to carry out between 200 and 300 merges @xcite . a productivity analysis of the units created in the first 350 merges can be used to cluster typologically similar languages @xcite . for instance, languages with the highest productivity score of the bpe subword units created in the first 350 steps are kalaallisut, barasano, apurinã and alamblak, all classified in linguistic typology as polysynthetic and concatenative languages. languages whose subword units created in the first 350 bpe steps get the lowest productivity score (i.e., vietnamese, thai, sango and yoruba) are classified in linguistic typology as analytic and mostly isolating. between, we find synthetic languages such as french, persian, greek and russian with medium productivity scores. assuming that these typological distinctions found in the productivity values of bpe subwords capture, at least approximately, some formal properties of languages, we try to exploit these properties for identifying function units. we thus conduct a productivity analysis similar to @xcite but with some modifications. we define two versions of the productivity score depending on how we define the context of bpe subwords. in the first version, the context is the word in which a given subword is found. this productivity score is the number of unique words w in the set of all space-separated words w that contain a given bpe vocabulary item v after w is tokenized. in the second version, we define the context more generally as a unique sequence of three subword units (trigram). to do this, we first tokenize each sentence with bpe, resulting in a sequence of subwords such as the uspanteko sentence below, tokenized with a bpe model trained to carry out 200 merges: we then create overlapping subword trigrams for each sentence, resulting in the following for the above sentence: foot_2 e -chaq g -chaq g -a g -a -l -a -l -aa ... once we have the set of subword trigrams t , the productivity score is the number of unique trigrams that contain a subword unit v. the advantage of this second, trigram-based productivity definition is that it also assigns a high productivity to function words (when the bpe token is a whole word), which are highly unproductive under the first definition. take, for example, the function word and in english. in the word-level definition of productivity, and would be assigned a low productivity, as it is unlikely to occur in many unique words. in a trigram context, however, and would presumably have a very high productivity if it were part of the bpe vocabulary, as it is likely to occur in many unique subword contexts. the two versions of the productivity score can be formally described as follows: standardized productivity scores to make various productivity scores comparable across the measures and different experimental settings, we center and scale the values so that the distribution of the scores in each given setting has a mean of zero and standard deviation of one. ## experiments for all our experiments, we use the data from all seven languages included in sigmorphon 2023 shared task on interlinear glossing @xcite listed in ## discussion in this study, we set out to quantify how sensitive bpe is to the function-content distinction. we do so with the goal of introducing this distinction to models during preprocessing. we find that, when using bpe models trained to carry out relatively few merges (namely, 50), many function units of a language can be identified in tokenized corpora by analyzing the units' productivity. with respect to the distinction between trigram and word-based definitions of productivity, our results seem to indicate that there is not a substantial difference between the two definitions. regardless of whether we include single characters, tables 4 and 5 (appendix b) indicate that both productivity definitions result in similar maximum f1 scores at 50 merges, and that the maximum occurs using a similar threshold. this lack of difference could be related to the selection of languages in our sample, as @xcite note that all of them demonstrate synthetic morphology to some degree. as for the difference in results when we do and do not include single characters in our calculations, we see that while the maximum f1 scores are higher on average for the conditions without single characters, there is less stability with respect to the threshold ζ at which these maxima occur. concerning the results on interlinear glossing, we see that our tokenization -which we used with the intention of including function units in the input data to the models -resulted in higher average performance in glossing function units for the smt model and the deepspin-3 model. the reason why we do not see improvements with the tü-cl model might be the fact that this model learns to segment the input words according to what will maximize the score on the final gloss predictions for each segment. with the bpe-50 tokenization, the smallest units in the segments of each word become subwords rather than characters. it is then possible that with bpe-50 rather than character input, we sometimes initialize the clusters that it learns to make in a way that actually hurts the glossing performance, perhaps because the bpe segmentation does not always occur at morphologically relevant positions. still, other studies have shown that using relatively small bpe vocabularies can be beneficial for performance in down-stream tasks. for instance, in training multilingual models for translation of spanish to eleven indigenous languages, @xcite found that using small, monolingual bpe models (trained to carry out 300 merges) outperformed the same model trained on a larger joint vocabulary. moreover, @xcite found that using relatively small vocabulary units (vocabulary sizes of 500) yielded improvements in the translation of instances of nonconcatenative morphology, such as vowel harmony and reduplication. ## conclusion in the present study, we explore the extent to which a bpe tokenizer can be used to extract a language's functional lexicon without any annotated data. our experiments on using a productivity analysis over languages' bpe subword vocabularies to find the function units ultimately led us to conclude that simply applying bpe models trained to carry out 50 merges is the best way to obtain a sample of a language's function units from bpe regardless of the morphological type of the language and the amount of available data. this method is not perfect, as it yields some false positives and leaves quite a few function units unrecognized. however, it is unsupervised and equally applicable across diverse languages. introducing a large portion of function units in the input by tokenizing the text with a languagespecific bpe model trained for 50 steps improved the performance of some models on the task of interlinear glossing. in future applications, we hope to explore how we might treat those function units differently during preprocessing in an effort to improve the models' representations for different tasks.