C ompound P iece: Evaluating and Improving Decompounding Performance of Language Models (2023)

Abstract: While many languages possess processes of joining two or more words to create compound words, previous studies have been typically limited only to languages with excessively productive compound formation (e.g., German, Dutch) and there is no public dataset containing compound and non-compound words across a large number of languages. In this work, we systematically study decompounding, the task of splitting compound words into their constituents, at a wide scale. We first address the data gap by introducing a dataset of 255k compound and non-compound words across 56 diverse languages obtained from Wiktionary. We then use this dataset to evaluate an array of Large Language Models (LLMs) on the decompounding task. We find that LLMs perform poorly, especially on words which are tokenized unfavorably by subword tokenization. We thus introduce a novel methodology to train dedicated models for decompounding. The proposed two-stage procedure relies on a fully self-supervised objective in the first stage, while the second, supervised learning stage optionally fine-tunes the model on the annotated Wiktionary data. Our self-supervised models outperform the prior best unsupervised decompounding models by 13.9% accuracy on average. Our fine-tuned models outperform all prior (language-specific) decompounding tools. Furthermore, we use our models to leverage decompounding during the creation of a subword tokenizer, which we refer to as CompoundPiece. CompoundPiece tokenizes compound words more favorably on average, leading to improved performance on decompounding over an otherwise equivalent model using SentencePiece tokenization.

## introduction decompounding is the task of separating compound words into their single word constituents. decompounding is used in user-facing tools such as dictionaries and morphological analyzers (altinok, † equal senior authorship. hard compound easy compound wind surfing spell binding winds f ur ing spell ing bind figure 2: examples of easy and hard compounds w.r.t. the t5 tokenizer (also used by flan ul2 20b). decompounding can come in two similar yet different task formats: (i) compound segmentation and (ii) compound normalization (ziering and van der plas, 2016). compound segmentation is the task of segmenting a word into its compound constituents, while preserving its surface form (e.g. @xmath0. compound normalization is the task of recovering the base form of each compound constituent (e.g. @xmath1. 1 most prior work on decompounding has focused on the few languages with excessively productive compound formation such as finnish, german and swedish @xcite @xcite . however, compound words occur in a large, diverse number of languages @xcite . yet, datasets which annotate compounds with their segmented or normalized form sparsely exist, even in languages with high compound usage. as the first contribution of this work, we aim to address this issue by introducing a dataset of 255k compound words and their normalized form as well as non-compound words covering 56 languages obtained from wiktionary ( www.wiktionary.org ). using our dataset, we then find that large language models (llms), which typically rely on subword-based tokenization @xcite , struggle with decompounding, as illustrated in figure 1 . performance is especially low for compounds where subword boundaries do not coincide with compound constituent boundaries; we term compounds with this property 'hard' compounds (figure 2 ). in order to create a more effective decompounding model, we then formulate compound segmentation and normalization as a sequence-to-sequence learning task @xcite and train a byte-level byt5 model @xcite using a two-stage framework. in the first stage, we use a novel self-supervised hyphen-prediction objective to learn compound segmentation without any labeled data. in the second stage, we turn the model into a compound normalization model via supervised training on our wiktionary data. in addition, we introduce a procedure to predict the segmentation of any compound word based on its normalized form, effectively making compound segmentation a subtask of normalization. finally, we demonstrate that decompounding has real-world applications by investigating compound segmentation for language model tokenization. we apply compound segmentation as pretokenization during training of a sentencepiece tokenizer @xcite , which results in fewer hard compounds while incurring no extra cost during training and inference of the language model (i.e. the only extra cost occurs during creation of the tokenizer). our stage 1 models outperform the best prior unsupervised models by 13.9% accuracy on average, while our (supervised) stage 2 models outperform all prior language-specific decompounding tools. furthermore, a model trained with a com-poundpiece tokenizer achieves a 5.5% improved performance on compound normalization over an otherwise equivalent sentencepiece model. ## results and discussion main compound segmentation results are shown in we further compare against language-specific and supervised methods in ## conclusion we systematically investigated word decompounding tasks of compound segmentation and normalization on a wide scale and in multilingual contexts. to this end, we introduced a dataset of 255k words including compounds and non-compounds across 56 languages from wiktionary, which allowed us to evaluate performance of llms on decompounding. we found that current llms' performance is limited due to hard compounds which arise when subword token boundaries do not coincide with compound constituent boundaries. we then introduced dedicated models for decompounding which use byte-level tokenization to entirely avoid hard compounds. finally, we used our decompounding models to create novel compoundpiece tokenizers, keeping the efficiency advantages of subword tokenization while strongly decreasing the amount of hard compounds; this increases the performance of compoundpiece models over comparable senten-cepiece models on the decompounding tasks.