Jamo-Level Subword Tokenization in Low-Resource K orean Machine Translation (2025)

Abstract: Subword tokenization, where text is represented in an intermediate form between full words and characters, is ubiquitous in modern NLP due to its ability to represent any input sentence with a small vocabulary. However for Korean, where there are 11,172 base characters (*syllables*) in its alphabet, it is difficult to have a vocabulary large enough to succinctly encode text while fitting within parameter-budget constraints. This motivates us to explore an alternative representation for Korean which relies on the decompositional nature of Korean syllables: a syllable can be uniquely decomposed into a sequence of two or three subcharacters (*jamo*), of which there are only 68.Using jamo as the basis for subword tokenization (e.g., byte-pair encoding) leads to shorter tokenized sequences with fewer vocabulary parameters, exposes the model to sub-syllable-level morphological information, and increases the amount of augmentation gained from subword regularization. We evaluate jamo-level subword tokenization on several Korean translation tasks and find that jamo-level subword models consistently outperform syllable- and byte-level models in low-resource and restricted-vocabulary settings.

## introduction modern language models struggle with languages such as chinese, japanese, and korean, where the large base character sets (hanzi, kanji/kana, and korean syllables, respectively) mean that a huge vocabulary parameter count is required to represent the base character set and the added subword tokens. however, unlike hanzi and kanji, the 11,172 modern korean syllables have a compositional example 1: example tokenizations for "공부합니다" (to study). from top to bottom: syllable-level, jamo-level, and the jamo-level tokenization mapped back to the syllable sequence. note that the jamo-level tokenization is able to cross syllable boundaries, while the syllablelevel tokenization cannot. we hypothesize that jamo-level subword tokenization should improve over syllable-level subword tokenization for three reasons. compared to syllable-level subword tokenization, jamo-level subword tokenization: in english↔korean and korean↔jeju-eo (a very-low-resource koreanic language spoken on korea's jeju island) translation tasks, we find that jamo-level models do not improve upon syllablelevel models in high resource settings. however, in low-resource and low-parameter settings (specifically, extremely small vocabulary budgets), jamo-level subword models greatly improve over both syllable-and byte-level subword models, especially when dropout is used. this supports our hypothesis about the benefits of jamo-level subword models. ## byte-pair encoding and subword regularization subword tokenization transforms free-form text into a sequence of tokens drawn from a finite vocabulary. the tokens occupy an intermediate granularity between full words (e.g., democratic or 대한민국) and character sequences (e.g., d e m o c r a t i c or 대 한 민 국), known as subwords (e.g., demo cratic or 대 한 민국). a key feature is that subword tokenizers are open vocabulary-any possible input text can be represented by them (i.e., they never produce out-of-vocabulary (oov) tokens). this is realized by including a comprehensive atomic vocabulary in the final subword vocabulary (e.g., the full alphabet for english or the set of all korean syllables). byte-pair encoding (bpe) is a simple and popular subword tokenization technique first introduced by @xcite and reintroduced by @xcite for neural machine translation. to construct the vocabulary, starting from the atomiccharacter-level representation of the training corpus, the model iteratively finds the most frequent co-occurring pair of tokens, merges them, and adds the merge to the subword vocabulary. this process repeats until the vocabulary reaches the desired size. for inference, starting from the character-level input, among all possible merges, the highest priority merge (the one that was added earliest to the merge vocabulary during training) is executed. this repeats until no possible merges remain. subword regularization, where tokenizations are sampled from a probability distribution, is often used during training to augment the dataset and improve model robustness, and has been shown to improve model quality across a wide variety of tasks, especially in low-resource machine translation @xcite . bpe-dropout @xcite ) is a variant of bpe's inference algorithm that injects randomness into the tokenization process as a form of subword regularization. ## hangul and jamo decomposition hangul, the modern korean writing system, is represented in unicode as a list of 11,172 unique syl- note that there are visual overlaps between some initial and final consonants (e.g., ㄱ), and that compatibility jamo is the minimal set of visually distinct jamo. to produce a decomposed jamo sequence from a syllable sequence, we concatenate the jamo from each triplet corresponding to the syllables. for a syllable without a final consonant, like 구 = (ㄱ, ㅜ, ∅), we omit the ∅ when writing out the jamo sequence foot_1 (i.e., 가 → (ㄱ, ㅏ, ∅) → ㄱㅏ). thus, every syllable maps to a length of two or three jamo sequence. example 1 demonstrates a syllable-level and jamo-level subword tokenization. hangul is represented in three different ways in unicode: precomposed syllables foot_2 , positional jamo foot_3 , and compatibility jamo foot_4 . all jamo (including ∅, which denotes the absence of a final consonant), organized by class, are shown in the syllable block (u+ac00-u+d7af) contains all 11,172 modern syllables, each represented by a single codepoint. a syllable can be decomposed into either of the jamo representations, outlined here and discussed in detail in appendix b. ## benefits of jamo-level tokenization one may ask, why even consider jamo-level tokenization, as syllable-level tokenization is the canonical unit of text? furthermore, byte-level bpe can be used to alleviate any space issues, and perhaps, as the vocabulary size grows, jamo-level tokens will just converge to syllable boundaries. tokenization length and small vocabularies especially at smaller vocabulary sizes, jamo-level bpe produces substantially shorter tokenized sequences compared to syllable level models, which is crucial to developing models that fit within some parameter or inference budget. as seen in figure 1 , compatibility and positional jamo produce shorter sequences than syllable-level subword tokenizers across all vocabulary sizes, until they eventually converge. when the vocabulary size is small, jamolevel models far outperform syllable-level models in compression ratio. this is especially true at the lowest end of the scale, where syllable-level models operate at essentially the character level (i.e., the entire vocabulary budget is taken up by the base syllable vocabulary, so no merges can be added). even with a vocabulary budget of just 500, jamo-level subword models produce 5% shorter sequences than syllable-level models with |v | = 2100, the smallest possible in our corpus. at |v | = 1000 and 1500, jamo-level models produce nearly 25% and 40% shorter sequences than a |v | = 2000 syllable model, respectively. having shorter sequence lengths is a common metric for tokenizer performance and is also an important consideration with attention-based language models that scale quadratically with sequence length. sub-syllable morphology morphemes in korean are not necessarily constrained to syllable boundaries. for example, the word 하기 (hagi/doing/gerund form) can be morphologically segmented into 하 (ha/to do/root) and 기 (gi/ing/nominalization). on the other hand, 합니다 (hab-ni-da/to do/honorific form) is morphologically segmented into 하 (ha/to do/root) and -ㅂ니 다 (b-nida/present tense honorific/ending). notice that the second morpheme contains an incomplete syllable (the final consonant ㅂ) which is part of the first syllable in 합니다 (similar to example 1). when modeling at the syllable level, this and similar jamo-level morphology is impossible to capture, as the tokenization is constrained to syllable boundaries, so either the ㅂ jamo information is lost due to being split off from -니다 or the entire sequence 합니다 is represented as a single subword token, and the underlying morphological information is not able to be shared with other sequences that also include the -ㅂ니다 morpheme. however, with jamo-level bpe, which is not constrained to syllable boundaries, it is possible to capture such morphological patterns. increased subword regularization subword regularization improves model robustness by sampling tokenizations during training, which augments the training set and breaks the model's conditioning on an exact, canonical tokenization. a key factor is the amount of augmentation (that is, the number of unique tokenized sequences) that are produced during stochastic tokenization, with the implication that a larger number of tokenizations should lead to a larger improvement in model quality and robustness @xcite . let w s be a sequence of korean syllables and w j the same sequence decomposed into jamo form (the argument works identically for positional and compatibility jamo). for simplicity, assume every subsequence of w s and w j is a token in the respective subword vocabularies. then, the stars and bars theorem says that tokenizing a sequence of length n into k tokens can be done in n-1 k-1 ways (wikipedia contributors, 2024) and the total number of ways to tokenize a sequence w is the maximum tokenized sequence lengths for a syllable and jamo sequence are |w s | and |w j |, respectively, corresponding to just their characterlevel sequences, which must be a valid tokenization. as ## experiments we compare syllable-level, byte-level, and jamolevel bpe tokenization on two translation tasks: english↔korean using the ai-hub news translation corpus foot_5 and korean↔jeju-eo @xcite (section 6.3). the full english↔korean dataset contains 800k sentence pairs, but we use a 200k sentence-pair subset given our focus on lowerresource settings (section 6.1). however, experimental results for the full corpus are given in appendix c and an analysis of a restricted-vocabulary experimental setting on the larger corpus is given in section 6.2.1. the korean↔jeju-eo corpus contains 180k sentence pairs. we use sentencepiece @xcite as the bpe tokenizer implementation and fairseq @xcite for training our language models. for each experiment, we fix an underlying transformer architecture and only vary the tokenizer according to the korean representation and vocabulary sizes. the source and target side tokenizers are trained separately and their parameters are not shared. all other training configurations are held equal in all experiments. the complete model and training information is given in appendix a. for each task, we compare the bleu, chrf (both via sacrebleu @xcite ), and comet @xcite ) scores of each model (using an average of 3 runs). all metrics are computed at the syllable level (after the jamo-level model outputs are recomposed to syllables). in the english↔korean corpus, there are ∼4600 unique characters. however, about ∼2500 of these are chinese, japanese, and korean (cjk) ideographs (hanzi/kanji/hanja), which appear in only a small fraction of sentences and with median frequency 1, but take up an outsized amount of the vocabulary if we include all observed symbols to avoid oov. to isolate our main focus (the representation of korean text), we remove all sentences with these ideographs. we analyze three axes: 1) input representation, 2) subword regularization, and 3) vocabulary size, and corpus size via the following experiments: for vocabulary size, the "full" vocabulary size was chosen arbitrarily without a hyperparameter sweep for a fair comparison. however, the "restricted" vocabulary size was chosen as the number of unique characters in the syllable-level corpus. in this setting, the syllable-level models act as character-level models (since there is no room in the vocabulary for merged tokens), while the other representations can still form additional subwords. ## analysis overall, we observe that using jamo-level subword tokenization performed on-par with or better than syllable-level tokenization in non-dropout settings and with large vocabularies. this is to be expected, especially at large vocabulary sizes where the jamolevel tokenizers converge to syllable-level tokens. one hypothesized benefit of using jamo-level subwords is that the increased amount of subword regularization would lead to better modeling. this was most clearly observed in the lowest-resource setting (korean↔jeju-eo), but we also observed it in the @xmath0. in the highest resource setting, dropout did not improve modeling quality at all, but this is in line with other research @xcite . for korean↔jeju-eo, not only did positional jamo-level models with dropout score better than all other models (with or without @xcite investigate korean tokenization in various natural language understanding tasks. the strategies compared include jamo and syllable character-level modeling, morphological segmentation provided by mecab-ko @xcite , syllable-level bpe, and word-level segmentation. in another work, @xcite consider jamo-level byte-pair encoding for jeju-eo in a similar way to what we explore here. however, this is in the context of text-to-speech, and the jamolevel subword encoding is only applied to the input side but not to generation. for a translation task, park and zhao (2020) used hierarchical syllable and jamo-level features, but also only for encoding. jamo-level modeling has been applied to many encoder-only tasks such as named entity recognition @xcite and sentence classification @xcite . for decoding, both @xcite and @xcite used jamo-level representations in character-level korean language-modeling tasks. however, their approaches do not apply to subword-level modeling. for chinese and japanese, which have large base vocabularies due to their use of ideographs, radicalbased decomposition has been explored as a possible sub-character method to reduce the required vocabulary budget and improve modeling @xcite @xcite . however, unlike korean, radicalbased decomposition is not lossless. ## conclusion we investigate jamo-level subword tokenization for korean machine translation based on three theoretical benefits-shorter tokenized sequences and better vocabulary allocation, exposure to sub-syllable morphological information, and larger space of tokenizations for subword regularization-and show that in two translation tasks and across multiple experimental settings, jamo-level models outperform syllable-level models and byte-level models. our experimental results support the hypothesized advantages of jamo-level subword modeling in that: 1) in small-vocabulary settings, jamolevel models far outperform syllable-level models (which essentially act like character-level models), 2) jamo-level models outperform byte-level models across the board (with the primary difference between them being that jamo models preserve subsyllable information that byte-level models do not and syllable-level models cannot), and 3) with the same dropout hyperparameters, jamo-level models both perform better than analogous syllable-and byte-level models and also exhibit a larger increase in performance over the non-dropout models as compared to syllable-and byte-level models. we conclude with the recommendation against simply defaulting to syllable-level or byte-level subword tokenization for korean nlp, as it can lead to poor tokenizations and loss of performance, especially in low-resource settings. we hope to have provided sufficient basis for further exploration into jamo-level subword tokenization with our work.