Parameter-Efficient K orean Character-Level Language Modeling (2023)

Abstract: Character-level language modeling has been shown empirically to perform well on highly agglutinative or morphologically rich languages while using only a small fraction of the parameters required by (sub)word models. Korean fits nicely into this framework, except that, like other CJK languages, it has a very large character vocabulary of 11,172 unique syllables. However, unlike Japanese Kanji and Chinese Hanzi, each Korean syllable can be uniquely factored into a small set of subcharacters, called jamo. We explore a “three-hot” scheme, where we exploit the decomposability of Korean characters to model at the syllable level but using only jamo-level representations. We find that our three-hot embedding and decoding scheme alleviates the two major issues with prior syllable- and jamo-level models. Namely, it requires fewer than 1% of the embedding parameters of a syllable model, and it does not require tripling the sequence length, as with jamo models. In addition, it addresses a theoretical flaw in a prior three-hot modeling scheme. Our experiments show that, even when reducing the number of embedding parameters by 99.6% (from 11.4M to just 36k), our model suffers no loss in translation quality compared to the baseline syllable model.

## introduction subword modeling has been used for korean to reduce the required vocabulary size due to its agglutinative nature and morphological richness. several works have characterized the many subword tokenization strategies for korean @xcite . the baseline approaches of syllable-and jamolevel modeling each have downsides. syllable models must have embeddings for all 11,172 syllables to provide full coverage, which requires an enormous number of parameters. on the other hand, jamo-level modeling requires fewer than 70 embeddings, but the sequence lengths become 3x longer, which greatly slows inference in attention-based models. here, we propose a three-hot model foot_0 that addresses both of these issues, as well as an issue in the architecture from @xcite . we emphasize that our goal is specifically to investigate parameter-efficient character-level modeling, and not to compare with (sub)word models. ## three-hot modeling naïve syllable-level and jamo-level modeling use one-hot encodings to represent each token. we propose a three-hot scheme, where jamo triplets representing syllables are consumed and generated at each step. figure 1 gives examples of each of the considered modeling schemes. ## parameter counts the syllable models contain several orders of magnitude more parameters than any other architecture. in the most extreme case, unshared syllable vs. shared, diagonal three-hot, the latter has only 36k 11.4m ≈ 0.32% as many embedding parameters. our underlying lm is a transformer model with 15.7m parameters on the target side foot_1 , not counting those for embedding or generation. thus, the unshared syllable-level embedding/decoding layers increase the target-side parameter count by 73%, while the unshared jamo, independent, and diagonal conditional three-hot increase it by less than 2%, and non-diagonal conditional three-hot by less than 6%. ## evaluation metrics since we are modeling at two different granularities, perplexity is not immediately comparable between the different architectures. to unify them, we use bits-per-jamo (bpj) as our granularity-agnostic metric. we use bleu and chrf to evaluate the quality of the translations produced by our models. since our models work on different granularities, we canonicalize their outputs for comparison. for syllable-level modeling, we split all syllables into their jamo subcharacters and leave all non-korean-syllable tokens as is. for three-hot models, triplets are flattened to a string of three characters. we remove the x pad from non-korean tokens and replace degenerate triplets with a special bad token. for all models, all jamo are converted to a for bleu, we use the standard n-gram order of 4, since this operates on the word-level (in our experiments, contiguous token sequences surrounded by whitespace). the default chrf character n-gram order is 6, which we interpret as meaning 6 syllables for korean. since we measure our metrics on decomposed jamo sequences, we use a jamo n-gram order of 18. ## experiments we use an english-korean news translation dataset from ai hub foot_2 with 400k sentences. we remove all sentence pairs that have more than 100 syllables on the korean side, leaving 365k , from which we select 5k and 5k for testing and validation. as our only goal is to evaluate the effectiveness of three-hot modeling compared to naïve syllableand jamo-level modeling, we use the same base transformer model architecture and hyperparameters in all experiments and only change the targetside embedding and decoding layers. the complete hyperparameter list is given in appendix b. in total, we train 30 models: unshared and shared weights for syllable, jamo, and three-hot independent modeling (6 total) and {unshared, shared} × {dense, diagonal} for each of the 6 conditional three-hot prediction orders (24 total). each model is trained for 50 epochs, and the epoch with the lowest bits-per-jamo on the validation set is used. for inference, we use a beam size of 15 for syllable models and 8 for jamo models. three-hot decoding is two-staged: an internal beam search constructs triplets jamo-by-jamo, and the highest probability triplets are added to the outer beam as syllables. for the independent three-hot models, we use beam size of 5 and an internal beam size of 3. for the conditional three-hot models, we use a beam size of 15 and an internal beam size of ## results and analysis independent three-hot modeling performs the worst across all metrics. on translation in particular, we suspect that it is because the model can generate high probability but meaningless triplets, especially in scenarios where several reasonable syllable continuations exist. the highest probability triplet may contain individual jamo that come from each of the true high probability syllable continuations, but which are combined in a way that forms a meaningless character. such triplets quickly saturate the beam with gibberish contexts and degrade the translation quality. for bits-per-jamo, conditional three-hot models vastly outperform all other architectures. however, this has some caveats. since the syllable model must predict all three jamo simultaneously, the bitsper-jamo is spread evenly throughout the component jamo. on the other hand, conditional three-hot modeling has a relatively difficult time predicting the first jamo, but the subsequent elements are increasingly easy due to the additional conditioning (appendix c lists bpj-per-jamo-class for each model). jamo models must deal with a longer context (and thus less focused attention) and the output layer can output any class of jamo or non-korean tokens at any time step, which makes modeling less structured and more difficult than conditional three-hot and syllable-level models. we observed that the independent three-hot models had the highest perplexity on v v , followed by v i , then v f . since this family generates jamo independently, it approximates the true entropy of the marginal subcharacter distributions. thus, we conjectured that (f, i, v) order would be the best and (v, i, f ) the worst for model accuracy, but found that no prediction order consistently outperformed the others. for translation, the bleu and chrf scores for jamo, syllable, and conditional three-hot models are extremely close. we found that, as we moved from unshared, dense three-hot models to shared, diagonal models, there was a slight loss in bpj performance, but even in the diagonal, shared case, all three-hot models perform on-par with syllable models on translation (± 0.3 bleu @xmath0.3 chrf), supporting our claims that the syllable models are massively overparameterized, and using fully dense w e and w h matrices in equation 1 is unnecessary. ## conclusion we presented a conditional three-hot decoder for korean character-level language models. our model addresses several issues with other korean (sub)character-level modeling schemes. compared to syllable-level models, it uses only a small fraction of the number of parameters, and, unlike jamolevel models, it does not triple the sequence length, avoiding the resulting inference time increase with attention mechanisms. it also addresses a theoretical flaw in a prior three-hot decoding scheme, where a syllable's subcharacters were generated independently. finally, we proposed several variants of our model to further reduce parameters. on a character-level translation task, we found that all of our conditional three-hot models perform on-par with jamo and syllable models, even when using as little as ∼0.3% of the embedding parameters of a syllable model. they also outperform the prior independent three-hot model in every metric. these results suggest that conditional three-hot modeling is an efficient and principled method of character-level korean language modeling.