Beyond Distribution: Investigating Language Models’ Understanding of S ino- K orean Morphemes (2025)

Abstract: We investigate whether Transformer-based language models, trained solely on Hangul text, can learn the compositional morphology of Sino-Korean (SK) morphemes, which are fundamental to Korean vocabulary. Using BERT_BASE and fastText, we conduct controlled experiments with target words and their “real” vs. “fake” neighbors—pairs that share a Hangul syllable representing the same SK morpheme vs. those that share only the Hangul syllable. Our results show that while both models—especially BERT—distinguish real and fake pairs to some extent, their performance is primarily driven by the frequency of each experimental word rather than a true understanding of SK morphemes. These findings highlight the limits of distributional learning for morpheme-level understanding and emphasize the need for explicit morphological modeling or Hanja-aware strategies to improve semantic representation in Korean language models. Our dataset and analysis code are available at: https://github.com/taeheejeon22/ko-skmorph-lm.

## introduction language models (lms) using distributional information have achieved considerable success, from traditional word embeddings to transformer-based architectures @xcite . while distribution is essential for semantic learning, human language processing also relies on additional resources beyond distribution-such as phonological and morphological information. these resources differ across languages: for example, understanding sino-korean (sk) morphemes foot_0 is critical for lexical semantics in korean. while sk words account for over 57% of the korean lexicon @xcite , native speakers often infer word meanings through sk morphemes without explicit instruction in hanja (korean chinese characters). for example (figure 1 ), mathematics learners unfamiliar with 진분 수 (眞分數 jin-bun-su 'proper fraction', where the numerator is smaller than the denominator) or 가분 수 (假分數 ga-bun-su 'improper fraction', where the numerator is larger) can infer their meanings using sk morpheme knowledge rather than context. even without prior exposure, knowing morphemes like 진 (眞 jin 'true') and 가 (假 ga 'false') enables semantic inference. psycholinguistic studies show that sk morpheme comprehension plays a crucial role in korean word recognition @xcite . native speakers'mental lexicons are closely linked to sk morpheme understanding, regardless of hanja literacy. 2 this raises an important question for ai: while lms aim to mimic human cognition, do they truly process language like humans? studies on lm linguistic capabilities have largely focused on syntax and semantics, with limited attention to morpheme-level understand-ing @xcite . while these works explore syntactic dependencies and meaning composition, morphological structure-especially in morphologically rich languages like korean-remains underexplored. sk morpheme comprehension requires syllablelevel representation and morphological awareness, but naive syllable tokenization underperforms morpheme-based and bpe approaches in korean nlp tasks @xcite . investigating whether lms acquire such knowledge tests their alignment with human language acquisition and generalization beyond surface co-occurrence. building on these points, we ask whether korean lms utilize sk morphemes in language processing. transformer-based lms typically adopt byte pair encoding (bpe) @xcite tokenizers. while full access to sk morphemes would require tokenizing each syllable, this is not feasible under bpe. thus, it is worth examining whether lms-despite lacking direct access to sk morphemes, which are essential for human-level understanding in korean-still make use of them during processing. for this purpose, we pose the research question: can a transformer-based lm, without direct access to sk morphemes, still learn their compositional morphology? we design experiments using a target sk word with a "real" neighbor (e.g., 생명 生命 saengmyeong 'life' & 생산 生産 saengsan 'production') that shares the same sk morpheme, 생 生 saeng, and a "fake" neighbor (e.g., 생명 生 命 & 생략 省略 saengnyak 'omission') that shares only the hangul syllable 생 saeng. using bidirectional encoder representations from transformers (bert) @xcite , a classic transformer model, alongside fasttext @xcite -which indirectly accesses sk morphemes via hangul syllablesyields mixed results. both bert and fasttext assign higher similarity to real than fake pairs, suggesting that bert may appear to handle the compositional morphology of sk morphemes even without explicit subword segmentation. however, further analysis reveals that bert's performance is not due to genuine morphological understanding, but rather to its reliance on word frequency in the training corpus. while neither model truly understands sk morphemes, bert's performance in particular is driven by exposure to frequent words, creating only the illusion of semantic understand-ing. in this paper, we contribute the following: ## general methodology this study examines whether an lm can learn compositional morphology purely from word distributions in hangul-only text. we construct word pairs to test whether a model can detect sk morphemes in cases of hangul homography. the idea is simple: we compare "real" pairs (same syllable and sk morpheme) and "fake" pairs (same syllable, different morphemes). a model that captures sk morphemes should assign higher similarity to real pairs. this section outlines shared methodological details across all experiments, including word pair selection (section 3.1), model selection and tokenization strategy (section 3.2), embedding extraction (section 3.3), and intrinsic evaluation approach (section 3.4). specific evaluation methods and metrics are discussed in section 4, where each experiment is detailed separately. ## experiments to investigate sk morpheme understanding in transformer-based lms without direct access to morpheme syllables, we conduct three experiments. experiment 1 (section 4.1) tests whether the models distinguish real from fake word pairs via cosine similarity. experiment 2 (section 4.2) compares top-k similar words for each pair to see if shared morphemes yield more overlap. experiment 3 (section 4.3) uses linear regression to test whether frequency-crucial in human language acquisition @xcite also affects lms. ## discussion although both bert and fasttext show relatively high accuracy overall (section 4.1), a closer look reveals that token frequency drives much of their performance, rather than a genuine grasp of sk morphemes. bert, in particular, assigns higher similarity to frequently seen words, but this reflects context-driven process that more strongly associates words appearing together often than true morphemic awareness. greater exposure to a word improves its contextual representation, but does not directly reveal its internal compositional morphology. more importantly, the models do not appear to learn this structure even indirectly: they process each word' s meaning in isolation rather than forming semantic clusters based on sk morpheme understanding. a central issue is that language models cannot explicitly encode the shared structures linking multiple words with the same sk morpheme. humans readily recognize that 국어 (國語 gug-eo 'national language'), 국립 (國立 gug-lib 'national'), and 국 사 (國史 gug-sa 'national history') share the mor-pheme 국 (國 gug 'nation') phonologically and orthographically, whereas lms fail to systematically detect such patterns. unlike humans, transformerbased models rely solely on contextual associations, overlooking information accessible through morphemic awareness. this limitation affects not only our morphemetokenized model but also typical bpe-based models. whereas humans refine their morphemic knowledge by repeatedly seeing words that share a syllable, lms trained solely on co-occurrence data struggle to internalize morphological structure. even large amounts of training data do not ensure a systematic grasp of morpheme-level structures, underscoring how purely distributional learning is insufficient for true understanding of compositional morphology of sk morphemes. to overcome these limitations, explicitly encoding shared morphemic information is key to improving lms'grasp of sk morphemes. current tokenization methods, such as bpe, rely on statistical segmentation rather than morphological principles, limiting their ability to capture systematic sk relationships. while subword models (e.g., fast-text) do include syllable-level access, our findings show that simply splitting by syllables is insufficient because one hangul syllable may map to multiple distinct hanja morphemes. moreover, @xcite demonstrate that syllable-level tokenization underperforms both morpheme-based and bpe approaches in many tasks. a more effective approach may involve linguistically informed tokenization. for example, @xcite propose a phonology-aware method that separates hangul onset-nucleus and coda, leading to improved performance over bpebased models. this highlights the need for alternative tokenization strategies that explicitly integrate sk morphemes rather than relying solely on character sequences. considering the above, incorporating hanjaaware models could enhance the performance of lms. research on human reading proficiency shows that less-proficient readers rely more on orthography, whereas more-proficient readers rely more on morphemes-likely using them to infer the meanings of unfamiliar words @xcite . similarly, explicitly integrating hanja information could help lms capture deeper morphemic structures. unlike human learners, who may find learning hanja challenging, lms face no such difficulty, making hanja incorporation a promising approach for advancing sk morpheme comprehension. more broadly, although we test sk words in korean, the issue is not language-dependent: it asks whether lms relying solely on distributional information-often effective without true morphological understanding-can be said to process language as humans do. this question is not confined to korean. japanese also has numerous sino-japanese compounds written in kanji (japanese chinese characters), with added complexity due to multiple readings. even in english, bpe tokenizers sometimes cut across morpheme boundaries (e.g., "@xmath0. in such cases, the model faces a challenge similar to that of a korean lm lacking understanding of sk morphemes. this underscores that the limitations we observed reflect a general property of distributional learning rather than an issue confined to korean nlp. finally, while our study restricted neighbors to those sharing the first syllable, future work should test whether overlap in other positions (e.g., final syllables) yields different patterns since the position of the target syllable may affect the results. in addition, although most fake pairs are contextually unrelated, some may co-occur in similar domains (e.g., 대결 對決 dae-gyeol 'match' and 대회 大會 dae-hoe 'competition'), potentially inflating similarity scores. 12 humans can readily distinguish these two types of similarity, contextual similarity vs. context-free similarity, but language models may conflate them; future studies should therefore compare model predictions with human judgments more directly. ## conclusion while modern lms successfully validate the core idea behind the distributional hypothesis @xcite , distribution alone does not fully explain word meaning. sino-korean morphemes in korean illustrate this gap. bert, a monumental transformer lm, does not seem to truly grasp their compositional morphology; rather, it learns each word separately, driven by frequency in training data. achieving deeper, human-like language understanding thus requires more than distributional cues.