How Do Transformer-Architecture Models Address Polysemy of K orean Adverbial Postpositions? (2022)

Abstract: Postpositions, which are characterized as multiple form-function associations and thus polysemous, pose a challenge to automatic identification of their usage. Several studies have used contextualized word-embedding models to reveal the functions of Korean postpositions. Despite the superior classification performance of previous studies, the particular reason how these models resolve the polysemy of Korean postpositions is not enough clear. To add more interpretation, for this reason, we devised a classification model by employing two transformer-architecture models—BERT and GPT-2—and introduces a computational simulation that interactively demonstrates how these transformer-architecture models simulate human interpretation of word-level polysemy involving Korean adverbial postpositions -ey, -eyse, and -(u)lo. Results reveal that (i) the BERT model performs better than the GPT-2 model to classify the intended function of postpositions, (ii) there is an inverse relationship between the classification accuracy and the number of functions that each postposition manifests, (iii) model performance is affected by the corpus size of each function, (iv) the models’ performance gradually improves as the epoch proceeds, and (vi) the models are affected by the scarcity of input and/or semantic closeness between the items.

## introduction polysemy, one type of ambiguity, occurs when one form delivers multiple, and yet related, meanings/functions @xcite . traditional word-embedding models have shown an unsatisfactory level of performance in polysemy interpretation(e.g., @xcite @xcite @xcite . this is mainly due to the technical nature of these models: they are static in that a single vector is assigned to each word @xcite @xcite . to overcome this issue, recent studies have proposed a contextualized word-embedding model which considers neighborhood information about a polysemous word on the basis of sequences of words around the target word @xcite @xcite . these models have achieved a good level of performance in many natural language processing tasks @xcite @xcite @xcite . among these models, transformerarchitecture models such as bidirectional encoder representations from transformer (bert; @xcite and generative pre-training 2 (gpt-2; @xcite shows the best performance for this task of polysemy interpretation @xcite @xcite . despite a good deal of research on transformerarchitecture models in english, very few studies have investigated transformer-architecture-based polysemy interpretation in languages that are typologically different from english. we therefore attend to korean, a subject-object-verb language with overt case-marking through a postposition-a bound morpheme which adds grammatical functions to a content word where it is attached @xcite . a korean postposition normally involves many-to-many associations between form and function. as such a postposition is polysemous @xcite . for example, an adverbial postposition -(u)lo (-ulo after a consonant) is interpreted as six major functions: criterion (crt), direction (dir), effector (eff), final state (fns), instrument (ins), and location (loc) @xcite . for instance, the following sentence involving the postposition -(u)lo as a marker of ins (instrument) as in (1). several studies have used transformerarchitecture models to address the word-level polysemy of korean adverbial postpositions (e.g., @xcite @xcite with the model performance (measured through a f-score) ranging from 0.776 @xcite to 0.856 @xcite . notably, the particular reason for the transformer architecture's superior performance over the others is somewhat unclear @xcite . to add more interpretation of the model performance, we devised a classification model by employing bert and gpt-2. in order to further understand how transformer-architecture models recognize word-level polysemy, we propose a transformer-architecture-based visualization system for polysemy interpretation of three adverbial postpositions, -ey, -eyse, and -(u)lo, which are frequently documented in the previous studies (e.g., @xcite @xcite @xcite . ## conclusion in this study, we made five major findings. first, bert performed better than gpt-2 in revealing the polysemy of korean postpositions. second, there was an inverse relation between the classification performance and the number of functions of each postposition. third, the model was affected by the corpus size of each function. fourth, the model was able to identify the intended functions of a postposition as the epoch progressed. fifth, these models were affected by the rarely occurring input and/or semantic closeness between the items, limiting the performance of two models in the given task to some extent. the findings of this study should be further verified by incorporating more postposition types that have similar degrees of polysemy that three adverbial postpositions demonstrate. future study will also benefit from considering other contextualized word-embedding models such as @xcite or electra @xcite to better ascertain the advantage of transformerarchitecture models in this kind of task. we believe our visualization system will con-tribute to extending the current understanding of how transformer-architecture models work for language tasks (particularly in non-english settings).