Do LLM s Agree with Humans on Emotional Associations to Nonsense Words? (2024)

Abstract: Understanding human perception of nonsense words is helpful to devise product and character names that match their characteristics. Previous studies have suggested the usefulness of Large Language Models (LLMs) for estimating such human perception, but they did not focus on its emotional aspects. Hence, this study aims to elucidate the relationship of emotions evoked by nonsense words between humans and LLMs. Using a representative LLM, GPT-4, we reproduce the procedure of an existing study to analyze evoked emotions of humans for nonsense words. A positive correlation of 0.40 was found between the emotion intensity scores reproduced by GPT-4 and those manually annotated by humans. Although the correlation is not very high, this demonstrates that GPT-4 may agree with humans on emotional associations to nonsense words. Considering that the previous study reported that the correlation among human annotators was about 0.68 on average and that between a regression model trained on the annotations for real words and humans was 0.17, GPT-4’s agreement with humans is notably strong.

## introduction nonsense words (hereinafter called "nonwords") are words that do not exist within the vocabulary of a language. although these words do not have any meaning, humans often associate specific impressions and feelings to their pronunciation and spelling @xcite . a well-known example is the bouba/kiki effect @xcite , in which people tend to associate pointy and round shapes with certain sounds. understanding such human perception of nonwords brings benefits especially in commerce, as it helps to devise new product, character, and brand names that match their characteristics. also, it can contribute to discovering how humans process words in general @xcite . however, investigating 入力 未知語 [alse, roice, dw "joy"に最も関連している 最も関連していないもの 回答 joyに対する 評価値 [2] most : roice least: dworth previous studies have discussed whether large language models (llms) can function as cognitive models of natural language @xcite , suggesting their usefulness in estimating the evoked impressions of nonwords in humans. @xcite evaluated the association between the sound and form of a nonword and the association between sound and gender in llms, namely chat-gpt (openai, 2022) and vicuna @xcite . they suggest the usefulness of llms for estimating the nonword impressions in humans such as the bouba/kiki effect. however, they have not revealed how llms work for emotions which is a core component for the meaning of a language vocabulary @xcite . @xcite constructed a large, high-quality, word-emotion association lexicon to contribute to the study of emotion analysis. based on their lexicon, @xcite nonwords. for each nonword, six emotion ratings (joy, sadness, anger, disgust, fear, and surprise) were assigned according to ekman's basic emotions @xcite . crowdsourced best-worstscaling annotations were used to collect ratings by 120 native english speakers. @xcite constructed a regression model to estimate the emotion ratings of nonwords. in the training phase, the regressor was trained on the emotion ratings of real words, then it was tested on those of nonwords. this regression model showed pearson's correlation coefficient of 0.17 at best. this indicates the traditional regression approach is insufficient to deal with the emotion rating prediction of nonwords. furthermore, since there has been no research focusing on emotion predictions through llms, how they can associate nonwords with emotions is elucidated. hence, the purpose of this study is to elucidate the relationship of the emotions evoked by nonwords between an llm and humans. our contributions can be summarized as: ## emotion ratings for nonwords to measure the correlation of nonword interpretation between an llm and humans, this section proposes a method to reproduce @xcite 's best-worst-scaling annotations using an llm. following their methodology, we focus on the six basic emotions. the annotation procedures by humans and an llm are contrasted in fig. 1 . ## correlation analysis: an experiment we conducted an experiment to investigate the correlation between humans and llms regarding the emotions evoked by nonwords. ## conclusion and future work do llms agree with humans on emotional associations to nonsense words? -yes, llms somewhat agree with humans. with the aim of elucidating the correlation between an llm's and humans' understanding of nonwords, our study used gpt-4 to reproduce the emotion ratings of @xcite 's study. we found a positive correlation of approximately 0.40 between gpt-4 and human ratings. this indicates that an llm can be useful to estimate the emotions evoked by nonwords for humans. gpt-4 suggests that the meaning of real words with similar spellings largely influences its interpretation of nonwords, and that it may utilize knowledge of sound symbolism regarding emotion. the existing analysis gathered from 120 persons, surely have different personae. in the future, we plan to assign more diverse personae to explore potential variations in ratings based on factors such as gender, age, and nationality. furthermore, investigating factors affecting the llm's nonword interpretation will also be promising.