LLM Neologism: Emergence of Mutated Characters due to Byte Encoding (2024)

Abstract: The process of language generation, which selects the most probable tokens one by one, may intrinsically result in output strings that humans never utter. We name this phenomenon “LLM neologism” and investigate it focusing on Japanese, Chinese, and Korean languages, where tokens can be smaller than characters. Our findings show that LLM neologism occurs through the combination of two high-frequency words with common tokens. We also clarify the cause of LLM neologism in the tokenization process with limited vocabularies. The results of this study provides important clues for better encoding of multibyte characters, aiming to prevent catastrophic results in AI-generated documents.

## introduction the text generation capabilities of llms have been improving year by year @xcite , and the sentences generated by llms have become indistinguishable from those written by humans. however, llms occasionally output non-existent words. although this is a rare phenomenon, its occurrence is a clear indication of an ai-generated sentence and thus should be avoided as much as possible. in this paper, we name this phenomenon llm neologism and investigate it thoroughly. the phenomenon is a type of hallucination. llm tends to cause hallucination, in which information that is not true is presented as if it were true @xcite . hallucination is divided into various types @xcite , but to the author's knowlegde, this type of hallucination is that has not been adressed in any previous paper. figure 1 shows the notion of llm neologism, where a non-existent japanese word "保 " is generated. we call such a word a neo-word. in languages where a single character can be split into multiple tokens, such as chinese, japanese, and korean, the generation of a neo-word is triggered by the mutation of token sequences of two frequentlyused words. additionally, the mixture of tokens in the prediction of output token sequences, those derived from two frequent words in the training data may be mutated. this results in a peculiar word (neo-word) that has a neo-char generated by the decoding of mixed byte codes. in section 2, we discuss the mechanism underlying llm neologism in more detail. in section 3, we explain the tokenization strategies in the existing llms, and in section 4, we present our observation of llm neologism in a systematic way. the main contribution of this work are as follows: here, we set up a hypothesis that a neo-word is generated from two frequent two-letter words that share the first letter and tend to appear in similar contexts. this explains the llm neologism in figure 1 . in the process of generating "保 険"('insurance'), after outputting its first two tokens foot_1 , it is impossible to guarantee the prediction of the code 0xba. instead, another token 0xbc, derived from "保証"('guarantee'), can have higher probability than 0xba, and this results in the generation of a neo-word that contains a neo-char. neo-words have been found on the web. a blog post foot_2 reported that chatgpt output gibberish japanese-like words that have never been seen before and that were subsequently used in a number the generated neo-char can be a very infrequently used character, causing a reduction in the naturality of the llm output and a critical problem of being revealed as ai-generated. for this reason, even if the rate of occurrence is not high, it is important to prevent llm neologism. in the next section, we discuss the tokenizer properties that are related to the occurrence of llm neologism. ## tokenizers in this section, we discusses the relationship between llm neologism and the underlying tokenization process. the generation of neo-words by a model depends on how characters are splitspecifically, on the tokenizer's vocabulary. many llms, such as llama2/3 (ai@meta, 2024), and gpt-3.5/4.0, use byte-pair-encoding (bpe) @xcite for tokenization. gpt-3.5 has 782 tokens for single or double bytes in its vocabulary, and llama2 has 256 tokens for a single byte. the combination of these model generated neo-word generated text or web text gpt yes web text llama2 yes web text elyza yes generated text elyza-fast no -granite no -swallow no - ## replication of llm neologism in this section, we list potential neo-words to determine the occurrence of llm neologism, and discuss its relationship with the tokenization. ## conclusion in this paper, we defined llm neologism and revealed its characteristics. we showed that neowords in japanese, chinese, and korean are generated from two frequent two-letter words that share a first letter and tend to appear in a similar context. neo-words are generated when a single character is split into multiple tokens, and we clarified that the likelihood of their generation depends on the tokenization method and the vocabulary. we demonstrated that neo-words in three languages appear in ai-generated texts, and showed that neo-words exist in context in a similar sense to constituent words. llm neologism is a tokenizer-dependent problem that occurs when a character is represented by multiple tokens. as stated by @xcite , there is no silver bullet solution that serves as a solution for all purposes. however, llm neologism is an essential issue to consider in the context of generating natural sentences in cjk languages. it is also known that llm can generate new words by mixing words in english 5 . it is a future challenge to generalize llm neologism in languages other than cjk.