NEO - BENCH : Evaluating Robustness of Large Language Models with Neologisms (2024)

Abstract: The performance of Large Language Models (LLMs) degrades from the temporal drift between data used for model training and newer text seen during inference. One understudied avenue of language change causing data drift is the emergence of neologisms – new word forms – over time. We create a diverse resource of recent English neologisms by using several popular collection methods. We analyze temporal drift using neologisms by comparing sentences containing new words with near-identical sentences that replace neologisms with existing substitute words. Model performance is nearly halved in machine translation when a single neologism is introduced in a sentence. Motivated by these results, we construct a benchmark to evaluate LLMs’ ability to generalize to neologisms with various natural language understanding tasks and model perplexity. Models with later knowledge cutoff dates yield lower perplexities and perform better in downstream tasks. LLMs are also affected differently based on the linguistic origins of words, indicating that neologisms are complex for static LLMs to address. We will release our benchmark and code for reproducing our experiments.

## introduction neologisms -recent word forms representing a new meaning, sense, or connotation @xcite -consistently surface as language changes. neologisms emerge to describe the ever-changing state of the world, such as new terms created during the covid-19 pandemic. while humans easily adapt to language change, large language models (llms) struggle with the misalignment of training data and new test data distributions @xcite . prior work on temporal language change @xcite @xcite observed model degradation when finetuning on older text and evaluating on newer data and named entities @xcite ; agar- doomscrolling and ways to prevent it 2023 2022 2021 2020 wal and nenkova, 2022; @xcite . however, as far as we are aware there has not been prior work that analyzes the robustness of llms on handling neologisms. we show that adding a neologism to text decreases machine translation quality by an average of 43% in a human evaluation ( §2), even for popular words emerging before 2020. in this paper, we present neo-bench, a new benchmark designed to test the ability of llms to understand and process neologisms. we combine multiple methods and online text corpora to collect a diverse set of 2,505 neologisms based on the linguistic taxonomy devised by @xcite : (i) lexical neologisms -words representing new concepts, e.g., "long covid"; (ii) morphological neologisms -blends of existing subwords, e.g., "doomscrolling"; and (iii) semantic neologisms -existing words that convey a new meaning or sense, e.g., "ice" (a term that refers to petrolor diesel-powered cars taking electric car charging spots). we estimate word prevalence over time with google trends to obtain trending neologisms. we also create 4 benchmark tasks to evaluate the 1 13885 figure 2 : a single neologism can dramatically affect model output, as shown by human evaluation of machine translation models on sentences containing neologisms and the same sentences with neologisms replaced by carefully chosen words that also fit in the context. oracle ensemble selects the best translation from all models. neo-bench evaluates a diverse set of llm capabilities on handling neologisms in various tasks. models must also understand compositionality for morphological neologisms, differentiate between word senses for semantic neologisms, and handle different contexts for lexical neologisms. ## motivation we start by using machine translation as an example to illustrate the significant challenge neologisms pose on state-of-the-art nlp systems. we manually collect 100 neologism words with sentential context from social media, news articles, and dictionaries. gpt-3.5, gpt-4 and commercial translation systems, e.g., google translate, foot_0 microsoft bing, foot_1 and deepl translator, foot_2 only managed to correctly translate about 34-47% of these 100 sentences that contain neologisms based on our manual inspection (figure 2 ; from english to chinese). in stark contrast, when replacing the neologism with a common word in these sentences, the percentage of correct translations rises substantially to 67-73%. we observe similar trends in open-source translation models, such as alma @xcite and m2m100 @xcite . one thing to note is that these replacement words are not exact synonyms, but words that have been carefully chosen to create a near-identical, semantically plausible sentence; because new words emerge in areas not occupied by existing words @xcite , true synonyms would often be verbose and incompatible with the sentence context. because the original sentences containing neologisms were collected in the wild, one might assume they would be even more natural in comparison to their modified counterparts, but yet, there is a large gap in translation quality between neologism and non-neologism words for all models. a closer look reveals that six typical types of errors are made in mistranslated model outputs, which include (ordered by severity): another interesting observation is that newer neologisms indeed show lower rates of good translations and often higher rates of mistranslations, as one may expect. figure 3 shows the percentage of good translations and mistranslations over time for varied models. compared to non-neologism sentences, models still yield lower rates of correct translations for neologisms that emerged before 2020. many neologisms use existing words to convey meanings, such that the poor performance of models is not wholly explained by the absence of these word forms in training data. we propose a novel benchmark ( §3) to systematically study the impact of neologisms on llms ( §4). ## neo-bench: a neologism benchmark we create neo-bench, a benchmark that consists of 2,505 neologisms (both words and phrases) that newly emerged around 2020-2023 and 4 intrinsic/extrinsic tasks ( ## key findings we utilize neo-bench tasks to evaluate the ability of various llms to adapt to neologisms. the following are our key findings: gpt-4's knowledge of neologisms is task specific. correct; and unknown if multiple tasks are incorrect. there are higher rates of correct translations for neologisms gpt-4 understands, as good translations drop by 20.3% if gpt-4 does not fully know a neologism's meaning. however, the rate of mistranslations are constant regardless of gpt-4 performance in other tasks, and gpt-4 only correctly translates 53.13% of neologisms it has complete knowledge of. gpt-4's knowledge of neologisms is compartmentalized and does not result in similarly high performance in machine translation compared to other neo-bench tasks, emphasizing the difficulty of translating neologisms. models perform worse on neologisms than preexisting words. for cloze questions in figure 5 , neologism answers are designed to be more natural as the original passages contained these neologisms, yet all models select a large portion (27.99% on average) of distractor answers. neologisms also have an average perplexity rank of 463 compared to distractor rankings of 45 in figure 7 , indicating much lower perplexity for pre-existing words. older llms perform significantly worse. in figure 5 , the average performance of gpt-j, bart, t5, and flan-t5 is 26.99% lower in cloze qa and 0 350 700 mistral olmo mistral-instr llama-2 olmo-instr llama-2 c llama-1 alpaca gpt-j 683 650 582 542 453 377 346 306 228 neologism ranking 0 250 500 33 80 29 67 82 24 38 28 21 substitute ranking 0 400 800 mistral olmo mistral-instr llama-2 olmo-instr llama-2 c llama-1 alpaca gpt-j 874 771 841 724 740 619 586 505 462 lexical 0 400 800 728 697 583 553 415 345 301 277 186 morphological 0 400 800 204 254 195 222 218 166 203 150 81 semantic figure 7: average rankings of neologisms and preexisting substitute terms compared to 5000 common words, sorted by model perplexities of texts filled in with each word. neologisms are separated by linguistic type: lexical, morphological, and semantic. perplexity rankings of older models increase drastically from 2019 until 2021. while neologisms are often gradually worked into a vocabulary (zhu and jurgens, 2021), we use trend lines to best estimate the date when a neologism becomes popular and report perplexity over time in figure 6 . newer models dip in 2020 but increase afterward as 52% of neologisms from this period are now conventionalized terms related to covid-19. larger models handle neologisms better. increasing the sizes of llama-2 and llama-2 chat leads to consistent improvements across both cloze question answering and definition generation. on average, lama-2 70b and llama-2 chat 70b yield 10.13% higher accuracy in cloze qa and 5.93% higher accuracy in definition generation than llama-2 7b and llama-2 chat 7b. instruction-tuning results in high neologism perplexities. in figure 7 , llama-1, llama-2, olmo, and mistral models have, on average, 125 lower neologism rankings than their instruct-tuned counterparts. instruct models are trained with dialogue @xcite , so uncommon generation is less desired. 5 linguistic taxonomy analysis we separate neo-bench task results by neologism linguistic structure: lexical, morphological, and semantic. figure 7 presents perplexity rankings, figure 9 reports human evaluation for mt, and figure 10 shows the results for the best models on cloze qa and definition generation. lexical neologisms produce the highest perplexities, but yield the best downstream results. on average, lexical neologisms have 226 higher rankings than other words, indicating higher relative model perplexities. figure 8 shows the distribution of characters per token of neologisms using the llama tokenizer, and the average number of characters per token for lexical, morphological, and semantic neologisms is 2.36, 2.98, and 3.24 respectively. lexical neologisms have more fragmented tokenizations, as these words have the highest proportion of 1-2 character tokens. lexical neologisms are less likely to be separated into long, common word roots or segments representing subword information, instead producing uncommon token sequences that result in higher neologism rankings and perplexity. in downstream tasks, however, lexical neologisms yield 0.6% higher cloze accuracy, 8.8% more correct definitions, and 21.5% more good translations than other neologisms. ## 2 neologism benchmark motivation 086 there are no currently no benchmarks to observe 087 the specific impact of neologisms on models. to 088 determine if neologisms have an impact on llms, 089 we evaluate machine translation (mt) models for 090 english-to-chinese translation. we collect 100 091 popular neologisms from 2020-2023 and construct input pairs for machine translation. each pair consists of a sentence containing a neologism and a near-identical, logically feasible sentence that replaces the neologism with a carefully constructed substitute term. these replacement words are not synonyms, as new words emerge in areas not occupied by existing words @xcite , so neologism synonyms are often verbose and incompatible with the sentence context. we test commercial systems: google translate 1 , microsoft bing 2 , and deepl translator 3 and open source models: alma 7b @xcite and m2m100 1.2b @xcite . mt output is manually annotated with hand-crafted labels of increasing error severity: while semantic neologisms yield high cloze qa accuracy, they also achieve the lowest percentages of correct definitions and translations. models produce popular definitions and literal translations based on a word's most common meaning, as the new sense of semantic neologisms is often nuanced and difficult to capture. ## related work temporal drift in llms. prior work has explored temporal data drift by creating temporal splits of training data data @xcite @xcite @xcite . new factual updates of concepts are studied with temporal splits of text corpora and qa datasets @xcite @xcite . other work has observed model degradation from new named entities @xcite @xcite . temporal degradation occurs during short-term crisis events where information changes quickly @xcite . studies have consistently found model degradation with perplexity and downstream tasks. there are no studies on model degradation from language change of neologisms, so we create a benchmark to evaluate models on neologisms with similar tasks. neologism collection. using reference texts as exclusion lists to filter common words from target corpora is the most documented method of neologism detection. target texts and exclusion lists include news articles @xcite , dictionaries @xcite @xcite , social media @xcite @xcite and other corpora @xcite . these texts are slow to curate, and semantic neologisms are filtered out. moreover, no resource collects general semantic neologisms. gsome resources measure word prevalence with time-series data of search queries to collect singleword neologism candidates @xcite or cybersecurity neologisms @xcite . they are limited in scope by collecting only one type of neologism based on rising popularity. other methods collect neologisms with new slang dictionary entries @xcite . dictionaries are slow to update, so new entries are often conventionalized words. there is no resource that uses time-series data to collect a variety of neologisms rising in prevalence. previous work has utilized search templates of explanation patterns to collect automatically neologisms @xcite . a few efforts have used neural methods to automatically detect one specific type of neologism, such as adjective-noun neologism pairs (mccrae, 2019), blend words @xcite , and grammatical neologisms @xcite , which are existing words with new parts of speech. no automatic resource collects neologisms from various topics and linguistic backgrounds. to address these limitations, neo-bench uses multiple methods to semi-automatically collect a variety of neologisms, including multiword, semantic, and prevalent neologisms. unseen words. rare words or typos are typically unseen in data when training a model but may show up during inference. prior work has measured model degradation from unseen words @xcite and used contextual subword embeddings @xcite @xcite @xcite , similar surface forms of common words @xcite , and morphological structure @xcite to represent unseen words. comparatively, the neologism lifecycle follows three stages: emergence, dissemination, and conventionalization @xcite . new words often become prevalent and drastically shift a language's distribution. semantic neologisms also use existing word forms and are not classified as unseen words. ## conclusion in this paper, we present neo-bench, a new benchmark to test the ability of llms to generalize on neologisms. we use several methods to collect a variety of neologisms, including prevalent, multiword, and semantic neologisms. in our experiments, we find that models struggle with neologisms in both perplexity and downstream tasks. machine translation is especially difficult, as translating neologisms often requires paraphrasing the sentence. current automatic metrics cannot measure translation quality, and human evaluation is still needed. neologisms also affect models differently based on linguistic structure, indicating that this phenomenon is complex for llms to address.