Unnatural Error Correction: GPT -4 Can Almost Perfectly Handle Unnatural Scrambled Text (2023)

Abstract: While Large Language Models (LLMs) have achieved remarkable performance in many tasks, much about their inner workings remains unclear. In this study, we present novel experimental insights into the resilience of LLMs, particularly GPT-4, when subjected to extensive character-level permutations. To investigate this, we first propose the Scrambled Bench, a suite designed to measure the capacity of LLMs to handle scrambled input, in terms of both recovering scrambled sentences and answering questions given scrambled context. The experimental results indicate that multiple advanced LLMs demonstrate the capability akin to typoglycemia, a phenomenon where humans can understand the meaning of words even when the letters within those words are scrambled, as long as the first and last letters remain in place. More surprisingly, we found that only GPT-4 nearly flawlessly processes inputs with unnatural errors, a task that poses significant challenges for other LLMs and often even for humans. Specifically, GPT-4 can almost perfectly reconstruct the original sentences from scrambled ones, decreasing the edit distance by 95%, even when all letters within each word are entirely scrambled. It is counter-intuitive that LLMs can exhibit such resilience despite severe disruption to input tokenization caused by scrambled text.

## introduction large language models (llms) demonstrate impressive proficiency across a range of tasks, with certain capabilities emerging as the models scale up in size-a phenomenon commonly known as emergent abilities. @xcite . as the llms the following sentence contains words with scrambled letters. please recover the original sentence from it. scrambled sentence: ojn amrh wno het 2023 meatsrs ermtnotuna no duysan ta atgsuau ntaanloi gflo bclu, gnclcinhi ish ifsrt nereg ecatkj nad ncedos raecer jroam. ## related works the most related works are the studies investigating the effects of word or sub-word level perturbations @xcite @xcite and the studies evaluating the robustness of llms @xcite . to the best of our knowledge, no existing studies have investigated llms' ability to handle character-level permutations, particularly those of an extremely high level that drastically change tokenization. our study aims to fill this gap. ## scrambled bench we propose two tasks to evaluate the capability of llms to handle scrambled text. ## conclusion in this study, we propose scrambled bench, a test suite to measure the ability of llms to handle scrambled text, including two tasks (i.e., scrambled sentence recovery and scrambled question answering) and construct scrambled datasets based on realtimeqa, dream and aqua-rat. despite the scrambled text drastically changes the tokenization, we demonstrate that advanced llms are capable of processing scrambled text to varying degrees. however, most llms have difficulty handling text that is scrambled to an extreme degree (i.e., 100% randomly scrambling). surprisingly, for both tasks, gpt-4 shows good results and outperforms other models by a large margin. for the scrambled sentence recovery task, gpt-4 can recover sentences by 95% edit distance reduction even in 100% randomly scrambling settings. for the scrambled question answering task, gpt-4 can maintain a very high proportion of its original accuracy using scrambled context.