Large Language Models Can Not Perform Well in Understanding and Manipulating Natural Language at Both Character and Word Levels? (2024)

Abstract: Despite their promising performance across various tasks, recent studies reveal that Large language models (LLMs) still exhibit significant deficiencies in handling several word-level and character-level tasks, e.g., word unscrambling and sentence editing, indicating urgent needs for substantial improvements in basic language understanding and manipulation. To address these challenges, it is crucial to develop large-scale benchmarks that can comprehensively assess the performance of LLMs in basic language tasks. In this paper, we introduce a bilingual benchmark, CWUM, to investigate the capabilities and limitations of LLMs in understanding and manipulating natural language at both character and word levels. CWUM consists of 15 simple text editing tasks, e.g., letter counting, word reversing, Chinese character inserting, etc. We conduct extensive experiments on eight advanced LLMs, including base models and instruction-tuned (chat) variants. The experimental results highlight significant failures of existing LLMs on CWUM tasks that humans can solve perfectly with 100% accuracy. On English tasks of CWUM, the average accuracy of GPT-4, LLaMA-3-70B, and Qwen-72B is 66.64%, 39.32%, and 33.16%, respectively, which lags far behind human performance. Instruction-tuning the base model does not lead to a distinct performance improvement, as the average accuracy of LLaMA-3-70B-Instruct on English tasks is only 1.44% higher than that of the base LLaMA-3-70B. Ultimately, we show that supervised fine-tuning (SFT) can enhance model performance on CWUM without compromising its ability to generalize across general tasks.

## introduction recently, large language models (llms) have demonstrated significant capabilities across a wide range of applications, including general natural language processing (nlp) and domain-specific tasks @xcite @xcite . reports indicate that llms have matched or even surpassed human performance in several areas. for example, llms outperform humans in specific language translation tasks, standardized reading comprehension tests, and logical reasoning assessments. additionally, llms excel at solving complex algebra and calculus problems in standardized mathematics tests and competitions. despite the promising performance across various tasks, recent studies propose that llms still exhibit significant deficiencies in handling several word-level and character-level tasks, e.g., word unscrambling and sentence editing @xcite . in simple tasks such as writing a sentence containing a specific word or choosing which of two words is longer, model performance is worse than that of elementary school students @xcite . this disparity indicates that while llms have made breakthroughs in higher-level language understanding and generation, substantial improvements are still needed for basic language understanding and manipulation. to address these challenges, it is crucial to develop large-scale benchmarks that can comprehensively assess the performance of llms in basic language tasks. a bilingual benchmark is particularly important as it allows for the evaluation of llms across different languages, revealing languagespecific deficiencies and providing a more comprehensive understanding of their capabilities and limitations. to this end, we propose a bilingual benchmark cwum, to evaluate the capacities and limitations of llms in understanding natural language at both character and word levels. specifically, cwum comprises 15 tasks focusing on text edition, including identification, insertion, reversal, and counting. in addition to evaluating model performance on each task of cwum, we investigate how model performance varies with increasing model size and shots. we also examine the impact of instruction tuning on model performance. to boost the confidence of model predictions, we employ a few-shot chain-of-thought (cot) prompt @xcite , which encourages the model to follow demonstrations that provide intermediate steps, such as identifying the letters constituting the input word or the words constituting the input sentence, before generating the final output. we evaluate the performance of eight advanced llms including both base models and instructiontuned (chat) variants, on the cwum benchmark. these models include llama-2 and llama-3 @xcite , @xcite , mistral @xcite , baichuan2 (baichuan, 2023), chatglm3 @xcite , yi @xcite , deepseek (deepseek-ai, 2024), and gpt-4 (openai, 2023). overall, we observe the following phenomena by comparing the testing accuracy of different models. (1) the tasks in the cwum benchmark pose a huge challenge to all evaluated models, resulting in a significant performance gap compared to human performance. as illustrated in figure 1 , human performance on the cwum benchmark is perfect (measured at 100%). even the best-performing model, gpt-4, achieves only 66.64% accuracy on english tasks and 78.20% on chinese tasks, respectively. the performance of representative open-source llama-3-70b on the english and chinese tasks is 39.92% and 30.02%, respectively, significantly lower than human performance. (2) instruction tuning does not lead to substantial performance improvement, e.g., the average accuracy of llama-3-70b-instruct and the base llama-3-70b on english tasks is 40.76% and 39.92%, respectively. (3) while model performance improves with increasing size, it remains unsatisfactory compared to human performance. additionally, by analyzing model predictions, we attribute the failures of llms on cwum to the following reasons. finally, we conduct experiments to explore whether supervised fine-tuning (sft) can improve model performance on cwum while maintaining its generalization ability on general tasks. we collect 160,000 training examples for eight english tasks from cwum and combine them with 520,000 general-purpose instruction-response pairs to construct the final sft dataset. fine-tuning llama-2-7b on this mixed dataset results in an 86% average accuracy improvement on all 10 english cwum tasks. additionally, on unseen general tasks including mmlu @xcite , hellaswag @xcite , wino-grande @xcite , and arc @xcite , the performance of the model finetuned on mixed data is comparable to that of the model fine-tuned on instruction data alone. ## related works large language models (llms) are increasingly significant in both research and daily life, making the evaluation of their capabilities a crucial issue. recently, substantial efforts have been made to develop benchmarks that assess llms from various perspectives. llms were originally designed to improve performance in natural language processing (nlp) tasks, including classification @xcite , question-answering @xcite , generation @xcite , etc. recently, evaluation research also trends to domain-specific tasks using datasets such as math @xcite and gsm8k @xcite for mathematical reasoning, ifeval @xcite for instruction following, and humaneval (chen et al., 2021) for code generation. beyond single-task datasets, large-scale benchmarks like mmlu @xcite , glue @xcite , and c-eval @xcite cover a wide range of tasks to pro-vide a comprehensive evaluation. furthermore, as llms are increasingly integrated into everyday activities, studies have begun to examine their robustness @xcite , ethical considerations and biases @xcite , and trustworthiness @xcite . these evaluations are vital to understanding the broader implications of llms and ensuring their reliable and ethical deployment. in addition to the work mentioned above, several studies focusing on the limitations of llms are drawing attention from the research community. @xcite investigates the reversal curse of llms, i.e., llms trained on "a is b" failing to learn "b is a". @xcite aims to study the order sensitivity of llms against options of multiple-choice questions and two approaches are presented to calibrate llms' predictions including majority vote and multiple evidence calibration (mec). to explore the limitations and predict the future behavior of llms, the beyond the imitation game benchmark (big-bench) @xcite compiles 204 tasks believed to exceed current models' capabilities. similar to our work, lmentry @xcite highlights substantial failures of llms on 25 tasks that are trivial for humans, e.g., writing a sentence containing a specific word or choosing which of two words is longer. unlike lmentry, we introduce cwum, a bilingual benchmark consisting of 15 character and word editing tasks, to evaluate the capabilities and limitations of existing llms in understanding and manipulating natural language at both character and word levels. this comprehensive approach aims to identify specific areas where llms fall short and provide insights for future model improvements. ## cwum cwum is a bilingual benchmark designed to assess the basic natural language comprehension abilities of current llms. it consists of 15 straightforward character-editing and word-editing tasks such as counting characters, reversing words, and identifying specific characters, which are tasks that an elementary student is generally expected to perform perfectly. each task consists of a training set and a test set. the simplicity of these tasks highlights the basic language understanding and manipulation capabilities of llms, providing a clear measure of their proficiency in handling fundamental linguistic operations. ## conclusion in this study, we introduce cwum, a novel bilingual benchmark designed to evaluate the capabilities and limitations of llms in understanding and manipulating natural language at both word and sentence levels. cwum comprises 15 text-editing tasks, including 10 in english and five in chinese, which are simple for humans but challenging for current llms. our comprehensive evaluation of eight advanced llms, including both base and instruction-tuned (chat) models, reveals significant deficiencies in their performance on these tasks. these findings suggest that while llms have made considerable progress, there is still a substantial gap to bridge in terms of achieving human-like proficiency in language understanding and manipulation. overall, cwum provides a valuable tool for assessing and guiding the development of future llms, emphasizing the need for more sophisticated mechanisms to handle the complexities of natural language at both character and word levels.