Concept-Reversed W inograd Schema Challenge: Evaluating and Improving Robust Reasoning in Large Language Models via Abstraction (2025)

Abstract: While Large Language Models (LLMs) have showcased remarkable proficiency in reasoning, there is still a concern about hallucinations and unreliable reasoning issues due to semantic associations and superficial logical chains. To evaluate the extent to which LLMs perform robust reasoning instead of relying on superficial logical chains, we propose a new evaluation dataset, the Concept-Reversed Winograd Schema Challenge (CR-WSC), based on the famous Winograd Schema Challenge (WSC) dataset. By simply reversing the concepts to those that are more associated with the wrong answer, we find that the performance of LLMs drops significantly despite the rationale of reasoning remaining the same. Furthermore, we propose Abstraction-of-Thought (AoT), a novel prompt method for recovering adversarial cases to normal cases using conceptual abstraction to improve LLMs’ robustness and consistency in reasoning, as demonstrated by experiments on CR-WSC.

## introduction reasoning serves as the cornerstone underpinning the efficacy and reliability of language models @xcite . while large language models (llms) have demonstrated remarkable proficiency in certain reasoning tasks @xcite , recent research has revealed that llms often experience issues with hallucinations and unreliable reasoning @xcite @xcite induced by semantic associations and superficial logical chain @xcite , especially under adversarial and long-tail scenarios @xcite . despite numerous methodologies proposed to enhance llms' reasoning capabilities, such as chain-of-thought (cot; @xcite and integration with tools and model @xcite @xcite , the robustness of their reasoning process still remains a concern @xcite @xcite . in this paper, we narrow down the scope of reasoning to the winograd schema challenge (wsc), a classic reasoning challenge first introduced as an alternative to the turing test, which requires commonsense knowledge and reasoning ability to solve. a winograd schema is a pair of sentences differing in one or two words with a highly ambiguous pronoun, resolved differently in the two sentences @xcite . an example is in the top corner of figure 1 , formulated as a coreference resolution task. when introduced initially, these tasks posed great challenges for machines, being non-google-proof -impossible to solve through simple word association using search engines @xcite . however, due to its small scale and the scaling up of llms, such a non-google-proof constraint is not considered hard anymore for llms, with gpt-3 achieving accuracies of 88.3% in the zero-shot setting @xcite . to introduce a novel turing test that can robustly evaluate llms regarding commonsense reasoning, we present the concept-reversed winograd schema challenge (cr-wsc). in addition to avoiding simple semantic associations of words, we create an adversarial dataset tailored specifically for llms, which is non-llm-proof : challenging to solve with llms. specifically, we first ask nlp experts to come up with different concept pairs that 1) has reversed attributes associated with the true answer (more semantically associated with the wrong answer), and 2) can cause a base llm to give a wrong answer. for example, in figure 1 , we replace the "father"-"son" pair with "bodybuilder"-"frail senior," such that the "frail senior" is more associated with the adjective "weak" in the context, which can lead an llm to link the pronoun "he" to the senior instead of the bodybuilder. next, we use the same idea to prompt an llm to develop difficult entity pairs at scale, using our annotated data as exemplars. the generated answers are then manually verified. while llms may encounter challenges from the adversarial dataset, their capability to conceptualize reasoning entities offers a promising avenue for fostering unbiased reasoning @xcite @xcite . for example, by conceptualizing "bodybuilder" to a personx and "frail senior" to a persony, llms will not be distracted by the adversarial word association and thus make the correct prediction. to conclude, first, we propose cr-wsc, an adversarial winograd schema challenge that requires the pairing entity to be non-llm-proof. second, we conduct evaluations using llms and find that cr-wsc is significantly harder than wsc, even though the reasoning rationale and logic behind it are the same. third, we propose a robust prompting method, called abstraction-of-thought (aot), to first abstract the adversarial question to a normalized reasoning question, thus facilitating robust reasoning. experimental results show that aot significantly improves reasoning performance and robustness. ## experiment in this section, we conduct a comprehensive array of experiments to validate the effectiveness of our proposed dataset and methods. ## conclusion to determine if llms truly understand reasoning or simply memorize questions, we introduce cr-wsc, a new dataset with confusing entities for coreference resolution. experiments show that even powerful llms struggle with cr-wsc, highlighting the need for more robust reasoning methods. we propose aot, a prompting technique that normalizes adversarial questions to improve llm reasoning ability in complex reasoning questions.