WSC +: Enhancing The W inograd Schema Challenge Using Tree-of-Experts (2024)

Abstract: The Winograd Schema Challenge (WSC) serves as a prominent benchmark for evaluating machine understanding. While Large Language Models (LLMs) excel at answering WSC questions, their ability to generate such questions remains less explored. In this work, we propose Tree-of-Experts (ToE), a novel prompting method which enhances the generation of WSC instances (50% valid cases vs. 10% in recent methods). Using this approach, we introduce WSC+, a novel dataset comprising 3,026 LLM-generated sentences. Notably, we extend the WSC framework by incorporating new ‘ambiguous’ and ‘offensive’ categories, providing a deeper insight into model overconfidence and bias. Our analysis reveals nuances in generation-evaluation consistency, suggesting that LLMs may not always outperform in evaluating their own generated questions when compared to those crafted by other models. On WSC+, GPT-4, the top-performing LLM, achieves an accuracy of 68.7%, significantly below the human benchmark of 95.1%.

## introduction as large language models (llms) continue to evolve, accurately assessing their common-sense reasoning capabilities becomes paramount. recent advancements highlight llms' capacity to recognize patterns from extensive text corpora, leading to strong results across various nlp benchmarks @xcite @xcite @xcite . yet, doubts persist regarding their true depth of comprehension, especially in common-sense reasoning @xcite @xcite ). an earlier benchmark, the winograd schema challenge (wsc) @xcite , was crafted mainly for pre-llm systems to assess this dimension in ai models. although early models struggled with intricate common-sense instances due to corpus limitations @xcite @xcite , contemporary llms, propelled by advances such as prompt engineering @xcite @xcite , show better performance. however, this success has sparked debates on the adequacy of current benchmarks in evaluating llms' real-world reasoning capabilities @xcite . manually constructing benchmarks for machine learning models often requires significant human effort and advanced information retrieval techniques @xcite . with the potential of llms in dataset generation, however, there is an opportunity to automate this process @xcite . still, designing tasks to deeply probe common-sense reasoning remains a hurdle @xcite , particularly evident when llms exhibit biases @xcite or unwarranted overconfidence @xcite : who is he?" (claude2's answer: the black boy). in both instances, although the models were given the option of selecting 'neither', they chose those specific answers, revealing inherent biases and overestimations in their reasoning abilities. addressing these intricacies, this work leverages prompt engineering to guide llms in crafting challenging instances, and subsequently analyzes how these models evaluate their own generated content. figure 1 presents a complete visual overview of our work. our key contributions are threefold: ## the wsc+ coreference task coreference resolution remains a cornerstone of understanding in natural language processing, with biases in llms' performance posing significant challenges to real-world applications @xcite . the wsc+ dataset seeks to address these biases in llms' pronominal coreference resolution. distinctively, wsc+ is primarily generated using an automated llm process, supplemented by human oversight for quality assurance and data filtration. each wsc+ instance is represented as @xmath0, a 1 , a 2 , p, r}, where: these examples illustrate the subtleties of the task, demonstrating how minor modifications can lead to distinct coreference resolutions. wsc+ categorizes questions into three types: traditional pairs: drawing inspiration from the original wsc, they adhere to criteria such as being resistant to simple search engine queries (often referred to as 'google-proof') @xcite . these questions test models on their ability to discern context and apply common sense. ambiguous pairs: designed to expose models' tendencies to overcommit in situations of uncertainty, these pairs gauge the capacity of models to recognize and appropriately handle ambiguity. for these questions, 'neither' entity is always the correct answer, underscoring the importance of models discerning uncertainty. offensive questions: aimed at detecting biases across various domains like religion, race, and gender, these questions challenge models with potentially biased or inappropriate situations (refer to appendix table ex. 12 for examples). offensive questions are not inherently derogatory, but choos-ing one of the subjects as a resolution could render the response inappropriate. foot_1 in the subsequent sections, we detail each component of the pipeline for constructing the wsc+ dataset, with instance examples shown in ## conclusion we presented a comprehensive analysis of large language models using the wsc+ dataset, which encompasses diverse and challenging scenarios. our tree-of-experts method offers a novel approach for improved wsc+ instance generation. notably, our study unveils inconsistencies in llms' generation-evaluation performance, emphasizing their reasoning challenges. our findings underline the importance of ongoing research on llm robustness, targeting reasoning disparities and ethical concerns.