T hai W inograd Schemas: A Benchmark for T hai Commonsense Reasoning (2025)

Abstract: Commonsense reasoning is one of the important aspects of natural language understanding, with several benchmarks developed to evaluate it. However, only a few of these benchmarks are available in languages other than English. Developing parallel benchmarks facilitates cross-lingual evaluation, enabling a better understanding of different languages. This research introduces a collection of Winograd Schemas in Thai, a novel dataset designed to evaluate commonsense reasoning capabilities in the context of the Thai language. Through a methodology involving native speakers, professional translators, and thorough validation, the schemas aim to closely reflect Thai language nuances, idioms, and cultural references while maintaining ambiguity and commonsense challenges. We evaluate the performance of popular large language models on this benchmark, revealing their strengths, limitations, and providing insights into the current state-of-the-art. Results indicate that while models like GPT-4 and Claude-3-Opus achieve high accuracy in English, their performance significantly drops in Thai, highlighting the need for further advancements in multilingual commonsense reasoning.

## introduction commonsense reasoning is an important challenge in artificial intelligence; however, most resources and benchmarks are in english, with only a few translations @xcite . the winograd schemas challenge (wsc) has emerged as a widely adopted benchmark for evaluating the commonsense reasoning capabilities of language models @xcite . wsc is featured in both glue and superglue @xcite . these benchmarks are widely used to evaluate a model's understanding of general language. significant progress has been made in developing commonsense reasoning models for high-resource languages like english. however, their performance on low-resource languages remains largely unexplored. this research gap hinders the development and fair evaluation of multilingual and crosslingual nlu systems. linzen (2020) argues for developing multilingual training data and benchmarks to avoid english-centric model development, suggesting composite scores that average performance across languages to better evaluate systems. there have been several one-to-one translations of the winograd schemas into various languages, including french (amsili and seminck, 2017), portuguese @xcite , mandarin @xcite , hebrew @xcite , hungarian @xcite , and russian @xcite . there are also various japanese translations and adaptations @xcite . before our translation, no thai version of the winograd schemas existed. this lack prompted us to take the initiative. the key contributions of our work are: ## experimental setup large language models were evaluated on thai and english winograd schemas to assess their thai language understanding and enable cross-linguistic comparisons of commonsense reasoning. the winograd schema challenge comes in two versions: wsc-273, which contains 273 questions, and wsc-285, an extended version with 12 additional questions, totaling 285 @xcite . in this study, the wsc-285 dataset was used. in addition to the main thai dataset, the models were evaluated on the thai-exact dataset, which consists of winograd schemas translated into thai using google translate, with hand corrections by the authors to address translation mistakes. this evaluation aimed to compare the effect of adapting thai names and contexts, as done in the main thai dataset, against the more literal translations in thaiexact. the models chosen for evaluation were typhoon @xcite , claude-3 (anthropic, 2024a), gpt-3.5 @xcite and gpt-4 @xcite , and c4ai command r+ @xcite . typhoon is a thai-specific model designed to handle thai language tasks. both gpt-3.5 and gpt-4 represent state-of-the-art generalpurpose language models, with gpt-4 offering improved reasoning and comprehension capabilities over its predecessor. claude-3 is another cuttingedge general-purpose language model excelling in similar tasks. c4ai command r+, on the other hand, is a multilingual model designed to support many languages. this selection was made to evaluate performance across thai-specific, multilingual, and general-purpose capabilities. further details about the models can be found in appendix b. ## result and discussion ## conclusion in conclusion, the thai winograd schemas benchmark represents a noteworthy step in evaluating commonsense reasoning capabilities within the thai language context. this novel dataset, meticulously developed and validated by native speakers and professional translators, aims to preserve linguistic and cultural nuances unique to thai. the comprehensive evaluation of state-of-the-art language models, including gpt-4, claude-3 variants, typhoon, and command r+ on both english and thai versions of the winograd schema challenge offers insights into their cross-lingual performance. the observed performance drop in thai suggests challenges these models may face in handling lowresource languages, indicating a need for further research and development in multilingual natural language processing.