S port QA : A Benchmark for Sports Understanding in Large Language Models (2024)

Abstract: A deep understanding of sports, a field rich in strategic and dynamic content, is crucial for advancing Natural Language Processing (NLP). This holds particular significance in the context of evaluating and advancing Large Language Models (LLMs), given the existing gap in specialized benchmarks. To bridge this gap, we introduce SportQA, a novel benchmark specifically designed for evaluating LLMs in the context of sports understanding. SportQA encompasses over 70,000 multiple-choice questions across three distinct difficulty levels, each targeting different aspects of sports knowledge from basic historical facts to intricate, scenario-based reasoning tasks. We conducted a thorough evaluation of prevalent LLMs, mainly utilizing few-shot learning paradigms supplemented by chain-of-thought (CoT) prompting. Our results reveal that while LLMs exhibit competent performance in basic sports knowledge, they struggle with more complex, scenario-based sports reasoning, lagging behind human expertise. The introduction of SportQA marks a significant step forward in NLP, offering a tool for assessing and enhancing sports understanding in LLMs. The dataset is available at https://github.com/haotianxia/SportQA

## introduction the dynamic and multifaceted world of sports, characterized by its fast pace, variety of types, abundance of strategies, and rich player narratives, presents a unique set of challenges for the sports understanding capabilities of large language models (llms). although llms have shown exceptional capabilities in many natural language processing (nlp) tasks such as natural language understanding (nlu) @xcite , information extraction @xcite , and question answering (qa) (zhao et al., 2023; li * corresponding authors. @xcite , their application in the sports domainwhich involves a complex blend of statistical data, narrative content, and strategic planning-remains underexplored. sports enthusiasts can easily answer questions like "who won the 2022 fifa world cup?" or "what is the record for the most points scored in an nba game?", but more complex queries like "why does the float serve appear more in low-level or young-age volleyball games than in high-level games?" require expert knowledge and experience. these challenges, ranging from general knowledge to expert-level analysis, underscore the need for a dedicated sports-focused questionanswering (qa) dataset to improve llms' comprehension and contextualization of sports information. to evaluate the sports understanding capabilities of llms, the sports qa task has been introduced with several datasets for evaluation. the bigbench sports understanding task (bench authors, 2023) focuses on factual sports knowledge, while liveqa @xcite is based on chinese livebroadcasting nba passages. these datasets illustrate certain aspects of sports understanding. however, they exhibit limitations in size, sports domain coverage, and depth of understanding. for instance, big-bench might ask llms to judge the plausibility of a statement like "tom brady threw a touchdown in the champions league final," merging an american football player with a soccer event. this approach, focusing more on superficial sports associations than on deep understanding of sports contexts and rules, leads to an incomplete assessment of llms' true sports understanding. to bridge this gap, we introduce sportqa, a comprehensive dataset crafted to challenge and evaluate llms in the domain of sports understanding. it consists of 70,592 multiple-choice questions, offering varying levels of difficulty, from straightforward historical facts to complex, scenario-based reasoning tasks that necessitate extensive sports knowledge and experience. as shown in figure 1 , humans' understanding of sports can be divided into three levels of difficulty: factual and historical knowledge (level-1), rules and tactics comprehension (level-2), and complex sports-scenario analysis (level-3). in our sportqa, level-1 understanding, covering 21,385 problems, allows humans to answer by recalling facts without needing additional sports expertise, such as knowing who won a specific olympic event. level-2, with 45,685 problems, suitable for sports enthusiasts, requires some expertise to understand diverse sports strategies and rules, like understanding soccer's offside rules. level-3, encompassing 3,522 problems, is designed for sports specialists with years of experience, featuring more complex scenarios. one example under this level would be determining the best technique for a volleyball player facing three blockers during a spike in an amateur game. level-1 and level-2 problems are presented as multiple-choice questions with one correct answer each, while level-3 problems follow a 'multiple select' format, allowing for one to four correct answers, and include both single and multi-hop questions in two levels of difficulty. overall, three levels of questions collectively provide a comprehensive set to evaluate llms' sports understanding across a spectrum of expertise. to gain deeper insights into the sports understanding challenges posed by sportqa, we extensively evaluated several popular recent llms, including llama2 @xcite , palm2 @xcite , gpt-3.5, and @xcite . the models were assessed through few-shot standard prompting, as well as chain-of-thought prompting @xcite . our findings show that gpt-4 outperforms in all levels, achieving an average accuracy of 82.16% in level-1, 75% in level-2, and 47.14% in level-3. however, in level-3, the most challenging level, gpt-4's performance is about 45% inferior compared to human experts' accuracy in the same test questions, indicating that its understanding of sports significantly lags behind human capabilities, suggesting considerable scope for improvement in this area. in summary, our contributions are threefold: ## sports understanding benchmark we introduce sportqa, a benchmark specifically designed to evaluate the understanding of sports for llms. developed in close collaboration with sports experts, questions in sportqa consist of three main levels of difficulty, each focusing on a distinct aspect of sports comprehension. level-1: foundational sports knowledge. questions in this level assess basic sports knowledge, focusing on factual recall and historical information. it provides a foundation for understanding the broader context of sports within llms. level-2: rules and tactics comprehension. this level's questions mainly evaluate llms' understanding of sports rules and strategies. covering 35 distinct sports types, it tests the models' ability to interpret game rules and employ strategic thinking, offering deeper insights into sports comprehension. additionally, it includes more complex fact-based historical questions to ensure a well-rounded assessment of sports knowledge. overall, the benchmark comprises a total of 70,592 questions. for each dataset and level, we provide a few-shot development set consisting of 5 questions per task, along with a separate test set for a thorough evaluation. ## experiments this section evaluates the performance of prevalent llms on our sportqa benchmark. we aim to assess the efficacy of these models in understanding complex sports scenarios across multiple tasks. we present the optimal result, derived from multiple iterations for each experimental condition. ## conclusion this study has introduced sportqa, a benchmark for assessing the understanding of sports in llms. unlike previous benchmarks that primarily focused on basic fact-recall or simplistic sports-related queries, sportqa delves deeper into the intricacies of sports knowledge, spanning historical facts, rules, strategies, and scenario-based reasoning. our evaluations reveal that while current llms like gpt-4 show promising capabilities in foundational knowledge and rule comprehension, their performance in complex, scenario-based reasoning remains a challenge and still lags behind human expertise. the results underscore the need for ongoing advancements in nlp and ai to achieve a deeper and more nuanced understanding of sports. improvements to logical reasoning and in-depth understanding in the context of sports will be a gateway toward allowing large language models to improve their performance and adaptability across a large range of real-world, diverse, and everchanging domains. sportqa serves as an essential tool for future research in this area, offering a structured framework to measure and enhance llms' capabilities in sports understanding.