From Data to Insights: The Power of LM ’s in Match Summarization (2024)

Abstract: The application of Natural Language Processing is progressively extending into many domains as time progresses. We are motivated to evaluate language model’s (LMs) capabilities in many real-world domains due to their significant potential. This study examines the use of LMs in sports, explicitly emphasizing their ability to convert data into text and their understanding of cricket. By examining cricket scorecards, a widely played sport on the Indian subcontinent and many other regions, we will evaluate the summaries produced by LMs from several viewpoints. We have collected concise summaries of the scorecards from the ODI World Cup 2023 and assessed them in terms of both factual accuracy and sports-specific significance. We analyze the specific factors that are included in the summaries and those that are excluded. Additionally, it analyzes prevalent mistakes concerning completeness, correctness, and conciseness. We are presenting our findings here and also our dataset and code are available https://github.com/satyawork/ODI-WORLDCUP.git

## introduction sports contribute over $500 billion annually to the global economy and are crucial for promoting physical health and reducing chronic diseases @xcite . they also foster social cohesion by bringing communities together and teaching essential values like teamwork and leadership @xcite . sports enhance personal development through skills such as discipline and resilience, shaping individuals physically and mentally @xcite . sports play a vital role in improving individual well-being and societal harmony globally. cricket's popularity surpasses many other sports due to its global reach, boasting 2.5 billion fans worldwide and massive viewership during events like the icc cricket world cup. its rich history and cultural significance in nations like india, pakistan, australia, and england contribute to its enduring appeal. the sport's diverse formats cater to different audiences, from traditional test matches to fast-paced twenty-20 games, accommodating varied preferences. additionally, prestigious leagues such as the ipl and bbl ensure year-round excitement and attract top international players. these factors collectively make cricket a powerhouse in sports, sustaining its widespread popularity and fan engagement globally. data-to-text generation involves transforming structured, non-linguistic input like tables, databases, tuples, or graphs into accurate textual descriptions automatically @xcite @xcite . this process is crucial in various real-world scenarios, such as creating weather forecasts based on meteorological data @xcite , summarizing biographical information @xcite , or generating sports summaries from game statistics @xcite . the objective is to convey pertinent details from the input data using natural language, ensuring the generated text faithfully and precisely reflects the original information. thus, achieving accuracy in representing the source data becomes paramount. several methods have been developed to tackle this challenge in data-to-text generation. these approaches utilize different strategies such as incorporating the structure of input data @xcite @xcite , employing neural templates @xcite , and emphasizing the arrangement of content @xcite . the rise of lms has brought about a profound shift in controllable text generation and data interpretation. recently, there has been a shift towards utilizing large-scale pre-trained models @xcite , which have shown notable improvements in fluency and the ability to generalize compared to earlier ap-proaches that did not use such models. in today's world, people regularly handle large amounts of organized data to make decisions and find information. it's crucial now more than ever to present this data in ways that are easy to understand and user-friendly @xcite . conveniently presenting data has sparked interest in techniques that convert intricate data tables into clear, meaningful narratives that meet the specific needs of users @xcite . these methods can be applied across various fields, such as game strategy planning, financial analysis, and human resources management. yet, current fine-tuned table-to-text models @xcite are often designed for specific tasks, restricting their flexibility for practical uses in different scenarios. recent studies show that lms can achieve performance comparable to state-of-the-art fine-tuned models in tasks like answering table questions and fact-checking. yet, there is still much uncharted territory regarding lms' ability to effectively generate text from tabular data to meet users' information needs. we have created a dataset named icc crick-world cup sourced from reputable sports analytics repositories, encompassing 12 distinct categories concerning the icc cricket world cup 2023. we tested several small-scale language models (slms) on this dataset to generate five types of summaries. these models were adopted for the resource constraint scenario. the resulting summaries underwent evaluation using various metrics supplemented by human assessment for enhanced accuracy. additionally, we have documented our findings on the performance of these summaries, highlighting instances of both success and areas needing improvement. this initiative introduces a cricket dataset structured with tables, challenging lms to derive meaningful insights from these data points. future advancements could leverage larger parameter lms to refine this approach, potentially extending its application to diverse domains beyond cricket. previously, significant advancements have been made in converting tabular data to text and summarizing it. two primary types of summaries can be generated from tabular data: this method allows for a more holistic summary of the entire dataset, capturing broader insights and trends. both methods have their distinct advantages and applications, enhancing the ability to derive meaningful narratives from structured data. in our work, we collected data points from reputable and reliable cricket sources to create a cricket summarization dataset. the dataset includes five types of summaries: an overall match summary, a bowling perspective summary, a batting perspective summary, and summaries from the viewpoints of team 1 supporters and team 2 supporters. these summaries were prepared by a few cricket experts. additionally, multiple llm models were used to generate summaries, which were then evaluated using various metrics. we also documented the mistakes made by the llms in different scenarios. ## related work natural language processing (nlp) for summarization focuses on automatically condensing large volumes of text into shorter, coherent summaries while retaining key information. this involves techniques like extractive and abstractive summarization, where models either select important sentences or generate new, concise summaries. nlpbased summarization is widely used in areas like news aggregation, research papers, and legal documents to provide quick and efficient insights. horasan and bilen (2020) researched the summarization of news articles about sports, demonstrating the impact of nlp in the sports domain. the study highlighted the increasing use of machine learning and deep learning techniques in sports analysis. @xcite showcased the application of machine learning for shot detection in cricket, further illustrating the integration of advanced computational methods in sports. additionally, other research efforts, such as those by @xcite have focused on analyzing videos and audios of cricket matches to derive insights. despite these advancements, there remains a scarcity of nlp-focused work specifically targeting cricket, indicating a potential area for further exploration and development. on the other hand, multiple researchers have extensively studied text summarization, as highlighted in a survey @xcite , which documented progress in this task and identified common pitfalls that lm;s encounter when summarizing long texts. base models like bert @xcite , t5 @xcite , roberta @xcite , and bart @xcite have shown promising results in summarization tasks . further, fine-tuned versions of these models have been applied to tabular data to generate various summaries. for instance, @xcite explored table-to-text generation with tabt5 model based on t5 pre-trained model, while @xcite and @xcite focused on pre-training techniques for table summarization and query-focused summarization of tabular data. large language models also show this type of capability. tabular data presents additional complexities compared to regular text summarization, and several studies have addressed table-to-text summarization. a systematic review by @xcite covers the datasets used for this task, such as totto @xcite by @xcite and others containing paired data of tables and their respective summaries. these datasets primarily consist of paired data, with some additional information based on specific cases. similarly, we generated a dataset of summaries along with a comparative analysis to explore this domain further. ## dataset we curated a comprehensive icc cricket world cup 2023 dataset from reputable sports analytics repositories, encompassing detailed records of each team's batting and bowling performances and powerplay logs. additionally, the dataset includes summarizations of match logs from various viewpoints: a general match summary, summaries from the perspectives of bowling and batting performances, and summaries specific to the team batting first and last. figure 1 shows an overview of dataset fields. these summaries were written by a writer with good knowledge of the 2023 odi world cup. also, all summaries are cross-verified for any errors, and it is confirmed that all noteworthy performances are included in the summary. each match entry in the dataset is accompanied by essential details such as match id, date-time, the team batting first, and the team batting last. the dataset comprises 48 matches, each with summaries and additional pertinent information. ## results our curated dataset of cricket world cup match summaries served as the gold standard because of multiple verifications and validation for evaluating summaries generated by different lms. these lm summaries were assessed using the previouslymentioned automated evaluation metrics, and the tabular result is present in the results indicate a correlation between high metric scores and similarity to the human-crafted summaries. lms such as llama2, llama3, and mistral achieved promising results in terms of similarity based on these metrics. phi3 mini and small performed averagely, while vicuna exhibited the lowest similarity scores. we conducted a human evaluation with a cricket expert to validate these findings further. we evaluate 1440 summaries generated by lm's. this evaluation aimed to assess the actual correctness and completeness of the summaries beyond the limitations of automatic metrics. the results of this human evaluation and the identified causes of errors in some summaries are discussed in detail below. the following are some common pitfalls and errors where the model struggles and its outputs are compromised: ## limitation although the summary generated could be flawed, cricket summarizing could have multiple limitations. this limitation starts from the input data set. several conditions significantly impact a cricket match, such as pitch conditions, weather, and pressure situations, yet the scorecard doesn't reflect these. the pitch can influence the match outcome substantially, but details about its condition are absent. similarly, weather conditions like rain or humidity, which can alter the course of a game, are not included. the pressure of the match situation, whether a tense chase or a dominant performance, isn't conveyed through numbers alone. player contributions in a cricket match extend beyond just batting or bowling performances. outstanding fielding efforts, 'like saving crucial runs' or 'taking spectacular catches,' can significantly impact the game's outcome. additionally, tactical decisions by the captain and support from the coaching staff, such as field placements, drs (decision review system) calls, or strategic bowling changes, are crucial but not documented in the scorecard. the scorecard also fails to capture team ## conclusion and future scope in this paper, we present an exploratory experiment designed to test the capability of language models to convert cricket scorecards into summaries. our study underscores the potential errors in the generated summaries, focusing on issues related to completeness, correctness, and conciseness. we include qualitative examples of typical errors and explore potential reasons for their occurrence. after extensive analysis, we conclude that lm summaries can contain errors, necessitating cross-verification. specifically, similar types of data can confuse models, and selecting appropriate adjectives can be challenging. further observations are detailed in the observations section. testing this hypothesis motivates future studies in table-to-text summarization within the sports domain. this work opens several promising areas for future research in domain-specific table summarization. analyzing the performance of high-parameter lms on such tasks could lead to improvements in robustness and accuracy. as datasets grow and incorporate summaries with varying word counts and output lengths, more nuanced human evaluation metrics may be developed, offering deeper insights from human assessments. with the increasing size and capabilities of models, the ongoing advancement of lms provides a valuable opportunity to enhance text summarization, translation, and content generation tasks. fine-tuning lms on domainspecific datasets, including sports, could unlock the potential for high-performing models that are less prone to errors than generic ones due to their deeper understanding of the specific domain.