M atch T ime: Towards Automatic Soccer Game Commentary Generation (2024)

Abstract: Soccer is a globally popular sport with a vast audience, in this paper, we consider constructing an automatic soccer game commentary model to improve the audiences’ viewing experience. In general, we make the following contributions: *First*, observing the prevalent video-text misalignment in existing datasets, we manually annotate timestamps for 49 matches, establishing a more robust benchmark for soccer game commentary generation, termed as *SN-Caption-test-align*; *Second*, we propose a multi-modal temporal alignment pipeline to automatically correct and filter the existing dataset at scale, creating a higher-quality soccer game commentary dataset for training, denoted as *MatchTime*; *Third*, based on our curated dataset, we train an automatic commentary generation model, named **MatchVoice**. Extensive experiments and ablation studies have demonstrated the effectiveness of our alignment pipeline, and training model on the curated datasets achieves state-of-the-art performance for commentary generation, showcasing that better alignment can lead to significant performance improvements in downstream tasks.

## introduction soccer, as one of the most popular sports globally, has captivated over 5 billion (fifa, 2023) viewers with its dynamic gameplay and intense moments. commentary plays a crucial role in improving the viewing experience, providing context, analysis, and emotional excitement to the audience. however, creating engaging and insightful commentary requires significant expertise and can be resourceintensive. in recent years, advancements in artificial intelligence, particularly in foundational visuallanguage models, have opened new possibilities for automating various aspects of content creation. this paper aims to develop an high-quality, automatic soccer commentary system. in the literature on video understanding, there has been relatively little attention on sports videos. pioneering work such as soccernet @xcite introduces the first soccer game dataset, containing videos of 500 soccer matches. subsequently, soccernet-caption @xcite compiles textual commentary data for 471 of these matches from the internet, establishing the first dataset and benchmark for soccer game commentary. however, upon careful examination, we observe that the quality of existing data is often unsatisfactory. for instance, as illustrated in figure 1 (left), since the textual commentaries are often collected from the text live broadcast website, there can be a delay with respect to the visual content, leading to prevalent misalignment between textual commentaries and video clips. in this paper, we start by probing the effect of the above-mentioned misalignment on the soccer game commentary systems. specifically, we manually correct the timestamps of commentaries for 49 matches in the soccernet-caption test set to obtain a new benchmark, termed as sn-caption-testalign. with manual check, we observe that these misalignments can result in temporal offsets for up to 152 seconds, with an average absolute offset of 16.63 seconds. as depicted in figure 1 (right), after manual correction, pre-trained off-the-shelf sn-caption model @xcite has exhibited large performance improvements, underscoring the effect of temporal alignment. to address the aforementioned misalignment issue between textual commentaries and visual content, we propose a two-stage pipeline to automatically correct and filter the existing commentary training set at scale. we first adopt whisperx @xcite to extract narration texts with corresponding timestamps from the background audio, which are then summarised into event descriptions by llama-3 (ai@meta, 2024) at fixed intervals. subsequently, we utilize llama-3 to select the goal! @xcite ([team]) picks up the ball inside the box and fires in a shot which is deflected past @xcite . he makes it 2:0. the percentage of ball possession is 66:34. we are about to witness a substitution. @xcite ([team]) for @xcite . the foul by @xcite ) is worthy of a card and a yellow is duly shown by @xcite . most appropriate time intervals based on the similarity between these timestamped event descriptions and textual commentaries. given such an operation only provides rough alignment, we further align the video and commentary by training a multi-modal temporal alignment model on a small set of manually annotated videos. our alignment pipeline enables to significantly mitigate the temporal offsets between the visual content and textual commentaries, resulting in an higher-quality soccer game commentary dataset, named matchtime. with such a curated dataset, we further develop a video-language model by connecting visual encoders with language model, termed as matchvoice, that enables to generate accurate and professional commentaries for soccer match videos. experimentally, we have thoroughly investigated the different visual encoders, demonstrating state-of-the-art performance in both precision and contextual relevance. to summarize, we make the following contributions: (i) we show the effect of misalignment in automatic commentary generation evaluation by manually correcting the alignment errors in 49 soccer matches, which can later be used as a new benchmark for the community, termed as sn-captiontest-align, as will be detailed in sec. 2; (ii) we further propose a multi-modal temporal video-text alignment pipeline that corrects and filters existing soccer game commentary datasets at scale, resulting in an high-quality training dataset for commentary generation, named matchtime, as will be detailed in sec. 3; (iii) we present a soccer game commentary model named matchvoice, establishing a new state-of-the-art performance for automatic soccer game commentary generation, as will be detailed in sec. 4. ## benchmark curation to probe the effect of misalignment on the performance of soccer game commentary models, we have manually annotated the timestamps of textual commentaries for 49 matches in the test set of soccernet-caption, resulting in a new benchmark, denoted as sn-caption-test-align. ## aligning commentary and videos in this section, we develop an automatic pipeline for aligning the timestamps of given textual commentaries to the corresponding video content in existing soccer game commentary datasets. in sec. 3.1, we start with the problem formulation for temporal alignment, and subsequently, in sec. 3.2, we elaborate on the details of our proposed multimodal temporal alignment pipeline. ## automatic soccer game commentary based on the curated dataset, we consider training a visual-language model for automatic commentary generation on given input video segments, termed as matchvoice. specifically, we start by describing the problem scenario, and followed by detailing on our proposed architecture. problem formulation. given a soccer game video with multiple clips, i.e., @xmath0, v 2 , . . . , v t }, our goal is to develop a visual-language model that generates corresponding textual commentary for each video segment, i.e., ĉi = ψ(v i ; θ 2 ), where θ 2 refers to the trainable parameters. visual encoder ❄ aggregator & mlp mlp … 🔥 🔥 𝐂 " ψ !"# ψ "$# 𝐕 : trainable parameters ❄ 🔥 … ❄ prefix tokens ## experiments in this section, we separately describe the experiment results for the considered tasks, namely, soccer commentary alignment (sec. 5.1), and automatic soccer commentary generation (sec. 5.2). ## conclusion in this paper, we consider a highly practical and commercially valuable task: automatically generating professional textual commentary for soccer games. specifically, we have observed a prevalent misalignment between visual contents and textual commentaries in existing datasets. to address this, we manually correct the timestamps of textual commentary in 49 videos in the existing dataset, establishing a new benchmark for the community, termed as sn-caption-test-align. building upon the manually checked data, we propose a multi-modal temporal video-text alignment pipeline that automatically corrects and filters existing data at scale, which enables us to construct a higher-quality soccer game commentary dataset, named matchtime. based on the curated dataset, we present matchvoice, a soccer game commentary model, which can accurately generate professional commentary for given match videos, significantly outperforming previous methods. extensive experiments have validated the critical performance improvements achieved through data alignment, as well as the superiority of our proposed alignment pipeline and commentary model.