Audio Commentary System for Real-Time Racing Game Play (2023)

Abstract: Live commentaries are essential for enhancing spectators’ enjoyment and understanding during sports events or e-sports streams. We introduce a live audio commentator system designed specifically for a racing game, driven by the high demand in the e-sports field. While a player is playing a racing game, our system tracks real-time user play data including speed and steer rotations, and generates commentary to accompany the live stream. Human evaluation suggested that generated commentary enhances enjoyment and understanding of races compared to streams without commentary. Incorporating additional modules to improve diversity and detect irregular events, such as course-outs and collisions, further increases the preference for the output commentaries.

## introduction live commentary enriches the spectator's experience in sports events and e-sports streams, but it is often unavailable for online videos or recordings of amateur sports due to a lack of skilled commentators. automatic generation techniques offer a potential solution to this problem. previous works focus on generating textbased commentaries using pre-stored tracked data @xcite . in contrast, we present a real-time automatic system for generating commentaries, specifically targeting race games inspired by the growing e-sports industry. live commentary generation typically involves tweet extraction @xcite , rulebased and keyword extraction from videos @xcite , and neural network-based datato-text approaches @xcite . our system combines utterance extraction and neural network-based methods. our system works with a physical controller for real-time gameplay in a racing game (assetto corsa). during gameplay, the system tracks the user's data, such as speed, steering rotation, and lap progress. then, the system generates candidates by a neural network-based generator and ranks them in terms of diversity. we also address the problem of limited coverage of rare events in the existing commentary generator. to mitigate these problems, our system detects course-outs and collisions and selects appropriate expressions from a predefined list of utterances. the current generation model is trained on an open japanese dataset @xcite , although it can be replaced with an english one. 1 we conducted an evaluation by human judges in terms of enjoyment. the results suggest that: 1) commentary usage enhances user immersion, 2) diversity is important for improving the overall quality of synthesized commentaries, and 3) identifying irregular events further gains the quality. ## architecture our system consists of five modules in a pipeline. 1: real-time data tracker assetto corsa allows custom functionality to be added to the game via plugins. thus, we develop a plugin which captures the relevant play data from the game's api. the data is sent to our data processing server. the server samples several features i.e., speed or other metrics, every 0.1 s and keeps the samples in a 10 s moving window. the collected features are sent to the candidate generator as a set of 100-element vectors. 2: candidate generator we extend an existing model @xcite , a multi-modal generator for textual and numerical inputs. for textual input, we consider the previous l utterances (we use @xmath0, while for numerical input, we use vectors produced from the realtime data tracker. textual input is encoded with bart's encoder (stockmark/bart-base-japanesenews), while the original used lstm. we use an mlp-based encoder for numerical inputs to obtain 728-sized vectors. another bart encoder processes two types of embeddings: 1) embeddings of the initial token in textual input, and 2) encoded tracked data. bart's decoder generates an utterance to get k candidates by a beam search. 3: utterance selection in our preliminary experiments, repetition of the same or similar utterances led to a decline in naturalness. thus, our system selects the candidate u cand least similar to the previous l utterances u i . the similarity is calculated as the exponentially weighted sum: l @xmath1, u cand ), where α = 0.2. we use bleu as sim(). ## experiments training: the candidate generator is trained with 34,897 gold utterance tuples, including previous utterances and tracked numerical data. validation is performed using 12,295 tuples. the batch size is 5, and adamw optimizer is used with a learning rate of 10 -5 . the best model is selected based on cross entropy loss on the validation set. evaluation: we assess spectator immersion in synthesized commentary and compare different commentaries. three models are compared: 1) using the best utterance in beam search, 2) incorporating a diversity module, and 3) combining the diversity and irregular event detection modules. four human evaluators rank these models in terms of enjoyment as an audience. ## results all the human judges agree that synthesized commentaries enhance immersion. in terms of enjoyment, the commentaries generated by the model with both diversity and irregular detection modules are ten times out of twelve judged better than the model that outputs the best utterance in beam search. this result suggests that these two modules are effective. the model that uses both diversity and irregular event detection modules was eight times out of twelve judged better than the model with only diversity models. thus, the irregular event detection helps to improve quality. ## conclusion we introduced a commentator for racing games, which generates real-time commentary based on tracked metrics. future possibilities include utilizing dialogue-styled commentary or enhancing live streams with explanatory graphics. 3