Generating Racing Game Commentary from Vision, Language, and Structured Data (2021)

Abstract: We propose the task of automatically generating commentaries for races in a motor racing game, from vision, structured numerical, and textual data. Commentaries provide information to support spectators in understanding events in races. Commentary generation models need to interpret the race situation and generate the correct content at the right moment. We divide the task into two subtasks: utterance timing identification and utterance generation. Because existing datasets do not have such alignments of data in multiple modalities, this setting has not been explored in depth. In this study, we introduce a new large-scale dataset that contains aligned video data, structured numerical data, and transcribed commentaries that consist of 129,226 utterances in 1,389 races in a game. Our analysis reveals that the characteristics of commentaries change over time or from viewpoints. Our experiments on the subtasks show that it is still challenging for a state-of-the-art vision encoder to capture useful information from videos to generate accurate commentaries. We make the dataset and baseline implementation publicly available for further research.

## introduction live commentary plays an important role in sports matches and video games; it makes spectators more excited, more immersed, and better informed about the matches or games (e.g., @xcite ), as in the example of racing game commentary "we are approaching the final long straight. i wonder who is going to win!". live commentary also enhances the value of online videos and home videos. however, providing a live commentary requires a certain level of commenting skills and knowledge of the target sports 1 https://kirt.airc.aist.go.jp/ racingcommentary or video games; the majority of online videos and home videos are left without live commentary. 2 the application of natural language generation technology would be a solution to this problem. thus, we select the racing game domain as an example, and propose a task of generating live commentary on it. examples of utterances in a live commentary for a racing game are shown in figure 1 . live commentaries should describe each important event in the race at the moment when the event occurs, within a short period of time. thus, we have to make decisions on when to speak and how long/elaborately to speak, in addition to what to say and how to say it, which have been long studied. the importance of each event would have to be assessed in the context of a competition in which participants are striving to win. it suggests that what to say for race commentary generation should be different from, for example, image captioning. in this sense, the task of live commentary generation contains inherent limitations that are not addressed in well-studied generation problems such as summary generation for basketball @xcite , and image/video captioning @xcite ; however, some techniques developed for such existing generation tasks are also useful for live commentary generation. as input, vision data, such as the videos shown in figure 1 , are common in many tasks. however, it is not a trivial task to capture important information from vision data, because many of the frames in a race would be similar to each other, unlike images for image captioning data. therefore, we propose the use of structured data, that is, telemetry data, including the positions and the speeds of cars, and the steering wheel angles. the figure 1 : translated examples of commentaries (original utterances in japanese are in brackets). for the commentaries on the top, the commentator is watching the race from the aerial viewpoint. for those at the bottom, the commentator is watching the race through a camera just behind the driver. we split the live commentary generation into two subtasks: the utterance timing identification and utterance generation. we propose multimodal models for these subtasks and also provide an empirical evaluation. our experiments suggest that the use of telemetry data works well for this task, whereas it is difficult for a state-of-the-art vision encoder to extract useful information from race videos, especially for utterance generation. our contributions are threefold: (1) we propose a novel task of automatically generating motor racing game commentaries, (2) we create a dataset and analyze its characteristics, and (3) we propose methods for this task and argue that combining multimodal data is challenging, which is worth exploring in depth. we make the dataset and baseline implementations publicly available to enhance further studies on this task. ## related work existing studies on commentary generation can be divided into real-time commentary @xcite and commentaries written afterwards @xcite . our focus is on the former. live commentary generation is formulated as the extraction of tweets @xcite , the combination of rules and keyword extraction from videos (kim and choi, 2020) and neural networkbased data-to-text @xcite . to generate commentary in real-time, we need to solve at least two tasks: timing identification and utterance generation tasks. however, existing studies focus on the latter, where the timings are given, for example, minute-by-minute updates @xcite . unlike baseball, the timing identification task for race commentary is not trivial because a race cannot be segmented simply. our setting can be considered as a combination of two different research topics: video captioning (kim and choi, 2020) and data-totext @xcite . various methods for encoding video frames have been actively studied @xcite ; commentaries often include comments that focus on the positional relation between cars, which requires a more fine-grained understanding of video frames. the performance of current vision encoders still needs to be evaluated. data-totext is the task of converting structured data into natural language, which has been applied to the domain of finance @xcite @xcite , weather forecast @xcite , a summary of sports matches @xcite and live sports commentary @xcite . the inputs used for existing studies are time-sequence numerical data @xcite , tables @xcite or simulated images @xcite . these models focus on neural networkbased approaches; however, data-to-text tasks have been studied for a long time (see a survey paper @xcite ) for details). existing studies on generation mostly focus on generating text from a single viewpoint, i.e. they generate objective descriptions of video frames in video captioning, and a data-to-text model generates a text that focuses on the main content of the input data. a few existing studies state that live commentaries change depending on the viewpoints of commentators. for example, @xcite found that the generated commentaries on soccer matches are not objective, and these are biased to mention more popular teams. the viewpoints are the key to characteristic commentaries, but most studies have ignored the difference caused by the viewpoints that our dataset addresses. ## dataset we describe the procedure used to create our dataset. we then show its statistics and the analysis to characterize the task. ## tasks and models we formulate a live commentary generation task and introduce the baseline models as shown in figure 4 . we report the performances of the baseline models for both subtasks to better understand the commentary generation task. ## experiments we conduct experiments for the two subtasks to further investigate the characteristics of the task. ## discussion we list the gold and the utterances generated by the model with learning rate 10 -4 in example 1: timestamp: 00:55 gold the player is now following very close to the car ahead. data+text now we're on turn 10, the player is now accelerating data+text+vision i want to step on the brakes firmly here. example 2: timestamp: 02:04 and 02:07 gold we are now approaching the chicane on turn 11 and 12. the player should properly use the curb and go on a straight line here, and the player showed stable race here. data+text the player should brake properly here. the player should brake properly here. the player's car and other cars. integrating vision information further reduces such utterances mentioning relative positions and other detailed information, and also makes utterances less specific. to generate utterances with detailed information, a model must accurately capture the information displayed in a small area of the image. however, it may be too hard for the model to, for example, capture the distance between the car driven by the player and the car just behind, or the drastic changes of speeds from the video frames shown in figure 1 , whereas telemetry data provides the useful information. from the perspective of studies on vision, methods to properly capture such features are worth exploring. we also observed that generated commentaries contain many repetitions of the same utterance, especially utterances generated by the model with vision information. the utterances in example 2 in we found errors in the name of a country e.g., nürburgring in germany was generated as nürburgring in italy. such errors are also known as a common problem in other generation tasks. ## future research directions finally, we discuss the future directions. we noticed that evaluation is very difficult for this task. only bleu scores of course cannot capture the correctness because this evaluation ignores the relation between a commentary and a race represented in. however, manually checking videos, language, structured data, and generated utterances would incur a huge labor cost. an exploration into correct and efficient automatic and manual evaluation methods that consider all vision, language, and structured data should be conducted in the future. for evaluation by using bleu, it might be helpful if we have multiple reference utterances for one timestamps. however, it is difficult to collect multiple utterances simultaneously in this task because different commentators give utterances at different timings. we leave them for an important future research direction. extensions of model would be considered as one of the main steps to produce better commentaries. however, more importantly, we need to explore an essential research question: "what is a good commentary?". further analysis of the characteristics that contributes to making commentaries better need to be conducted. ## conclusion in this paper, we proposed the task of generating commentaries for motor racing games. our analysis reveals that the characteristics of utterances change over time in a race, and such changes are also caused by differences in viewpoints. they also show that combining vision, language and structured data is challenging, which worth studying in depth. for future work, exploring better methods to combine vision, language, and structured data will be a promising direction for future work. we release the data to enhance further studies on generation tasks from multimodal inputs.