X iaomingbot: A M ultilingual R obot N ews R eporter (2020)

Abstract: This paper proposes the building of Xiaomingbot, an intelligent, multilingual and multimodal software robot equipped with four inte- gral capabilities: news generation, news translation, news reading and avatar animation. Its system summarizes Chinese news that it automatically generates from data tables. Next, it translates the summary or the full article into multiple languages, and reads the multi- lingual rendition through synthesized speech. Notably, Xiaomingbot utilizes a voice cloning technology to synthesize the speech trained from a real person’s voice data in one input language. The proposed system enjoys several merits: it has an animated avatar, and is able to generate and read multilingual news. Since it was put into practice, Xiaomingbot has written over 600,000 articles, and gained over 150,000 followers on social media platforms.

## introduction the wake of automated news reporting as an emerging research topic has witnessed the development and deployment of several robot news reporters with various capabilities. technological improvements in modern natural language generation have further enabled automatic news writing in certain areas. for example, gpt-2 is able to create fairly plausible stories @xcite . bayesian generative methods have been able to create descriptions or advertisement slogans from structured data @xcite . summarization technology has been exploited to produce reports on sports news from human commentary text @xcite . while very promising, most previous robot reporters and machine writing systems have limited * the work was done while the author was an intern at bytedance ai lab. ## system architecture the xiaomingbot system includes four components working together in an pipeline, as shown in figure 1. the system receives input from data table containing event records, which, depending on the domain, can be either a sports game with time-line information, or a financial piece such as tracking stock market. the final output is an animated avatar reading the news article with a synthesized voice. figure 2 illustrates an example of our xiaomingbot system. first, the text generation model generates a piece of sports news. then, as is shown on the top of the figure, the text summarization module trims the produced news into a summary, which can be read by users who prefer a condensed abstract instead of the whole news. next, the machine translation module will translate the summary into the language that the user specifies, as illustrated on the bottom right of the figure. relying on the text to speech (tts) module, xiaomingbot can read both the summary and its translation in different languages using the same voice. finally, the system can visualize an animated character with synchronized lip motion and facial expression, as well as lifelike body and clothing. ## news generation in this section, we will first describe the automated news generation module, followed by the news summarization component. ## news translation in order to provide multilingual news to users, we propose using a machine translation system to translate news articles. in our system, we pre-trained several neural machine translation models, and employ state of the art transformer big model as our nmt component. the parameters are exactly the same with @xcite . in order to further improve the system and speed up the inference, we implemented a cuda based nmt system, which is 10x faster than the tensorflow in the 35th minute, alavés mubarak received a yellow card. approach 1 . furthermore, our machine translation system leverages named-entity (ne) replacement for glossaries including team name, player name and so on to improve the translation accuracy. it can be further improved by recent machine translation techniques @xcite . we use the in-house data to train our machine translation system. for chinese-to-english, the dataset contains more than 100 million parallel sentence pairs. for chinese-to-japanese, the dataset contains more than 60 million parallel sentence pairs. ## multilingual news reading in order to read the text of the generated and/or translated news article, we developed a text to speech synthesis model with multilingual capability, which only requires a small amount of recorded voice of a speaker in one language. we developed an additional cross-lingual voice cloning technique to clone the pronunciation and intonation. our cross-lingual voice cloning model is based on tacotron 2 (j. shen, 2018), which uses an attentionbased sequence-to-sequence model to generate a sequence of log-mel spectrogram frames from an 1 https://github.com/bytedance/byseqlib input text sequence @xcite . the architecture is illustrated in figure 4 , we made the following augmentations on the base tacotron 2 model: for chinese tts, we used hundreds of speakers from internal automatic audio text processing toolkit, for english, we used libritts dataset @xcite , and for japanese we used jvs corpus which includes 100 japanese speakers. as for input representations, we used phoneme with tone for chinese, phoneme with stress for english, and phoneme with mora accent for japanese. in our experiment, we recorded 2 hours of chinese voice data from an internal female speaker who speaks only chinese for this demo. ## synchronized avatar animation synthesis we believe that lifelike animated avatar will make the news broadcasting more viewer friendly. in this section, we will describe the techniques to render the animated avatar and to synchronize the lip and facial motions. ## conclusion in this paper, we present xiaomingbot, a multilingual and multi-modal system for news reporting. the entire process of xiaomingbot's news reporting can be condensed as follows. first, it learns how to write news articles based on a template based table2text technology, and summarize the news through an extraction based method. next, its system translates the summarization into multiple languages. finally, the system produces the video of an animated avatar reading the news with synthesized voice. owing to the voice cloning model that can learn from a few chinese audio samples, xiaomingbot can maintain consistency in intonation and voice projection across different languages. so far, xiaomingbot has been deployed online and is serving users. the system is but a first attempt to build a fully functional robot reporter capable of writing, speaking, and expressing with motion. xiaomingbot is not yet perfect, and has limitations and room for improvement. one such important direction for future improvement is the expansion of areas that it can work in, which can be achieved through a promising approach of adopting model based technologies together with rule/template based ones. another direction for improvement is to further enhance the ability to interact with users via a conversation interface.