Document-aligned J apanese- E nglish Conversation Parallel Corpus (2020)

Abstract: Sentence-level (SL) machine translation (MT) has reached acceptable quality for many high-resourced languages, but not document-level (DL) MT, which is difficult to 1) train with little amount of DL data; and 2) evaluate, as the main methods and data sets focus on SL evaluation. To address the first issue, we present a document-aligned Japanese-English conversation corpus, including balanced, high-quality business conversation data for tuning and testing. As for the second issue, we manually identify the main areas where SL MT fails to produce adequate translations in lack of context. We then create an evaluation set where these phenomena are annotated to alleviate automatic evaluation of DL systems. We train MT models using our corpus to demonstrate how using context leads to improvements.

## introduction the quality of machine translation (mt) for written text and monologue has vastly improved due to the increased amount of available parallel corpora and recent neural network technologies. however, there is much room for improvement in the context of dialogue or conversation translation. one typical case is the translation from a pro-drop language to a nonpro-drop language where correct pronouns must be supplemented according to the context. the omission of the pronouns occurs more frequently in spoken language than written language. recently, context-aware mt models attract attention from many researchers @xcite to solve this kind of problem, however, there are almost no parallel conversation corpora with context information except the rather noisy open subtitles corpus @xcite . a document and sentence-aligned conversation parallel corpus should be advantageous to push mt research in this field to the next stage. in this paper, we introduce a newly constructed documentaligned (da) japanese-english conversation corpus, which contains three sub-corpora: business scene dialogue (bsd @xcite ), japanese translation of ami meeting corpus (ami @xcite ) and japanese translation of ontonotes 5.0 (on @xcite ). the corpus contains multi-person conversations in various situations: business scenes, meetings under specific themes, broadcast conversations and telephone conversations. we supplement the original bsd part with additional data, increasing its size by almost three times. we also enrich the corpus with speaker information and other useful meta-data, and separate balanced versions of development and evaluation data sets. ## related work there are many ready-to-use parallel corpora for training mt systems, but most of them are in written languages such as web crawl, patents @xcite , scientific papers @xcite . even though some parallel corpora are in spoken language, they are mostly monologues @xcite or contain a lot of noise @xcite . most of the mt evaluation campaigns such as wmt foot_0 , wat foot_1 adopt the written language, monologue or noisy dialogue parallel corpora for their translation tasks. among them, there is only one clean, dialogue parallel corpus @xcite adopted by iwslt foot_2 in the conversational speech translation task. jparacrawl @xcite is a recently announced large english-japanese parallel corpus built by crawling the web and aligning parallel sentences. its size is impressive, but it is composed of noisy web-crawled data and has many duplicate sentences. compared to our corpus, jparacrawl does not have meta-information and is not da. @xcite evaluate what modern mt systems struggle with when translating from english into russian and construct new development and evaluation sets based on human evaluation. the sets target linguistic phenomena -dexis, ellipsis and lexical cohesion. the authors also provide code for a context-aware nmt toolkit that improves upon translating these phenomena. in contrast, our development/evaluation sets contain complete documents of consecutive sentences, not broken up into only the sentences requiring context. ## corpus description our corpus consists of 3 sub-corpora, each of which originates from different sources -bsd, ami, and on. bsd was newly constructed, while ami and on are translations of the existing english versions of these corpora. detailed statistics of the sub-corpora are provided in tables 1 and 2 . bsd consists of the scenes mentioned in ## machine translation experiments the conversation corpus alone is not big enough to train real-world nmt systems (as demonstrated by @xcite ). however, by increasing the size of the high-quality bsd corpus, we managed to train reasonable nmt systems. the full statistics of our data are shown in ## human evaluation we translated the evaluation set in both directions using our baseline nmt and performed a two step human evaluation similar to @xcite . after that, we analysed the remaining sentences to determine which truly require context. we used yahoo! japan crowdsourcing 7 for the human evaluation. evaluation quality was guaranteed using screening questions which were indistinguishable from the real questions. only those who correctly answered all the screening questions were considered valid evaluators. each sentence was evaluated by 5 different evaluators. in the first step, evaluators were asked to mark each sentence individually as ok or not good (ng), where ok meant that the general meaning of the original sentence was transferred to the translation, whereas ng meant that the translation is completely unusable. in the second step, we used only the consecutive pairs of sentences, which were both marked as ok in the first step by at least three evaluators, and asked evaluators to mark them as ok if the corresponding translations made sense in context of each other. we calculated the free-marginal kappa @xcite values for the evaluations to measure agreement between evaluators. the results (overall agreement -67%, freemarginal kappa -0.34) show moderate agreement, which is common for crowdsourcing. ## conclusion we presented a document-aligned parallel corpus of english-japanese conversations intended for training and evaluation of mt systems. we describe the corpus in detail and indicate which linguistic phenomena are challenging for mt. in our @xmath0: results of the second step of the crowdsourcing human evaluation compared to @xmath1. the first row shows sentence pair totals and the last two rows show sentence pairs, where both sentences were marked as "good" individually, evaluated in context of each other as either good or bad pairs. prev. source: when did you say you' d answer me? prev. conc.+f.: when did they say they will reply? source: they said early next week. base.: he told me early next week. conc.+f.: i said it early next week . evaluation set we marked examples, which can have multiple contrasting translations when tackled on the sentence-level. the release will include the full bsd corpus and japanese translations of ami and on along with instructions on how to align them. the original source language, speaker, scene, document, ambiguity type will also be included. in the future we plan to model speakers and origin languages in mt, as it can help capture broader context @xcite and more precise pronoun translations @xcite . we are also interested in experimenting with modelling the scene information within the training data to produce more appropriate translations for each of the politeness settings.