Evaluation Scheme of Focal Translation for J apanese Partially Amended Statutes (2021)

Abstract: For updating the translations of Japanese statutes based on their amendments, we need to consider the translation “focality;” that is, we should only modify expressions that are relevant to the amendment and retain the others to avoid misconstruing its contents. In this paper, we introduce an evaluation metric and a corpus to improve focality evaluations. Our metric is called an Inclusive Score for DIfferential Translation: (ISDIT). ISDIT consists of two factors: (1) the n-gram recall of expressions unaffected by the amendment and (2) the n-gram precision of the output compared to the reference. This metric supersedes an existing one for focality by simultaneously calculating the translation quality of the changed expressions in addition to that of the unchanged expressions. We also newly compile a corpus for Japanese partially amendment translation that secures the focality of the post-amendment translations, while an existing evaluation corpus does not. With the metric and the corpus, we examine the performance of existing translation methods for Japanese partially amendment translations.

## introduction in the world's globalized society, governments must quickly announce their statutes worldwide to facilitate international trade, economic investments, legislation support, and so on. the japanese government addressed this issue in april 2009 by launching the japanese law translation database system (jlt) @xcite where it announces the english translations of japanese statutes. however, as of january 2020, only 23.4% (163/697) of the translated statutes in jlt correspond to their latest versions @xcite . after amending a statute, its translation must be promptly updated to avoid creating confusion among international readers. unfortunately, statutory sentences are much tougher to translate than ordinary sentences because the former are highly technical, complex, and long. furthermore, when translating statutory sentences that are partially modified by an amendment, we must consider focal translations. that is, we should only modify expressions that are changed by the amendment without changing the others. for example, consider the following sentence: "申立ては、事故の事実を示して、書面 でこれをしなければならない。" (the request shall be made in a document stating the facts of the accident.) its amendment rewrote "事故" (jiko; accident) to "海難" (kainan; marine accident). the following revision satisfies the focality requirement: "the request shall be made in a document stating the facts of the marine accident" because it contains minimum modifications. on the other hand, although "the ::::::: petition shall be made in a document ::::::::: however, we argue that two matters from their study must be improved: the evaluation metric amendment sentence in an amendment act (act no. 34 of 2019) in article 164, ① delete paragraph 4, ② delete the latter part of paragraph 3, ③ replace "the parents of" with "(limited to a child of 15 years of age or older)" in item (i) of the same paragraph, ④ replace "set forth in the preceding item" with "who exercises parental authority over …" in item (ii) of the same paragraph, ⑤ delete item (iii) of the same paragraph, ⑥ regard the same paragraph as paragraph 6 of the same article, ⑦ add the following paragraph next to the same paragraph: 7 … of special adoption eligibility … (omitted) translation figure 1 : amendment sentence and the dataset. their metric consists of two factors: (1) the n-gram recall of expressions unaffected by amendments and (2) a redundant penalty for lengthy outputs. although with this metric we can evaluate how completely the method retained expressions irrelevant to the amendment, we cannot evaluate how adequately it translated expressions relevant to the amendment. the second is the dataset they used for their experiments. their translation examples of partially amended statutory sentences are from amendment-versioncontrolled bilingual statutes in jlt. however, translations in jlt are not always focal. therefore, their reported scores do not seem accurate. in this paper, we solve these two matters. for the first, we introduce another metric for focality called the inclusive score for differential translation (isdit), which incorporates n-gram precision between the output and the reference instead of a redundant penalty. with this modification, the metric simultaneously evaluates the translation quality of both the changed and unchanged expressions that indicate the quality of the focal translation. for the second, we compile a corpus that secures focality between pre-and postamendment translations and achieve it by asking professional human translators to translate focal post-amendment translations. this paper makes the following contributions to amended statutory sentence translation tasks: this paper is organized as follows. in section 2, we clarify the background of our study. in section 3, we explain related work. in section 4, we describe our proposal and present our evaluation experiments and discussions in section 5. finally, we summarize and conclude in section 6. ## background in this section, we clarify the background of our study. first, we introduce the partial amendment process in japanese legislation from the viewpoint of document modification and then we identify our study objective in the process. ## related work we describe related work in this section. we overview the suitable machine translation methods for partially amended sentences in section 3.1. we discuss metrics and data in sections 3.2 and 3.3. ## proposal in this section, we propose an evaluation scheme for japanese partially amendment translations. our evaluation scheme includes a new evaluation metric isdit and a differential translation corpus that secures the focality of its examples. ## experiment we experimentally evaluated the machine translation methods with our new resources. ## summary we proposed a better evaluation scheme for japanese partially amendment translations and developed a new metric called isdit that assesses the translation quality of both changed and unchanged expressions. we also compiled a corpus that secures the focality of translation. using our corpus, we observed the characteristics of translation methods and isdit. our future work will increase the size of our corpus so that it can be used for neural network training, considering the publicization of the corpus. we will also identify the best weighting of the two factors in isdit. third, we will consider applications of isdit to other domains of versioncontrolled documents such as contracts, technical documents, and product manuals.