Translation from Historical to Contemporary J apanese Using J apanese T5 (2023)

Abstract: This paper presents machine translation from historical Japanese to contemporary Japanese using a Text-to-Text Transfer Transformer (T5). The result of the previous study that used neural machine translation (NMT), Long Short Term Memory (LSTM), could not outperform that of the work that used statistical machine translation (SMT). Because an NMT model tends to require more training data than an SMT model, the lack of parallel data of historical and contemporary Japanese could be the reason. Therefore, we used Japanese T5, a kind of large language model to compensate for the lack of data. Our experiments show that the translation with T5 is slightly lower than SMT. In addition, we added the title of the literature book from which the example sentence was extracted at the beginning of the input. Japanese historical corpus consists of a variety of texts ranging in periods when the texts were written and the writing styles. Therefore, we expected that the title gives information about the period and style, to the translation model. Additional experiments revealed that, with title information, the translation from historical Japanese to contemporary Japanese with T5 surpassed that with SMT.

## introduction this paper develops a translation system from historical japanese to contemporary japanese. in japan, there are a large number of historical literature including untranslated ones. although the experts are able to translate them, it is not only difficult but also time-consuming to translate them manually. therefore, translation systems of historical japanese are necessary. to our knowledge, two works have been done for translation from historical japanese to contemporary japanese (see section 2). @xcite used statistical machine translation (smt), which presented a translation system whose bleu score @xcite was 28.02, the highest in the previous study. @xcite used neural machine translation (nmt), long short term memory (lstm), and the bleu score of their system was 19.95. we believe that the lack of translation data is why the nmt, specifically, translation with lstm, could not outperform the smt. an nmt model tends to require more training data than an smt model does but the parallel data of historical and contemporary japanese are limited; they are 86,684 sentence pairs. therefore, we used textto-text transfer transformer (t5) @xcite , which is a pre-trained large language model (llm), to compensate for the lack of data. after the release of the gpt @xcite and the bert @xcite in 2018, llms have achieved state-of-the-art results in various tasks of natural language processing. we hypothesise that the poor performance due to the limited training data could be complemented by the use of large pre-trained models (see section 3). in addition, we focused on the diversity of the historical texts. the japanese historical corpus consists of a variety of texts ranging in periods when the texts were written and the writing styles. for example, the complete collection of japanese classical literature published by @xcite foot_0 contains older texts written in the 800s and relatively new texts written in the 1800s. in addition, the collection consists of texts of various styles, including diaries, poems, folk stories, and so on (see section 4). therefore, it is not surprising if the diversity of the texts gives a negative influence on the performance of the translation system. @xcite reported that giving speakers' ids to persona-based dialogue generation improves the dialogue quality when the training data consists of data from multiple speakers. we believe that this is because the information of the speakers alleviates the negative influence of the diversity of the train-ing data. instead of speakers' ids, we utilized the titles of the literature book from which the example sentences were extracted. we compared our methods with the previous studies using bleu and bert scores and analysed the quality of the translations (see sections 5, 6 and 7 ). the contributions of this paper are as follows: research on machine translation from historical japanese into contemporary japanese has been reported by hoshino et al. (2014) who used smt and takaku et al. (2020) who used lstm. the bleu scores were 28.02 for smt and 19.95 for lstm. we used t5 to complement the lack of parallel data that is required for the training of the model. an example of a previous study of translation using t5 is work reported by emezue and dossou (2021). they used multilingual t5 (mt5) to translate african languages such as kosa, yoruba, and igbo into english and french. the number of data on these african languages is limited; kosa, the largest corpus has 158,660 monolingual sentences and 137,000 parallel data with english. they reported that the bleu score of the kosa-english translation was 30.25. their research has shown that t5 is effective for low-resource language translation. @xcite used t5 for machine translation to aid bilingual data-to-text generation and semantic parsing. they showed the machine translation using t5 improved performances of generation and parsing tasks. we gave the title of the literature book from which the example sentence was extracted to the translation model. this is inspired by @xcite , who gave speakers' ids to the personabased dialogue generation system. the dialogue quality improved when the training data consisted of data from multiple speakers; the utterances output from the system became more speaker-specific. we expected that the information of the speakers would alleviate the negative influence of the diversity of the training data. instead of speakers' ids, we input the titles of the literature book to alleviate the negative influence of the diversity of the training sentences in period and style. in addition, @xcite added an extra token to backtranslation data for noising techniques foot_1 . they inform the model which data is back-translated to alleviate the noise's effect. instead, we added the title of the literature book to inform the model the period and writing style information. in addition, there is research on translation related to digital humanities such as @xcite , @xcite , and @xcite . ## translation using t5 t5 is a pre-trained model based on transformer @xcite trained with colossal clean crawled corpus (c4), which is a cleaned version of common crawl's web crawl corpus. using such high-quality data and careful tuning of the model, t5 achieved state-of-the-art performance on 26 various tasks in 2019. in this paper, we fine-tuned pre-trained japanese t5 to translate historical japanese into contemporary japanese. we aim to compensate for the lack of data using a pre-trained llm because the amount of parallel corpus of historical and contemporary japanese is limited. we used sonoisa/t5base-japanese foot_2 for japanese t5. this model was trained with japanese wikipedia dump data, japanese oscar corpus, and japanese cc-100: monolingual datasets from web crawl data. although it is mostly trained with contemporary japanese, we used it for transformation from historical japanese to contemporary japanese. therefore, this method could be deemed as a diachronic domain adaptation. we employed this method because @xcite reported that the performance of japanese word sense disambiguation of historical japanese significantly improved when they used diachronic domain adaptation using japanese bert trained with contemporary japanese. in addition, we gave the titles of the literature books from which the example sentences were extracted, to the translation model: the fine-tuned t5. we set a title and colon as a prefix of the historical japanese sentence. ## data we used a parallel corpus extracted from the complete collection of japanese classical literature published by shogakukan by @xcite . this corpus consists of historical japanese sentences paired with manually translated contemporary japanese sentences. we used 50 pieces of literature listed in tables 3 and 4 . however, they are 60 books because some literature has more than one volume and hōgen monogatari and heiji monogatari were compiled together in one book 4 . the corpus contains a total of 86,684 sentence pairs, some of which have multiple contempo- historical japanese total number of sentences 86,684 vocabulary size 49,200 number of tokens 2,774,745 contemporary japanese total number of sentences 86,684 vocabulary size 45,690 number of tokens 3,611,783 as the parallel corpus did not have the title information, we searched the sentences of the source language, i.e., historical japanese, in the complete collection of japanese classical literature, the original dataset, and automatically identified it. in this process, a few errors happened especially for titles year style chikamatsu-monzaemon-shū (collection of chikamatsu-monzaemon's stories) 1703-1720 playbook genji monogatari (the tale of genji) before 1001 tale gikeiki (the chronicle of yoshitsune) before 1400 war chronicle heichū monogatari (tales of heichū) around 1000 poem tales heiji monogatari (the tale of heiji) mid-13th century war chronicle heike monogatari (the tale of the heike) before 1309 epic account hōgen monogatari (the tale of hōgen) around 1220 war chronicle hōjōki (square-jō record) 1212 essay imose yamaonna teikin 1771 playbook ise monogatari (the tales of ise) around 900? poem tales izumi shikibu nikki (the diary of lady izumi) 1008 poetic diary jikkinsyo 1252? folktales kagerō nikki (the mayfly diary) 974 poetic diary kaikou 1764 playbook kanameishi 1663 folktales kindaishōka 1209 essay on poetry kokin wakashū (collection of japanese poems of ancient and modern times) 905 anthology of the poetry kokin wakashū appendix 905 anthology of the poetry kokka-hachiron 1742 essay on poetry konjaku monogatarishū (anthology of tales from the past) around 1120 folktales koraifūteishō 1197 essay on poetry maigetsu-shō 1219 essay on poetry makura no sōshi (the pillow book) 1001 essay murasaki shikibu nikki (the diary of lady murasaki) 1010 diary matsuranomiya monogatari (the tale of the matsura palace) 1201 tale mutsuwaki (chronicle of mutsu) late 11th century war chronicle nihon ryōiki (miraculous stories from the japanese buddhist tradition) 822 folktales nii-manabi-iken 1811 essay on poetry ochikubo monogatari (the tale of ochikubo) around 1000 tale ōkagami (the great mirror) 1119 history book otogi monogatari 1678 folktales sanukinosuke nikki (the diary of sanukinosuke) 1109 diary sarashina nikki (the sarashina diary) 1020 nonfiction narrative shasekishū (sand and pebbles) 1283 buddhist text shōbōgenzō zuimonki (the treasury of the true dharma eye: record of things heard) 1235-1238 dharma talks shōmonki, masakadoki (chronicle of masakado) before 1099 war chronicle soga monogatari (the revenge of the soga brothers) before 1285 war chronicle taketori monogatari (the tale of the bamboo cutter) around 900 fictional prose narrative takitsuke moekui keshizumi 1677 folktales ## experiments we conducted two kinds of experiments: (1) the experiment using the parallel data and (2) the experiment using the title-added parallel data. we used sonoisa/t5-base-japanese and t5 tokenizer in the hugging face library for japanese t5 and its tokenizer. we conducted a grid search for the learning rate, epoch number, and repetition penalty in both experiments as shown in we measured bleu scores and the similarities between translations using sentence-bert @xcite for evaluation. for the calculation of bleu score, we used sacreblue @xcite 5 . we calculated the similarities between translations for reference because bleu scores do not directly reflect the meanings of the sentences. we used sonoisa/sentence-bert-base-jamean-tokens-v2 to calculate the similarities 6 . we used default settings for the hyperparameters of the sentence-bert. ## translation analysis in addition, in the second example, the origi-nal sentence says "ここなる人々" literally means "people staying here." the translations of our systems are correct if the sentence was translated literally. however, the reference translation rephrased the word to "my sons." this is an additional explanation by a translator. therefore, we believe that the bleu scores of our systems tend to decrease because the reference translations are idiomatic. to evaluate these systems, literal translations are necessary. moreover, some japanese verbs totally change the word when an honorific form is used. it is like the past tense of irregular verbs in english. in addition, some japanese verbs have some variant honorific forms. in the first example, two verbs, which mean bring someone and order, are written in normal form in the reference translation. however, when the t5 translation with the parallel data was used for the experiments, two verbs, which means bring someone and say, were written in honorific form. when the title-added parallel data was used, a verb that means "say" was written in another honorific form and the other two verbs, which mean bring someone and follow, were written in normal form. in the second example, "お は せ ま し か ば," which means "were still alive" with old honorific form in the original sentence was translated into "生 き て い ら っ し ゃ っ た ら," in the reference translation, which means the same but with contemporary honorific form. however, it was translated into "ご存命でいらっしゃったら" using our systems instead of "生きていらっしゃったら." "ご存命でいらっしゃったら" means "were still alive" again, and this is another expression that means the same with a different contemporary honorific form. in other words, these two expressions mean the same, however, the ways of honorification are different. these kinds of variations of expressions also tend to decrease the bleu scores. because we do not have the translation examples of the other systems, we cannot compare the results directly 8 . however, we believe that nmt tends to learning rate epoch number repetition penalty bleu the parallel data 0.0002 10 1.0 27.50 the title-added parallel data 0.0005 5 1.5 28.67 lstm @xcite 19.95 smt @xcite 28.02 ## conclusion and future work in this paper, we translated historical japanese into contemporary japanese by fine-tuning japanese t5, which is a large-scale pre-trained model. we expected that t5 would be useful for the translation because it compensates for the lack of parallel data. in addition, we proposed to add a book title from which the source sentence was extracted at the beginning of the input sentences. because chj, the historical text corpus we used, ranges in periods when the texts were written and styles, we expected that the title of the books could give those kinds of information to the translation systems. the experiments revealed that t5 and giving the title of the books are effective for the translation; bleu scores and the sentence-bert similarities of our system outperformed those of previous studies, which used smt and lstm, respectively. according to the analysis of the translation examples, we observed that the translations of our systems are more literal than the reference translations. however, the bleu scores tend to decrease because the reference translations are idiomatic, rather than literal translation. the evaluation using the literal translations by human annotators is our future work. in addition, further experiments and analyses should be done to investigate the effectiveness of our method. for example, comparison to the setting where random titles were used and the setting where only periods and genres are given should be explored. moreover, we are planning to compare our method to large language models, such as chatgpt. sentence bleu sbert original 具して参れ」と宣へば、常葉を具して参りけり。 english translation (someone,the subject in omitted here) brought tokiha (this is a woman's name.) to him because (he, the subject is omitted here. ) said "bring her to me." reference 連れて来い」と命ずるので、伊藤武者は常葉を連れてきた。 english translation ito musha (this is a warrior's name.) brought tokiha to him because (he, the subject is omitted here. ) ordered as "bring her to me." the parallel data 連れて参れ」と言われるので、常葉を連れて参った。 21.98 0.81 english translation (he, the subject is omitted here. ) brought tokiha to him because (he, the subject is omitted here again. ) said "bring her to me." the title-added parallel data ついて来い」とおっしゃるので、常葉を連れて行った。 32.04 0.81 english translation (he, the subject is omitted here. ) brought tokiha to him because (he, the subject is omitted here again. ) said "follow me." sentence bleu sbert original 故殿おはせましかば、ここなる人々も、 かかるすさびごとにぞ、心は乱らまし」とうち泣きたまふ。 english translation (the subject is omitted here.) cried saying " if the late lord were still alive, i'm sure even the people staying here would have had to worry their heads about these whimsical play things." reference 故殿が生きていらっしゃったら、こちらの息子たち だってこうした気まぐれな遊び事で頭を 悩ませていたことでしょうに」とお泣きになる。 english translation (the subject is omitted here.) cried saying " if the late lord were still alive, i'm sure even my sons here would have had to worry their heads about these whimsical play things." the parallel data 亡き殿がご存命でいらっしゃったら、ここにいる 人々も、こうした遊び事に心を 奪われていたでしょうに」とお泣きになる。 35.16 0.83 english translation (the subject is omitted here.) cried saying " if the late lord were still alive, i'm sure even the people staying here would have been fascinated by these play things." the title-added parallel data 亡き殿がご存命でいらっしゃったら、ここにいる 人たちも、こうした慰みごとのために心が乱れ たことでしょう」とお泣きになる。 28.26 0.74 english translation (the subject is omitted here.) cried saying " if the late lord were still alive, i'm sure even the people staying here would have been disturbed by these comforts."