Controlling J apanese Machine Translation Output by Using JLPT Vocabulary Levels (2022)

Abstract: In Neural Machine Translation (NMT) systems, there is generally little control over the lexicon of the output. Consequently, the translated output may be too difficult for certain audiences. For example, for people with limited knowledge of the language, vocabulary is a major impediment to understanding a text. In this work, we build a complexity-controllable NMT for English-to-Japanese translations. More particularly, we aim to modulate the difficulty of the translation in terms of not only the vocabulary but also the use of kanji. For achieving this, we follow a sentence-tagging approach to influence the output. Controlling Japanese Machine Translation Output by Using JLPT Vocabulary Levels.

## introduction in the natural language processing research, text simplification aims to find variants of a text which convey the same meaning but are expressed in a simpler form. this process includes modifications such as reducing the length, decreasing the use of infrequent words, etc. simplification systems are useful for helping certain populations such as children, non-native speakers, and people with a low level of literacy or language disorders @xcite . in this work, we apply simplification to the translation task. in particular, we aim to control the lexicon complexity of english-to-japanese neural machine translation (nmt) models. the output generated by an nmt system in japanese may be too difficult to understand for a person with more limited knowledge of the language. an example of this is the use of kanji ideograms. certain kanji are learned in the later stages of education 1 , which causes some people not to be entirely familiarized with all of them. this implies that both vocabulary and kanji may represent an accessibility problem. for modulating the translation, our approach is inspired by other works consisting of influencing the generation of sentences for certain domains or languages. this can be achieved by including a tag at the beginning of each sentence stating how the output should be. for our approach, we use tags to indicate the level of lexicon complexity expected in the output. we first insert a token at the beginning of each training sentence according to the complexity of the japanese target side. then, at decoding time, we can influence the output by using such tags. this paper describes such an approach, and explores the following research questions (rq): rq1: can the vocabulary complexity of the output be controlled adding tags in the source sentences? the addition of tags in the source sentences to control the output of the nmt models has been explored not only for different domains @xcite but also for different languages @xcite . we want to explore these techniques for japanese translation and investigate how could it be used to control the complexity level of the output. rq2: how much does the output level agree with the complexity level indicated in the input? although adding tags could bias the complexity of the translation, it has limitations. for example, some translations may require the use of complex vocabulary despite the restrictions. we analyze to what extent the complexity of the sentences generated by the nmt corresponds to those indicated in the input. ## japanese language and jlpt the japanese language has three writing systems foot_0 : hiragana (46 characters); katakana (46 characters); and kanji (more than 2000 characters). hiragana is mainly used for native japanese words whereas katakana is used for foreign words or onomatopoeia. for example, the translation for the word "hat" is ぼうし (read as "boushi") which is written in hiragana. alternatively, some people may use the term borrowed from english "ハット" ("hatto") which is a transliteration of "hat". as it is a loanword, it is written in the katakana syllabary. despite that, native japanese speakers would use more frequently the kanji ideogram 帽子 (which is also read as "boushi") for "hat". although it is possible to fully express in japanese using hiragana or katakana exclusively, kanji is usually used. despite that, as there exists more than 2000 kanji, a japanese learner would assimilate them gradually, and therefore be more comfortable using hiragana for writing or reading certain words. a popular criterion to measure the level of proficiency in japanese for non-native speakers is the japanese language proficiency test (jlpt). it is a five-level grading system that ranges from jlpt 5 (the most basic) to jlpt 1 (the most advanced). these five levels are also referred to as n5, n4, n3, n2, and n1. 3 in this work we use both notations, "jlpt" or "n" , indistinctly. additionally, we may refer to higher levels to those jlpt levels closer to n1, and lower levels to those closer to n5. ## related work in the text simplification field, several approaches alter the complexity of the lexicon. for example, @xcite propose replacing difficult words with a simpler synonym. furthermore, @xcite perform the complex-word replacement applied for japanese language. alternatively, @xcite build a monolingual nmt system to transform sentences into a simplified version in the same language. @xcite propose a similar monolingual sequence-to-sequence system with several levels of complexity in english. these are based on the grade level of us education system. similarly to our work, they control the complexity by using special tokens for each grade. performing text simplification in combination with translation has also been explored by @xcite . they focus on using automatically simplified sentences as the input of an nmt model. regarding the complexity-controllable translation, @xcite aim to produce translations based on different levels established by the common european framework of reference for languages (cefr). shardlow and alva-manchego (2022) also performs combinations of simplification and translation (translate then simplify, simplify then translate and direct) to generate simplified translations. there are previous works that use tags to control the output. @xcite extract different characteristics that measure the complexity and include them as tags in the source to condition the output. similarly, @xcite also use a tagging system, training the model with a dataset where the same sentences have been rewritten at different complexity level. finally, @xcite use two tags (i.e. "simple" and "complex") to classify the sentences by difficulty. some difference with our research is that we use a five-tag system based on the jlpt framework. in addition, as we explore the japanese language, the definition of complexity also considers spelling. therefore, depending on the writing system, some words may have different complexity levels. ## complexity-controllable translation our proposal consists of building an english-to-japanese nmt model with a controllable lexicon complexity. in this work, the complexity is measured based solely on the vocabulary of the different jlpt levels. there are two main processes involved: (i) determine the jlpt level of a sentence (section 4.1); and (ii), include the complexity level in the training word jlpt level 友達 3 ともだち 5 ## experiments we build nmt models in the english-to-japanese direction using marian nmt @xcite . these models consist of a transformer @xcite model with 6 layers both in the encoder and 6 in the decoder. we train it for a maximum of 500k steps (18 epochs). we use one of the biggest english-japanese corpus, jparacrawl v3.0 @xcite , as train set (25.7m sentences) and 10k randomlyselected sentences from tatoeba @xcite as dev set. for the experiments, we build two models. one model is built with plain data without any modification that serves as a reference for comparison purposes. the second model is built by including tags as described in section 4.2. we use kytea @xcite to split sentences and extract the vocabulary of the japanese side. for testing the models, we randomly selected 5000 sentences from tatoeba (from those not included in the dev set). this dataset is built for educational purposes and therefore there are sentences of different complexities. first, we translate these sentences with the plain model. then, for the model that uses tags, we replicated each sentence five times and added a different tag (from n1 to n5) to each of them. by doing this we encourage the model to produce translations of different levels for each input. this means that we generate six alternative translations from a single test set. one output is the translation of the plain nmt model trained without tags ("no-tag" output). the other five outputs correspond to the translation when one of the tags is added at the beginning of the sentence. in the following, we name each output with the tag added in the source. for example, we refer as n2 output to the translations when the tag "n2" was added in the input sentences. ## experimental results we divide the analysis of the results of the experiments into four different sections: (i) section 6.1, where we explore the simplification capabilities of the nmt model (rq1); (ii) section 6.2, where we analyze the agreement between the output level and that stated in the input (rq2); (iii) section 6.3, where we investigate the translation quality (rq3); and (iv), section 6.4, where we provide translation examples that illustrate the effect of constraining complexity in the output. 6.1 rq1: can the vocabulary complexity of the output be controlled adding tags in the source sentences? the tags used to modulate the complexity are based on the vocabulary and kanji in japanese. accordingly, we explore whether the outputs of the model are simplified in terms of these. first, we explore the usage of kanji. for someone with limited knowledge of japanese, it is expected the kanji comprehension to be lower also. therefore, the outputs in the lower levels should contain a smaller proportion of kanji. in figure 1 we present what is the proportion of kanji in the outputs. in the reference 27.5% of the characters are kanji, which is similar to the output of the nmt model with no tags. this is also the proportion in the outputs of higher levels of jlpt (in fact, the n1 output has a slightly higher usage of kanji than the plain model). the proportion of kanji decreases gradually as lower jlpt levels are stated in the input. for the n5 output, the percentage of characters that are kanji is just 14.8%. therefore we can say that in terms of kanji usage, the inclusion of tags is beneficial to decrease the complexity. in addition to that, we compare the vocabulary sizes of the translations. in general, the more restricted the generation is, the lower the size of the vocabulary is expected to be. in figure 2 we present the number of distinct words in each output. in the plot, we see that the size of the vocabulary for the model with no tags is similar to those of higher levels such as n1 or n2. the number of words tends to decrease the more restricted the complexity is. the n5 output seems to be an exception to that, as the vocabulary size exceeds that of the n1 output. however, upon inspection of the translations, we discovered that many words were just copied directly from the source instead of being a translation. we decided to include in that plot the size of vocabulary after removing the alphanumeric terms (e.g. english words, numbers) from the output as it may distort the analysis. under these circumstances, we observe that the number of words also decreases for the n5 output. in this case, the vocabularies of the translations range from 4400 words (for less restricted outputs such as no-tag or n1) to 3800 words (for n5 output). consequently, the sizes of the vocabularies also indicate that the complexity tag is useful to limit the diversity of words. for answering the second rq, we want to estimate whether the outputs match the levels stated on the test set. therefore, we classified the outputs of the model (25000 sentences) and compare them to the level that was prepended as a tag on the source side. in figure 3 we present the heatmap of the confusion matrix. we found that only 38% of the sentences in the output had an exact match with the proposed tag on their source side. most of the sentences in the output of the model are predicted to be jlpt 3 level as can be seen in figure 4 . in addition, we also see that many disagreements occur in sentences where lower complexity is expected. many sentences in the n5 output include kanji of more advanced levels. we find two main reasons for that. one reason is that certain terms may not exist in the vocabulary of lower jlpt levels. for example, even when attaching an n5 tag to a sentence, the model may not be able to translate difficult concepts such as "monopolize" or "corruption" that do not exist in the vocabulary of such a low level. another reason is that the model does not have enough information to generate an adequate translation. for example, the word "window" is a basic word that would be categorized as jlpt 5 (lowest complexity level) if it is spelled in hiragana as "ま ど". however, in the n5 output, this was spelled using the kanji "窓" instead, which is considered to be in level jlpt 3. this occurs because it is unlikely to find the hiragana spelling in most texts. upon the inspection of the training data, we did not find any occurrence of translation of "window" spelled in hiragana. answering this rq, we find that the model is not very accurate in terms of generating translation in the jlpt level as expected in the input. alternatively, one may consider that the knowledge of vocabulary should be cumulative. in the experiments, we used the hardest word to tag the sentence, which implies that sentences classified as jlpt 1 or 2, also contain the vocabulary of lower levels. this is coherent with the problem of different literacy levels, as an advanced reader is also capable of reading sentences with simpler vocabulary. in such a case, we could consider that the output should be at either the same or lower level than that stated in the input. then, the number of correctly classified sentences ascends to 62%. despite that, the problem of lower-level sentences containing difficult kanji remains. ## conclusion and future work in this work, we have used the addition of tags to control the complexity of the output of an english-to-japanese mt model. the complexity has been established based on the vocabulary and kanji of jlpt exams. our results show that the complexity of the lexicon in the translation can be modulated with these tags. despite that, although it can be influenced to a certain extent, the output may contain vocabulary of higher levels than that stated. this is not only because in the lower levels the vocabulary is too limited, but also because of the lack of translation occurrences in the train data. we have also shown that restricting the output harms the translation quality. none of the outputs obtained using a complexity tag was better than that of a model trained without any restriction. in addition, enforcing too much simplicity causes the model not to be able to translate accurately and in some cases, it ends up copying words from the source. one limitation of this work is that the classification of difficulty is decided solely based on the vocabulary. in future work, we want to expand this to also consider other factors such as the grammar or length of the sentences. another aspect that we want to investigate is using alternative configurations. for example, text simplification or paraphrasing models @xcite could be included to change the distribution of the complexity in the training data.