Modeling Target-side Inflection in Placeholder Translation (2021)

Abstract: Placeholder translation systems enable the users to specify how a specific phrase is translated in the output sentence. The system is trained to output special placeholder tokens and the user-specified term is injected into the output through the context-free replacement of the placeholder token. However and this approach could result in ungrammatical sentences because it is often the case that the specified term needs to be inflected according to the context of the output and which is unknown before the translation. To address this problem and we propose a novel method of placeholder translation that can inflect specified terms according to the grammatical construction of the output sentence. We extend the seq2seq architecture with a character-level decoder that takes the lemma of a user-specified term and the words generated from the word-level decoder to output a correct inflected form of the lemma. We evaluate our approach with a Japanese-to-English translation task in the scientific writing domain and and show our model can incorporate specified terms in a correct form more successfully than other comparable models.

## introduction over the last several years, neural machine translation (nmt) has pushed the quality of machine translation to near-human performance @xcite . however, due to its end-to-end nature, this comes with the cost of losing a certain degree of control over the produced translation, which once was explicitly modeled, for example, in the form of phrase table @xcite in statistical machine translation (smt). in practice, users often want to specify how certain words are translated in order to ensure the consistency of document-level translation or to guarantee the model to produce the correct translation for words that may be underrepresented in the training corpus such as proper nouns, technical terms, or novel words. given this motivation, a line of previous research has investigated placeholder translation @xcite . with a source sentence where certain words are replaced with a special placeholder token, the model produces a translation with the special placeholder token in an appropriate position, and then that placeholder token is replaced with a pre-specified term in a post-processing step. although this approach ensures that certain words appear in the translation, one limitation is that the user must specify the term that fits in the context surrounding the placeholder token, or specifically, the term should be properly inflected according to the syntactic structure of specified translation: 管理 → controlling source: フローセンサーの原理は浮遊式流量計のテーパー管内フロートの位置を差 動トランスで検出し,これの電圧制御により流量を[verb]する。 reference: the sensor controls the flow rate by detecting the position of the float in the tepered tube with a a differential transformer and @xcite it with the obtained voltage. system output: the principle of the flow sensor is that the position of the float in the taper tube of the floating flowmeter is detected by the differential transformer, and the flow rate is @xcite by this voltage control. the produced translation. to illustrate the problem, we show an actual output from a normal placeholder translation model in japanese to english translation in the system is supposed to translate the word 管理 into controlling as in the reference, but the output has a different grammatical construction and thus the progressive form controlling is invalid in this context; instead, controlled should be injected in the placeholder. the appropriate word form is difficult to predict, especially in translation between grammatically distant languages, such as japanese and english. as manually correcting the inflection in post-editing significantly hurts the convenience of placeholder translation, we need a way to automatically handle inflection. one possible approach to this problem is the code-switching methods, in which certain words in the source sentence are replaced with the specific target words, and the model is encouraged to include those specific words in the translation. this approach is flexible in that the model can inflect the specified words according to the context @xcite , but less faithful to the lexical constraints, often ignoring the specified terms ( §5). to address this problem, we propose a model that automatically inflects a pre-specified term according to the context of the produced translation. we extend the sequence-to-sequence encoder and decoder with an additional character-level decoder that predicts the inflected form of the pre-specified term. our approach combines the advantages of both the placeholder and the code-switching methods: the faithfulness to lexical constraints and the flexibility of dynamically deciding the word form in the output. we test our approach with a japanese-to-english translation task in the scientific-writing domain @xcite , where the translation of technical terms poses a challenge to a vanilla nmt system. the results show that the proposed method can include the specified term in the appropriately inflected form in the translation with higher accuracy than a comparable code-switching method. we also perform a careful error analysis to understand the weaknesses of each system and suggest directions for future work. ## approach the proposed model builds upon a sequence-to-sequence (seq2seq) model with an attention mechanism. specifically, we use the transformer model @xcite . in the normal placeholder translation, the model is trained to generate placeholder tokens [placeholder] when the source sentence includes them. then the placeholder tokens are replaced with user-provided terms in post-processing. we extend the model to be able to handle inflection. specifically, we consider the scenario where lemmas are provided as a specified term. on top of the (sub)word-level decoder, we stack a character-level decoder to generate the content of the placeholder token. the characterlevel decoder has to predict the correct inflected form of the specified lemma in the surrounding context. specifically, given the target tokens {w 1 , ..., w t } that contain a placeholder token and the specified lemma that consists of l characters c @xmath0, .., c l }, the character decoder generates the inflected form c inf @xmath1, .., c ′ l ′ }. we model the generation process with a decoder with attention mechanism (fig. 1 ). we first summarize the contextual information on the placeholder token by a context encoder. specifically, we feed the embeddings of the target tokens {w 1 , ..., w t } into another transformer encoder to contextualize the placeholder token (eq. 1). then, the contextualized representation of the placeholder token h p and the character embeddings of the specified lemma {c 1 , .., c l } with positional encoding @xcite are concatenated to form key-value vectors for decoder attention (eq. 2). finally, the key-value vectors are passed to the characterlevel transformer decoder and it generates the inflected form ## experimental setups we evaluate the proposed model with several baselines to show how well the model can produce the appropriately inflected form of a given lemma. ## conclusion and future work in this study, we point out that the traditional placeholder translation method embeds the specified term into the generated translation without considering the context of the placeholder token, which potentially leads to grammatically incorrect translations. to address this shortcoming, we proposed a flexible placeholder translation model that handles inflection when the specified term is given in the form of a lemma. in the experiment of the japanese-to-english translation task, we showed that the proposed model can inflect user-specified terms more accurately than the code-switching method. future work includes testing the proposed method on morphologically-rich languages or extending the model to handle more than one placeholder in a sentence. also, the proposed model still has room for improvement to learn inflection. it is possible that we can improve the model by exploiting monolingual corpora in the target language to provide additional training signals for learning the correct inflection in context.