Neural Machine Translation with Synchronous Latent Phrase Structure (2021)

Abstract: It is reported that grammatical information is useful for machine translation (MT) task. However, the annotation of grammatical information requires the highly human resources. Furthermore, it is not trivial to adapt grammatical information to MT since grammatical annotation usually adapts tokenization standards which might not be suitable to capture the relation of two languages, and the use of sub-word tokenization, e.g., Byte-Pair-Encoding, to alleviate out-of-vocabulary problem might not be compatible with those annotations. In this work, we propose two methods to explicitly incorporate grammatical information without supervising annotation; first, latent phrase structure is induced in an unsupervised fashion from a multi-head attention mechanism; second, the induced phrase structures in encoder and decoder are synchronized so that they are compatible with each other using constraints during training. We demonstrate that our approach produces better performance and explainability in two tasks, translation and alignment tasks without extra resources. Although we could not obtain the high quality phrase structure in constituency parsing when evaluated monolingually, we find that the induced phrase structures enhance the explainability of translation through the synchronization constraint.

## introduction although machine translation (mt) has achieved improved performance using neural machine translation (nmt), the translation qualities for distant languages are still poor @xcite . as a way to tackle the problem, statistical mt (smt) incorporates synchronous grammar to achieve more linguistically accurate translations, in which complex structural relations between source and target languages are expressed using phrase structure @xcite . the synchronous grammar expresses the complex relationships between source and target languages and incorporates phrase structure to enable more linguistically accurate translation. a similar idea could be employed for nmt to achieve improved performance on those distant language pairs. however, grammatical information annotation demands high human resources. in addition, such grammatical annotation is done on word-level granularities, which might not be the best tokenization for mt tasks due by language mismatch or out-of-vocabulary problem, and often sub-word tokenization, e.g., byte-pair-encoding (bpe) @xcite , is employed to alleviate the problem. as a result, it is difficult to incorporate grammatical information into nmt that handle multiple languages simultaneously. recently, there have been researches on unsupervised learning of phrase structure without relying on human annotations. although these phrase structures learned in an unsupervised fashion are very close to the human annotation @xcite , there exists no model which incorporates phrase structures as latent information to improve the performance and explainability of translation. in this work, we introduce an approach to incorporate the phrase structure explicitly into transformer @xcite . the approach can split into two steps; first, latent phrase structures are induced in an unsupervised fashion for the source and target sides @xcite ; second, the two induced latent phrase structures are synchronously agreed with each other through an attention mechanism @xcite . experiments on german-english and japanese-english show that our synchronous latent structures have achieved better performance on translation and alignment tasks. we also show that the induced phrase structures and synchronous structures can enhance the explainability of translation through our detailed analysis in word alignment task. ## synchronous latent phrase structure in this section, we present the synchronous latent phrase structure. this proposed method is split into two steps. one is latent phrase structure induction (lpsi) and the other is synchronous constraint. figure 1 shows the flow of synchronizing japanese source and english target syntactic distances. ## experiments we train our proposed models using the training objective in equation 11 and evaluate them on three tasks: translation, constituency parsing, and word alignment. we implement models within the fairseq sequence modeling toolkit @xcite . ## conclusion this paper introduces the approach to improve the performance and explainability of mt. in the mt task, our model improves the quality of translation even through distant language pairs. in the alignment task, we demonstrate that synchronous constraint for syntactic distance can produce high precisional alignments to interpret mt hypothesis. currently, our approach induces the poor latent phrase structure constructed with the previous work. to achieve the more high performance and explainability of mt, we would like to investigate other syntactic structures and a translation model which can induce better latent phrase structure.