A Study of Syntactic Multi-Modality in Non-Autoregressive Machine Translation (2022)

Abstract: It is difficult for non-autoregressive translation (NAT) models to capture the multi-modal distribution of target translations due to their conditional independence assumption, which is known as the “multi-modality problem”, including the lexical multi-modality and the syntactic multi-modality. While the first one has been well studied, the syntactic multi-modality brings severe challenges to the standard cross entropy (XE) loss in NAT and is understudied. In this paper, we conduct a systematic study on the syntactic multi-modality problem. Specifically, we decompose it into short- and long-range syntactic multi-modalities and evaluate several recent NAT algorithms with advanced loss functions on both carefully designed synthesized datasets and real datasets. We find that the Connectionist Temporal Classification (CTC) loss and the Order-Agnostic Cross Entropy (OAXE) loss can better handle short- and long-range syntactic multi-modalities respectively. Furthermore, we take the best of both and design a new loss function to better handle the complicated syntactic multi-modality in real-world datasets. To facilitate practical usage, we provide a guide to using different loss functions for different kinds of syntactic multi-modality.

## introduction traditional neural machine translation (nmt) models predict each target token conditioned on previous generated tokens in an autoregressive way @xcite , resulting in high latency in inference. non-autoregressive translation (nat) models generate all the target tokens in parallel @xcite , significantly reducing inference latency. a disadvantage of nat is that it suffers from the multi-modality problem @xcite when a source sentence corresponds to multiple correct translations @xcite . there are two types of multi-modalities: the lexical one and the syntactic one. the former one has been adequately studied @xcite @xcite , while the latter one brings severe challenges to the widely used cross entropy (xe) loss in nat. with standard xe loss, the generated tokens are required to be strictly aligned with ground truth tokens in the same positions, which fails to provide positive feedback for correctly predicted words on different positions as shown in fig. 1a . therefore, advanced loss functions are introduced to provide better feedback for nat training: connectionist temporal classification (ctc) loss @xcite considers all possible monotonic alignments between a generated sequence and the ground truth; aligned cross-entropy (axe) loss @xcite selects the best monotonic alignment; and order-agnostic cross entropy (oaxe) loss @xcite calculates the xe loss with the best alignment based on maximum bipartite matching algorithm. even if with those advanced loss functions, we find they do not perform consistently across datasets and languages. in addition, diverse grammar rules in natural language @xcite implies the existence of different kinds of syntactic multi-modality. inspired by @xcite ; @xcite ; @xcite , we categorize the syntactic multi-modality into two sub types: the long-range and short-range ones. the long-range multi-modality is mainly caused by long-range word order diversity (e.g., an adverbial of place may appear at the beginning or the end of a sentence). the short-range multi-modality is mainly caused by short-range word order diversity (e.g., an adverb may appear either in front of or behind the corresponding verb) and optional words (e.g., in some languages, determiners and prepositions may be optional @xcite ). based on the above categorization of syntactic multi-modality, we fur-ther ask two research questions: (1) which kinds of syntactic multi-modality do these loss functions excel at respectively? (2) how to better address this problem by taking advantage of different loss functions? in this paper, we conduct a systematic study to answer these questions: ## analyses on synthesized datasets to make fine-grained analyses on the syntactic multi-modality problem, we first categorize it into long-range and short-range types, where the longrange one is mainly caused by long-range word order diversity, and the short-range one is mainly caused by short-range word order diversity and opin the first iteration, "sen" consists of ("np", "vp") as the solid lines. in the second iteration, "np" consists of ("dt", "rb", "jj", "n") and "vp" consists of ("v", "np", "rb") as the dash lines. in the third iteration, "np" consists of ("dt", "jj", "n") as the dot-and-dash lines. ## analyses on real datasets though carefully designed, the synthesized sentence pairs consisting of numbers are still different from the real sentence pairs. therefore, in this section, we validate the findings in section 3 based on real datasets. considering that different types of syntactic multi-modality are highly coupled in the real corpus, we conduct experiments on carefully selected sub-datasets from a corpus, to approximately decompose the syntactic multi-modality. in the following, we first show the details of the approach to decompose the syntactic multi-modality, and then provide the analytical results based on the real datasets. analytical approach. in order to decompose the long-range and short-range types of syntactic multi-modality, we select sentences that only contain subject and verb phrases from a corpus, and divide them into two sub-datasets according to the relative order of subject and verb (i.e., subject first that denoted as "sv", or verb first that denoted as "vs"). meanwhile, we only consider the declarative sentence pairs in the corpus to eliminate the word order difference caused by mood. following this method, the long-range multi-modality is eliminated in each sub-dataset (i.e., "sv" and "vs"), which can be used to evaluate the effect of short-range multi-modality. to analyse the longrange multi-modality, we can adjust the degree of long-range word order diversity by sampling data from the two sub-datasets with varied ratios, while roughly keeping the degree of short-range word or- v" among the sentences with both "rb v" and "v rb" orders are shown in column "rb v". the percentage of sentences with "jj n" among the sentences with both "jj n" and "n jj" orders are shown in column "jj n". analytical results. we keep the total number of sentence pairs in the training set as 0.2m (i.e., the number of russian sentences in the "vs" subdataset), and change the ratio of sentence pairs sampled from two sub-datasets (i.e., "sv" and "vs"). the results are shown in in order to demonstrate whether we have decomposed the long-and short-range syntactic multimodalities, we verify whether the degree of shortrange multi-modality remains almost the same when varying the degree of long-range multimodality. we evaluate the short-range syntactic diversity based on the relative order between: 1) adverb and verb ("rb v"); 2) adjective and noun ("jj n"). as shown in ## better solving the syntactic multi-modality problem as shown in previous sections, the ctc and the oaxe loss functions are good at dealing with shortand long-range syntactic multi-modalities respectively. while in real-world corpora, different types of multi-modalities usually occur together and vary in different languages. accordingly, it may be better to use different loss functions for different languages. in this section, we first introduce a new loss function named combined ctc and oaxe (coco), which takes advantage of both ctc and oaxe to better handle the long-range and shortrange syntactic multi-modalities simultaneously, and then provide a guideline on how to choose the appropriate loss function for different scenarios. ## conclusion in this paper, we conduct a systematic study on the syntactic multi-modality problem in nonautoregressive machine translation. we first categorize this problem into long-range and short-range types and study the effectiveness of different loss functions on each type. considering the different types are usually entangled in real-world datasets, we design and construct synthesized datasets to control the degree of one type of multi-modality without changing another for analyses. we find that ctc loss is good at short-range syntactic multimodality while oaxe loss is better at the longrange one. these findings are further verified on real-world datasets with our designed analytical approach. based on these analyses, we propose a coco loss that can better handle the complicated syntactic multi-modality in real-world datasets, and a practical guide to use different loss functions for different kinds of syntactic multi-modality: coco loss is preferred when the word order of target language is relatively flexible while ctc loss is preferred when target language is with strict word order. our study in this paper can facilitate better understanding of the multi-modality problem and provide insights to better solve this problem in non-autoregressive translation. besides, there still remain some open problems that need future investigation. for example, we generally consider longrange and short-range types for syntactic multimodality, while there may be more fine-gained categorizations on the syntactic multi-modality due to the complex syntax of natural language.