Non-Autoregressive Translation by Learning Target Categorical Codes (2021)

Abstract: Non-autoregressive Transformer is a promising text generation model. However, current non-autoregressive models still fall behind their autoregressive counterparts in translation quality. We attribute this accuracy gap to the lack of dependency modeling among decoder inputs. In this paper, we propose CNAT, which learns implicitly categorical codes as latent variables into the non-autoregressive decoding. The interaction among these categorical codes remedies the missing dependencies and improves the model capacity. Experiment results show that our model achieves comparable or better performance in machine translation tasks than several strong baselines.

## introduction non-autoregressive transformer (nat, @xcite @xcite is a promising text generation model for machine translation. it introduces the conditional independent assumption among the target language outputs and simultaneously generates the whole sentence, bringing in a remarkable efficiency improvement (more than 10× speed-up) versus the autoregressive model. however, the nat models still lay behind the autoregressive models in terms of bleu @xcite for machine translation. we attribute the low-quality of nat models to the lack of dependencies modeling for the target outputs, making it harder to model the generation of the target side translation. a promising way is to model the dependencies of the target language by the latent variables. a line of research works @xcite @xcite introduce latent variable modeling to the non-autoregressive transformer and improves translation quality. the latent variables could be regarded as the springboard to bridge the modeling gap, introducing more informative decoder inputs than the previ-ously copied inputs. more specifically, the latentvariable based model first predicts a latent variable sequence conditioned on the source representation, where each variable represents a chunk of words. the model then simultaneously could generate all the target tokens conditioning on the latent sequence and the source representation since the target dependencies have been modeled into the latent sequence. however, due to the modeling complexity of the chunks, the above approaches always rely on a large number (more than 2 15 , @xcite of latent codes for discrete latent spaces, which may hurt the translation efficiencythe essential goal of non-autoregressive decoding. @xcite introduce syntactic labels as a proxy to the learned discrete latent space and improve the nats' performance. the syntactic label greatly reduces the search space of latent codes, leading to a better performance in both quality and speed. however, it needs an external syntactic parser to produce the reference syntactic tree, which may only be effective in limited scenarios. thus, it is still challenging to model the dependency between latent variables for non-autoregressive decoding efficiently. in this paper, we propose to learn a set of latent codes that can act like the syntactic label, which is learned without using the explicit syntactic trees. to learn these codes in an unsupervised way, we use each latent code to represent a fuzzy target category instead of a chunk as the previous research @xcite . more specifically, we first employ vector quantization @xcite to discretize the target language to the latent space with a smaller number (less than 128) of latent variables, which can serve as the fuzzy word-class information each target language word. we then model the latent variables with conditional random fields (crf, @xcite . to avoid the mismatch of the training and inference for latent variable modeling, we propose using a gated neural network to form the decoder inputs. equipping it with scheduled sampling @xcite , the model works more robustly. experiment results on wmt14 and iwslt14 show that cnat achieves the new state-of-theart performance without knowledge distillation. with the sequence-level knowledge distillation and reranking techniques, the cnat is comparable to the current state-of-the-art iterative-based model while keeping a competitive decoding speedup. ## background neural machine translation (nmt) is formulated as a conditional probability model p(y|x), which models a sentence @xmath0, y 2 , • • • , y m } in the target language given the input @xmath1, x 2 , • • • , x n } from the source language. @xcite proposes non-autoregressive transformer (nat) for machine translation, breaking the dependency among target tokens, thus achieving simultaneous decoding for all tokens. for a source sentence, a non-autoregressive decoder factorizes the probability of its target sentence as: ## approach in this section, we present our proposed cnat, an extension to the transformer incorporated with non-autoregressive decoding for target tokens and autoregressive decoding for latent sequences. in brief, cnat follows the architecture of latent transformer @xcite , except for the latent variable modeling (in § 3.1 and § 3.2) and inputs initialization (in § 3.3). ## experiments datasets. we conduct the experiments on the most widely used machine translation benchmarks: wmt14 english-german (wmt14 en-de, 4.5m pairs) foot_0 and iwslt14 german-english (iwslt14, 160k pairs) foot_1 . the datasets are processed with the moses script @xcite , and the words are segmented into subword units using byte-pair encoding @xcite . we use the shared subword embeddings between the source language and target language for the wmt datasets and the separated subword embeddings for the iwslt14 dataset. ## conclusion we propose cnat, which implicitly models the categorical codes of the target language, narrowing the performance gap between the nonautoregressive decoding and autoregressive decoding. specifically, cnat builds upon the latent transformer and models the target-side categorical information with vector quantization and conditional random fields (crf) model. we further employ a gated neural network to form the decoder inputs. equipped with the scheduled sampling, cnat works more robust. as a result, the cnat achieves a significant improvement and moves closer to the performance of the transformer on machine translation. bingzhen wei, mingxuan wang, hao zhou, junyang lin, and xu sun. 2019. imitation learning for nonautoregressive neural machine translation. in proceedings of the 57th annual meeting of the association for computational linguistics, pages 1304-1312, florence, italy. association for computational linguistics. xuewen yang, yingru liu, dongliang xie, xin wang, and niranjan balasubramanian. 2019. latent partof-speech sequences for neural machine translation. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 780-790, hong kong, china. association for computational linguistics. a non-indo-european translation dataset. we apply the cnat to the non-indo-european translation tasks on the ldc chinese-english 5 (denote as ldc zh-en, 1.30m sentence pairs) and mt02 test set of nist zh-en dataset. we use nlpirictclas 6 and moses tokenizer for chinese and english tokenization, respectively. model bleu transformer 28.05 nat 12.31 cnat 22.16 for each latent variable, we list the top 3 frequent pos tags and their corresponding percentages.