Isotropy-Enhanced Conditional Masked Language Models (2023)

Abstract: Non-autoregressive models have been widely used for various text generation tasks to accelerate the inference process but at the cost of generation quality to some extent. To achieve a good balance between inference speedup and generation quality, iterative NAR models like CMLM and Disco are proposed. Researchers have made much follow-up progress based on them, and some recent iterative models can achieve very promising performance while maintaining significant speedup. In this paper, we give more insights into iterative NAR models by exploring the anisotropic problem, i.e., the representations of distinct predicted target tokens are similar and indiscriminative. Upon the confirmation of the anisotropic problem in iterative NAR models, we first analyze the effectiveness of the contrastive learning method and further propose the Look Neighbors strategy to enhance the learning of token representations during training. Experiments on 4 WMT datasets show that our methods consistently improve the performance as well as alleviate the anisotropic problem of the conditional masked language model, even outperforming the current SoTA result on WMT14 EN \rightarrow DE.

## introduction transformer-based models @xcite with auto-regressive (ar) decoding paradigm have successfully applied to various text generation tasks, such as machine translation @xcite ), text summarization @xcite , dialogue systems @xcite , with exciting performance being achieved. however, the low inference efficiency prohibits their usage (especially with current super-large transformer models as backbones) since the ar paradigm predicts the tokens one by one in left-to-right order. to speed up model inference, many non-autoregressive methods @xcite @xcite @xcite explore predicting target tokens in parallel in recent years. as a result of parallel decoding, they inevitably neglect the internal dependency of the target tokens, causing the generation quality falls behind their ar counterparts to some extent. to achieve a better trade-off between inference speedup and generation quality, many iterative nar models @xcite @xcite have been proposed recently. they adopt multiple decoding iterations to generate final results, where the intermediate result generated from the last iteration can provide useful target information and then be fed into the model for refinements in the next iteration. excitingly, iterative decoding brings significant performance improvements for nar models and even outperforms the vanilla autoregressive transformer. among them, one of the most competitive and widely-used models is cmlm @xcite , which adopts uniform masking during training and a mask-predict strategy for decoding. based on cmlm, many advanced methods, such as training strategy @xcite , criterion @xcite , and inference strategy @xcite , have been explored and achieved better performance. despite significant progress, some critical problems of nar methods still remain challenging, e.g., repetition and reliance on distillation. to disclose more insights into the enhancement of the iterative nar models, this paper explores from a different perspective, i.e., the anisotropic problem, which has been revealed recently in ar models @xcite . this can cause the degeneration of ar neural language models with the anisotropic distribution of token representations d yd ]x w fd wh yd ox fu xu l d yd]xw fdwhyd oxfuxul (a) iter.1 on @xmath0. d yd ]x w fd wh yd ox fu xu l d yd]xw fdwhyd oxfuxul (b) iter.10 on @xmath1. ( v j le w hl q h 0 hq j h 3 dw lh q wh q (v jlew hlqh 0hqjh 3dwlhqwhq (c) iter.1 on @xmath2. ( v j le w hl q h 0 hq j h 3 dw lh q wh q (v jlew hlqh 0hqjh 3dwlhqwhq (d) iter.10 on @xmath3. (taken from the output layer of the model), in which the high similarity of different tokens can mislead the model to generate repetitive tokens at different steps. naturally, we wonder whether the repetition problem of nar models is also closely associated with the anisotropic representations. we take the typical iterative nar model cmlm @xcite , and present the results in figure 1 , the same phenomenon in ar model @xcite also holds for iterative nar models, i.e., the token similarity is close to each other (figure 1 (a) and (c)). besides, decoding with multiple iterations (figure 1 (b) and (d)) do not alleviate the dense representation of token similarity for a large margin. therefore, the anisotropic problem of the iterative nar models is worth more exploration. to alleviate the anisotropic problem of cmlm, we first test the effectiveness of the existing method based on contrastive learning @xcite . however, we found that directly adopting the existing method does not obtain consistent improvements on various datasets. then, we analyze the potential reason and attribute it to the failure to learn the token representation well during training, which is a widely recognized problem for all nat models. the reason why "failure to learn representations" is not conducive to improving performance will be discussed in section 2.3. motivated by the particularly dense representation of adjacent tokens, we propose a simple yet effective strategy named look neighbors to enhance the dependency of adjacent tokens and auxiliarily learn better repre-sentations. due to learning better representations, look neighbors can supplement contrastive learning well during the training of cmlm. we conduct comprehensive experiments to evaluate our methods. results on wmt14 en↔de and wmt16 en↔ro datasets all demonstrate that our methods outperform the vanilla cmlm model, achieve comparable performance compared with strong iterative nar baselines. besides, the explorations on the token representation of cmlm also give a new insight into the iterative nar models, indicating the anisotropic problem, which has been well learned in ar models recently, also affects the performance and further developments for nar models. ## look neighbors in this section, we introduce our enhancement method, look neighbors, which improves the abil-ity to learn token representations during training. as mentioned in section 2.3, the failure to learn the token representations well for cmlm prevents the superior effects of adopting contrastive learning. the instability of the representations in nar models has also been explored in recent nar papers @xcite . they propose the regularization method to normalize the token representations. unlike these regularization methods, we try the enhance the learning of token representations. specifically, we incorporate the representations of other tokens to make predictions during training, thus improving the ability to learn token representations. as shown in figure 1 , the cosine similarity is denser in adjacent tokens, indicating that cmlm has more difficulty in understanding and distinguishing the adjacent representations of target tokens. as a result, we mainly incorporated the representations of adjacent tokens rather than all other tokens. the most similar exploration is explored in lava-nat @xcite . they propose a look-around decoding strategy to enhance the inference schedule but without enhancing the training process. we propose the look neighbors, where the model integrates a neighbor-aware self-attention into every decoder layer and concatenates the representations from the last decoder layer of the left and right neighbors to assist in predicting the target tokens. firstly, we formally introduce our neighboraware self-attention as follows: ## conclusion in this paper, we first verify that the strong and representative iterative nar model, cmlm, suffers from the anisotropic problem by analyzing the changes in the cosine similarity of token representations. to alleviate the anisotropic problem, we introduce the contrastive learning method to make the token representations discriminative and further propose a look neighbors strategy to enhance token representation learning during training. extensive experiments on several widely-used translation datasets indicate that our proposed methods have a great effect on alleviating the anisotropic problem and can consistently improve the performance of iterative nar models. we hope that our explorations can give new insights into the iterative nar models and promote their development.