Fast and Accurate Deep Bidirectional Language Representations for Unsupervised Learning (2020)

Abstract: Even though BERT has achieved successful performance improvements in various supervised learning tasks, BERT is still limited by repetitive inferences on unsupervised tasks for the computation of contextual language representations. To resolve this limitation, we propose a novel deep bidirectional language model called a Transformer-based Text Autoencoder (T-TA). The T-TA computes contextual language representations without repetition and displays the benefits of a deep bidirectional architecture, such as that of BERT. In computation time experiments in a CPU environment, the proposed T-TA performs over six times faster than the BERT-like model on a reranking task and twelve times faster on a semantic similarity task. Furthermore, the T-TA shows competitive or even better accuracies than those of BERT on the above tasks. Code is available at https://github.com/joongbo/tta .

## introduction a language model is an essential component of many natural language processing (nlp) applications ranging from automatic speech recognition (asr) @xcite to neural machine translation (nmt) @xcite @xcite . recently, the bidirectional encoder representations from transformers (bert) @xcite and its variations have led to significant improvements in learning natural language representation and have achieved state-of-the-art performances on various downstream tasks such as the general language understanding evaluation (glue) benchmark @xcite and question answering @xcite . bert continues to succeed in various unsupervised tasks, such as the n -best list reranking for asr and nmt @xcite , con-firming that deep bidirectional language models are useful in unsupervised applications as well. however, concerning its applications to unsupervised learning tasks, bert is significantly inefficient at computing language representations at the inference stage @xcite . during training, bert adopts the masked language modeling (mlm) objective, which is to predict the original word of the explicitly masked word from the input sequence. following the mlm objective, each contextual word representation should be computed by a two-step process: masking a word in the input and feeding the result to bert. during the inference stage, this process is repeated n times to obtain the representations of all the words within a text sequence @xcite @xcite , resulting in a computational complexity of o(n 3 ) foot_0 in terms of the number of words n. hence, it is necessary to reduce the computational complexity when applying the model to situations where the inference time is critical, e.g., mobile environments and real-time systems @xcite . considering this limitation of bert, we submit a new research question: "can we construct a deep bidirectional language model with a minimal inference time while maintaining the accuracy of bert?" in this paper, in response to the above question, we propose a novel bidirectional language model named the transformer-based text autoencoder (t-ta), which has a reduced computational complexity of o(n 2 ) when applying the model to unsupervised applications. the proposed model is trained with a new learning objective named language autoencoding (lae). the lae objective, which allows the target labels to be the same as the text input, is to predict every token in the input sequence simultaneously without merely copying the input to the output. to learn the proposed objective, we devise both a diagonal masking operation and an input isolation mechanism inside the t-ta based on the transformer encoder @xcite . these components enable the proposed t-ta to compute contextualized language representations at once while maintaining the benefits of the deep bidirectional architecture of bert. we conduct a series of experiments on two unsupervised tasks: n -best list reranking and unsupervised semantic textual similarity. first, by conducting runtime experiments in a cpu environment, we show that the proposed t-ta is 6.35 times faster than the bert-like model in the reranking task and 12.7 times faster in the unsupervised semantic textual similarity task. second, despite its faster inference time, the t-ta achieves competitive performances relative to bert on reranking tasks. furthermore, the t-ta outperforms bert by up to 8 points in pearson's r on unsupervised semantic textual similarity tasks. ## related works when referring to an autoencoder for language modeling, sequence-to-sequence learning approaches have been commonly used. these approaches encode a given sentence into a compressed vector representation, followed by a decoder that reconstructs the original sentence from the sentence-level representation @xcite @xcite . to the best of our knowledge, however, none of these approaches consider an autoencoder that encodes word-level representations (such as bert) without an autoregressive decoding process. many studies have been performed on neural network-based language models for word-level representations. distributed word representations were proposed and attracted considerable interest, as they were considered to be fundamental building blocks for nlp tasks @xcite @xcite . subsequently, researchers explored contextualized representations of text where each word has a different representation depending on the context @xcite . most recently, a transformer-based deep bidirectional model was proposed and applied to various supervised-learning tasks with remarkable success @xcite . for unsupervised tasks, researchers have adopted recently developed language-representation models and investigated their effectiveness; a typical example is the n -best list reranking for asr and nmt tasks. in particular, studies have integrated leftto-right and right-to-left language models @xcite @xcite to outperform conventional unidirectional language models @xcite in these tasks. furthermore, bert-based approaches have been explored and have achieved significant performance improvements on these tasks because bidirectional language models yield the pseudo-log-likelihood of a given sentence, and this score is useful in ranking the n-best hypotheses @xcite @xcite . another line of research involves reducing the computation time and memory consumption of bert. @xcite proposed parameterreduction techniques, factorized embedding parameterization and cross-layer parameter sharing and reported 18 times fewer parameters and a 1.7-fold increase in the training time. similarly, @xcite presented a method to pretrain a smaller model that can be fine-tuned for downstream tasks and achieved 1.4 times fewer parameters with a 1.6fold increase in the inference time. however, none of these studies developed methods that directly revise the bert architecture to reduce the computational complexity during the inference stage. ## language model baselines in a conventional language modeling task, the ith token x i is predicted using its preceding context @xmath0, . . . , x i-1 ]; throughout this paper, this objective is known as causal language modeling (clm) following @xcite . as shown in figure 1a , we can obtain (left-to-right) contextualized language representations h @xmath1, . . . , h c n ] after feeding the input sequence to the clm-trained language model only once, where recently, bert @xcite was designed to enable the full contextualization of language representations by using the mlm objective, in which some tokens from the input sequence are randomly masked; the objective is to predict the original tokens at the masked positions using only their context. as in figure 1b , we can obtain a contextualized representation of the i-th token h m @xmath2, where ## experiments to evaluate the proposed method, we conduct a series of experiments. we first evaluate the contextual language representations obtained from the t-ta on n -best list reranking tasks. we then apply our method to unsupervised semantic textual similarity (sts) tasks. the following sections will demonstrate that the proposed model is much faster than bert during the inference stage (section 5.2) while showing competitive or even better accuracies than those of bert on reranking tasks (section 5.3) and sts tasks (section 5.4). ## conclusion in this work, we propose a novel deep bidirectional language model, namely, the t-ta, to eliminate the computational overload of applying bert to unsupervised applications. experimental results on n -best list reranking and unsupervised sts tasks demonstrate that the proposed t-ta is significantly faster than the bert-like approach, and its encoding ability is competitive with (or even better than) that of bert. shows the average runtimes of the unilm and the t-ta for the number of words in a sentence. since we use subword tokens, the number of words n w and the number of tokens n can be different (n @xmath3.