Pre-training Universal Language Representation (2021)

Abstract: Despite the well-developed cut-edge representation learning for language, most language representation models usually focus on specific levels of linguistic units. This work introduces universal language representation learning, i.e., embeddings of different levels of linguistic units or text with quite diverse lengths in a uniform vector space. We propose the training objective MiSAD that utilizes meaningful n-grams extracted from large unlabeled corpus by a simple but effective algorithm for pre-trained language models. Then we empirically verify that well designed pre-training scheme may effectively yield universal language representation, which will bring great convenience when handling multiple layers of linguistic objects in a unified way. Especially, our model achieves the highest accuracy on analogy tasks in different language levels and significantly improves the performance on downstream tasks in the GLUE benchmark and a question answering dataset.

## introduction in this paper, we propose universal language representation (ulr) that uniformly embeds linguistic units in different hierarchies in the same vector space. a universal language representation model encodes linguistic units such as words, phrases or sentences into fixed-sized vectors and handles multiple layers of linguistic objects in a unified way. ulr learning may offer a great convenience when confronted with sequences of different lengths, especially in tasks such as natural language understanding (nlu) and question answering (qa), hence it is of great importance in both scientific research and industrial applications. as is well known, embedding representation for a certain linguistic unit (i.e., word) enables linguistics-meaningful arithmetic calculation among different vectors, also known as word analogy @xcite . for example: in fact, manipulating embeddings in the vector space reveals syntactic and semantic relations between the original symbol sequences and this feature is indeed useful in true applications. for example, "london is the capital of england" can be formulized as: then given two documents one of which contains "england" and "capital", the other contains "london", we consider them relevant. while a ulr model may generalize such good analogy features onto free text with all language levels involved together. for example, eat an onion : vegetable :: eat a pear : fruit. ulr has practical values in dialogue systems, by which human-computer communication will go far beyond executing instructions. one of the main challenges of dialogue systems is dialogue state tracking (dst). it can be formulated as a semantic parsing task @xcite , namely, converting natural language utterances with any length into unified representations. thus this is essentially a problem that can be conveniently solved by mapping sequences with similar semantic meanings into similar representations in the same vector space according to a ulr model. another use of ulr is in the frequently asked questions (faq) retrieval task, where the goal is to answer a user's question by retrieving question paraphrases that already have an answer from the database. such task can be accurately done by only manipulating vectors such as calculating and ranking vector distance (i.e., cosine similarity). the core is to embed sequences of different lengths in the same vector space. then a ulr model retrieves the correct question-answer pair for the user query according to vector distance. in this paper, we propose a universal language representation learning method that generates fixedsized vectors for sequences of different lengths based on pre-trained language models @xcite @xcite . we first introduce an efficient approach to extract and prune meaningful n-grams from unlabeled corpus. then we present a new pre-training objective, minimizing symbol-vector algorithmic difference (misad), that explicitly applies a penalty over different levels of linguistic units if their representations tend not to be in the same vector space. to investigate our model's ability of capturing different levels of language information, we introduce an original universal analogy task derived from google's word analogy dataset, where our model significantly improves the performance of previous pre-trained language models. evaluation on a wide range of downstream tasks also demonstrates the effectiveness of our ulr model. overall, our ulr-bert reaches the highest average accuracy on the universal analogy dataset and obtains 1.1% gain over google bert on the glue benchmark. extensive experimental results on a question answering task verifies that our model can be easily applied to real-world applications in an extremely convenient way. ## related work previous language representation learning methods such as word2vec @xcite , glove @xcite , laser @xcite , infersent @xcite and use @xcite focus on specific granular linguistic units, e.g., words or sentences. later proposed elmo @xcite , openai gpt @xcite , bert @xcite and xlnet @xcite learns contextualized representation for each input token. although such pre-trained language models (prlms) more or less are capable of offering universal language representation through their general-purpose training objectives, all the prlms devote into the contex-tualized representations from a generic text background and pay little attention on our concerned universal language presentation. as a typical prlm, bert is trained on a large amount of unlabeled data including two training targets: masked language model (mlm), and next sentence prediction (nsp). albert @xcite is trained with sentence-order prediction (sop) as a replacement of nsp. structbert @xcite combines nsp and sop to learn inter-sentence structural information. nevertheless, roberta @xcite and spanbert @xcite show that single-sequence training is better than the sentence-pair scenario. besides, bert-wwm @xcite , structbert @xcite , spanbert @xcite ) perform mlm on higher linguistic levels, augmenting the mlm objective by masking whole words, trigrams or spans, respectively. electra @xcite further improves pre-training through a generator and discriminator architecture. the aforementioned models may seemingly handle different sized input sequences, but all of them focus on sentence-level specific representation still for each word, which may cause unsatisfactory performance in real-world situations. there are a series of downstream nlp tasks especially on question answering which may be conveniently and effectively solved through ulr like solution. actually, though in different forms, these tasks more and more tend to be solved by our suggested ulr model, including dialogue utterance regularization @xcite , question paraphrasing @xcite , measuring qa similarities in faq tasks @xcite . ## model as pre-trained contextualized language models show their powerfulness in generic language representation for various downstream nlp tasks, we present a bert-style ulr model that is especially designed to effectively learn universal, fixed-sized representations for input sequences of any granularity, i.e., words, phrases, and sentences. our proposed pre-training method is furthermore strengthened in three-fold. first, we extract a large number of meaningful n-grams from monolingual corpus based on point-wise mutual information to leverage the multi-granular structural information. second, inspired by word and phrase representation and their compositionality, we introduce a novel pre-training objective that directly models the input sequences and the extracted n-grams through manipulating their representations. finally, we implement a normalized score for each n-gram to guide their sampling for training. ## implementation of ulr pre-training this section introduces our ulr pre-training details. as for the pre-training corpus, we download the english wikipedia corpus foot_0 and pre-process with process wiki.py foot_1 , which extracts text from xml files. when processing paragraphs from wikipedia, we find that a large number of entities are annotated with special marks, which may be useful for our task. therefore, we identify all the entities and treat them as high-quality n-grams. then, we remove punctuation marks and characters in other languages based on regular expressions, and finally get a corpus of 2,266m words. as for n-gram pruning, pmi scores of all ngrams with a maximum length of @xmath0. we manually evaluate the extracted n-grams and find more than 50% of the top 2000 n-grams contain 2 ∼ 3 words, and only less than 3% n-grams are longer than 4. although a larger n-gram vocabulary can cover longer ngrams, it will cause too many meaningless n-grams at the same time. therefore, we empirically retain the top 3000 n-grams for each document. finally, we randomly sample 10m sentences from the entire corpus to reduce training time. during pre-training, bert packs sentence pairs into a single sequence and use the special [cls] token as sentence-pair representation. however, our misad training objective requires singlesentence inputs. thus in our experiments, each input is an n-ngram or a single sequence with a maximum length of 128. special tokens [cls] and [sep] are added at the front and end of each input, respectively. instead of training from scratch, we initialize our model with the officially released checkpoints of bert @xcite , al-bert @xcite and electra @xcite . we use adam optimizer (kingma and ba, 2017) with initial learning rate of 5e-5 and linear warmup over the first 10% of the training steps. batch size is 64 and dropout rate is 0.1. each model is trained for one epoch over 10m training examples on four nvidia tesla p40 gpus. ## ablation study in this section, we explore to what extent does our model benefit from the misad objective and sampling strategy, and further confirm that our pretraining procedure improves the model's ability of encoding variable-length sequences. ## conclusion this work formally introduces universal language representation learning to enable unified vector operations among different language hierarchies. for such a purpose, we propose three highlighted ulr learning enhancement, including the newly designed training objective, minimizing symbolvector algorithmic difference (misad). in detailed model implementation, we extend bert's pre-training objective to a more general level, which leverages information from sequences of different lengths in a comprehensive way. in addition, we provide a universal analogy dataset as a task-independent evaluation benchmark. overall experimental results show that our proposed ulr model is generally effective in a broad range of nlp tasks including natural language question answering and so on.