Contextual Representation Learning beyond Masked Language Modeling (2022)

Abstract: Currently, masked language modeling (e.g., BERT) is the prime choice to learn contextualized representations. Due to the pervasiveness, it naturally raises an interesting question: how do masked language models (MLMs) learn contextual representations? In this work, we analyze the learning dynamics of MLMs and find that it adopts sampled embeddings as anchors to estimate and inject contextual semantics to representations, which limits the efficiency and effectiveness of MLMs. To address these problems, we propose TACO, a simple yet effective representation learning approach to directly model global semantics. To be specific, TACO extracts and aligns contextual semantics hidden in contextualized representations to encourage models to attend global semantics when generating contextualized representations. Experiments on the GLUE benchmark show that TACO achieves up to 5x speedup and up to 1.2 points average improvement over MLM.

## introduction in the age of deep learning, the basis of representation learning is to learn distributional semantics. the target of distributional semantics can be summed up in the so-called distributional hypothesis @xcite : linguistic items with similar distributions have similar meanings. to model similar meanings, traditional representation approaches @xcite (e.g., word2vec) model distributional semantics by defining tokens using context-independent (ci) dense vectors, i.e., word embeddings, and directly aligning the representations of tokens in the same context. nowadays, pre-trained language models (ptms) @xcite @xcite expand static embeddings into contextualized representations where each token has two kinds of representations: contextindependent embedding, and context-dependent enc. 𝑥 ! (#) 𝑥 % (#) 𝑥 & (#) -= @xmath0, ) ( , ) positive negative ⇒ ⇒ enc. ## proposed approach: taco to address the challenges of mlm, we propose a new method taco to combine global anchors and local anchors. we first introduce tc, a tokenalignment contrastive loss which explicitly models global semantics in section 3.1, and combine tc with mlm to get the overall objective for training our taco model in section 3.2. ## conclusion in this paper, we propose a simple yet effective objective to learn contextualized representation. tak-ing mlm as an example, we investigate whether and how current language model pre-training objectives learn contextualized representation. we find that the mlm objective mainly focuses on local anchors to align contextualized representations, which harms global dependencies modeling due to an "embedding bias" problem. motivated by these problems, we propose taco to directly model global semantics. it can be easily combined with existing lm objectives. by combining local and global anchors, taco achieves up to 5× speedups and up to 1.2 improvements on glue score. this demonstrates the potential of taco to serve as a plug-and-play approach to improve contextualized representation learning. transformers: state-of-the-art natural language processing. in proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, emnlp 2020 -demos, online, november 16-20, 2020, pages 38-45. association for computational linguistics. zhilin yang, zihang dai, yiming yang, jaime g. carbonell, ruslan salakhutdinov, and quoc v. le. 2019. xlnet: generalized autoregressive pretraining for language understanding. in advances in neural information processing systems 32: annual conference on neural information processing systems 2019, neurips 2019, december 8-14, 2019, vancouver, bc, canada, pages 5754-5764. yukun zhu, ryan kiros, rich zemel, ruslan salakhutdinov, raquel urtasun, antonio torralba, and sanja fidler. 2015. aligning books and movies: towards story-like visual explanations by watching movies and reading books. in proceedings of the ieee international conference on computer vision, pages 19-27.