L ex F it: Lexical Fine-Tuning of Pretrained Language Models (2021)

Abstract: Transformer-based language models (LMs) pretrained on large text collections implicitly store a wealth of lexical semantic knowledge, but it is non-trivial to extract that knowledge effectively from their parameters. Inspired by prior work on semantic specialization of static word embedding (WE) models, we show that it is possible to expose and enrich lexical knowledge from the LMs, that is, to specialize them to serve as effective and universal “decontextualized” word encoders even when fed input words “in isolation” (i.e., without any context). Their transformation into such word encoders is achieved through a simple and efficient lexical fine-tuning procedure (termed LexFit) based on dual-encoder network structures. Further, we show that LexFit can yield effective word encoders even with limited lexical supervision and, via cross-lingual transfer, in different languages without any readily available external knowledge. Our evaluation over four established, structurally different lexical-level tasks in 8 languages indicates the superiority of LexFit-based WEs over standard static WEs (e.g., fastText) and WEs from vanilla LMs. Other extensive experiments and ablation studies further profile the LexFit framework, and indicate best practices and performance variations across LexFit variants, languages, and lexical tasks, also directly questioning the usefulness of traditional WE models in the era of large neural models.

## introduction probing large pretrained encoders like bert @xcite revealed that they contain a wealth of lexical knowledge @xcite . if type-level word vectors are extracted from bert with appropriate strategies, they can even outperform traditional word embeddings (wes) in some lexical tasks @xcite @xcite . however, both static and contextualized wes ultimately learn solely from the distributional word co-occurrence signal. this source of signal is known to lead to distortions in the induced representations by conflating meaning based on topical relatedness rather than authentic semantic similarity @xcite @xcite . this also creates a ripple effect on downstream applications, where model performance may suffer @xcite @xcite . our work takes inspiration from the methods to correct these distortions and complement the distributional signal with structured information, which were originally devised for static wes. in particular, the process known as semantic specialization (or retrofitting) injects information about lexical relations from databases like wordnet @xcite or the paraphrase database @xcite into wes. thus, it accentuates relationships of pure semantic similarity in the re-fined representations @xcite @xcite . our goal is to create representations that take advantage of both 1) the expressivity and lexical knowledge already stored in pretrained language models (lms) and 2) the precision of lexical finetuning. to this effect, we develop lexfit, a versatile lexical fine-tuning framework, illustrated in figure 1 , drawing a parallel with universal sentence encoders like sentencebert @xcite ). 1 our working hypothesis, extensively evaluated in this paper, is as follows: pretrained encoders store a wealth of lexical knowledge, but it is not straightforward to extract that knowledge. we can expose this knowledge by rewiring their parameters through lexical fine-tuning, and turn the lms into universal (decontextualized) word encoders. compared to prior attempts at injecting lexical knowledge into large lms @xcite , our lexfit method is innovative as it is deployed post-hoc on top of already pretrained lms, rather than requiring joint multi-task training. moreover, lexfit is: 1) more efficient, as it does not incur the overhead of masked language modeling pretraining; and 2) more versatile, as it can be ported to any model independently from its architecture or original training objective. finally, our results demonstrate the usefulness of lexfit: we report large gains over wes extracted from vanilla lms and over traditional we models across 8 languages and 4 lexical tasks, even with very limited and noisy external lexical knowledge, validating the rewiring hypothesis. the code is available at: https://github.com/cambridgeltl/lexfit . ## from language models to (decontextualized) word encoders the motivation for this work largely stems from the recent work on probing and analyzing pretrained language models for various types of knowledge they might implicitly store (e.g., syntax, world knowledge) @xcite . here, we focus on their lexical semantic knowledge @xcite , with an aim of extracting high-quality static word embeddings from the parameters of the input lms. in what follows, we describe lexical fine-tuning via dual-encoder networks ( §2.1), followed by the we extraction pro-1 these approaches are connected as they are both trained via contrastive learning on dual-encoder architectures, but they provide representations for a different granularity of meaning. ## experimental setup languages and language models. our language selection for evaluation is guided by the following (partially clashing) constraints @xcite ): a) availability of comparable pretrained monolingual lms; b) task and evaluation data availabil-ity; and c) ensuring some typological diversity of the selection. the final test languages are english (en), german (de), spanish (es), finnish (fi), italian (it), polish (pl), russian (ru), and turkish (tr). for comparability across languages, we use monolingual uncased bert base models for all languages (@xmath0, 12 attention heads, hidden layer dimensionality is 768), available (see the appendix) via the huggingface repository @xcite . ## results and discussion the main results for all four tasks are summarized in tables 1 2 3 4 , and further results and analyses are available in §4.1 (with additional results in the appendix). these results offer multiple axes of comparison, discussed in what follows. ## conclusion and future work we proposed lexfit, a lexical fine-tuning procedure which transforms pretrained lms such as bert into effective decontextualized word encoders through dual-encoder architectures. our experiments demonstrated that the lexical knowledge already stored in pretrained lms can be further exposed via additional inexpensive lexfiting with (even limited amounts of) external lexical knowledge. we successfully applied lexfit even to languages without any external human-curated lexical knowledge. our lexfit word embeddings (wes) outperform "traditional" static wes (e.g., fasttext) across a spectrum of lexical tasks across diverse languages in controlled evaluations, thus directly questioning the practical usefulness of the traditional we models in modern nlp. besides inducing better static wes for lexical tasks, following the line of lexical probing work @xcite , our goal in this work was to understand how (and how much) lexical semantic knowledge is coded in pretrained lms, and how to 'unlock' the knowledge from the lms. we hope that our work will be beneficial for all lexical tasks where static wes from traditional we models are still largely used @xcite .