Probing Pretrained Language Models for Lexical Semantics (2020)

Abstract: The success of large pretrained language models (LMs) such as BERT and RoBERTa has sparked interest in probing their representations, in order to unveil what types of knowledge they implicitly capture. While prior research focused on morphosyntactic, semantic, and world knowledge, it remains unclear to which extent LMs also derive lexical type-level knowledge from words in context. In this work, we present a systematic empirical analysis across six typologically diverse languages and five different lexical tasks, addressing the following questions: 1) How do different lexical knowledge extraction strategies (monolingual versus multilingual source LM, out-of-context versus in-context encoding, inclusion of special tokens, and layer-wise averaging) impact performance? How consistent are the observed effects across tasks and languages? 2) Is lexical knowledge stored in few parameters, or is it scattered throughout the network? 3) How do these representations fare against traditional static word vectors in lexical tasks 4) Does the lexical information emerging from independently trained monolingual LMs display latent similarities? Our main results indicate patterns and best practices that hold universally, but also point to prominent variations across languages and tasks. Moreover, we validate the claim that lower Transformer layers carry more type-level lexical knowledge, but also show that this knowledge is distributed across multiple layers.

## introduction and motivation language models (lms) based on deep transformer networks @xcite , pretrained on unprecedentedly large amounts of text, offer unmatched performance in virtually every nlp task @xcite . models such as bert @xcite , roberta @xcite , and t5 @xcite replaced task-specific neural architectures that relied on static word embeddings (wes; @xcite @xcite , where each word is assigned a single (type-level) vector. while there is a clear consensus on the effectiveness of pretrained lms, a body of recent research has aspired to understand why they work @xcite . state-of-the-art models are "probed" to shed light on whether they capture task-agnostic linguistic knowledge and structures @xcite @xcite ; e.g., they have been extensively probed for syntactic knowledge @xcite @xcite and morphology @xcite . in this work, we put focus on uncovering and understanding how and where lexical semantic knowledge is coded in state-of-the-art lms. while preliminary findings from @xcite and @xcite suggest that there is a wealth of lexical knowledge available within the parameters of bert and other lms, a systematic empirical study across different languages is currently lacking. we present such a study, spanning six typologically diverse languages for which comparable pretrained bert models and evaluation data are readily available. we dissect the pipeline for extracting lexical representations, and divide it into crucial components, including: the underlying source lm, the selection of subword tokens, external corpora, and which transformer layers to average over. different choices give rise to different extraction configurations (see we run experiments and analyses on five diverse lexical tasks using standard evaluation benchmarks: lexical semantic similarity (lsim), word analogy resolution (wa), bilingual lexicon induction (bli), cross-lingual information retrieval (clir), and lex-ical relation prediction @xcite . the main idea is to aggregate lexical information into static type-level "bert-based" word embeddings and plug them into "the classical nlp pipeline" @xcite , similar to traditional static word vectors. the chosen tasks can be seen as "lexico-semantic probes" providing an opportunity to simultaneously 1) evaluate the richness of lexical information extracted from different parameters of the underlying pretrained lm on intrinsic (e.g., lsim, wa) and extrinsic lexical tasks (e.g., relp); 2) compare different type-level representation extraction strategies; and 3) benchmark "bert-based" static vectors against traditional static word embeddings such as fasttext @xcite . our study aims at providing answers to the following key questions: q1) do lexical extraction strategies generalise across different languages and tasks, or do they rather require language-and taskspecific adjustments?; q2) is lexical information concentrated in a small number of parameters and layers, or scattered throughout the encoder?; q3) are "bert-based" static word embeddings competitive with traditional word embeddings such as fasttext?; q4) do monolingual lms independently trained in multiple languages learn structurally similar representations for words denoting similar concepts (i.e., translation pairs)? we observe that different languages and tasks indeed require distinct configurations to reach peak performance, which calls for a careful tuning of configuration components according to the specific task-language combination at hand (q1). however, several universal patterns emerge across languages and tasks. for instance, lexical information is predominantly concentrated in lower transformer layers, hence excluding higher layers from the extraction achieves superior scores (q1 and q2). further, representations extracted from single layers do not match in accuracy those extracted by averaging over several layers (q2). while static word representations obtained from monolingual lms are competitive or even outperform static fasttext embeddings in tasks such as lsim, wa, and relp, lexical representations from massively multilingual models such as multilingual bert (mbert) are substantially worse (q1 and q3). we also demonstrate that translation pairs indeed obtain similar representations (q4), but the similarity depends on the extraction configuration, as well as on the typological distance between the two languages. ## lexical representations from pretrained language models classical static word embeddings @xcite @xcite are grounded in distributional semantics, as they infer the meaning of each word type from its co-occurrence patterns. however, lm-pretrained transformer encoders have introduced at least two levels of misalignment with the classical approach @xcite . first, representations are assigned to word tokens and are affected by the current context and position within a sentence @xcite . second, tokens may correspond to subword strings rather than complete word forms. this begs the question: do pretrained encoders still retain a notion of lexical concepts, abstracted from their instances in texts? analyses of lexical semantic information in large pretrained lms have been limited so far, focusing only on the english language and on the task of word sense disambiguation. @xcite showed that senses are encoded with finer-grained precision in higher layers, to the extent that their representation of the same token tends not to be self-similar across different contexts @xcite . as a consequence, we hypothesise that abstract, type-level information could be codified in lower layers instead. however, given the absence of a direct equivalent to a static word type embedding, we still need to establish how to extract such type-level information. in prior work, contextualised representations (and attention weights) have been interpreted in the light of linguistic knowledge mostly through probes. these consist in learned classifier predicting annotations like pos tags @xcite and word senses @xcite @xcite , or linear transformations to a space where distances mirror dependency tree structures @xcite . 1 in this work, we explore several unsupervised word-level representation extraction strategies and configurations for lexico-semantic tasks (i.e., probes), stemming from different combinations of the components detailed in ## results and discussion a summary of the results is shown in figure 2 for lsim, in figure 3a for bli, in figure 3b for clir, in figure 4a and figure 4b for relp, and in figure 4c for wa. these results offer multiple axes of comparison, and the ensuing discussion focuses on the central questions q1-q3 posed in §1. 6 monolingual versus multilingual lms. results across all tasks validate the intuition that languagespecific monolingual lms contain much more lexical information for a particular target language than massively multilingual models such as mbert or xlm-r @xcite . we see large drops between mono.* and multi.* configurations even for very high-resource languages (en and de), and they are even more prominent for fi and tr. encompassing 100+ training languages with limited model capacity, multilingual models suffer from the "curse of multilinguality" @xcite : they must trade off monolingual lexical information coverage (and consequently monolingual performance) for a wider language coverage. 7 how important is context? another observation that holds across all configurations concerns the usefulness of providing contexts drawn from external corpora, and corroborates findings from prior work @xcite : iso configurations cannot match configurations that average subword embeddings from multiple contexts (aoc-10 and aoc-100). however, it is worth noting that 1) perfor- average over layers @xmath0.1 0.2 0.3 0.4 bli scores (mrr) (a) summary bli results @xmath1.025 0.050 0.075 0.100 0.125 0.150 clir scores (map) (b) summary clir results figure 3: summary results for the two cross-lingual evaluation tasks: (a) bli (mrr scores) and (b) clir (map scores). we report average scores over all language pairs; individual results for each language pair are available in the appendix. thick solid horizontal lines denote performance of standard fasttext vectors in exactly the same cross-lingual mapping setup. @xmath2.65 0.67 0.69 0.71 0.73 micro-averaged f 1 scores (a) relp: english @xmath3.60 0.62 0.64 0.66 0.68 micro-averaged f 1 scores (b) relp: german @xmath4.15 0.20 0.25 0.30 0.35 0.40 analogy scores (precision@1) (c) wa: english figure 4: micro-averaged f 1 scores in the relp task for (a) en and (b) de. the scores with 768-dim vectors randomly initalized via xavier init (glorot and bengio, 2010) are 0.473 (en) and 0.512 (de); (c) en wa results. in relp, it is notable that 'bert-based' embeddings can recover more lexical relation knowledge than standard ft vectors. these findings reveal that pretrained lms indeed implicitly capture plenty of lexical type-level knowledge (which needs to be 'recovered' from the models); this also suggests why pretrained lms have been successful in tasks where this knowledge is directly useful, such as ner and pos tagging @xcite . finally, we also note that gains with aoc over iso are much more pronounced for the under-performing multi.* configurations: this indicates that mono models store more lexical information even in absence of context. how important are special tokens? the results reveal that the inclusion of special tokens [cls] and [sep] into type-level embedding extraction deteriorates the final lexical information contained in the embeddings. this finding holds for different languages, underlying lms, and averaging across various layers. the nospec configurations consistently outperform their all and withcls counterparts, both in iso and aoc-{10, 100} settings. 8 our finding at the lexical level aligns well with prior observations on using bert directly as a sentence encoder @xcite @xcite : while [cls] is useful for sentence-pair classification tasks, using [cls] as a sentence representation produces inferior representations than averaging over sentence's subwords. in this work, we show that [cls] and [sep] should also be fully excluded from subword averaging for type-level word representations. how important is layer-wise averaging? averaging across layers bottom-to-top (i.e., from l 0 to l 12 ) is beneficial across the board, but we notice that scores typically saturate or even decrease in some tasks and languages when we include higher layers into averaging: see the scores with *.avg(@xmath5.avg(@xmath6, e.g., for fi lsim; en/de relp, and summary bli and clir scores. this hints to the fact that two strategies typically used in prior work, either to take the vectors only from the embedding layer l 0 @xcite or to average across all layers @xcite , extract suboptimal word representations for a wide range of setups and languages. the sweet spot for n in *.avg(@xmath7, as peak scores are obtained with different n-s. whereas averaging across all layers generally hurts performance, the results strongly suggest that averaging across layer subsets (rather than selecting a single layer) is widely useful, especially across bottom-most layers: e.g., @xmath8.iso.nospec yields an average score of 0.561 in lsim, 0.076 in clir, and 0.432 in bli; the respective scores when averaging over the 6 top layers are: 0.218, 0.008, and 0.230. this evidence implies that, although scattered across multiple layers, type-level lexical information seems to be concentrated in lower transformer layers. we investigate these conjectures further in §4.1. ## further discussion and conclusion what about larger lms and corpora? aspects of lm pretraining, such as the number of model parameters or the size of pretraining data, also impact lexical knowledge stored in the lm's parameters. our preliminary experiments have verified that en bert-large yields slight gains over the en bert-base architecture used in our work (e.g., peak en lsim scores rise from 0.518 to 0.531). in a similar vein, we have run additional experiments with two available italian (it) bert-base models with identical parameter setups, where one was trained 9 previous work has empirically validated that sentence representations for semantically similar inputs from different languages are less similar in higher transformer layers @xcite . in figure 5 , we demonstrate that this is also the case for type-level lexical information; however, unlike sentence representations where highest similarity is reported in lowest layers, figure 5 suggests that highest cka similarities are achieved in intermediate layers l5-l8. on 13gb of it text, and the other on 81gb. in en (bert-base)-it bli and clir evaluations we measure improvements from 0.548 to 0.572 (bli), and from 0.148 to 0.160 (clir) with the 81gb it model. in-depth analyses of these factors are out of the scope of this work, but they warrant further investigations. opening future research avenues. our study has empirically validated that (monolingually) pretrained lms store a wealth of type-level lexical knowledge, but effectively uncovering and extracting such knowledge from the lms' parameters depends on several crucial components (see §2). in particular, some universal choices of configuration can be recommended: i) choosing monolingual lms; ii) encoding words with multiple contexts; iii) excluding special tokens; iv) averaging over lower layers. moreover, we found that type-level wes extracted from pretrained lms can surpass static wes like fasttext @xcite . this study has only scratched the surface of this research avenue. in future work, we plan to investigate how domains of external corpora affect aoc configurations, and how to sample representative contexts from the corpora. we will also extend the study to more languages, more lexical semantic probes, and other larger underlying lms. the difference in performance across layers also calls for more sophisticated lexical representation extraction methods (e.g., through layer weighting or attention) similar to meta-embedding approaches @xcite @xcite . given the current large gaps between monolingual and multilingual lms, we will also focus on lightweight methods to enrich lexical content in multilingual lms @xcite .