M irror W i C : On Eliciting Word-in-Context Representations from Pretrained Language Models (2021)

Abstract: Recent work indicated that pretrained language models (PLMs) such as BERT and RoBERTa can be transformed into effective sentence and word encoders even via simple self-supervised techniques. Inspired by this line of work, in this paper we propose a fully unsupervised approach to improving word-in-context (WiC) representations in PLMs, achieved via a simple and efficient WiC-targeted fine-tuning procedure: MirrorWiC. The proposed method leverages only raw texts sampled from Wikipedia, assuming no sense-annotated data, and learns context-aware word representations within a standard contrastive learning setup. We experiment with a series of standard and comprehensive WiC benchmarks across multiple languages. Our proposed fully unsupervised MirrorWiC models obtain substantial gains over off-the-shelf PLMs across all monolingual, multilingual and cross-lingual setups. Moreover, on some standard WiC benchmarks, MirrorWiC is even on-par with supervised models fine-tuned with in-task data and sense labels.

## introduction pretrained language models (plms) such as bert @xcite and roberta @xcite provide dynamic contextual representations; they induce token-level lexical representations that capture the impact of the word's context on its embedding. recent studies have assessed the plms by probing into their off-the-shelf representation/feature space @xcite @xcite . while off-the-shelf plms already offer a useful contextualised lexical semantic space, their contextualised representation spaces suffer from instability and anisotropy @xcite ., * equal contribution. due to the fat-tailed nature of pandemic risk… due t_fat-tailed nature of _mic risk… a review article in nature reported this finding. the end doesn justify the m j nature nature nature due to the fat-tailed nature of pandemic risk… due t[mask] fat-tailed nature of [mask]mic risk… f( ) f( ) f( ) f( ) random span deletion …… push pull human interaction with nature has played a major role. the end doesn't always justify the means. ## mirrorwic: methodology baseline wic representations. prior work directly extracts word-in-context representations from the parameters of the off-the-shelf plms. the most effective (empirically validated) strategy is 1) averaging the representations from the top four plm's layers, and 2) then taking either the first constituent subword from the plm's vocabulary to represent the target word, or further averaging the representations of the word's constituent subwords @xcite . ## experimental setup wic evaluation. we evaluate mirrorwic on a range of context-sensitive lexical semantic tasks in monolingual english settings, as well as in multilingual and cross-lingual settings. for english, we evaluate on two similarity-based tasks: usim and cosimlex; two word-in-context classification tasks: wic and wic-tsv; and oneshot word sense disambiguation (wsd). usim @xcite measures the similarity between two instances of the same word occurring in two different sentential contexts. cosimlex (armendariz et al., 2020) measures the change in similarity between two different words appearing in two different contexts: paragraphs. we follow the standard evaluation protocol, computing the cosine similarity of the contextual word representations and comparing them against human-elicited scores via spearman's rank correlation (ρ). the wic classification task (pilehvar and camacho-collados, 2019) challenges a model to make a binary decision on whether or not the same target word has the same meaning in two different contexts. the wic-tsv (tsv) task @xcite extends the original wic to multiple domains with three different subtasks. in tsv-1, the task is to decide if the intended sense of the target word in the context matches the target sense described by the definition. in tsv-2, the model must identify if the intended sense (in the context) is the hyponym of the provided hypernyms. tsv-3 combines the previous two subtasks (see @xcite for further details). the wsd task @xcite requires a system to select the correct label for a given target word in context from a candidate set of all possible meanings for this target word. to evaluate the feature space of the models in wsd, we create a one-shot setting where we provide one context example foot_2 per label and perform nearest neighbour search over contextual word representations from the candidate labels. we directly test the models on the concatenated all test set from @xcite without access to training and development data. we also perform multilingual and cross-lingual evaluation on xl-wic @xcite and am2ico @xcite . xl-wic provides wic-style evaluations in multiple languages. am2ico extends xl-wic to lower-resource languages, adds more difficult adversarial examples, and enables cross-lingual evaluations. for brevity, we show results for four typologically diverse languages both from xl-wic (zh, ko, hr, et); and four languages in am2ico (zh, ka, ja, ar). 4 for wic, tsv, xl-wic and am2ico, our main experiments follow the unsupervised method from pilehvar and camacho-collados (2019): we compute cosine similarity between the contextual word representations in each pair, and search for a threshold to divide true (i.e., same meaning) and false instances on the development set in each task. 5 we report accuracy scores in the main paper, while additional area-under-curve (auc) scores are available in app. §a.1. underlying plms. we experiment with several standard input plms for english, but we remind the reader that the mirrorwic framework is applicable with a wide range of plms: 1) bert @xcite as a standard choice for wic representation learning and evaluation @xcite ; 2) roberta @xcite as an optimised and improved plm; and 3) deberta @xcite as a more recent plm that achieves state-of-the-art results in a range of natural language understanding tasks @xcite . 6 for all non-english experiments, unless noted otherwise, we rely on multilingual bert (mbert) as the underlying plm (see app. §a.2). fine-tuning details. we largely follow the mir-rorbert fine-tuning setup @xcite , using 10k sentences randomly drawn from wikipedia as the mirrorwic fine-tuning corpus. for monolingual models, we sample 10k sentences from the corresponding wikipedia of that language. for cross-lingual models, we sample 5k sentences from english wikipedia and 5k from wikipedia of each target language. we train all models with adamw @xcite with a learning rate of 2e-5 for 1 epoch. the τ in eq. ( 1 ) is set to 0.04. we set k (random span masking rate) to 10, 0 and 1 for bert, roberta and deberta respectively. the respective dropout rates are 0.4, 0.3 and 0.3 for bert, roberta and deberta. all hyperparameters are tuned on the development set of wic and kept unchanged for all other experiments. we refer the reader to the appendix ( ## conclusion we proposed mirrorwic, a fully unsupervised approach for eliciting word-in-context representations from pretrained language models (plms), requiring only raw sentences as input, and disposing of labelled data and sense inventories. we showed that mirrorwic is plm-agnostic and languageagnostic, yielding substantial performance boosts in context-aware lexical semantic tasks in english, multilingual and cross-lingual setups and demonstrating that additional wic knowledge can be exposed from the plms. we then delved into the inner-working of mirrorwic, demonstrating that the performance improvement strongly correlates with metrics such as isotropy score and intra-sentence word similarity. in future work, we will also look into weakly supervised approaches that combine self-supervision with external senserelated knowledge.