SLICE : Supersense-based Lightweight Interpretable Contextual Embeddings (2020)

Abstract: Contextualised embeddings such as BERT have become de facto state-of-the-art references in many NLP applications, thanks to their impressive performances. However, their opaqueness makes it hard to interpret their behaviour. SLICE is a hybrid model that combines supersense labels with contextual embeddings. We introduce a weakly supervised method to learn interpretable embeddings from raw corpora and small lists of seed words. Our model is able to represent both a word and its context as embeddings into the same compact space, whose dimensions correspond to interpretable supersenses. We assess the model in a task of supersense tagging for French nouns. The little amount of supervision required makes it particularly well suited for low-resourced scenarios. Thanks to its interpretability, we perform linguistic analyses about the predicted supersenses in terms of input word and context representations.

## introduction the form-meaning association relating words to their senses is a fundamental component of human languages. hence, lexical semantics, that is, the representation of the meaning of words, is an important research topic in computational linguistics. processing word meaning is essential for the (compositional) interpretation of larger units such as phrases and sentences. therefore, computational lexical semantics is, explicitly or implicitly, at the core of higher-level nlp tasks such as textual understanding, information extraction, and automatic summarisation. much effort has been put in the manual and semi-automatic construction of resources encoding lexical semantics (i.e., word meaning). these include semantic lexicons with inventories of possible senses that lexical units can assume (e.g., wordnet) and sense-annotated corpora specifying which of these senses are employed in context (e.g., semcor). alternatively, real-numbered vectors can encode contextual cooccurrence, acting as a proxy for a lexical unit's semantics. this principle has guided the development of numerous distributional semantic models, that is, semantic vector representations inferred from corpus co-occurrences, e.g., @xcite . advances in neural networks shifted the focus of computational semantics to representation learning, so as to obtain vectors as by-products of neural networks @xcite , in this booming field, a myriad of models have emerged, efficiently learned from corpora, benefiting from high-performance neural architectures and libraries. thus, vector representations, rebranded word embeddings, have become the dominant technique to represent lexical units, at the core of state-of-the-art neural approaches. traditional static embeddings, such as word2vec and fasttext, assume that each word's meaning can be represented as a single vector, independently of its context. while generic and reusable, these models usually conflate the different meanings of a given unit into a single vector @xcite . contextual models, such as elmo, gpt-2, bert, and their variants, encode each word's occurrence as a context-dependent vector, assuming that each context corresponds to a different sense @xcite . in short, while static models create one generic embedding per lexical unit, contextual models provide a fine-grained distinct representation for each occurrence. both models, but especially the latter, are increasingly complex and opaque @xcite , requiring advanced techniques to help humans understand their strengths and limitations @xcite . given this landscape, we introduce slice: an alternative semantic model which constitutes a trade-off between static, interpretable symbolic senses and contextual word embeddings. we propose a weakly supervised technique to build dense low-dimensional embeddings whose dimensions represent coarsegrained semantic classes i.e., supersenses such as animate entity and natural object (sec. 3). our lightweight model embeds both lexical units and their contexts into the same semantic space. thus, words and their contexts are represented as two compact vectors of directly interpretable scores, one per supersense, automatically learned from a non annotated corpus. our embeddings are assessed in a supersense tagging setting (sec. 4). thanks to the model's interpretability, we are able to perform a rich linguistic analysis of the results, providing insights to understand the model's predictions (sec. 5). ## related work our work is positioned at the crossroads of word and sense embeddings, interpretable semantic representations, and weakly supervised semantic classification. we briefly review a sample of relevant work on these topics. ## contextual and lexical signatures the heart of slice consists in a series of binary classifiers, one per supersense. each classifier takes as input a context c and produces a score that indicates how likely c could be associated to a given supersense s i . this score, noted cs i (c), is called a context score. a context c can be associated to a d-dimensional vector, called its signature cs(c) = (cs 1 (c), . . . , cs d (c)) t , where d is the number of different supersenses. 2 the classifiers are also used to model the overall tendency of a word w to occur in contexts that are representative of a given supersense s i . this information is modeled by the lexical scores ls i (w), computed by aggregating the context scores of all occurrences of w in a large corpus. a word w is therefore associated to a d-dimensional vector, also called its signature foot_2 ls(w) = (ls 1 (w), . . . , ls d (w)) t . such vectors can be compared to word embeddings produced by deep learning methods. the difference, however, is that each dimension of a word signature corresponds to an interpretable supersense. as stated above, slice relies on classifiers that themselves require, to be trained, supersenseannotated corpora. such corpora, when they exist, usually are of limited size and do not allow to build reliable context and word signatures. this is why we propose a semi-supervised method not requiring annotated corpora, but a list of representative words for each supersense, easier and cheaper to constitute. ## experimental setup we describe in this section the data we have used to build contextual and lexical signatures and the data we will use in the following section to evaluate our method. supersense tagset to guarantee a sufficient scope (with respect to the size of the seed lists) and to simplify analysis, we grouped wordnet supersenses into six coarser categories for our experiments on french nouns (details in appendix a). animate entity (ani) includes nouns referring to living and animate entities, namely persons (e.g., agriculteur 'farmer') and animals (e.g., chiot 'puppy'); natural object (nat) is for nouns referring to natural entities (e.g., étoile 'star'), plants (e.g., peuplier 'poplar tree') and body parts (e.g., hanche 'hip'); manufactured object (man) is composed of nouns denoting human-made or transformed entities: artifacts (e.g., chronomètre 'stopwatch') or built places (e.g., isoloir 'voting booth'); informational object (inf) includes nouns denoting abstract objects having informational contents (e.g., théorie 'theory'), those referring to knowledge areas (e.g., ethnologie 'ethnology') and financial assets (e.g., budget 'budget'); dynamic situation (dyn) gathers nouns denoting things that happen or that are carried out, like actions (e.g., ablation 'removal'), activities (e.g., cyclisme 'cycling') and events (e.g., explosion 'outbreak'); stative situation (sta) is for nouns that denote properties (e.g., dignité 'dignity'), states (e.g., endettement 'easement') and feelings (e.g., tendresse 'tenderness'). seeds we used data provided by the wolf, a french lexical resource automatically built from the princeton wordnet @xcite , to draw up the six seed lists s i . a list of monosemous french nouns has been extracted from this resource and then we manually selected 200 nouns for each coarser category described above. for example, the seed list for the dyn class, contains nouns manually selected from the wolf nouns having only one supersense among those that denote dynamic situations. selecting monosemous seeds for pseudo-annotation of the corpus can bias the classifiers, which never encounters polysemous words at training time, but only in test data. however, this should not be a problem, as the we learn to classify contexts, not words. that is, the absence of polysemous words among the seeds should not be problematic, assuming that most polysemous words in context are disambiguated. corpus and preprocessing experiments have been conducted on the frwac corpus, which contains about 1.6 billion words crawled from the web @xcite . the corpus has been pos tagged, lemmatised and morphologically analysed by an in-house parser trained on the french corpora of universal dependencies @xcite . the corpus is divided into 55 parts of about 1m sentences each. part 54 is used as development corpus for early stopping, all other parts are used for training. positive and negative seed sets s i and s - i are split into a training set (80% of the lemmas) and a development (20% of the lemmas). the training seeds are used to annotate the training corpus, while the development seeds are used to annotate the development corpus. this is a deterministic process: each occurrence of a word in s i (resp. s - i ) is annotated as 1 (resp. 0). we artificially balance the number of training contexts in each corpus c i to avoid biases related to different distributions of positive and negative examples. given the seed lists s i and s - i , we count the total number of occurrences of lemmas from each list, n i and n - i in c. if n @xmath0, all sentences containing a lemma from s i are added to c i . then, sentences containing lemmas from s - i are randomly added until at least n i occurrences from s - i appear in c i . if n - @xmath1. all other sentences are discarded. evaluation data the frsemcor corpus was used for evaluation @xcite . 7 it contains manual annotations for more than 12,000 nouns in the sequoia treebank, a corpus of 3,009 sentences from different sources including morphological and syntactic annotations @xcite . noun tokens have been annotated with 24 supersenses adapted from the wordnet supersense tagset. 8 for this experiment, we used 7,188 annotated nouns: 5,160 have been used for training, 1,015 for development and 1,013 for evaluation. ## supersense tagging we have evaluated slice on a supersense tagging task because our model produces interpretable senses that can be directly compared to senses used in semantically annotated corpora (frsemcor, in our case). our model produces, for every word in context, a description of the context through the context signature, and a description of the word usage through its lexical signature. comparing different ways to combine these two pieces of information is interesting from a linguistic point of view since it can lead to interesting analyses of complex linguistic phenomena such as polysemy, multi-facet nouns, and unusual contexts (e.g., manufactured objects man in contexts typical of animate beings ani). as a comparison point for the performances reached by slice, we have used a simple baseline, which selects for every noun occurrence its most frequent supersense (mfs) in the training corpus. when the word does not occur in the training corpus, the most frequent supersense across all words is selected. this crude method gives better results as the training corpus grows, since the coverage grows with size of the training corpus and selecting the most frequent supersense is a good heuristic @xcite . we also compare our model to a state-of-the-art model in other wsd tasks: a french-specific version of bert called flaubert @xcite . we use the 1024-dimensional embeddings available in flaubert-large as part of the huggingface library. 9 for each target noun, we obtain its contextualised embedding from the top layer and provide it to an mlp identical to the one described in section 5.2. tokenisation incompatibilities due to bpe encoding are rare (e.g., 50/1,013= occurrences in the test corpus); they are resolved by taking the noun's last subtoken before the word separator as its embedding. in our model, the decision to tag word w in context c with a given supersense is taken based on the lexical signature of w and the context signature of c. 10 they are combined to yield a word-in-context signature ψ(ls(w), cs(c)), which is also d-dimensional. the component corresponding to the highest score is selected as the predicted supersense for w in c: the main missing part in this model is the nature of function ψ that combines lexical and contextual signatures. we discuss in the two following sections two instantiations of function ψ. ## conclusions we have presented a method to learn interpretable embeddings using as weak supervision a list of seed nouns for each supersense. we use the occurrences of seed (prototypical) nouns to train a classifier which associates contexts to supersenses. the context scores are aggregated to generate a single lexical score per supersense. each of these scores are seen as an interpretable dimension of a dense word embedding. we have evaluated our method on a supersense tagging task to predict in-context coarse supersenses. in addition to a good performance with very little training data, our method's interpretability allows us to analyse the results in terms of the (supersense) dimensions of the input embeddings. moreover, our model is considerably faster and lighter than state-of-the-art contextualised embeddings, e.g., we represent inputs as a set of 12 scores whereas flaubert uses 1024-dimensional opaque vectors. we have also built and released the lexicon containing the 10k most frequent french nouns of the frwac and their corresponding embeddings. we hope that this resource can be complementary to existing embeddings and lexical semantic resources. the lexicon, along with the seed lists, predictions and evaluation data are freely available. 13 we have applied our method to nouns only, and our embeddings are 6-dimensional (one per coarse supersense), certainly lacking expressive power to cover the full range of semantic distinctions. in theory, there is nothing that prevents us from increasing the number of dimensions (e.g., to cover the traditional wordnet supersenses), and to experiment with other parts of speech (e.g., verbs and adjectives). in practice, the list of seeds for some supersenses may be too small (e.g., time), and we lack annotated corpora to evaluate the method for other pos in french. the sensitivity of the method to the number of seed elements needs to be studied in more detail in the future. another issue that remains open is the integration of embeddings with different pos tags: should we build a different model per pos (with different interpretable dimensions) or one single models in which inapplicable dimensions are empty? as future extensions, we envisage integrating our embeddings in other downstream tasks such as semantic parsing. we would like to generalise our method to other syntactic and semantic categories, e.g., can we build interpretable embeddings in which each dimensions represents a given pos using seed lists of verbs, nouns, etc.? transformer models are a promising alternative to recurrent neural networks to focus on relevant contexts for the classifiers.