Unsupervised Distillation of Syntactic Information from Contextualized Word Representations (2020)

Abstract: Contextualized word representations, such as ELMo and BERT, were shown to perform well on various semantic and syntactic task. In this work, we tackle the task of unsupervised disentanglement between semantics and structure in neural language representations: we aim to learn a transformation of the contextualized vectors, that discards the lexical semantics, but keeps the structural information. To this end, we automatically generate groups of sentences which are structurally similar but semantically different, and use metric-learning approach to learn a transformation that emphasizes the structural component that is encoded in the vectors. We demonstrate that our transformation clusters vectors in space by structural properties, rather than by lexical semantics. Finally, we demonstrate the utility of our distilled representations by showing that they outperform the original contextualized representations in a few-shot parsing setting.

## introduction human language 1 is a complex system, involving an intricate interplay between meaning (semantics) and structural rules between words and phrases (syntax). self-supervised neural sequence models for text trained with a language modeling objective, such as elmo @xcite , bert @xcite , and roberta @xcite , were shown to produce representations that excel in recovering both structure-related information @xcite @xcite as well as in semantic information @xcite . in this work, we study the problem of disentangling structure from semantics in neural language pairs of words are represented by the difference between their transformation f , which is identical for all words. the pairs of words in the anchor and positive sentences are lexically different, but structurally similar. the negative example presented here is especially challenging, as it is lexically similar, but structurally different. ## related work the problem of disentangling different sources of variation has long been studied in computer vision, and was recently applied to neural models @xcite @xcite . such disentanglement can assist in learning representations that are invariant to specific factors, such as pose-invariant face-recognition @xcite or style-invariant digit recognition @xcite . from a generative point of view, disentanglement can be used to modify one aspect of the input (e.g., "style"), while keeping the other factors (e.g., "content") intact, as done in neural image style-transfer @xcite . in nlp, disentanglement is much less researched. in controlled natural language generation and style transfer, several works attempted to disentangle factors of variation such as sentiment or age of the writer, with the intention to control for those factors and generate new sentences with specific properties @xcite @xcite , or transfer existing sentences to similar sentences that differ only in the those properties. the latter goal of style transfer is often realized by learning representations which are invariant to the controlled attributes @xcite . another main line of work which is relevant to our approach is that of probing. the concept, originally introduced by @xcite and @xcite , relies on training classifiers (probes) to expose symbolic linguistic information that is encoded in the model. a large body of works have shown sensitivity to both semantic @xcite and syntactic @xcite @xcite @xcite information. @xcite demonstrated that it is possible to train a linear transformation, under which squared euclidean distance between transformed contextualized word vectors correspond to the distances between the respective words in the syntactic tree. li and eisner (2019) have used a variational estimation method @xcite of the information-bottleneck principle @xcite to extract word embeddings that are useful to the end task of parsing. while impressive, those works presuppose a specific syntactic structure (e.g. annotated parse tree) and use this linguistic signal to learn the probe in a supervised manner. this approach can introduce confounding between extracting information and learning it by the probe @xcite @xcite . in contrast, we aim to expose the structural information encoded in the network in an unsupervised manner, without pre-supposing an existing syntactic annotation scheme. ## method our goal is to learn a function f : r @xmath0, which operates on contextualized word representations x and extracts vectors f (x) which make the structural information encoded in x more salient, while discarding as much lexical information as possible. in the sentences "maple syrup is delicious" and "neural networks are interesting", we want to learn a function f such that f (v 3 syrup ) ≈ f (v 1 networks ), where v i word is the contextualized vector representation of the word in sentence i. we also want f (v 4 syrup ) ≈ f (v 2 networks ), while keeping f (v 1 networks ) ≈ f (v 2 networks ). moreover, we would like the relation between the words "maple" and "delicious" in the third sentence, to be similar to the relation between "neural" and "interesting" in the first sentence: pair(v 3 maple , v 3 delicious ) ≈ pair(v 1 neural , v 1 interesting ). operatively, we represent pairs of words (x, y) by the difference between their transformation f (x)f (y), and aim to learn a function f that preserves: the choice to represent pairs this way was inspired by several works that demonstrated that nontrivial semantic and syntactic relations between uncontextualized word representations can be approximated by simple vector arithmetic @xcite . to learn f , we start with groups of sentences such that the sentences within each group are known to share structure but differ in lexical semantics. we call the sentences in each group structurally equivalent. figure 2 shows an example of two structurally equivalent sets. acquiring such sets is challenging, especially if we do not assume a known syntactic formalism and cannot mine for sentences based on their observed tree structures. to this end, we automatically generate the sets starting with known sentences and sampling variants from a language model ( §3.1). our sentence-set generation procedure ensures that words from the same set that share an index also share their structural function. we call such words corresponding. we now proceed to learn a function f to map contextualized vectors of corresponding words (and the relations between them, as described above) to neighbouring points in the space. we train f such that the representation assigned to positive pairs -pairs that share indices and come from the same equivalent set -is distinguished from the representations of negative pairs -challenging pairs that come from different sentences, and thus do not share the structure of the original pair, but can, potentially, share their lexical meaning. we do so using triplet loss, which pushes the representations of pairs coming from the same group closer together ( §3.3). figure 1 sketches the network. ## experiments and analysis we have trained the syntactic transformation f in a way that should encourage it to retain the structural information encoded in contextualized vectors, but discard other information. we assess the representations the model acquired in an unsupervised manner, by evaluating the extent to which the local neighbors of each transformed contextualized vector f (x) share known structural properties, such as grammatical function within the sentence. for the baseline, we expect the neighbors of each vector to share a mix of semantic and syntactic properties. for the transformed vectors, we expect the neighbors to share mainly syntactic properties. finally, we demonstrate that in a few-shot setting, our representations outperform the original elmo representation, indicating they are indeed distilled from syntax, and discard other information that is encoded in elmo vectors but is irrelevant for the extraction of the structure of a sentence. corpus for training the transformation f , we rely on 150,000 sentences from wikipedia, tokenized and pos-tagged by spacy @xcite . the pos tags are used in the equivalent set generation to filter replacement words. apart from pos tagging, we do not rely on any syntactic annotation during training. the evaluation sentences for the experiments mentioned below are sampled from a collection of 1,000,000 original and unmodified wikipedia sentences (different from those used in the model training). in the original elmo representation most states -apart from those characterized by a specific partof-speech, such as amod (adjectives, in orange) or nummod (numbers, in light green) -do not fit well into a single cluster. in contrast, the syntactically transformed vectors are more neatly clustered, with some clusters, such as direct objects (brown) and prepositional-objects (blue), that are relatively separated after, but not before, the transformation. interestingly, some functions that used to be a single group in elmo (like the adjectives in orange, or the noun-compounds in green) are type text q1 in this way of thinking, an impacting projectile goes into an ice-rich layer -but no further. n they generally have a pre-engraved rifling band to engage the rifled launch tube, spin-stabilizing the projectile, hence the term "rifle". nt to achieve a large explosive yield, a linear implosion weapon needs more material, about 13 kgs. ## conclusion we propose an unsupervised method for the distillation of structural information from neural contextualized word representations. we used a process of sequential bert-based substitution to create a large number of sentences which are structurally similar, but semantically different. by controlling for structure while changing lexical choice, we learn a metric under which pairs of words that come from structurally-similar sentences are close in space. we demonstrated that the representations acquired by this method share structural properties with their neighbors in space, and show that with a minimal supervision, those representations outperform elmo in the task of few-shots parsing. the method is a first step towards a better disentanglement between various kinds of information that is represented in neural sequence models. the method used to create the structurally equivalent sentences can be useful by its own as a dataaugmentation technique. in future work, we aim to extend this method to allow for a more soft alignment between structurally-equivalent sentences.