Analyzing Encoded Concepts in Transformer Language Models (2022)

Abstract: We propose a novel framework ConceptX, to analyze how latent concepts are encoded in representations learned within pre-trained lan-guage models. It uses clustering to discover the encoded concepts and explains them by aligning with a large set of human-defined concepts. Our analysis on seven transformer language models reveal interesting insights: i) the latent space within the learned representations overlap with different linguistic concepts to a varying degree, ii) the lower layers in the model are dominated by lexical concepts (e.g., affixation) and linguistic ontologies (e.g. Word-Net), whereas the core-linguistic concepts (e.g., morphology, syntactic relations) are better represented in the middle and higher layers, iii) some encoded concepts are multi-faceted and cannot be adequately explained using the existing human-defined concepts.

## introduction contextualized word representations learned in deep neural network models (ddns) capture rich concepts making them ubiquitous for transfer learning towards downstream nlp. despite their revolution, the blackbox nature of the deep nlp models is a major bottle-neck for their large scale adaptability. understanding the inner dynamics of these models is important to ensure fairness, robustness, reliability and control. a plethora of research has been carried out to probe dnns for the linguistic knowledge (e.g. morphology, syntactic and semantic roles) captured within the learned representations. a commonly used framework to gauge how well linguistic information can be extracted from these models is the probing framework @xcite , where they train an auxiliary classifier using representations as features to predict the property of interest. the performance of the classifier reflects the amount of knowledge learned within representations. to this end, the researchers have analyzed what knowledge is learned within the representations through relevant extrinsic phenomenon varying from word morphology @xcite to high level concepts such as syntactic structure @xcite and semantics @xcite @xcite or more generic properties @xcite . in this work, we approach the representation analysis from a different angle and present a novel framework conceptx. in contrast to relying on the prediction capacity of the representations, we analyze the latent concepts learned within these representations and how knowledge is structured, using an unsupervised method. more specifically, we question: i) do the representations encode knowledge inline with linguistic properties such as word morphology and semantics? ii) which properties dominate the overall structure in these representations? iii) does the model learn any novel concepts beyond linguistic properties? answers to these questions reveal how deep neural network models structure language information to learn a task. our inspiration to use the term concept comes from "concept based explanation" in computer vision @xcite @xcite . @xcite defined a concept as "a class containing certain objects as elements, where the objects have certain properties". we define an encoded concept as a cluster of contextaware latent representations of words, where the representations are encoder layer outputs. our framework clusters contextualized representations using agglomerative hierarchical clustering @xcite . the resulting clusters represent encoded concepts, captured within the learned representations (please see @xcite 1 for illustration). we then use a novel align-figure 1 : conceptx: i) extract representations from trained model, ii) cluster the representations to obtain encoded concepts, iii) align the concepts to human-defined concepts ment function that measures the amount of overlap between encoded concepts and a range of predefined categories (that we call as human-defined concepts in this paper). we experimented with affixes, casing, morphological, syntactic, semantic, wordnet @xcite , and psycholinguistic concepts (liwc pennebaker et al. ( 2001 )). the use of such a diverse set of human-defined concepts enables us to cover various abstractions of language. in figure 3 we present a few examples of human-defined concepts that were aligned with the encoded concepts. we carry out our study on seven pre-trained transformer models such as bert @xcite and xlm-roberta @xcite , with varying optimization functions, architectural details and training data. some notable findings emerging from our analysis are as follows: our contributions in this paper are as follow: i) we present conceptx, a framework that interprets encoded concepts in the learned representation by measuring their alignment to the human-defined concepts. ii) we provide a qualitative and quantitative evidence of how knowledge is structured within deep nlp models with respect to a large suite of human-defined concepts. ## related work most of the work done on interpretability in deep nlp addresses two questions in particular: (i) what linguistic (and non-linguistic) knowledge is learned within contextualized representations, concept analysis and (ii) how this information is utilized in the decision making process, attribution analysis @xcite . the former thrives on post-hoc decomposability, where we analyze representations to uncover linguistic phenomenon that are captured as the network is trained towards any nlp task @xcite @xcite @xcite and the latter characterize the role of model components and input features towards a specific prediction @xcite @xcite . our work falls into the former category. previous studies have explored visualization methods to analyze the learned representations @xcite , attention heads @xcite , language compositionality @xcite etc. a more commonly used framework analyzes representations by correlating parts of the neural network with linguistic properties, by training a classifier to predict a feature of interest @xcite @xcite . several researchers used probing classifiers for investigating the contextualized representations learned from a variety of neural language models on a variety of character- @xcite , word- @xcite or sub-sentence level @xcite linguistic tasks. rather than analyzing the representations as a whole, several researchers also explored identifying salient neurons within the model that capture different properties @xcite @xcite or are salient for the model irrespective of the property @xcite . our work is inline with @xcite , who analyzed latent concepts learned in pre-trained models. @xcite used a binary classification task to induce latent concepts relevant to a task and showed the presence of linguistically motivated and novel concepts in the representation. however, different from them, we analyze representations in an unsupervised fashion. @xcite used human-in-the-loop to analyze latent spaces in bert. our framework uses human-defined concepts to automatically generate explanations for the latent concepts. this enabled us to scale our study to many transformer models. in a similar work, @xcite applied manifold analysis technique to understand the amount of information stored about object categories per unit. our approach does away from the methodological limitations of probing framework such as complexity of the probes, effect of randomness etc @xcite . however, it is important to mention that the two frameworks are orthogonal and complement each other. ## methodology a vector representation in the neural network model is composed of feature attributes of the input words. we group the encoded vector representations using a clustering approach discussed below. the underlying clusters, that we term as the encoded concepts, are then matched with the human-defined concepts using an alignment function. formally, consider a neural network (nn) model m with l encoder layers {l 1 , l 2 , ...l l , ..., l l }, with h hidden nodes per layer. an input sentence consisting of m words w 1 , w 2 , ...w i , ..., w m is fed into a nn. for each input word i, we compute the node output (after applying the activation func-tions) y l h (w i ) of every hidden node @xmath0, ..., h} in each layer l, where -→ y l (w i ) is the vector representation composing the outputs of all hidden nodes in layer l for w i . our goal is to cluster representations -→ y l , from a large training data to obtain encoded concepts. we then align these with various human-defined concepts to obtain an explanation of them to build an understanding of how these concepts are represented across the network. ## analysis in this section, we analyze the encoded concepts by aligning them with the human-defined concepts. ## conclusion we presented conceptx, a novel framework for analyzing the encoded concepts within deep nlp models. our method uses unsupervised clustering to discover latent concepts within the contextualized representations and then aligned these concepts with a suite of human-defined concepts to generate explanations for them. our results illuminate how dnns structure language information. a few notable findings are: i) lower layers capture shallow linguistic concepts, ii) whereas the abstract linguistic concepts such as morphology and semantics are preserved higher in the network, iii) the extent of alignment varies across different models and different human-defined concepts, iv) we found that novel explanations and an improved coverage of concepts can be achieved via compositionality. pro anaphoric & deictic pronouns: he, she, i, him not negation: not, no, neither, without def definite: the, loit, derde nec necessity: must, should, have to has possessive pronoun: my, her pos possibility: might, could, perhaps, alleged, can ref reflexive & reciprocal pron.: herself, each other dsc (discourse) emp emphasizing pronouns: himself sub subordinate relations: that, while, because act (speech act) coo coordinate relations: so, {,}, {;}, and gre greeting & parting: hi, bye app appositional relations: {,}, which, {(}, -itj interjections, exclamations: alas, ah but contrast: but, yet hes hesitation: err nam (named entity) que interrogative: who, which, ? per person: axl rose, sherlock holmes att (attribute) gpe geo-political entity: paris, japan quc concrete quantity: two, six million, twice gpo geo-political origin: parisian, french quv vague quantity: millions, many, enough geo geographical location: alps, nile col colour: red, crimson, light blue, chestnut brown org organization: ikea, eu ist intersective: open, vegetarian, quickly art artifact: ios 7 sst subsective: skillful surgeon, tall kid hap happening: eurovision 2017 pri privative: former, fake uom unit of measurement: meter, $, %, degree celsius deg degree: 2 meters tall, 20 years old ctc contact information: 112, info@mail.com int intensifier: very, much, too, rather url url: http://pmb.let.rug.nl rel relation: in, on, 's, of, after lit literal use of names: his name is john sco score: 3-0, grade a nth other names: