On the Transformation of Latent Space in Fine-Tuned NLP Models (2022)

Abstract: We study the evolution of latent space in fine-tuned NLP models. Different from the commonly used probing-framework, we opt for an unsupervised method to analyze representations. More specifically, we discover latent concepts in the representational space using hierarchical clustering. We then use an alignment function to gauge the similarity between the latent space of a pre-trained model and its fine-tuned version. We use traditional linguistic concepts to facilitate our understanding and also study how the model space transforms towards task-specific information. We perform a thorough analysis, comparing pre-trained and fine-tuned models across three models and three downstream tasks. The notable findings of our work are: i) the latent space of the higher layers evolve towards task-specific concepts, ii) whereas the lower layers retain generic concepts acquired in the pre-trained model, iii) we discovered that some concepts in the higher layers acquire polarity towards the output class, and iv) that these concepts can be used for generating adversarial triggers.

## introduction the revolution of deep learning models in nlp can be attributed to transfer learning from pre-trained language models. contextualized representations learned within these models capture rich linguistic knowledge that can be leveraged towards novel tasks e.g. classification of covid-19 tweets @xcite , disease prediction @xcite or natural language understanding tasks such as squad @xcite and glue @xcite . despite their success, the opaqueness of deep neural networks remain a cause of concern and has spurred a new area of research to analyze these models. a large body of work analyzed the knowledge learned within representations of pre-trained models @xcite @xcite @xcite and showed the presence of core-linguistic knowledge in various parts of the network. although transfer learning using pre-trained models has become ubiquitous, very few papers @xcite @xcite have analyzed the representations of the fine-tuned models. given their massive usability, interpreting fine-tuned models and highlighting task-specific peculiarities is critical for their deployment in real-word scenarios, where it is important to ensure fairness and trust when applying ai solutions. in this paper, we focus on analyzing fine-tuned models and investigate: how does the latent space evolve in a fine-tuned model? different from the commonly used probing-framework of training a post-hoc classifier @xcite , we opt for an unsupervised method to analyze the latent space of pre-trained models. more specifically, we cluster contextualized representations in high dimensional space using hierarchical clustering and term these clusters as the encoded concepts @xcite . we then analyze how these encoded concepts evolve as the models are fine-tuned towards a downstream task. specifically, we target the following questions: i) how do the latent spaces compare between base foot_0 and the fine-tuned models? ii) how does the presence of core-linguistic concepts change during transfer learning? and iii) how is the knowledge of downstream tasks structured in a fine-tuned model? we use an alignment function @xcite to compare the concepts encoded in the finetuned models with: i) the concepts encoded in their pre-trained base models, ii) the human-defined concepts (e.g. parts-of-speech tags or semantic properties), and iii) the labels of the downstream task towards which the model is fine-tuned. we carried out our study using three pre-trained transformer language models; bert @xcite , xlm-roberta @xcite and albert @xcite , analyzing how their representation space evolves as they are finetuned towards the task of sentiment analysis (sst-2, @xcite , natural language inference (mnli, @xcite and hate speech detection (hsd, @xcite . our analysis yields interesting insights such as: ## methodology our work builds on the latent concept analysis method @xcite for interpreting representational spaces of neural network models. we cluster contextualized embeddings to discover encoded concepts in the model and study the evolution of the latent space in the fine-tuned model by aligning the encoded concepts of the fine-tuned model to: i) their pre-trained version, ii) the humandefined concepts and iii) the task-specific concepts (for the task the pre-trained model is fine-tuned on). figure 1 presents an overview of our approach. in the following, we define the scope of concept and discuss each step of our approach in detail. ## analysis language model pre-training has been shown to capture rich linguistic features @xcite ) that are redundantly distributed across the network @xcite . we analyze how the representational space transforms when tuning towards a downstream task: i) how much knowledge is carried forward and ii) how it is redistributed, using our alignment framework. ## adversarial triggers the discovery of polarized concepts in the sst-2 and hsd tasks, motivated us to question: whether the fine-tuned model is learning the actual task or relying on lexical triggers to solve the problem. adversarial examples have been used in the literature to highlight model's vulnerability @xcite . we show that our polarity concepts can be used to generate such examples using the following formulation: let c ec (+ve) = {c + 1 , c + 2 , . . . , c + n } be a set of latent concepts that are identified to have a strong affinity towards predicting positive sentiment in m } be the sentences in a dev-set that are predicted as negative by the model. we compute the flipping accuracy of each concept c + x using the following function: ## related work a plethora of papers have been written in the past five years on interpreting deep nlp models. the work done in this direction can be broadly classified into: i) post-hoc representation analysis that encode the contextualized embedding for the knowledge learned @xcite @xcite and ii) causation analysis that connect input features with model behavior as a whole and at a level of individual predictions @xcite @xcite ). 10 our work mainly falls in the former category although we demonstrated a causal link between the encoded knowledge and model predictions by analyzing the concepts in the final layers and demonstrating how they can be used to generate adversarial examples with lexical triggers. recent work @xcite formally attempts to bridge the gap by connecting the two lines of work. relatively less work has been done on interpreting fine-tuned models. @xcite analyzed the heads encoding negation scope in finetuned bert and roberta models. @xcite ; @xcite analyzed linguistic knowledge in pre-trained models and showed that while fine-tuning changes the upper layers of the model, but does not lead to "catastrophic forgetting of linguistic phenomena". our results resonate with their findings, in that the higher layers learn task-specific concepts. ## conclusion we studied the evolution of latent space of pretrained models when fine-tuned towards a downstream task. our approach uses hierarchical clustering to find encoded concepts in the representations. we analyzed them by comparing with the encoded concepts of base model, human-defined concepts, and task-specific concepts. we showed that the latent space of fine-tuned model is substantially different from their base counterparts. the humandefined linguistic knowledge largely vanishes from the higher layers. the higher layers encode taskspecific concepts relevant to solve the task. moreover, we showed that these task-specific concepts can be used in generating adversarial examples that flips the predictions of the model up to 91% of the time in the case of albert hate speech model. the discovery of word-level task-specific concepts suggest that the models rely on lexical triggers and are vulnerable to adversarial attacks.