Can LLM s Facilitate Interpretation of Pre-trained Language Models? (2023)

Abstract: Work done to uncover the knowledge encoded within pre-trained language models rely on annotated corpora or human-in-the-loop methods. However, these approaches are limited in terms of scalability and the scope of interpretation. We propose using a large language model, ChatGPT, as an annotator to enable fine-grained interpretation analysis of pre-trained language models. We discover latent concepts within pre-trained language models by applying agglomerative hierarchical clustering over contextualized representations and then annotate these concepts using ChatGPT. Our findings demonstrate that ChatGPT produces accurate and semantically richer annotations compared to human-annotated concepts. Additionally, we showcase how GPT-based annotations empower interpretation analysis methodologies of which we demonstrate two: probing frameworks and neuron interpretation. To facilitate further exploration and experimentation in the field, we make available a substantial ConceptNet dataset (TCN) comprising 39,000 annotated concepts.

## introduction a large body of work done on interpreting pretrained language models answers the question: what knowledge is learned within these models? researchers have investigated the concepts encoded in pre-trained language models by probing them against various linguistic properties, such as morphological @xcite , syntactic @xcite @xcite , and semantic @xcite tasks, among others. much of the methodology used in these analyses heavily rely on either having access to an annotated corpus that pertains to the linguistic concept of interest @xcite @xcite , or involve human-inthe-loop @xcite @xcite to facilitate such an analysis. the use of pre-defined linguistic concepts restricts the scope of interpretation to only very general linguistic concepts, while human-inthe-loop methods are not scalable. we circumvent this bottleneck by using a large language model, chatgpt, as an annotator to enable fine-grained interpretation analysis. generative pre-trained transformers (gpt) have been trained on an unprecedented amount of textual data, enabling them to develop a substantial understanding of natural language. as their capabilities continue to improve, researchers are finding creative ways to leverage their assistance for various applications, such as question-answering in financial and medical domains @xcite , simplifying medical reports @xcite , and detecting stance @xcite . we carry out an investigation of whether gpt models, specifically chatgpt, can aid in the interpretation of pre-trained language models (plms). a fascinating characteristic of neural language models is that words sharing any linguistic relationship cluster together in high-dimensional spaces @xcite . recent research @xcite @xcite has built upon this idea by exploring representation analysis through latent spaces in pre-trained models. building on the work of @xcite we aim to identify encoded concepts within pre-trained models using agglomerative hierarchical clustering @xcite on contextualized representations. the underlying hypothesis is that these clusters represent latent concepts, capturing the language knowledge acquired by the model. unlike previous approaches that rely on predefined concepts @xcite or human annotation @xcite to label these concepts, we leverage the chatgpt model. figure 1 : chatgpt as an annotator: human annotation or taggers trained on pre-defined concepts, cover only a fraction of a model's concept space. chatgpt enables scaling up annotation to include nearly all concepts, including the concepts that may not have been manually annotated before. our findings indicate that the annotations produced by chatgpt are semantically richer and accurate compared to the human-annotated concepts (for instance bert concept net). notably, chat-gpt correctly labeled the majority of concepts deemed uninterpretable by human annotators. using an llm like chatgpt improves scalability and accuracy. for instance, the work in @xcite was limited to 269 concepts in the final layer of the bert-base-cased @xcite model, while human annotations in @xcite were confined to 100 keys per layer. using chatgpt, the exploration can be scaled to the entire latent space of the models and many more architectures. we used gpt to annotate 39k concepts across 5 pre-trained language models. building upon this finding, we further demonstrate that gpt-based annotations empowers methodologies in interpretation analysis of which we show two: i) probing framework @xcite , ii) neuron analysis @xcite . probing framework we train probes from gptannotated concept representations to explore concepts that go beyond conventional linguistic categories. for instance, instead of probing for named entities (e.g. ne:per), we can investigate whether a model distinguishes between male and female names or probing for "cities in the southeastern united states" instead of ne:loc. ## methodology we discover latent concepts by applying clustering on feature vectors ( §2.1). they are then labeled using chatgpt ( §2.2) and used for fine-grained interpretation analysis ( §2.3 and 2.4). a visual representation of this process is shown in figure 1 . ## experimental setup latent concept data we used a subset of the wmt news 2018 dataset, containing 250k randomly chosen sentences (≈5m tokens). we set a word occurrence threshold of 10 and restricted each word type to a maximum of 10 occurrences. this selection was made to reduce computational and memory requirements when clustering highdimensional vectors. we preserved the original embedding space to avoid information loss through dimensionality reduction techniques like pca. consequently, our final dataset consisted of 25,000 word types, each represented by 10 contexts. ## concept-based interpretation analysis now that we have established the capability of large language models like chatgpt in providing rich semantic annotations, we will showcase how these annotations can facilitate extensive finegrained analysis on a large scale. ## related work with the ever-evolving capabilities of the llms, researchers are actively exploring innovative ways to harness their assistance. prompt engineering, the process of crafting instructions to guide the behavior and extract relevant knowledge from these oracles, has emerged as a new area of research @xcite @xcite @xcite . recent work has established llms as highly proficient annotators. @xcite carried out evaluation of gpt-3's performance as a data annotator for text classification and named entity recognition tasks, employing three primary methodologies to assess its effectiveness. @xcite showed that gpt-3 as an annotator can reduce cost from 50-96% compared to human annotations on 9 nlp tasks. they also showed that models trained using gpt-3 labeled data outperformed the gpt-3 fewshot learner. similarly, @xcite showed that chatgpt achieves higher zero-shot accuracy compared to crowd-source workers in various annotation tasks, encompassing relevance, stance, topics, and frames detection. our work is different from previous work done using gpt as annotator. we annotate the latent concepts encoded within the embedding space of pre-trained language models. we demonstrate how such a large scale annotation enriches representation analysis via application in probing classifiers and neuron analysis. ## conclusion the scope of previous studies in interpreting neural language models is limited to general ontologies or small-scale manually labeled concepts. in our research, we showcase the effectiveness of large language models, specifically chatgpt, as a valuable tool for annotating latent spaces in pre-trained language models. this large-scale annotation of latent concepts broadens the scope of interpretation from human-defined ontologies to encompass all concepts learned within the model, and eliminates the human-in-the-loop effort for annotating these concepts. we release a comprehensive gptannotated transformers concept net (tcn) consisting of 39,000 concepts, extracted from a wide range of transformer language models. tcn empowers the researchers to carry out large-scale interpretation studies of these models. to demonstrate this, we employ two widely used techniques in the field of interpretability: probing classifiers and neuron analysis. this novel dimension of analysis, previously absent in earlier studies, sheds light on intricate aspects of these models. by showcasing the superiority, adaptability, and diverse applications of chatgpt annotations, we lay the groundwork for a more comprehensive understanding of nlp models.