Reverse Probing: Evaluating Knowledge Transfer via Finetuned Task Embeddings for Coreference Resolution (2025)

Abstract: In this work, we reimagine classical probing to evaluate knowledge transfer from simple source to more complex target tasks. Instead of probing frozen representations from a complex source task on diverse simple target probing tasks (as usually done in probing), we explore the effectiveness of embeddings from multiple simple source tasks on a single target task. We select coreference resolution, a linguistically complex problem requiring contextual understanding, as focus target task, and test the usefulness of embeddings from comparably simpler tasks tasks such as paraphrase detection, named entity recognition, and relation extraction. Through systematic experiments, we evaluate the impact of individual and combined task embeddings. Our findings reveal that task embeddings vary significantly in utility for coreference resolution, with semantic similarity tasks (e.g., paraphrase detection) proving most beneficial. Additionally, representations from intermediate layers of fine-tuned models often outperform those from final layers. Combining embeddings from multiple tasks consistently improves performance, with attention-based aggregation yielding substantial gains. These insights shed light on relationships between task-specific representations and their adaptability to complex downstream tasks, encouraging further exploration of embedding-level task transfer. Our source code is publicly available under https://github.com/Cora4NLP/multi-task-knowledge-transfer.

## introduction language models have exhibited superior performance in most areas of nlp applications, including natural language inference @xcite , question answering @xcite , commonsense reasoning @xcite , and others. since the establishment of language models with partial superhuman performance, research has aimed to pinpoint which types of knowledge are exactly encoded by such language models. one technique in the field of explainable artificial intelligence for evaluating the presence of such types of knowledge is probing @xcite @xcite . probing involves adding linear classifiers on top of representations extracted from a pre-trained model, which are trained on simple tasks for predicting a feature of choice, such as syntactic structures @xcite , entity types @xcite , or specific types of commonsense knowledge @xcite . a main intuition behind probing is to evaluate to what degree the representations that are learned from the complex source task can be re-purposed to solve a new, simpler task @xcite . in our work we decide to reverse this paradigm (thus reverse probing) and investigate how different source task embeddings, from a model trained on simple tasks, can be adapted to a new, more complex target task. in other words, we try to answer the question: can we reuse knowledge from simpler tasks for a more complex task? such a recycling of knowledge is not only interesting to deepen our understanding of what type of knowledge is encoded in language models, but it also results in more energyefficient deep learning, by reusing network weights and representations. we choose coreference resolution @xcite as our target task because solving coreference is -up to date -a challenging nlp problem that even newer large language models struggle with @xcite . coreference resolution involves understanding of context, what counts as a valid mention and which mentions refer to the same entity. solving coreference may require different types of linguistic knowledge. our goal is to find out which types of information from which source task models are useful and how this information can be combined and/or adapted to work for the target task. to isolate the effects of single tasks, we rely on small language models, in our case bert @xcite . such models do not possess sophisticated in-context abilities and require finetuning steps in order to perform well on tasks. our research questions are as follows: ## reverse probing the goal of our framework is to evaluate the transferability of knowledge embedded in representations from simpler source tasks to a complex target task. figure 1 gives an overview. let @xmath0, s 2 , . . . , s k } be a set of source tasks with models pre-trained on simpler nlp tasks, and t be the target task (coreference resolution in this case). m s is a pre-trained model fine-tuned on source task s. we then define h l s to be the output embeddings from layer l of m s . for each source task s @xmath1, we extract embeddings h s from layer l of the corresponding source task model m s (figure 1 , embedding extraction). we either take the output at a single or multiple consecutive layers. note that these may be also intermediate layers (model truncation). optionally, we apply l 2 normalization. secondly, we aggregate token embeddings from different source task models by using an aggregation function a to combine embeddings across layers and models. the aggregation is done tokenwise, so that every token can be represented as a combination of different model outputs. we define a to be either the mean of all vectors, i.e. as alternative, we use a simple attention mechanism for the combination (figure 1 , embedding aggregation): in some experiments we use only a single model. in this case the mean corresponds to the original embedding of the source model and attention simply means self-attention. next, the aggregated token embeddings are passed to the target task head that includes several layers with trainable weights (figure 1 , training target task layers which follow the coref-hoi implementation by @xcite ). figure 1 shows the probing workflow with four different source task models. each source model is pre-trained separately on a corresponding dataset as described in section 3. the models are based on bert-base-cased contextualized embeddings with different task-specific heads and their weights are frozen. given that the source models cannot update their weights during probing, our conjecture is that those models that perform better on the coreference task "out-of-the-box" contain some useful information that is relevant for the target task. ## tasks in this section we describe the target task, the source tasks and their respective training data. ## experiments and results in this section we describe our experiments with various source models and probe them on the coreference resolution task ( §4.2). we also evaluate different embedding aggregation methods ( §4.3), measure the effects of using intermediate layer output and normalization ( §4.4), vary the embedding context from several hidden layers ( §4.5) and compare the performance of multiple vs single models ( §4.6). ## related work apart from the work on probing that was presented in the introduction, our work is closely related to the idea of transfer learning @xcite , one of the ubiquitous paradigms in modern nlp. the idea of transfer learning is to train a model on a task a and then transfer the weights to a task b, either with or without further finetuning. this is the basis of most modern language models, which are pretrained and then applied or evaluated on specific downstream tasks @xcite @xcite . the pretraining data of large language models often contains samples from various natural language tasks, which renders most language models as multi-task learners @xcite . multi-task learning describes a paradigm where a model is simultaneously trained on a range of tasks. while this concept is related to the work presented here, the main difference is that several source tasks are mixed together during (pre-)training usually, which (a) mean aggregation (b) attention aggregation figure 6: single models: full vs normalized vs truncated is not the case with our work. in some sense, our work is related to research around task arithmetics @xcite @xcite , which has the goal to explicitly compute task representations in networks, e.g. as differences to a random initalization, and implement transfer learning by means of difference vector arithmetics. in contrast, our work concentrates on hidden representations, rather than the parameters of the network. ## conclusion in this project we "reversed" the classical probing and investigated how different source task embeddings contribute to a target task (coreference resolution). our experiments with paraphrase detection (mrpc), named entity recognition (ner), relation extraction (re) and extractive question answering (qa) as source tasks show there are quite different embedding representations that achieve different scores on the target task ranging from 35.63 f1 (ner) to 61.16 f1 (mrpc) for single models. moreover, we found that the best performing embeddings were typically not the outputs of the last hidden layer but rather the representations generated at lower layers. mrpc was found to be the best source model, whereas, surprisingly, ner performed the worst. we also explored different combinations of source models and found that two or more models typically outperform single ones. we considered mean and attention-based embedding aggregation methods and demonstrated the effectiveness of attention. for single models, we also showed that combining the outputs of several hidden layers instead of only one layer is beneficial. however, combining the outputs of all available layers is not necessarily a good strategy and usually the best scores can be achieved by combining only the outputs of the last 4 hidden layers that possibly contain more high-level, semantic information important for the coreference task. in the future it would be interesting to experiment with more types of embedding manipula- (a) mean aggregation (b) attention aggregation figure 8: single and combined (2x) models