SOI Matters: Analyzing Multi-Setting Training Dynamics in Pretrained Language Models via Subsets of Interest (2025)

Abstract: This work investigates the impact of multi-task, multi-lingual, and multi-source learning approaches on the robustness and performance of pretrained language models. To enhance this analysis, we introduce Subsets of Interest (SOI), a novel categorization framework that identifies six distinct learning behavior patterns during training, including forgettable examples, unlearned examples, and always correct examples. Through SOI transition heatmaps and dataset cartography visualization, we analyze how examples shift between these categories when transitioning from single-setting to multi-setting configurations. We perform comprehensive experiments across three parallel comparisons: multi-task vs. single-task learning using English tasks (entailment, paraphrase, sentiment), multi-source vs. single-source learning using sentiment analysis datasets, and multi-lingual vs. single-lingual learning using intent classification in French, English, and Persian. Our results demonstrate that multi-source learning consistently improves out-of-distribution performance by up to 7%, while multi-task learning shows mixed results with notable gains in similar task combinations. We further introduce a two-stage fine-tuning approach where the second stage leverages SOI-based subset selection to achieve additional performance improvements. These findings provide new insights into training dynamics and offer practical approaches for optimizing multi-setting language model performance.

## introduction deep learning has revolutionized natural language processing (nlp), with transformer-based models @xcite achieving remarkable success across various tasks. these architectures primarily fall into two categories: decoder-only models, such as gpt-2 @xcite , and encoder-only models, including bert @xcite and roberta @xcite . recently, decoder-only large language models (llms) have gained prominence, notably following the success of chatgpt. while newer opensource llms, such as the llama3 family @xcite , facilitate human-friendly interactions across diverse tasks, they do not consistently outperform traditional models @xcite . furthermore, current benchmarks for evaluating llms emphasize general capabilities such as comprehension and reasoning, frequently neglecting specialized nlp tasks like text classification and named entity recognition. recent research by @xcite indicates that smaller, fine-tuned encoderonly pretrained language models (plms), such as roberta, can match or exceed the performance of larger llms across various specialized datasets. although closed-source llms like gpt-4o @xcite can occasionally surpass plms with extensive prompt engineering, smaller open-source models offer substantial advantages regarding speed, cost-effectiveness, and transparency. therefore, systematic analyses of plm training dynamics remain crucial, even as llms increasingly dominate nlp research. motivated by this, we systematically investigate the impact of multi-task, multi-lingual, and multi-source learning approaches on the robustness and performance of plms. multi-task learning, which leverages shared knowledge across related tasks, has shown considerable promise for enhancing model generalization and robustness, particularly under constraints of limited data and computational resources. concurrently, multi-source learning exploits diverse data origins to provide models with a broader understanding of target problems, while multi-lingual learning enables the acquisition of language-agnostic knowledge, significantly improving cross-lingual transfer and performance on low-resource languages. despite these advances, a key challenge in training plms is handling forgettable examples, samples that are difficult or out-of-scope, leading models to frequently oscillate between correct and incorrect predictions during training. although fine-tuning on these challenging examples has proven beneficial in enhancing model robustness @xcite , systematic analysis of their underlying learning patterns is currently lacking. to address this gap, we introduce subsets of interest (soi), a novel framework for categorizing dataset samples based on distinct learning behaviors observed during training. specifically, soi consists of six categories: unlearned examples (une), always correct examples (ace), 1time forgettable examples (1t-f rge), at least 2-times forgettable examples (≥ 2t-f rge), early-learned examples (ele), and late-learned examples (lle). collectively, these subsets enable detailed insights into the dynamics of model learning behaviors under single-or multisetting configurations, spanning different tasks, languages, and sources. furthermore, we investigate the potential of soi subsets to enhance out-ofdistribution performance through second-stage finetuning strategies based on various soi combinations. the key contributions of this work are as follows: first, we introduce the soi framework, systematically classifying training samples into distinct learning behavior subsets (section 4.1). second, we visualize model learning dynamics via dataset cartography and soi transition heatmaps, offering intuitive insights into sample-level training behaviors (sections 4.2 and 4.3). third, we provide a comprehensive comparative analysis of multi-task, multi-lingual, and multi-source learning methods, evaluating their impacts on both in-distribution (id) and out-of-distribution (ood) performances (subsection 5.1). lastly, we extend our ood evaluations through second-stage fine-tuning on strategically chosen subsets derived from soi analyses, demonstrating additional performance gains (subsection 5.2). 2 2 our code is publicly available at this github repository. it builds upon the implementation provided here, adapting hugging face's transformers library for multi-setting training. ## related work in recent years, extensive research has focused on developing multilingual models as well as models capable of performing multiple nlp tasks simultaneously. multi-task learning leverages shared representations to jointly optimize model performance across various related tasks, enhancing model generalization, robustness, and computational efficiency. early foundational work by @xcite introduced multi-task learning concepts to nlp, illustrating that training multiple tasks concurrently could lead to better feature generalization and more robust representations. subsequent studies have widely adopted transfer learning techniques @xcite , demonstrating how pretrained language model knowledge can significantly enhance performance on various downstream nlp tasks. multi-lingual learning, another promising direction, enables models to gain language-agnostic knowledge to understand, generate, and generalize textual information across multiple languages. @xcite introduced xlm-r, a robust cross-lingual plm trained on diverse multilingual data, significantly improving performance on low-resource languages and facilitating effective cross-lingual transfer. understanding model behavior at the individual example level represents another critical aspect in training language models. the phenomenon of example forgetting, instances where models oscillate between correct and incorrect predictions during training, has been thoroughly investigated by @xcite . their work demonstrated that fine-tuning models specifically on these challenging, forgettable examples can significantly enhance model robustness and generalization on task-specific ood datasets. complementary to this perspective, @xcite proposed dataset cartography, a visualization technique characterizing training samples based on prediction confidence and variability metrics. their method categorizes data into easy-to-learn, hard-to-learn, and ambiguous regions, providing intuitive insights into model behavior throughout training. they conclude that training the model from scratch on the ambiguous region achieves the best id and ood performances compared to other scenario cases, including training on hard-to-learn and forgetting examples. inspired by these foundational works, our study introduces the subsets of interest (soi) framework, extending beyond previous categorizations with a finer-grained, analytical perspective. instead of limiting analysis to three regions, soi systematically classifies training examples into six distinct learning subsets based on their dynamic behaviors during training. our comprehensive categorization enriches existing analytical tools, offering nuanced insights into model ood generalization capability across various multi-task, multi-lingual, and multisource training scenarios. ## experiments setup in this section we introduce three parallel experimental comparisons: multi-task vs. single-task learning, multi-source vs. single-source learning, and multi-lingual vs. single-lingual learning. for each comparison, we conducted similar experiments to evaluate both performance and generalizability. our experimental framework encompasses various tasks, languages, datasets, and a unified model architecture detailed below. ## subsets of interest in this section, we present a comprehensive framework for analyzing deep learning models through the lens of training dynamics. we introduce the concept of subsets of interest (soi), a novel categorization system that partitions training examples based on their unique learning patterns observed during the training process. our analysis unfolds in three complementary parts: first, we formally define the six distinct soi categories and their characteristics; second, we employ dataset cartography to visualize how these subsets manifest in the confidence-variability space; and third, we introduce transition heatmaps to track how examples migrate between soi categories under different training configurations. together, these components provide a systematic approach to understanding and analyzing the complex dynamics of neural network training. ## results & analyses in this section, we present our experimental results and analyses, focusing on the impact of different multi-setting configurations on id and ood performance, relative to their corresponding singlesetting baselines. ## conclusion & future work in this work, we conducted a comprehensive investigation into the effects of multi-task, multi-source, and multi-lingual training on plms, emphasizing the learning dynamics through the introduction of soi. by leveraging soi transition heatmaps and dataset cartography, we provided novel insights into how different training configurations influence both id and ood performance. our results reveal that multi-source learning consistently enhances ood generalization, while multi-task and multilingual learning exhibit more nuanced behavior, offering benefits primarily when task or language similarities exist. the proposed two-stage fine-tuning approach, particularly when guided by soi-based sample selection, showed further gains in ood performance, especially in multi-task settings. to sum up, our work highlights the potential of multisetting configurations in creating more adaptable, robust plms capable of generalizing across tasks, languages, and sources. while our study focused on encoder-based plms, future work could apply the soi framework to large decoder-based language models, such as gpt-style models, to gain insights into their training behaviors and generalization capabilities. additionally, expanding beyond pairwise combinations to train models on multiple (three or more) tasks, sources, or languages simultaneously could provide a deeper understanding of scaling trends in multi-setting learning. another direction involves investigating curriculum learning strategies where training is staged according to soi categories. tables 1, 2, and 3 summarize experimental results from initial (first stage) and soi-guided fine-tuning (second stage). the initial fine-tuning trains plms separately (single-setting) or jointly (multi-setting) on id datasets; id columns report in-distribution evaluations, while ood columns show out-of-distribution performance. the soi-guided fine-tuning (second stage) further optimizes multi-setting models using targeted subsets strategically selected via soi transition heatmaps (section 5.2.1), with improvements measured under the second-stage ood columns. to interpret these tables, first compare single-setting to multi-setting performances from the initial fine-tuning, then evaluate the additional gains obtained from the subsequent soi-guided (second stage) fine-tuning. of heatmaps for reference: figures 7, 8, and 9 show how samples transition happen between different soi categories when moving from single-mode to multi-mode learning. each cell indicates the number of samples that moved from one category to another, with rows representing the initial (singlemode) categories and columns showing the final (multi-mode) categories.