MT 2 ST : Adaptive Multi-Task to Single-Task Learning (2025)

Abstract: We propose MT2ST , a general and efficient framework for accelerating multi-task training by progressively transitioning to single-task optimization. Unlike conventional multi-task learning (MTL) or single-task fine-tuning (STL), MT2ST dynamically adjusts the training focus via two complementary strategies: Diminish , which gradually down-weights auxiliary losses, and Switch , which explicitly switches to the primary task at a scheduled point. We demonstrate the effectiveness of MT2ST across three key paradigms: representation learning, transformers, and diffusion models, covering both unimodal (text/image) and multimodal (vision-language) tasks. Extensive experiments show that MT2ST significantly improves training efficiency—achieving up to 56% FLOPs compression—while maintaining or surpassing task performance. These results suggest MT2ST as a general-purpose solution for scalable and adaptive multi-task training. Although this work is general-purpose, it is especially suitable for multimodal settings such as VQA or vision-language retrieval, where auxiliary pretraining (e.g., masked language modeling or contrastive learning) often diverges from final objectives. We include a VQA case study and outline its efficiency for multimodal retrieval.

## introduction the rapid evolution of large-scale models in machine learning (ml), particularly in natural language processing (nlp), computer vision (cv), and speech recognition, has brought tremendous advances in task performance but also increased the demand for computational efficiency. as models grow in parameter size and data requirements, efficient training strategies have become indispensable for scalable deployment and practical adaptation. among these, the training of taskspecific embeddings remains a fundamental com-ponent, serving as the backbone for semantic representation in both unimodal and multimodal applications @xcite . a major trade-off emerges in the choice of training paradigm: single-task learning (stl) vs. multi-task learning (mtl). stl enables highfidelity adaptation to a specific task objective, often yielding superior precision. however, it lacks inductive bias and representation reuse, limiting generalization. in contrast, mtl introduces auxiliary tasks that can guide shared representation learning, promoting robustness and faster convergence, especially in low-resource regimes @xcite . nevertheless, mtl is not without cost: task interference, gradient conflict @xcite , and heterogeneous learning dynamics can degrade both convergence speed and final task performance @xcite @xcite . to address this dilemma, we propose the multi-task to single-task (mt2st) framework-an adaptive training strategy that combines the strengths of mtl and stl by dynamically shifting the training focus from a multi-task setup to a single-task objective. as illustrated in figure 1 , mt2st is based on a key insight: shared learning in the early stages of training helps build generalized representations, but over time, specialization is necessary to maximize performance on the main task. mt2st incorporates two strategies for controlling this transition: our approach is simple, lightweight, and does not require architecture modifications, making it compatible with most encoder-decoder or encoder-only models. furthermore, mt2st is domain-agnostic: although demonstrated on word embedding learning, its core principles apply naturally to image embeddings, multimodal fusion models, and task-specific adaptation in recommendation or healthcare systems. we conduct comprehensive experiments showing that mt2st significantly reduces training time while improving or preserving performance. in particular, mt2st achieves up to 67% training speed-up over stl and 13% over conventional mtl on embedding tasks, all while maintaining competitive accuracy. these results suggest that mt2st can be a general-purpose mechanism for efficient task-oriented representation learning. contributions to summarize, our contributions are as follows: ## mt2st deployment in this section, we formally describe how mt2st is deployed across three representative paradigms: representation learning, transformer-based architectures, and diffusion models. we focus on the formulation of adaptive learning weights γ k (t) and present unique integration strategies in each context. to avoid redundancy, core mechanisms such as task weighting decay and switching dynamics already discussed in §3 are omitted. ## experiments and applications we evaluate the proposed mt2st framework to answer the following research questions: q1: how do the diminish and switch strategies impact training efficiency and performance? q2: what are the effects of mt2st across various models and architectures? q3: can mt2st generalize across modalities such as vision, text, and multimodal systems? ## conclusion in this work, we propose mt2st, a general and adaptive multi-task to single-task training framework designed to accelerate model convergence while preserving or even improving final task performance. mt2st introduces two complementary strategies-diminish and switch-that enable smooth or staged transitions from multi-task sharing to single-task specialization. we evaluate mt2st across a wide spectrum of models and modalities, including classical representation learners, transformer-based architectures, and diffusion models. empirical results on text, image, and multimodal tasks show that mt2st consistently improves accuracy while reducing training time and computational overhead. our analysis highlights mt2st as a practical and modular framework for efficient optimization across diverse ai systems. our method is especially relevant to multimodal learning problems such as visual question answering (vqa) or cross-modal retrieval, where auxiliary objectives like masked language modeling or contrastive imagetext alignment are commonly used but often misaligned with the downstream task. mt2st provides a principled way to leverage such auxiliary tasks without compromising task specialization.