Enhancing Small Language Models for Cross-Lingual Generalized Zero-Shot Classification with Soft Prompt Tuning (2025)

Abstract: In NLP, Zero-Shot Classification (ZSC) has become essential for enabling models to classify text into categories unseen during training, particularly in low-resource languages and domains where labeled data is scarce. While pretrained language models (PLMs) have shown promise in ZSC, they often rely on large training datasets or external knowledge, limiting their applicability in multilingual and low-resource scenarios.Recent approaches leveraging natural language prompts reduce the dependence on large training datasets but struggle to effectively incorporate available labeled data from related classification tasks, especially when these datasets originate from different languages or distributions. Moreover, existing prompt-based methods typically rely on manually crafted prompts in a specific language, limiting their adaptability and effectiveness in cross-lingual settings.To address these challenges, we introduce RoSPrompt, a lightweight and data-efficient approach for training soft prompts that enhance cross-lingual ZSC while ensuring robust generalization across data distribution shifts. RoSPrompt is designed for small multilingual PLMs, enabling them to leverage high-resource languages to improve performance in low-resource settings without requiring extensive fine-tuning or high computational costs. We evaluate our approach on multiple multilingual PLMs across datasets covering 106 languages, demonstrating strong cross-lingual transfer performance and robust generalization capabilities over unseen classes.

## introduction zero-shot classification (zsc) is a task in nlp where a model classifies inputs into classes that it has not seen during training. this task is crucial in real-world scenarios where some classes are underrepresented with little or no labeled data. traditionally, two approaches have dominated the landscape: entailment-based and similarity-based approaches. entailment-based approaches @xcite focus on understanding relationships between sentences, particularly determining the level of entailment between the document and the potential class labels. this method requires the model to have a deep understanding of language structure and logic. on the other hand, similarity-based approaches focus on computing the similarity between the input and labels of each class, even if the model has never encountered them during training. this method often relies on embeddings or vector representations of text, allowing the model to make inferences based on how closely the input aligns with class descriptors @xcite . however, these methods face inherent drawbacks, as they depend on natural language inference or semantic text similarity datasets that require considerable effort to develop and are susceptible to potential biases @xcite . in light of this, and with the acknowledgment of the extensive knowledge embedded in general pre-trained language models (plms) and the potential to extract it, a novel paradigm has arisen: prompting. prompting reformulates a task as a cloze-style task using a natural language prompt, retrieves the model's masked or next token prediction, and maps it to the right class via a verbalizer, while requiring little to no training data. nevertheless, traditional prompting methods are hindered not only by manual effort and inherent biases of the individuals creating the prompts and verbalizers, but also by other factors such as the order of examples in the prompt during in-context learning @xcite . to address this, @xcite developed an automated system for generating prompts and verbalizers using a limited number of training samples. furthermore, @xcite introduced a technique that eliminates the need for training data by automatically creating a verbalizer using an external knowledge base. motivated by the goal of elim-inating the need for any additional data, @xcite proposed a method that forms a verbalizer using only the plm's embedding space, without requiring any training data or external knowledge base. this approach, while efficient and effective in various zsc tasks, shares a limitation with the methods of @xcite and @xcite : it relies on language-specific prompts which introduce a language bias, making the method less effective in multilingual scenarios. moreover, despite the high efficiency and appeal of methods that operate without existing data, their inability to leverage even a minimal amount of available data from a similar classification task in a high-resource language, can be seen as a significant limitation in our data-abundant world. to address these shortcomings, we suggest to transform the language-specific hard prompts into trainable soft prompts @xcite , which can then be fine-tuned. however, directly adopting the conventional soft prompt tuning (spt) setup leads to overfitting on the seen classes ( §7), therefore, does not generalize under data distribution shifts. in response to this constraint, we introduce robust soft prompts (rosprompt), a novel method for cross-lingual zero-shot topic classification through few-shot spt, which exhibits robust out-of-distribution generalization and strong crosslingual transfer performance. rosprompt not only retains the efficiency and effectiveness of leveraging the knowledge of plms but also enhances it by incorporating small sets of existing data. by doing so, we aim to broaden the applicability of zsc in a multilingual context, ensuring more accurate topic classification across diverse languages and datasets. specifically, our approach (a) enables the training of soft prompts, which are better suited for zsc tasks compared to hand-crafted, natural language hard prompts. to showcase the efficacy of our proposed approach, we conduct a comprehensive evaluation using three distinct types of multilingual language models (encoder-only, decoder-only, and encoderdecoder) and three diverse datasets, encompassing 106 languages, thereby highlighting the versatility and applicability of our method in cross-lingual scenarios. given an input sequence x and the set of c potential classes c, we define the two main components of spt: if we denote the function performed by the model as f , with its parameters θ (which are frozen during spt), the logits over the vocabulary v for the next token in the sequence are given by: the predicted class will then be ŷ = arg max @xmath0, even without task-specific fine-tuning. their technique, npprompt, involves adding a natural language prompt to the input example, prompting the model to fill in the [mask] for bert-based models, or predict the next token for autoregressive and seq2seq models, which are then used for the final classification of the sample. nevertheless, their strategy is primarily designed for english, as the prompts employed are in english. applying their method to additional languages would necessitate the engineering of new prompts specific to those languages. furthermore, despite the appeal of their zero-shot framework, particularly when there is a lack of finetuning data, it falls short by not accommodating the use of existing labeled data when it is available. therefore, we suggest to extend their method by transforming the natural language prompt into a trainable soft prompt @xcite , enabling its training through any available topic classification data in the source language for subsequent zero-shot topic classification in any target language. ## rosprompt we describe our technique as a hybrid of spt @xcite and npprompt @xcite . spt excels in data efficiency but is sensitive to data distribution shifts, needing unique prompts for each topic classification dataset. on the other hand, npprompt uses one prompt for various data distributions but fails to leverage existing data. our strategy combines their strengths, using a single, robust soft prompt for different data distributions and enhancing data utilization (figure 1 ). train set 1 train set n . . . test set 1 test set n . . . train set test set 1 test set n . . . soft prompt 1 soft prompt n soft prompt test set 1 test set n ## experimental setup we provide a general description of the datasets for training and evaluation, along with the models used in our experiments. further specific details about the experimental setup can be found in appendix a. ## results for each of the three models, our experimental findings are presented in additionally, we observe that while larger generative llms slightly outperform the smaller rosprompt-enhanced llms on high-resource languages, they significantly underperform, often worse than the random baseline, on low-resource languages, highlighting the effectiveness of our method in such scenarios. ## ablation study to illustrate the individual contributions of each component in our training method, we carry out an ablation study. we assess the efficacy of our original method against variants lacking the loss penalty, contrastive label smoothing, and/or multilingual labels. the outcomes of this study, presented in this could potentially be attributed to xlm-r's enhanced code-switching capabilities @xcite , making it more efficient at using multilingual label tokens during training compared to xglm and mt0. ## generalized zero-shot learning in our initial experiments, training (seen) and evaluation (unseen) classes were distinct with merely minimal overlap. in contrast, the generalized zero-shot learning (gzsl) settings, which mirror realworld situations more closely, involve evaluating on a mix of both seen and unseen classes. models in this setting often struggle with overfitting to seen classes and fail to perform well on unseen classes @xcite . therefore, we aim to investigate whether our method is also efficient under gzsl settings. for this, we fine-tune the soft prompt on a subset of classes from a dataset, then test it on the entire set of classes. considering the potential variability resulting from the specific choice of seen and unseen classes, we repeat this process four times for each dataset and model, each time with a different subset of seen classes. we then average the f1 scores for seen and unseen classes and present them in ## contextualizing our approach in this study, we acknowledge that comparing our approach with npprompt may not constitute an entirely fair comparison. rosprompt uses a small dataset for training, while npprompt directly leverages a plm without additional fine-tuning. however, it is important to emphasize that the intent of our research is not to demonstrate rosprompt's performance superiority over npprompt. instead, our objective is to illustrate how rosprompt's methodology can effectively improve cross-lingual transfer capabilities of natural language prompts. this aspect is vital as our findings indicate that merely converting hard prompts to soft prompts and then fine-tuning them using the standard spt approach results in non-robust prompts which are ineffective for generalized zsc. additionally, while our paper focuses on topic classification, we believe that our approach could be equally effective for other types of classification tasks as well. nonetheless, we emphasize the significance of zero-shot learning in topic classification, where classes often change more frequently over time or across domains, unlike in more stable tasks like sentiment analysis, where classes show less variation. furthermore, we want to emphasize the threefold efficiency of our approach: 1) it is data efficient, requiring only a small number of labeled training samples from any comparable classification task; 2) it is computationally efficient as fewer than 0.1% of parameters are fine-tuned compared to full-model fine-tuning, reducing training time by approximately 50% in our experiments; 3) it is memory-efficient, as for n training processes, besides the resulting n prompts that take up a few hundred kbs at most, only one model copy is stored, in contrast to full-model fine-tuning where each model occupies several gbs of storage. moreover, while our method is theoretically applicable to larger models with billions of parameters, our primary target is smaller llms, which are often sufficient for tasks like zero-shot classification but need more focused guidance. these smaller multilingual models also excel in lowresource languages, where larger english-centric models, as we demonstrate, are less effective. ## conclusion in this paper, we introduced rosprompt, a novel approach for cross-lingual zero-shot topic classification. it combines the advantages of few-shot spt with the extensive knowledge acquired by language models in their pre-training phase. our training method is designed for computational efficiency and incorporates three key components to enhance the standard spt methodology, contributing to rosprompt's cross-lingual abilities and resilience to data distribution shifts.