Statement-Tuning Enables Efficient Cross-lingual Generalization in Encoder-only Models (2025)

Abstract: Large Language Models (LLMs) excel in zero-shot and few-shot tasks, but achieving similar performance with encoder-only models like BERT and RoBERTa has been challenging due to their architecture. However, encoders offer advantages such as lower computational and memory costs. Recent work adapts them for zero-shot generalization using Statement Tuning, which reformulates tasks into finite templates. We extend this approach to multilingual NLP, exploring whether encoders can achieve zero-shot cross-lingual generalization and serve as efficient alternatives to memory-intensive LLMs for low-resource languages. Our results show that state-of-the-art encoder models generalize well across languages, rivaling multilingual LLMs while being more efficient. We also analyze multilingual Statement Tuning dataset design, efficiency gains, and language-specific generalization, contributing to more inclusive and resource-efficient NLP models. We release our code and models.

## introduction large language models (llms) have shown great capabilities in zero-shot and few-shot settings @xcite @xcite . however, these capabilities are often more difficult to observe in encoder-only models like bert @xcite and roberta @xcite due to their architectural design. these models are typically pretrained with a masked language modeling (mlm) objective and are finetuned by adding task-specific layers to enable their usage on a downstream task. these task-specific layers block the extension of these models to new tasks in a few-shot or zero-shot manner. despite these difficulties, applying encoder models for zero-shot task generalization offers several advantages. first, encoder models are typically more lightweight than llms and therefore require less computational power and memory. further, encoder models excel at generating contextual embeddings that better capture semantic meaning. for instance, recent work @xcite demonstrated that decoder-only llms perform worse on word meaning comprehension than encoder-only models. encoder-only models are also more efficient in inference for tasks such as sequence labeling due to their architecture @xcite . to enable encoder model usage in zero-shot task generalization, @xcite introduced statement-tuning. this technique converts tasks into a set of statements with finite templates, training on an encoder-only model, roberta, to discriminate between potential statements and derive final results. this method demonstrates the feasibility of using encoder models, typically specialized for specific tasks, to handle various unseen natural language understanding (nlu) tasks, similar to zero-shot prompting in decoder models. it shows competitive performance compared to large language models (llms) with significantly fewer parameters and training data, highlighting its potential for zero-shot learning. however, the original approach focused only on english, raising questions about its applicability in multilingual settings and its ability to generalize to new tasks and languages. in this work, we aim to explore the adaptation of statement-tuning for multilingual nlp tasks. specifically, we investigate whether encoder models can achieve zero-shot cross-lingual generalization similar to decoder-only llms @xcite . given the multilingual setting, it is crucial to emphasize the importance of lowcompute solutions. speakers and users of lowresource languages often lack the computational resources necessary to utilize memory-intensive llms. therefore, our method, which leverages efficient encoder models, is particularly important as it offers a more accessible and inclusive approach to zero-shot text classification in these contexts @xcite . hence our contributions are as follows: finally, @xcite propose statement-tuning to show that through templatebased data augmentation, much smaller roberta models can be finetuned on limited data to match or even exceed the zero-shot nlu capabilities of several llms of up to 70b parameters on monolingual classification tasks. while elshabrawy et al. (2025) focused only on english, we studied whether statement-tuning method is possible in other languages. additionally, we explore the efficiency of the approach in more detail and offer insight on the effect of pretraining data on the performance of statement-tuning. ## method: multilingual statement-tuning in this section, we outline the steps involved in statement-tuning. ## results and analysis in this section, we derive insights from our experimental results about the cross-lingual zero-shot generalization capabilities of encoder models. ## conclusion while large generative models dominate multilingual nlp, the potential of encoder-only models for cross-lingual generalization remains underexplored. we show that a well-designed finetuning setup enables state-of-the-art pretrained encoderonly models to match, or even surpass, generative models in three of four unseen cross-lingual nlu tasks, despite using far fewer parameters. additionally, these models generalize across languages even when finetuned only on a monolingual multitask dataset, leveraging their multilingual pretraining. our findings position encoder-only models as a memory-efficient alternative for multilingual multitask nlu. future work can further optimize finetuning, extend cross-lingual generalization, and refine encoder architectures for large-scale multilingual learning.