T o P ro: Token-Level Prompt Decomposition for Cross-Lingual Sequence Labeling Tasks (2024)

Abstract: Prompt-based methods have been successfully applied to multilingual pretrained language models for zero-shot cross-lingual understanding. However, most previous studies primarily focused on sentence-level classification tasks, and only a few considered token-level labeling tasks such as Named Entity Recognition (NER) and Part-of-Speech (POS) tagging. In this paper, we propose Token-Level Prompt Decomposition (ToPro), which facilitates the prompt-based method for token-level sequence labeling tasks. The ToPro method decomposes an input sentence into single tokens and applies one prompt template to each token. Our experiments on multilingual NER and POS tagging datasets demonstrate that ToPro-based fine-tuning outperforms Vanilla fine-tuning and Prompt-Tuning in zero-shot cross-lingual transfer, especially for languages that are typologically different from the source language English. Our method also attains state-of-the-art performance when employed with the mT5 model. Besides, our exploratory study in multilingual large language models shows that ToPro performs much better than the current in-context learning method. Overall, the performance improvements show that ToPro could potentially serve as a novel and simple benchmarking method for sequence labeling tasks.

## introduction as multilingual pretrained language models (mplms) continue to evolve @xcite @xcite @xcite , zero-shot cross-lingual transfer methods are gaining increasing popularity within the multilingual nlp domain @xcite . please give the pos tags of the sentence: "works as stated!". the pos tags of the sentence: "works as stated!" are: ??? the pos tag of "works" is "verb". the pos tag of "as" is "cconj". the pos tag of "stated" is "verb". the pos tag of "!" is "punct". ## related work mplms and cross-lingual transfer the progress in mplms has established them as the basis for cross-lingual transfer. mplms typically adopt the architecture of monolingual transformerbased language models and are pretrained on extensive unlabeled multilingual corpora. examples of mplms are mbert @xcite , xlm-r @xcite , mt5 @xcite , glot500 @xcite , etc. empirical studies @xcite have showcased the remarkable crosslingual prowess of mplms which are fine-tuned on english training datasets, and then used to predict on test datasets in other languages. several benchmark datasets such as xtreme @xcite , xtreme-r @xcite , and taxi1500 @xcite have been created to assess the capabilities of multilingual models. the increasing popularity of prompt learning has drawn the attention of researchers towards prompt-based methods for cross-lingual transfer @xcite . diverging from previous studies centered on sentence-level classification tasks, our work applies prompt-based fine-tuning to tokenlevel sequence labeling tasks. mllms and in-context learning bloomz and mt0 @xcite stand out as two representative multilingual models in the era of llms. both are fine-tuned on the xp3 dataset which contains multi-lingual multi-task prompts. bloomz is built upon bloom @xcite while mt0 is built upon mt5 @xcite . @xcite demonstrated that llms like gpt-3 can acquire task-solving icl abilities. the emergence of mllms opens up the possibility for conducting zero-shot cross-lingual icl, as demonstrated by recent benchmarking efforts, for example mega @xcite and buffet @xcite . however, current icl methods using text-to-text prompting with a fixed output template for sequence labeling tasks "consistently exhibit extremely poor performance" @xcite when applied to mllms, failing to exploit their real cross-lingual transfer abilities. contrary to that, our proposed topro method better reflects the potential of mllms on token-level tasks. ## topro for fine-tuning problem formulation in prompt-based learning, there is a pattern-verbalizer pair (pvp) @xcite ) consisting of (i) a prompt pattern which converts the input text into a cloze-style question with a mask token, and (ii) a verbalizer which maps the labels onto representative words from the lm's vocabulary. this aligns well with the nature of text classification tasks where one label is predicted based on the input text. as figure 2 shows, the input text x of a sentiment analysis task can be reformulated with a prompt pattern p (•) into a prompted input representation p (x) = " works as stated! in summary, the product was @xcite . " the prompt p (x) is processed by the lm to determine the most likely verbalizer word in the masked position. the label corresponding to this verbalizer is the prediction which is evaluated against the gold standard. however, in sequence labeling tasks, each token of the input should receive a label. thus, it is not possible to apply this type of prompt pattern with one mask token directly for token classification. ## conclusion in our work, we introduce topro for token-level sequence labeling tasks, a novel and simple method that adopts the basic framework of prompting from sentence classification tasks and applies the prompt template to each token in a sentence. we evaluate the topro-based fine-tuning for zero-shot cross-lingual transfer and compare it to vanilla finetuning and prompt-tuning baselines. we apply topro with three mplms on two representative sequence labeling tasks: ner and pos tagging. our experiments show that topro outperforms the baselines with the mplms and achieves sota performance with mt5. we further discovered that the performance improvement of topro is generally more obvious in the cross-lingual context, especially for languages that are linguistically very different from the source language english, highlighting its cross-lingual ability. additionally, we applied the topro method to mllms and noticed better performances of topro as well, compared to existing benchmarking work. overall, topro shows a noticeable performance improvement and could serve as a potential benchmark for sequence labeling tasks for future studies in prompt-based learning.