Comparing Prompt-Based and Standard Fine-Tuning for U rdu Text Classification (2023)

Abstract: Recent advancements in natural language processing have demonstrated the efficacy of pre-trained language models for various downstream tasks through prompt-based fine-tuning. In contrast to standard fine-tuning, which relies solely on labeled examples, prompt-based fine-tuning combines a few labeled examples (few shot) with guidance through prompts tailored for the specific language and task. For low-resource languages, where labeled examples are limited, prompt-based fine-tuning appears to be a promising alternative. In this paper, we compare prompt-based and standard fine-tuning for the popular task of text classification in Urdu and Roman Urdu languages. We conduct experiments using five datasets, covering different domains, and pre-trained multilingual transformers. The results reveal that significant improvement of up to 13% in accuracy is achieved by prompt-based fine-tuning over standard fine-tuning approaches. This suggests the potential of prompt-based fine-tuning as a valuable approach for low-resource languages with limited labeled data.

## introduction recent advancements in natural language processing (nlp) have highlighted the efficacy of pretrained language models (plms) in various downstream tasks, including text classification. plms, such as @xcite and @xcite , have revolutionized nlp by pre-training on extensive textual data to acquire language understanding and common knowledge. however, optimizing the performance of models for various languages, especially low-resource languages like urdu and roman urdu, presents challenges that need to be addressed. standard fine-tuning usually requires large amounts of taskspecific labeled examples to adapt the parameters of plms for robust performance. more recently, prompt-based fine-tuning @xcite has emerged as a promising alternative for improving classification accuracy in low-resource language contexts @xcite @xcite @xcite . prompt-based fine-tuning combines a small set of annotated examples (few shot) with carefully designed language-specific prompts tailored to the task. these prompts explicitly guide the models, providing crucial context and information for precise predictions. this paper presents an empirical evaluation of prompt-based fine-tuning with traditional standard fine-tuning for text classification in urdu and roman urdu. the objective is to determine whether prompt-based fine-tuning surpasses the performance of standard fine-tuning of multilingual plms in few shot setting. to do this, we conduct experiments on five diverse datasets spanning different domains and encompassing various classification tasks. these datasets are carefully selected to represent the challenges and nuances of urdu and roman urdu text classification. additionally, we utilize three pre-trained multilingual transformers (1) bert-multilingual @xcite , (2) distilbert @xcite , and (3) xlm-roberta @xcite . these transformers have been widely used and proven effective in various nlp tasks across multiple languages. by incorporating numerous transformers, we aim to evaluate the robustness and generalizability of prompt-based fine-tuning across different architectures and language models. our findings reveal a significant improvement of up to 13% in classification accuracy achieved by prompt-based fine-tuning over traditional approaches. this improvement highlights the effectiveness of prompt-based methods in capturing the complex linguistic characteristics and nuances inherent in urdu texts and demonstrates its potential as a promising alternative for low-resource languages where limited labeled data is available. to the best of our knowledge, there are no published works on prompt engineering or prompt-based finetuning for urdu text classification. our work comparing prompt and standard fine-tuning in urdu can seed further research in this direction. ## literature review in the past few years, social media platforms have experienced an enormous surge in users. with the rise in digital media usage, there is an increasing demand for automated text classification in urdu. recently, various transfer learning and data augmentation approaches have been investigated for text classification, such as those described by @xcite @xcite . these methods use pre-trained language models and fine-tune them on smaller datasets to enhance their performance on specific tasks. however, a recent technique in natural language processing called prompt engineering has recently gained attention from researchers @xcite . designing prompts to guide language models improves model predictions by effectively utilizing contextual information. this technique achieves better results with minimal data, including short learning or zero-shot learning. various types of prompts are used for text classification, namely human designed prompts @xcite , schema prompts @xcite , and null prompts @xcite . while prompt engineering research has primarily focused on the english language, there has been recent work exploring its effectiveness in other languages. for example, @xcite conducted research to determine if prompt engineering could improve text classification in chinese. their results showed that the use of prompts yielded positive results. similarly, @xcite applied prompt engineering techniques to korean and found that it improved performance for various text classification tasks, including topic and semantic classification, even with few-shot learning. however, no known work has been done to study prompt learning for the urdu language. ## experimental design in this section, we describe the datasets and the experimental setup employed to compare the performance of prompt-based and standard fine-tuning approaches for text classification in urdu. fine-tuning models on limited amounts of labeled data can introduce instability in execution and result in substantial performance variations depending on the choice of data splits @xcite . to generate robust results, we adopt a careful and comprehensive approach, as outlined below. we aim to fine-tune a pre-trained language model l (standard and prompt-based) on task d with label space y for urdu and roman urdu text classification. our goal is to develop effective learning strategies that generalize well to an unseen test set (x test in , y test ) ∼ d test . in the fewshot setting, we have limited training examples per class. let k denote the number of training examples per class, and |y | denote the total number of classes in the task. thus, the few-shot training set d train consists of k @xmath0, where now, given the language model l, our process begins by converting the input x in into a token sequence x, which is then mapped to a sequence of hidden vectors h @xmath1. for example, in a binary sentiment classification task, we can construct a prompt using the input @xmath2. the prompt formulation would be in the literature, various templates are utilized for prompt-based classification tasks @xcite . however, we find that "yeh [mask] hai." template is performing better for urdu and roman urdu. we do not mention the results for other templates due to lack of space in the paper. the language model l is then responsible for determining whether it is more suitable to fill in the [mask] position with "khubsurat" (beautiful) or "fazool" (useless) as depicted in figure 1 . this prompt-based methodology enables the model to autonomously complete prompts and make sentiment/classification predictions, enabling more efficient and accurate classification. to construct the training and development sets for each dataset, we select k labeled examples cls head yeh jaga bohat pyari hai. [sep] [sep] . ‫ایپری‬ ‫تہب‬ ‫ہگج‬ ‫ہی‬ ‫ےہ‬ label: positive label: negative 𝐿𝑎𝑏𝑒𝑙 𝑠𝑝𝑎𝑐𝑒 𝑌 [cls] yeh jaga bohat pyari hai. yeh hai. [sep] [sep] ‫ےہ‬ ‫ہی‬ . ‫ہی‬ ‫ہگج‬ ‫تہب‬ ‫ایپری‬ ‫ےہ‬ [cls] 𝐿𝑎𝑏𝑒𝑙 𝑚𝑎𝑝𝑝𝑖𝑛𝑔 𝑀(𝑌) input template (a) standard fine-tuning (b) prompt-based fine-tuning input template [cls] mlm head mlm head [mask] [mask] (label: positive) ‫وخوصبرت‬ (label: negative) ‫وضفل‬ khubsurat (label: positive) fazool (label: negative) [cls] cls head yeh jaga bohat pyari hai. [sep] [sep] . ‫ایپری‬ ‫تہب‬ ‫ہگج‬ ‫ہی‬ ‫ےہ‬ label: positive label: negative 𝐿𝑎𝑏𝑒𝑙 𝑠𝑝𝑎𝑐𝑒 𝑌 [cls] yeh jaga bohat pyari hai. ye hai. [sep] [sep] ‫ےہ‬ ‫ہی‬ . ‫ہی‬ ‫ہگج‬ ‫تہب‬ ‫ایپری‬ ‫ےہ‬ [cls] 𝐿𝑎𝑏𝑒𝑙 𝑚𝑎𝑝𝑝𝑖𝑛𝑔 𝑀(𝑌) input template (a) standard fine-tuning (b) prompt-based fine-tuning input template [cls] mlm head mlm head [mask] [mask] (label: positive) ‫وخوصبرت‬ (label: negative) ‫وضفل‬ khubsurat (label: positive) fazool (label: negative) [cls] : "this place is very beautiful."), ( cls head yeh jaga bohat pyari hai. [sep] [sep] . ‫ایپری‬ ‫تہب‬ ‫ہگج‬ ‫ہی‬ ‫ےہ‬ label: positive label: negative 𝐿𝑎𝑏𝑒𝑙 𝑠𝑝𝑎𝑐𝑒 𝑌 [cls] yeh jaga bohat pyari hai. ye hai. [sep] [sep] ‫ےہ‬ ‫ہی‬ . ‫ہی‬ ‫ہگج‬ ‫تہب‬ ‫ایپری‬ ‫ےہ‬ [cls] 𝐿𝑎𝑏𝑒𝑙 𝑚𝑎𝑝𝑝𝑖𝑛𝑔 𝑀(𝑌) input (a) standard fine-tuning (b) prompt-based fine-tuning input template [cls] mlm head mlm head [mask] [mask] (label: positive) ‫وخوصبرت‬ (label: negative) ‫وضفل‬ khubsurat (label: positive) fazool (label: negative) [cls] : it is [mask]), (yeh [mask] hai : it is [mask]) ( [cls] yeh jaga bohat pyari hai. ye hai. [sep] [sep] ‫ےہ‬ ‫ہی‬ . ‫ہی‬ ‫ہگج‬ ‫تہب‬ ‫ایپری‬ ‫ےہ‬ [cls] 𝐿𝑎𝑏𝑒𝑙 𝑚𝑎𝑝𝑝𝑖𝑛𝑔 𝑀(𝑌) input template (b) prompt-based fine-tuning input template mlm head mlm head [mask] [mask] (label: positive) ‫وخوصبرت‬ (label: negative) ‫وضفل‬ khubsurat (label: positive) fazool (label: negative) : beautiful), ( i. [sep] sitive gative 𝑒𝑙 𝑠𝑝𝑎𝑐𝑒 𝑌 [cls] yeh jaga bohat pyari hai. ye hai. [sep] [sep] ‫ےہ‬ ‫ہی‬ . ‫ہی‬ ‫ہگج‬ ‫تہب‬ ‫ایپری‬ ‫ےہ‬ [cls] 𝐿𝑎𝑏𝑒𝑙 𝑚𝑎𝑝𝑝𝑖𝑛𝑔 𝑀(𝑌) input template ng (b) prompt-based fine-tuning input template mlm head mlm head [mask] [mask] (label: positive) ‫وخوصبرت‬ (label: negative) ‫وضفل‬ khubsurat (label: positive) fazool (label: negative) [cls] we also refer to this number as splits. the remaining examples from d train are reserved for the test set (with no labels). to ensure fair evaluation, we perform multiple rounds of testing. we select samples randomly from the unlabeled test set not used in each round's training d train and development d dev sets. this process is repeated five times, allowing for a comprehensive evaluation of the models' performance across different test sets. we evaluate classification performance using accuracy and macro f1-score. we report the mean and standard deviation of each measure over the five runs and for different splits. this approach allows us to draw robust conclusions regarding the effectiveness of prompt-based and standard finetuning approaches for text classification. prompt-based fine-tuning involves using language-specific prompts tailored to urdu and roman urdu to provide additional context to the models during training. this helps capture the intricacies of the languages and improve text classification performance. no prompts are used in standard fine-tuning, and the models rely solely on labeled examples. comparing these approaches allows us to assess the impact of prompts in leveraging language-specific knowledge. we used the hugging face foot_0 library for standard fine-tuning of models, while for prompt-based fine-tuning, we employed the openprompt foot_1 library. openprompt is an open-source framework designed explicitly for prompt learning, providing a comprehensive set of tools and resources for this approach. our fine-tuning process (standard and prompt-based) utilizes a learning rate of 2e -5. the optimization method is adamw (an adam optimizer variant), and the loss function is cross-entropy loss. the number of epochs for each training is 10. our study utilizes three pre-trained multilingual transformers, namely (1) xlm-roberta-base, (2) bert-base-multilingual-cased, and (3) distilbertbase-multilingual-cased, which are publicly available on huggingface library. ## results and discussion on the rued dataset, prompt-based fine-tuned xlm-roberta outperformed other models with the highest accuracy of 35.6% for splits 32. however, for splits 8 and 16, prompt fine-tuned distil-bert and bert-multilingual models performed better, respectively. in the case of the uod dataset, the results were more varied. prompt-based finetuned xlm-roberta outperformed other models for splits 4 and 16. however, for splits 8 and 32, prompt fine-tuned bert-multilingual yielded better results compared to the other models. in our experiments, we consistently observed that prompt fine-tuned models outperformed standard fine-tuned models achieving up to 13% absolute improvement and 5.44% average improvement in accuracy across all tasks. additionally, it is noteworthy that as the data splits increased, the models' accuracy also increased, as evident from figures 2 , 3 , 4 , 5 , and 6 . these figures show the variation of accuracy with split size for uod, rud, rued, ruhsold, and uned datasets, respectively, under prompt-based and standard fine-tuning of the respective best-performing models. in general, accuracy improves with an increase in split size. however, for prompt-based fine-tuned models, the increase in accuracy was more significant compared to standard fine-tuned models. it is also worth noting that standard fine-tuning almost always lags behind prompt-based fine-tuning for all models and datasets for up to 32 splits, confirming that for the limited number of labeled examples, prompt-based fine-tuning is preferable. 20 30 40 50 60 70 80 90 4 8 16 32 accuracy (%) k-splits standard fine-tune prompt-based fine-tune figure 3: accuracy comparison for standard vs. prompt-based fine-tuning for rud dataset, where k-split denotes the number of instances per class. 20 22 24 26 28 30 32 34 36 38 4 8 16 32 accuracy (%) k-splits standard fine-tune prompt-based fine-tune figure 4: accuracy comparison for standard vs. prompt-based fine-tuning for rued dataset, where k-split denotes the number of instances per class. 0 5 10 15 20 25 30 35 40 45 50 4 8 16 accuracy (%) k-splits standard fine-tune prompt-based fine-tune figure 6: accuracy comparison for standard vs. prompt-based fine-tuning for uned dataset, where k-split denotes the number of instances per class. we present additional experimental results in the appendices. appendix a gives the standard deviations corresponding to the mean accuracy values reported in ## conclusion in this paper, we compare prompt-based fine-tuning with standard fine-tuning for urdu and roman urdu text classification when restricted to dozens of training examples only. we perform experi-ments using several multilingual pre-trained language models on different classification datasets. regardless of the training set size or the specific classification task, prompt-based fine-tuning consistently outperforms standard fine-tuning, highlighting its robustness and generalizability. the clear prediction superiority of prompt-based approaches, coupled with its generally lower computational cost, makes it an attractive alternative to traditional fine-tuning methods for low-resource languages. the insights gained from this study can inspire future research and encourage the adoption of prompt-based techniques in other low-resource languages.