Are Large Language Models Actually Good at Text Style Transfer? (2024)

Abstract: We analyze the performance of large language models (LLMs) on Text Style Transfer (TST), specifically focusing on sentiment transfer and text detoxification across three languages: English, Hindi, and Bengali. Text Style Transfer involves modifying the linguistic style of a text while preserving its core content. We evaluate the capabilities of pre-trained LLMs using zero-shot and few-shot prompting as well as parameter-efficient finetuning on publicly available datasets. Our evaluation using automatic metrics, GPT-4 and human evaluations reveals that while some prompted LLMs perform well in English, their performance in on other languages (Hindi, Bengali) remains average. However, finetuning significantly improves results compared to zero-shot and few-shot prompting, making them comparable to previous state-of-the-art. This underscores the necessity of dedicated datasets and specialized models for effective TST.

## introduction text style transfer (tst) involves rewriting text to incorporate additional or alternative stylistic elements while preserving its overall semantics and structure @xcite . although style transfer has garnered increased research interest @xcite , it usually requires a substantial amount of labeled training examples, either as parallel text data @xcite or non-parallel text data of a single style @xcite . recent survey papers have identified a need for new methods that reduce the training data requirements and expand the scope of styles supported @xcite . this makes llm prompting a compelling option and a few works explore it in tst @xcite , but llm's usefulness, particularly in multilingual and diverse stylistic contexts and with new open llms, requires further exploration. this paper aims at evaluating llms on tst systematically. we focus on two popular subtasks of tst, sentiment transfer @xcite ) and text detoxification @xcite , and three languages: english, hindi, and bengali. we evaluate the llms using zero-shot and few-shot prompting. additionally, we investigate parameter-efficient finetuning @xcite . using automatic metrics as well as human evaluation and reference-free gpt-4-based evaluation (kocmi and federmann, 2023), we compare our results to previous stateof-the-art (sota), i.e., smaller language models specifically trained on the same dedicated datasets. our findings indicate that gpt-3.5 as well as a few open llms show promising results, but do not surpass previous sota. while the performance of open llms on prompting is weaker, finetuning leads to significantly improvements, aligning closely with gpt-3.5 and sota performance. this highlights the necessity of dedicated datasets and models tailored for tst tasks. foot_0 2 related work tst typically involves training on pairs of texts that share content but differ in style. a standard sequence-to-sequence supervised training approach is particularly challenging due to the limited availability of parallel data @xcite . tst methods are thus often unsupervised @xcite @xcite , which leads to high data requirements @xcite . prompt-based methods have become popular recently, with llm's ability to solve various downstream tasks @xcite , including tst @xcite @xcite . while these previous works achieved some success using non-instruction tuned models such as gpt-3, lamda or gpt-j, a comprehensive evaluation using different-sized instruction-tuned llms and prompting as well as finetuning is still needed. ## conclusion we evaluated the efficacy of llms for text style transfer, focusing on sentiment transfer and text detoxification across english, hindi, and bengali. we analyzed llms under zero-shot and few-shot prompting as well as with parameter-efficient finetuning. our findings indicate that while some open llms exhibit promising performance in english, their multilingual capabilities are still limited. however, finetuning demonstrates significant improvements, aligning the performance of these models with previous state-of-the-art systems. our study underscores the importance of tailored datasets and targeted models (even smallsize) for this task. in the future, we aim to expand our experiments to include more styles and languages. we will also look into alternative finetuning methods @xcite and advanced prompting techniques @xcite , to further improve performance.