Multilingual Text Style Transfer: Datasets & Models for I ndian Languages (2024)

Abstract: Text style transfer (TST) involves altering the linguistic style of a text while preserving its style-independent content. This paper focuses on sentiment transfer, a popular TST subtask, across a spectrum of Indian languages: Hindi, Magahi, Malayalam, Marathi, Punjabi, Odia, Telugu, and Urdu, expanding upon previous work on English-Bangla sentiment transfer. We introduce dedicated datasets of 1,000 positive and 1,000 negative style-parallel sentences for each of these eight languages. We then evaluate the performance of various benchmark models categorized into parallel, non-parallel, cross-lingual, and shared learning approaches, including the Llama2 and GPT-3.5 large language models (LLMs). Our experiments highlight the significance of parallel data in TST and demonstrate the effectiveness of the Masked Style Filling (MSF) approach in non-parallel techniques. Moreover, cross-lingual and joint multilingual learning methods show promise, offering insights into selecting optimal models tailored to the specific language and task requirements. To the best of our knowledge, this work represents the first comprehensive exploration of the TST task as sentiment transfer across a diverse set of languages.

## introduction text style transfer (tst) is an evolving field within natural language processing that has gained prominence for its capacity to modify the style of a given text while preserving its fundamental content @xcite . notably, tst has primarily been explored in english, leaving a significant gap in linguistic diversity and a lack of comprehensive resources for effective multilingual style transfer. this research aims to bridge this gap by extending the boundaries of tst to include other diverse indian languages: hindi, magahi, malayalam, marathi, punjabi, odia, telugu, and urdu. we work with sentiment transfer and use the english dataset of @xcite , who experimented with english and bangla. we have extended the scope by adding eight new languages to the dataset. we manually translated the english dataset into other languages to maintain the style, content, and structural alignment, prioritizing naturalness in the target language (details in section 3.2). we created new multilingual tst datasets using human annotators. they serve as counterparts to the refined english dataset @xcite in a well-established linguistic context. in addition, we tested several standard models (see section 4) to validate and assess the quality and usefulness of the language-specific datasets. our contributions are summarized as follows: ## related work tst typically involves training on pairs of texts that share content but differ in style. for example, @xcite used a sequence-to-sequence model with a pointer network to transform modern english into shakespearean english. meanwhile, @xcite employed minimal parallel data and integrated various low-resource methods for tst. however, this approach is particularly challenging due to the limited availability of parallel data @xcite . to reduce the need for parallel data, two main strategies have been used: (i) simple text replacement, where specific style-related phrases are explicitly identified and substituted @xcite . (ii) implicitly disentangling style from content through latent representations, using techniques like back-translation and autoencoding @xcite @xcite @xcite . however, non-parallel approaches often produce mixed results and require significant amounts of stylized non-parallel data, which can be scarce for many styles @xcite . in our experiments in section 4, we evaluate both approaches using low-resource parallel data and non-parallel approaches. multilingual style transfer is a relatively unexplored area in prior research. @xcite presented a multilingual formality style transfer benchmark, xformal, including languages like chinese, russian, latvian, estonian, and french. moreover, @xcite focused on altering formality in various indian languages. to the best of our knowledge, we are the first to explore text sentiment transfer within the domain of tst for the languages under consideration. we follow both above works by evaluating models on our benchmark in multilingual as well as crosslingual setups. ## dataset preparation we decided to base our effort on the yelp dataset of @xcite , as it offered a suitable size, parallel structure, and a relevant domain for our efforts. the dataset consists of 1,000 styleparallel sentences, i.e., negative and positive counterparts, with otherwise identical or similar meanings, from the domain of restaurant reviews. 500 sentences were originally written as positive and manually transferred to negative, the other 500 went in the opposite direction. the data is available in english and bengali, with english originally based on @xcite . however, the english data are not identical, as @xcite revised the texts to address issues like inconsistencies, spelling errors, inaccuracies in sentence sentiment, compromised linguistic fluency, omitted context, and improper sentiment adjustments. we translated the english dataset into eight indian languages to serve the aims of our experiment. in the following subsections, we briefly overview the tst task's language selection process in section 3.1. we also explore the manual style-translation process and the challenges encountered in section 3.2. ## models our experimental models use five methodologies (sections 4.1-4.5): parallel, non-parallel, crosslingual, shared multilingual learning and prompted llms. the first three methods are adopted from @xcite , and we only briefly summarize them. the last two are newly introduced for this task. ## evaluation metrics the evaluation process comprises three critical dimensions: sentiment transfer accuracy, content retention, and linguistic fluency. we employed our fine-tuned classifiers to calculate sentiment transfer accuracy (acc) (see section 5.3). in line with previous studies @xcite @xcite , we evaluate content preservation through the bleu score @xcite and embedding similarity (cs) @xcite when compared to the input sentences. the embedding similarity (cs) is computed using labse sentence embeddings @xcite in combination with cosine similarity. similarly to @xcite and @xcite , we derive a single comprehensive score for the two important measures of tst, sentiment transfer accuracy and content preservation, by calculating the arithmetic mean (avg) @xcite of acc, bleu, and cs. while this is not ideal, as the scores' sensitivities are different, it allows us to easily compare with an accuracy-preservation tradeoff. assessing linguistic fluency, particularly for all the indian languages, presents a challenge due to the absence of robust evaluation tools for indian languages @xcite . earlier research cautioned against using perplexity (ppl) as a measure of fluency, as it tends to favor awkward sentences with commonly used words @xcite . with this in mind, we still present a basic fluency evaluation using ppl with a multilingual gpt (mgpt) model @xcite . all experiments were conducted separately for positive-to-negative and negative-to-positive sentiment transfer tasks. the metric results were then averaged and presented in this paper. as automated metrics for language generation may not correlate well with human judgments @xcite , we also run a small-scale human evaluation with expert annotators, i.e., the same linguists that were involved in the dataset creation process, on a random sample of 50 sentences from the test set for selected models (equally split to both positive-to-negative and negative-topositive sentiment transfer tasks). the outputs are rated on a 5-point likert scale for style transfer accuracy, content preservation, and fluency (for details, see appendix b). ## conclusion in this study, we address the problem of text style transfer, primarily focusing on multilingual tst in indian languages. this work provides useful resources for tst in eight languages, explores various benchmark models, and presents an analysis of experimental results for all these languages. furthermore, it is worth noting that our presented datasets are style-parallel and parallel across the languages, making them consistent and comparable for the tst task. in future work, we plan to explore a wider range of style attributes and incorporate more languages, leveraging our existing methodologies and framework, which can be adapted to any style attribute given the availability of parallel data. models @xmath0. → first time i came in, i knew i just wanted something new. hi: पहली बार जब मैं आया तो मु झे पता था िक मैं बस यहाँ से जाना चाहता था। → पहली बार जब मैं अं दर आया, तो मु झे पता था िक मु झे बस कु छ नया चािहए। mag: जब हम पिहला बार ऐली,तऽ हमरा पता हल िक हम बस िनकलल चाहली। → पिहला बार हम अं दर ऐली, हमरा पता हल िक हम बस कु छ नया चािहत िह । mr: जे व्हा मी पिहल्यां दा आत आलो ते व्हा मला मािहत होते क मला फक्त िनघायचे आहे . → पिहल्यां दा मी आत आलो ते व्हा मला मािहत होतं क मला काहीतरी नवीन हवं आहे . ml: ആദ്യമായി ഞാൻ വന്നേപ്പാൾ എനിക്ക് േപാകണെമന്ന് അറിയാമായിരുന്നു. → ആദ്യമായി ഞാൻ വന്നേപ്പാൾ, എനി-ക്ക് പുതിയ എെന്തങ്കിലും േവണെമന്ന് അറിയാമായിരുന്നു. pa: ਪਿਹਲੀ ਵਾਰ ਜਦੋਂ ਮੈਂ ਅੰ ਦਰ ਆਇਆ ਤਾਂ ਮੈ ਨੂ ੰ ਪਤਾ ਸੀ ਿਕ ਮੈਂ ਬੱਸ ਛੱਡਣਾ ਚਾਹੁ ੰ ਦਾ ਸੀ। → ਪਿਹਲੀ ਵਾਰ ਜਦੋਂ ਮੈਂ ਅੰ ਦਰ ਆਇਆ, ਮੈ ਨੂ ੰ ਪਤਾ ਸੀ ਿਕ ਮੈਂ ਕੁ ਝ ਨਵਾਂ ਚਾਹੁ ੰ ਦਾ ਹਾਂ । or: ପ୍ର ଥମ ଥର ମଁ ୁ ଭ ତରକ ଆସି ଲ ମଁ ୁ ଜାଣି ଥ ଲ େଯ ମଁ ୁ ଛାଡି ବାକ ଚାହଁୁ ଛି । → ପ୍ର ଥମ ଥର ମଁ ୁ ଭ ତରକ ଆସି ଲ , ମଁ ୁ ଜାଣି ଲ େଯ ମଁ ୁ କି ଛି ନ ଆ ଚାହଁୁ ଛି । ur: ‫ں۔‬ ‫ف‬ ‫م‬ ‫رآ‬ → ‫م‬ ، ‫رآ‬ ‫ا‬ ‫ر‬ ‫ں۔‬ ‫ف‬ te: మొదటిసారి లోపలికి చచాచ్ను, నాకు తె లుసు నేను వె ళిళ్పోవాలనుకుంటునాన్ని. → మొదటిసారి నేను లోపలి వచిచ్న డు నాకు తె లుసు నేనేదో కొంత కొతత్ దనం కోరుకుంటునాన్. thank you amanda, i will be back ! → no thanks amanda, i won't be back ! hi: धन्यवाद अमां डा, मैं वापस आऊ ं गा! → भाड़ में जाओ अमां डा, मैं वापस नहीं आऊ ँ गा! mag: धन्यवाद अमां डा, हम बापस आएम! → नऽ, धन्यवाद अमां डा, हम बापस नऽ आएम! mr: धन्यवाद अमां डा, मी परत ये ईन! → नाही धन्यवाद अमां डा, मी परत ये णार नाही! ml: നന്ദി അമാൻഡ, ഞാൻ മടങ്ങിവരും! → ഇല്ല നന്ദി അമാൻഡ, ഞാൻ തിരിെക വരില്ല! pa: ਧੰ ਨਵਾਦ ਅਮਾਂ ਡਾ ਵਾਪਸ ਆਵਾਂ ਗਾ! → ਕੋ ਈ ਧੰ ਨਵਾਦ ਨਹੀਂ ਅਮਾਂ ਡਾ, ਮੈਂ ਵਾਪਸ ਨਹੀਂ ਆਵਾਂ ਗਾ! or: ଧନ୍ୟବାଦ ଅମା ା, ମଁ ୁ େଫରି ଆସି ବି ! → ନା ଧନ୍ୟବାଦ ଅମା ା, ମଁ ୁ େଫରି ବି ନାହିଁ ! ur: ! ‫آؤں‬ ‫وا‬ ‫ا،‬ ‫ا‬ ‫آپ‬ → ‫وا‬ ‫ا،‬ ‫ا‬ ! ‫آؤں‬ te: ధనయ్వాదాలు అమండ, నేను వె నకిక్ వసాత్ ను. → ధనయ్వాదాలు కాదు అమండ, నేను వె నకిక్ రాను. parallel en: first time i came in i felt i felt right at home. hi: पहली बार जब मैं आया तो मु झे पता था िक मैं बस यहाँ से जाना चाहता था। mag: जब हम पिहला बार ऐली,तऽ हमरा पता हल िक हम बस िनकलल चाहली। mr: जे व्हा मी पिहल्यां दा आत आलो ते व्हा मला मािहत होते क मी खू प चां गले काम करायचे आहे . ml: ആദ്യമായി ഞാൻ വന്നേപ്പാൾ എനിക്ക് േപാകണെമന്ന് അറിയാമായിരുന്നു. pa: ਪਿਹਲੀ ਵਾਰ ਜਦੋਂ ਮੈਂ ਅੰ ਦਰ ਆਇਆ ਤਾਂ ਮੈ ਨੂ ੰ ਪਤਾ ਸੀ ਿਕ ਮੈਂ ਬੱਸ ਛੱਡਣਾ ਚਾਹੁ ੰ ਦਾ ਸੀ। or: ପ୍ର ଥମ ଥର ମଁ ୁ ଭ ତରକ ଆସି ଲ ମଁ ୁ ଜାଣି ଥ ଲ େଯ ମଁ ୁ ଛାଡି ବାକ ଚାହଁୁ ଛି । ur: ‫ں۔‬ ‫ف‬ ‫م‬ ‫رآ‬ te: మొదటిసారి లోపలికి చచాచ్ను. en: thank you amanda, i will not be back! hi धन्यवाद अमां डा, मैं वापस नहीं जाऊ ं गा! mag: धन्यवाद अमां डा, हम बापस आएम! mr: धन्यवाद अमां डा, मी परत ये णार नाही. ml: അമാൻഡ, ഞാൻ തിരിെക വരില്ല! pa: ਧੰ ਨਵਾਦ ਅਮਾਂ ਡਾ ਵਾਪਸ ਨਹੀਂ ਆਵਾਂ ਗਾ! or: ଧନ୍ୟବାଦ ଅମା ା, ମଁ ୁ େଫରି ଆସି ବି ନାହିଁ । ur: ‫۔‬ ‫آؤں‬ ‫وا‬ ‫ا،‬ ‫ا‬ ‫آپ‬ te: ధనయ్వాదాలు అమండ, నేను వె నకిక్ వసాత్ ను. joint en: first time i came in i knew i wanted to stay. hi पहली बार जब मैं आया तो मु झे पता था िक मैं बस यहाँ से जाना चाहता था। mag: जब हम पिहला बार ऐली,तऽ हमरा पता हल िक हम बस िनकलल चाहली। mr: जे व्हा मी पिहल्यां दा आत आलो ते व्हा मला मािहत होते क मला फक्त िनघायचे आहे . ml: ആദ്യമായി ഞാൻ വന്നേപ്പാൾ എനിക്ക് േപാകണെമന്ന് അറിയാമായിരുന്നു. pa: ਪਿਹਲੀ ਵਾਰ ਜਦੋਂ ਮੈਂ ਅੰ ਦਰ ਆਇਆ ਤਾਂ ਮੈ ਨੂ ੰ ਪਤਾ ਸੀ ਿਕ ਮੈਂ ਬੱਸ ਛੱਡਣਾ ਚਾਹੁ ੰ ਦਾ ਸੀ। or: ପ୍ର ଥମ ଥର ମଁ ୁ ଭ ତରକ ଆସି ଲ ମଁ ୁ ଜାଣି ଥ ଲ େଯ ମଁ ୁ ଛାଡି ବାକ ଚାହଁୁ ଛି । ur: ‫ں۔‬ ‫ف‬ ‫رآ‬ te: మొదటిసారి లోపలికి చచాచ్ను, నాకు తె లుసు నేను వె ళిళ్పోవాలనుకుంటునాన్ని. en: sorry amanda, i will not be back. hi धन्यवाद अमां डा, मैं वापस नहीं आऊ ं गा! mag: धन्यवाद अमां डा, हम बापस नऽ आएम! mr: धन्यवाद अमां डा, मी परत ये णार नाही! ml: നന്ദി അമാൻഡ, ഞാൻ തിരിെക വരില്ല! pa: ਧੰ ਨਵਾਦ ਅਮਾਂ ਡਾ ਵਾਪਸ ਆਵਾਂ ਗਾ! or: ଧନ୍ୟବାଦ ଅମା ା, ମଁ ୁ େଫରି ଆସି ବି ! ur:! ‫آؤں‬ ‫وا‬ ، ‫آپ‬ te: ధనయ్వాదాలు అమండ, నేను వె నకిక్ రాను. gpt3.5 en: first time i came in i knew i never wanted to leave. hi पहली बार जब मैं आया तो मु झे पता था िक मैं यहाँ रुकना चाहता था। mag: जब हम पिहला बार ऐली,तऽ हमरा पता हल िक हम बस िनकलल चाहली। mr: जे व्हा मी पिहल्यां दा आत आलो ते व्हा मला मािहत होते क मला आवडतं य आहे . ml: ആദ്യമായി ഞാൻ വന്നേപ്പാൾ എനിക്ക് േപാകണെമന്ന് അറിയാമായിരുന്നു. pa: ਪਿਹਲੀ ਵਾਰ ਜਦੋਂ ਮੈਂ ਅੰ ਦਰ ਆਇਆ ਤਾਂ ਮੈ ਨੂ ੰ ਪਤਾ ਸੀ ਿਕ ਮੈਂ ਬੱਸ ਆਗੇ ਬਢ਼ਣਾ ਚਾਹੁ ੰ ਦਾ ਸੀ। or: ପ୍ର ଥମ ଥର ମଁ ୁ ଭ ତରକ ଆସି ଲ ମଁ ୁ ଜାଣି ଥ ଲ େଯ ମଁ ୁ ଛାଡି ବାକ ur: ‫۔‬ ‫وں‬ ‫اس‬ ‫ا‬ ‫م‬ ‫اور‬ ‫رآ‬ te: మొదటిసారి లోపలికి చచాచ్ను, నాకు తె లుసు నేను వె ళిళ్పోవాలనుకుంటునాన్ని. en: thank you amanda, i will never be back ! hi धन्यवाद अमां डा, मैं वापस नहीं आऊ ं गा। mag: धन्यवाद अमां डा, हम बापस नऽ आएम। mr: धन्यवाद अमां डा, मी परत ये ईन! ml: നന്ദി അമാൻഡ, ഞാൻ മടങ്ങിവരുന്നില്ല! pa: ਖੇ ਦ ਅਮਾਂ ਡਾ ਵਾਪਸ ਆਵਾਂ ਗਾ! or: ଧନ୍ୟବାଦ ଅମା ା, ମଁ ୁ େଫରି ଆସି ବି ନାହିଁ । ur:! ‫آؤں‬ ‫وا‬ ‫ا،‬ ‫ا‬ ‫آپ‬ te: ధనయ్వాదాలు అమండ, నేను వె నకిక్ రాను.