Low-Resource Text Style Transfer for B angla: Data & Models (2023)

Abstract: Text style transfer (TST) involves modifying the linguistic style of a given text while retaining its core content. This paper addresses the challenging task of text style transfer in the Bangla language, which is low-resourced in this area. We present a novel Bangla dataset that facilitates text sentiment transfer, a subtask of TST, enabling the transformation of positive sentiment sentences to negative and vice versa. To establish a high-quality base for further research, we refined and corrected an existing English dataset of 1,000 sentences for sentiment transfer based on Yelp reviews, and we introduce a new human-translated Bangla dataset that parallels its English counterpart. Furthermore, we offer multiple benchmark models that serve as a validation of the dataset and baseline for further research.

## introduction text style transfer (tst) aims to modify the style of a given text while preserving its underlying content @xcite @xcite , see figure 1 ). prior research in text style transfer has primarily focused on the english language, overlooking languages with limited resources, such as bangla. this work aims to close this gap specifically for bangla and explores the text sentiment transfer task, which is a prominent subtask 1 of tst @xcite @xcite . bangla, also referred to as bengali, is mostly spoken in the indian regions of west bengal, assam, and tripura and is the mother tongue of about 97.2 million speakers as per the 2011 census report of india. 2 it is one of the 22 scheduled (official) @xcite indian languages and the national language of bangladesh. syntactically, bangla 1 moving forward, we will use the terms "style transfer" and "sentiment transfer" interchangeably in this paper. source style style transfer: example: style / sentiment -{neg , pos} the food is tasteless. the food is delicious. neg -pos pos -neg content content target style figure 1 : an example of sentiment transfer as a tst task in english and bangla. adapted from our previous paper @xcite . the unique challenges posed by the lowresource nature of bangla require specifically tailored innovative approaches. to achieve tst in bangla, we build upon an existing english dataset of 1,000 sentences for this task adapted from yelp reviews by @xcite . however, upon careful examination, we found that the quality of the original english dataset did not meet the standards we aimed to establish. to address this problem, we manually checked and modified the english dataset to improve its quality. subsequently, we adapted the curated english dataset to the bangla language, ensuring alignment in both content and structure. importantly, we introduce a novel bangla dataset, crafted by human annotators, serving as a parallel counterpart to the refined english dataset. furthermore, to facilitate the evaluation, we provide benchmark models capable of assessing the efficacy of text style transfer on our datasets. this paper marks a significant contribution to the field, as it not only pioneers text style transfer in the bangla language but also provides a foundation for future research endeavors in multilingual text style transfer. our work not only broadens the scope of text style transfer to include a low-resource language but also underscores the importance of dataset quality. our data and experimental code are released on github. 3 our contributions are summarized as follows: ## related work existing works in tst are mostly aimed at the english language and can be broadly classified into below categories: tst with parallel data tst can be modeled as a sequence-to-sequence task and trained on pairs of texts with similar content but different styles. here, @xcite used a sequence-to-sequence model with a pointer network to translate modern english into shakespearean english. @xcite leveraged minimal parallel data and incorporated various low-resource methods to explore the tst task. however, this approach to tst is inherently challenging due to the scarcity of parallel data @xcite . non-parallel approaches to tst two main strategies were employed to avoid reliance on parallel data: (i) straightforward text replacement, where style-specific phrases are explicitly identified and replaced @xcite , (ii) implicit style-content disentanglement via latent representations through techniques such as back-translation and autoencoding @xcite @xcite @xcite . adversarial learning was shown to improve the results of both approaches @xcite @xcite . despite a lot of progress, non-parallel approaches tend to produce mixed results and often require large amounts of non-parallel data, which is not readily available for many styles, limiting their practical applicability in low-resource settings @xcite . multilingual style transfer remains a relatively uncharted territory in prior research. in a comprehensive survey conducted by @xcite , only one work of tst was identified in languages such as chinese, russian, latvian, estonian, and french. additionally, they introduced an evaluation dataset for formality transfer, encompassing french, brazilian portuguese, and italian. another study focused on formality transfer across various indic languages @xcite . existing work has primarily concentrated on resource-rich languages, leaving languages like bangla understudied in the domain of tst. the only previous work on bangla known to us is the experiment of palash et al. (2019), who used a small amount of non-parallel data to train an autoencoder, with largely negative results. ## dataset creation we utilized the yelp dataset @xcite , which is publicly available and has been used by prior tst experiments. it consists of user-generated content in the form of reviews for hospitality establishments. for each review sentence that is originally positive or negative, a parallel sentence has been created where the sentiment has been flipped but sentiment-independent content retained as much as possible. the dataset is in english. 500 sentences have been transferred from negative to positive and another 500 from positive to negative. ## models our models are categorized into three approaches: parallel, non-parallel (not using parallel training data), and cross-lingual, i.e., without using our bangla dataset for training. for an overview of the methodologies, see figure 2 . ## experimetal settings each dataset consists of 500 positive-to-negative and 500 negative-to-positive sentences (see section 3). to maintain a consistent approach across all our experiments, we have divided these datasets into 400 examples for training, 100 for development, and 500 for testing purposes. we used the mbart-large-50 model @xcite ) from the huggingface library @xcite for both english and bangla. to optimize model performance, hyperparameter tuning was performed, resulting in the selection of a learning rate of 1e-5 and a batch size of 3. dropout was applied with a rate of 0.1 across the network. additionally, l2 regularization with a strength of 0.01 was introduced. training ran over 5 epochs. for machine translation in the bt reconstruction experiment, we use the facebook nllb-200-3.3b models @xcite from hugging-face. for our msf experiments and for evaluating sentiment transfer accuracy, we fine-tuned a classifier based on the bert-base multilingual cased model @xcite , using the same training set as our primary task. this finetuned classifier achieves accuracy rates of 87.0% for english and 83.0% for bangla. in the msf process, we employ a threshold of 0.5 to selectively filter style lexicon. ## conclusion in this study, we delved into the challenging domain of text style transfer primarily for the bangla language, addressing the scarcity of resources in it. this work contributes essential resources and benchmark models for both, bangla and english. future work involves exploring further underrepresented languages in the multi-lingual tst re-search.