Few-shot Controllable Style Transfer for Low-Resource Multilingual Settings (2022)

Abstract: Style transfer is the task of rewriting a sentence into a target style while approximately preserving content. While most prior literature assumes access to a large style-labelled corpus, recent work (Riley et al. 2021) has attempted “few-shot” style transfer using only 3-10 sentences at inference for style extraction. In this work we study a relevant low-resource setting: style transfer for languages where no style-labelled corpora are available. We notice that existing few-shot methods perform this task poorly, often copying inputs verbatim. We push the state-of-the-art for few-shot style transfer with a new method modeling the stylistic difference between paraphrases. When compared to prior work, our model achieves 2-3x better performance in formality transfer and code-mixing addition across seven languages. Moreover, our method is better at controlling the style transfer magnitude using an input scalar knob. We report promising qualitative results for several attribute transfer tasks (sentiment transfer, simplification, gender neutralization, text anonymization) all without retraining the model. Finally, we find model evaluation to be difficult due to the lack of datasets and metrics for many languages. To facilitate future research we crowdsource formality annotations for 4000 sentence pairs in four Indic languages, and use this data to design our automatic evaluations.

## introduction style transfer is a natural language generation task in which input sentences need to be re-written into a target style, while preserving semantics. it has many applications such as writing assistance @xcite , controlling generation for attributes ## the universal rewriter (ur) model we will start by discussing the universal rewriter (ur) model from @xcite , upon which our proposed diffur model is built. at a high level, the ur model extracts a style vector s from an exemplar sentence e, which reflects the desired target style. this style vector is used to style transfer an input sentence x. concretely, consider f enc , f dec to be encoder & decoder transformers initialized with mt5 @xcite , which are composed to form the model f ur . the ur model extracts the style vector using the encoder representation of a special @xcite token prepended to e, and adds it to the input x representations for style transfer, ## evaluation automatic evaluation of style transfer is challenging @xcite @xcite , and the lack of resources (such as evaluation datasets, style classifiers) make evaluation trickier for indic languages. to tackle this issue, we first collect a small dataset of formality and semantic similarity annotations in four indic languages (section 5.1). we use this dataset to guide the design of an evaluation suite (section 5.2-5.6). since automatic metrics in generation are imperfect @xcite , we complement our results with human evaluation (section 5.7). ## conclusion we present a recipe for building & evaluating controllable few-shot style transfer systems needing only 3-10 style examples at inference, useful in low-resource settings. our methods outperform prior work in formality transfer & code-mixing for 7 languages, with promising qualitative results for several other attribute transfer tasks. future work includes further improving systems for some attributes, and studying style transfer for languages where little / no translation data is available.