Adapter- TST : A Parameter Efficient Method for Multiple-Attribute Text Style Transfer (2023)

Abstract: Adapting a large language model for multiple-attribute text style transfer via fine-tuning can be challenging due to the substantial amount of computational resources and labeled data required for the specific downstream task. In this paper, we address this challenge by introducing Adapter-TST, a framework that freezes the pre-trained model’s original parameters and enables the development of a multiple-attribute text style transfer model. Using BART as the backbone model, Adapter-TST utilizes different neural adapters to model different types of attribute information, similar to a plug-in connected to BART. Our method allows control over multiple attributes (e.g. sentiment, tense, active or passive voice) and configures the adapters’ architecture to generate multiple outputs in respect to attributes or compositional editing on the same sentence. We evaluate the proposed model on both traditional sentiment transfer and multiple-attribute transfer tasks. The experiment results demonstrate that Adapter-TST outperforms all the state-of-the-art baselines with significantly less computational resources. We have also empirically shown that each adapter is able to characterize specific stylistic attributes effectively and can be configured to perform compositional editing.

## introduction motivation. text style transfer (tst) is a popular natural language generation task that aims to change the stylistic properties (e.g., sentiment, formality, tense, voice) of the text while preserving the style-independent content @xcite . existing studies explore performing text style transfer on attributes like age, or gender @xcite , sentiment @xcite @xcite , formality @xcite , politeness @xcite , and ## methodology this section proposes adapter-tst, which adds neural adapters into each transformer layer to capture different attribute information for multipleattribute tst. we first introduce the adapter structure used in adapter-tst and its parameter efficiency. subsequently, we explain how the adapters are configured for different multiple-attribute tst settings, namely, multiple stylistic attribute outputs and compositional editing. finally, we describe the training objectives of adapter-tst. ## case study we conducted case studies to showcase the style transferred outputs of both the adapter-tst and style transformer models. randomly sampled examples and their corresponding outputs are presented in ## conclusion in this paper, we introduced a parameter-efficient framework, adapter-tst with different neural adapters to capture different attribute information for multiple-attribute tst tasks. during training, the original parameters of bart-large were frozen, and only the adapters' parameters were optimized to relax the dependence on computational resources and supervised data. we conducted extensive ex-periments on traditional sentiment transfer and multiple-attribute tst tasks. the automatic and human-based evaluation results showed that the attribute-specific adapters in adapter-tst is able to capture relevant stylistic attributes to transfer the style while preserving the original content successfully. our case studies also demonstrated that adapter-tst was able to generate high-quality text in the target style. for future work, we will continue to improve tst models' ability to model multiple attributes in terms of quality and efficiency. we will also explore plugging adapter-tst on other plms and evaluate its effectiveness.