Toward the Modular Training of Controlled Paraphrase Adapters (2024)

Abstract: Controlled paraphrase generation often focuses on a specific aspect of paraphrasing, for instance syntactically controlled paraphrase generation. However, these models face a limitation: they lack modularity. Consequently adapting them for another aspect, such as lexical variation, needs full retraining of the model each time. To enhance the flexibility in training controlled paraphrase models, our proposition involves incrementally training a modularized system for controlled paraphrase generation for English. We start by fine-tuning a pretrained language model to learn the broad task of paraphrase generation, generally emphasizing meaning preservation and surface form variation. Subsequently, we train a specialized sub-task adapter with limited sub-task specific training data. We can then leverage this adapter in guiding the paraphrase generation process toward a desired output aligning with the distinctive features within the sub-task training data. The preliminary results on comparing the fine-tuned and adapted model against various competing systems indicates that the most successful method for mastering both general paraphrasing skills and task-specific expertise follows a two-stage approach. This approach involves starting with the initial fine-tuning of a generic paraphrase model and subsequently tailoring it for the specific sub-task.

## introduction paraphrase generation aims to produce sentences that maintain high semantic similarity with the source sentence, while deviating enough from it on surface form. commonly used sequence-tosequence models encounter challenges in generating diverse paraphrase outputs @xcite . as a result, recent research in paraphrase generation has shifted toward controlled generation methods. these approaches condition the model on predefined qualities to produce specific outputs, aiming to overcome this limitation. exploring approaches to controlled text generation has both theoretical and practical implications. it can influence the theoretical understanding of automatic language generation and offer practical applications across various domains and industries. through leveraging controlled text generation, models can for instance be steered to produce language that better follows user preferences @xcite . with enough surface form variation, paraphrasing can be useful in question answering @xcite , data augmentation @xcite , and machine translation @xcite , among other tasks. even if trained to perform certain paraphrase transformations, recent controlled paraphrase generation systems are limited in flexibility. incorporating an additional control feature necessitates retraining the entire model. to overcome this limitation, we make the assumption that paraphrase generation essentially behaves in a modular manner. to evaluate our assumption, we propose the training of a modular system for controlled paraphrase generation through initial fine-tuning or broader task adapters @xcite followed by more specialized sub-task adapters. hence, in contrast to standard fine-tuning of all parameters of a model, we initially train the model to perform the necessary paraphrasing skills, namely meaning preservation and surface form variation, and further refine the model in a modular way to produce outputs that encompass some desired paraphrase nuances. we focus on english paraphrasing, incorporating one specific aspect of paraphrasing, namely antonym substitution @xcite . we select this paraphrase operation due to the availability of a specialized test suite designed for evaluating paraphrase models on sentence pairs that incorporate antonym substitution @xcite , enabling systematic comparison of various experimental setups. however, our proposed approach could as well be applied to other paraphrase phenomena and languages where paraphrase data is available. ## previous research common methods for automatic paraphrase generation rely on sequence-to-sequence modeling, often leveraging machine translation @xcite @xcite , or monolingual parallel data @xcite . these models, however, often struggle with generating sufficient variation @xcite . as a result, increased emphasis has been given to generating controlled paraphrases, specifically targeting variations across predefined dimensions. there has been significant research attention directed toward controlled paraphrasing in various granularity levels, from aiming to produce lexical variation by providing synonym substitutions @xcite to the generation of syntactically controlled paraphrases @xcite @xcite . while these approaches are constrained by concentrating solely on one level of detail, diverse paraphrasing encompasses multiple levels of granularity. to acknowledge this limitation, @xcite use dictionaries to perform word-level and phrase-level paraphrasing, obtaining more variation. @xcite train a multilingual nmt model with control tokens related to various aspects of paraphrasing. it is still however an open question how the control tokens interplay. in addition, a critical limitation arises with these models: they lack modularity, wherein all control tokens exert simultaneous influence on the output, making it impossible to selectively deactivate any subset of control features during the inference process or flexibly adapt the model to new features. in contrast, we propose the training of a modular controlled paraphrase generation model leveraging adapter transformations @xcite . in addition to being widely studied for crosslingual transfer (e.g., @xcite and nmt @xcite , modular and parameter efficient fine-tuning has been explored in other sequence-to-sequence tasks. @xcite use a modification of trainable adapter blocks @xcite to adapt mt outputs for new languages and domains. @xcite leverage prefix-tuning for generating syntactically controlled paraphrases. in contrast to the previous work on modular fine-tuning, our focus lies in the modular training paradigm specific to paraphrasing. we delve into training specialized sub-task adapters within this single task. these adapters are supposed to capture task and sub-task specific information, and are to be assembled to produce controlled paraphrasing toward an intended output. ## data we use the english partition of the opusparcus paraphrase dataset @xcite for alternately fine-tuning the full model or training a generic paraphrase adapter. the training data in opusparcus was automatically constructed and organized to prioritize the most probable paraphrastic sentence pairs at the beginning, with decreasing likelihood of being paraphrases as the data progresses. hence, we select the first 1 000 000 sentence pairs from the corpus as training data, denoted as t , comprising approximately of 95% of true paraphrases @xcite , and use the sentence pairs annotated as paraphrases from the opusparcus development set for tuning the models. moreover, within the training set t , we extract a specialized subset t n ⊂ t consisting of 12 870 examples. we use the first 12 000 examples as training data and save the final 870 examples to serve as a development set for tuning the specialized systems. this subset exclusively comprises instances where an explicit negation token is present in the target but absent in the source, and is used for training a dedicated sub-task adapter as a part of a broader paraphrasing task. we aim to extract sentences that demonstrate interesting paraphrastic relationships through the use of negation or negated antonymy, as opposed to sentences that negate the intended meaning. we release the taskspecific data in https://github.com/teemuvh/ controlled-paraphrase-adapters . ## experiments our objective is to incrementally train and assemble a modular system for controlled paraphrase generation. we undertake training and assessment across several models. to start, we establish a baseline by fine-tuning flan-t5-base 1 (chung et al., 2022) using a set of 1 000 000 paraphrase pairs (t ) sourced from the english partition of the opusparcus training set. furthermore, we fine-tune a sepa-original candidates you're not fat. you're not thin., you're fat., you're thin. it's not fair. it's not unfair., it's fair., it's unfair. this is not a good idea. this is not a bad idea., this is a good idea., this is a bad idea. it is not safe. it is not dangerous., it is safe., it is dangerous. in all adapter experiments, we leverage the adapter-transformers library @xcite . we optimize modular fine-tuning by utilizing the bottleneck adapter (houlsby et al., 2019) configuration proposed in @xcite in conjunction with the base model. we then proceed to train two task adapters: one using the entire training dataset (t ) for a broad paraphrasing task, and another using a subset (t n ) of the data for a specific controlled paraphrasing sub-task. finally, we explore incremental adapter training by enhancing the base model with the paraphrase adapter. we then freeze the weights of the base model and the paraphrase adapter and proceed to train an additional sub-task adapter. this adapter not only benefits from the paraphrase adapter's information but also focuses on learning more specific paraphrasing transformations incorporating negation and antonym substitution. we train each system on a single gpu for 3 epochs with a batch size of 128, and 5e-5 learning rate. we evaluate the models on a dedicated test suite designed for paraphrase detection within sequences incorporating negated antonyms @xcite . the test suite is intended to be used to evaluate models on a difficult paraphrase detection task involving sequences with high lexical overlap. examples of the data are provided in ## results fully fine-tuning the model with the filtered subset (t n ) of the training data (neg-ft), thus highlighting paraphrasing through negation and antonymy, consistently produces higher bleu scores on the task compared to both the base model and models trained solely on t . adapter training on top of the base model using t n (neg-adapt) results in even higher bleu scores. parameter efficient fine-tuning has been shown to be effective in low-resource scenarios (e.g., karimi mahabadi et al., 2021), which might explain why the adapter method achieves higher bleu scores compared to full fine-tuning when trained specifically for the given paraphrasing sub-task. initiating training by fine-tuning a generic paraphrase model, followed by further fine-tuning with the specific sub-task data yields a subpar model (para-ft+neg-ft). similarly, training an extensive paraphrase adapter before introducing a specialized sub-task adapter (para-adapt+neg-adapt) results in a model which barely surpasses the base model's performance when evaluated against the reference using bleu. comparing the outputs to the input sentences however shows that the incrementally adapted model achieves similar bleu scores as the base model by trying to produce variation rather than simply duplicating the input sentence, as indicated by the lower s-bleu score of the adapted model. the best bleu scores are obtained by fully finetuning the base model leveraging all 1 000 000 paraphrase examples and training a specialized subtask adapter on top of the refined model (para-ft+neg-adapt). we hypothesize that the initial finetuning steers the model toward generating outputs that highly resemble the input, reflected in a relatively high s-bleu. subsequent adapter training on a smaller scale then refines the model's proficiency in paraphrase operations involving negation and negated antonyms, as indicated by the highest bleu. the relationship between the obtained bleu and s-bleu is presented in figure 1 . a robust paraphrase model would typically demonstrate a balance between a higher bleu score and a lower s-bleu score, positioning itself toward the lower the best performing models are assumed to show far to the right, reflecting a high bleu with respect to the reference, and at around 25 % s-bleu, which is the bleu value of the reference with respect to the source. that is, an oracle model that would produce the desired reference sentences would obtain @xmath0.90 %. to summarize the results, the base model along with the models subjected to plain fine-tuning or adaptation with the more generic paraphrase data exhibit poor performance, highlighted by the base model's high s-bleu, and the low bleu scores achieved by the fine-tuned or adapted models. incorporating specialized training for the intended paraphrasing task, either through fine-tuning or adaptation, is essential for success in the task. however, the results obtained with the models specifically trained for paraphrasing through negation or negated antonymy remain somewhat inconclusive. further analysis is necessary to determine the optimal training configuration for assembling general paraphrasing capabilities with specialized sub-task capabilities. additionally, we hypothesize that parameter-efficient fine-tuning is better suited in scenarios involving limited data. however, the limited training data is also more task-specific, so it is still too early to draw general conclusions. ## conclusions we propose the training of a modular paraphrase generation model that is built incrementally. this model starts by fine-tuning on a robust pretrained language model to learn the general requirements of paraphrase generation, namely meaning preservation and surface form variation. subsequently, we train a specialized sub-task adapter with a limited number of sub-task specific training data to guide the paraphrase generation process toward a desired output. we compare the model involving fine-tuning followed by sub-task adaptation to several counterparts, including a base model without further training, as well as differently fine-tuned or adapted systems. when assessing on a dedicated test set involving paraphrasing with negation or negated antonyms, we find that the most effective approach for learning both general paraphrasing abilities and sub-task specific expertise is achieved by fully fine-tuning a model for paraphrasing and then tailoring it to the specific sub-task through modular updates. in future work, we wish to delve deeper into modularity for controlled paraphrasing. we intend to expand the model's capabilities by incrementally training it to encompass additional paraphrasing nuances, such as syntactic or lexical variation. furthermore, we would like to assess how varying the size and task-specificity of the training data impacts the results. finally, we would like to extend our approach to a multilingual setup.