Linguistically-Controlled Paraphrase Generation (2025)

Abstract: Controlled paraphrase generation produces paraphrases that preserve meaning while allowing precise control over linguistic attributes of the output. We introduce LingConv, an encoder-decoder framework that enables fine-grained control over 40 linguistic attributes in English. To improve reliability, we introduce a novel inference-time quality control mechanism that iteratively refines attribute embeddings to generate paraphrases that closely match target attributes without sacrificing semantic fidelity. LingConv reduces attribute error by up to 34% over existing models, with the quality control mechanism contributing an additional 14% improvement.

## introduction controllable text generation (ctg) aims to produce text with specified linguistic attributes @xcite . a sub-task, controlled paraphrase generation (cpg), aims to generate paraphrases that satisfy desired attributes while preserving meaning. cpg has applications in text simplification @xcite @xcite , toxicity control @xcite , data augmentation @xcite , and creating linguistically challenging data @xcite . the key challenge is to balance attribute adherence with semantic fidelity. prior work in cpg typically controls a small number of attributes, often less than three @xcite @xcite . large language models (llms) such as llama @xcite , while powerful, struggle with precise and simultaneous control over many attributes via prompting @xcite . in addition, decoding-time methods that use two male and two female clowns are walking along the street. four clowns are walking down the pavement and waving. four clowns are walking down the road, waving as they go. figure 1 : we aim to transform a given sentence into multiple paraphrases, each satisfying distinct linguistic attributes. our model takes a source sentence and a set of target linguistic attributes and generates a paraphrase optimized to satisfy the target attributes. here we show three paraphrases with different linguistic attributes generated for the source sentence. linguistic features identified using the spacy "en_core_web_sm", with stop-word list from explosion ai (2025). we introduce lingconv, a novel encoderdecoder cpg model that offers fine-grained control over 40 linguistic attributes spanning lexical, syntactic, discourse, and semantic aspects (see @xcite . lingconv integrates attribute embeddings directly into the decoder and employs a robust inference-time quality control (qc) mechanism. this qc mechanism iteratively refines outputs using linguistic attribute and semantic consistency classifiers, guided by a line-search algorithm to ensure close alignment with target attributes without sacrificing meaning. figure 1 shows an example of our model's capability. our contributions are: we demonstrate through extensive experiments that lingconv outperforms strong baselines by up to 34% in attribute control, with the qc mechanism providing a further 14% improvement. cpg has the potential to generate data that challenges existing models from a linguistic perspective, produce text with varying levels of linguistic complexity for language learners @xcite @xcite @xcite or data augmentation @xcite , and make text accessible through language simplification @xcite . the main challenge in cpg is to generate text that preserves the meaning of the source and satisfies the desired linguistic attributes. while existing work has explored this balance, most work has focused on a limited set of attributes, as discussed below. accommodating a wider array of linguistic attributes in cpg is crucial because it improves the flexibility and engagement for diverse audiences including language learners. lingconv is an encoder-decoder cpg model that offers fine-grained control over 40 linguistic attributes spanning lexical, syntactic, discourse, and semantic aspects (see appendix b). it integrates attribute embeddings directly into the decoder and employs a novel inference-time quality control (qc) mechanism that iteratively refines outputs using linguistic attribute and semantic con-sistency classifiers to ensure close alignment with target attributes without sacrificing meaning. extensive experiments demonstrate that ling-conv outperforms strong baselines by up to 34% in attribute control, with the qc mechanism providing a further 14% improvement. we also show the utility of our approach in data augmentation, where attribute-controlled synthetic data can be tailored to improve downstream task performance. analysis reveals which linguistic attributes are easier or harder to control and the factors influencing controllability. ## related work controllable text generation (ctg) and controlled paraphrase generation (cpg) have seen significant advances, with early works focusing on controlling a small set of attributes such as formality @xcite @xcite . most prior cpg approaches are limited to manipulating up to three attributes simultaneously @xcite @xcite , often relying on discrete control tokens or prompt-based strategies, which can be imprecise and lack fine-grained control. decoding-time control methods using attribute classifiers @xcite are typically slow and struggle with highdimensional attribute spaces, and quality control at inference time is rarely addressed. llms such as llama @xcite and t5 @xcite demonstrate strong general-purpose generation, but prompt-based control remains coarse and unreliable for fine-grained attribute manipulation. @xcite show that including a textual syntactic constraint in paraphrase generation produces syntactically diverse outputs. other approaches have explored keyword exclusion (kajiwara, 2019), using discriminator networks to enforce diversity @xcite , and following exemplar syntax @xcite . fset @xcite improves quality and diversity by retrieving similar paraphrase pairs and applying their edits to the source sentence. variational autoencoders (vaes) were used to disentangle semantic and syntactic representations to generate diverse paraphrases @xcite @xcite . models that condition on explicit syntactic sketches via hierarchical discrete codes have also been proposed @xcite . gcpg @xcite concatenates con- paraphrase generator linguistic attribute predictor (pre-trained & frozen ) target text translation loss (cross-entropy) source text unfortunately, i cannot swim. prediction swimming is not within my scope of capabilities. target attributes yes prediction complexity prediction logits straight-through gradients exists semantic equivalence predictor (pre-trained & frozen ) quality control of target indices at inference time missing features imputation figure 2: lingconv architecture: the paraphrase generator extends the t5 model by incorporating linguistic attributes into the decoder inputs. linguistic attributes of the source (l s ) and target (l t ) are embedded and fused with the generation using element-wise addition to the decoder inputs. in addition, the linguistic attribute predictor estimates attributes of the generated text, which facilitates backpropagation of the linguistic attribute error. during inference, the quality control mechanism iteratively adjusts inputs to guide outputs towards desired attributes. the semantic equivalence predictor (se) receives as input the source sentence and the candidate generation t, as in algorithm 1 (line 25), to assess semantic similarity. the model is trained with a dual objective of semantic equivalence and linguistic attribute adherence. alternative approaches to ctg include energybased models that sample from a latent space @xcite @xcite . gradient-based methods like pplm @xcite ) steer generation using an attribute classifier at inference time but are often slow. fudge @xcite improves efficiency by re-weighting the next-token probability based on the desired attribute. existing work has focused on specific types of control. qcpg @xcite controls for three abstract attributes (semantic similarity, syntactic variation, lexical variation), while others focus on keyword presence @xcite . syntactically-controlled paraphrase generation has been explored by manipulating abstract meaning representation (amr) trees @xcite , reordering parse tree segments @xcite , or using constituency parse templates @xcite . other methods disentangle semantics and syntax by adding sentence parse trees or amr trees as features @xcite . in summary, previous works have primarily focused on a narrow set of linguistic attributes and often lack robust quality control mechanisms at inference time. in addition, llms are powerful general-purpose generation, but achieving finegrained, multi-attribute control is still a major challenge. our work addresses these gaps by introducing a model capable of controlling a large, diverse set of linguistic attributes simultaneously, complemented by a novel inference-time quality control mechanism to ensure both attribute adherence and semantic fidelity. ## experiments we evaluate lingconv on mrpc, sts-b, and qqp, using bertscore and mean squared error (mse) of attribute adherence (see appendix f). ## results ## conclusion we present a model for controllable text generation, offering control over 40 linguistic attributes and an effective mechanism for quality control at inference time, yielding a 12% improvement in output quality. we introduce the "novel target challenge", where models generate paraphrases based on attributes from an "irrelevant" sentence. the setting evaluates models' adaptability to novel attributes and acts as a robust test for controlled paraphrase generation models. in addition, we evaluate the model on the downstream application of generating synthetic data for augmentation. our model generates paraphrases that boost performance and can be used to mitigate dataset biases. future work can investigate mechanisms to handle contradictory or noisy attribute specifications to enable the model to resolve conflicts and prioritize constraints, and extend lingconv beyond english to multilingual and low-resource settings, where new attribute extractors and cross-lingual transfer will be needed.