Analyzing the Effect of Linguistic Instructions on Paraphrase Generation (2025)

Abstract: Recent work has demonstrated that large language models can often generate fluent and linguistically correct text, adhering to given instructions. However, to what extent can they execute complex instructions requiring knowledge of fundamental linguistic concepts and elaborate semantic reasoning? Our study connects an established linguistic theory of paraphrasing with LLM-based practice to analyze which specific types of paraphrases LLMs can accurately produce and where they still struggle. To this end, we investigate a method of analyzing paraphrases generated by LLMs prompted with a comprehensive set of systematic linguistic instructions. We conduct a case study using GPT-4, which has shown strong performance across various language generation tasks, and we believe that other LLMs may face similar challenges in comparable scenarios. We examine GPT-4 from a linguistic perspective to explore its potential contributions to linguistic research regarding paraphrasing, systematically assessing how accurately the model generates paraphrases that adhere to specified transformation rules. Our results suggest that GPT-4 frequently prioritizes simple lexical or syntactic alternations, often disregarding the transformation guidelines if they overly complicate the primary task.

## introduction large language models (llms) can, without doubt, generate fluent and linguistically correct language with relevance to given prompts @xcite . however, to what extent can they follow complex linguistic instructions and execute them in a meaningful way? to this end, we propose a systematic approach for analyzing llms in performing explicit, theoretically grounded paraphrase transformations in english, using a validated list of 25 linguistic operations @xcite . it is necessary to have knowledge of fundamental linguistic concepts to follow those specialized instructions. this study provides insight into the capabilities and limitations of llms when faced with such a demanding task. extending our understanding on the connections between linguistically grounded theories of paraphrasing and the practical abilities of llms, we hope to improve paraphrasing performance with explicit linguistic operations, with potential applications in text simplification @xcite , computerassisted language learning @xcite , machine translation @xcite and automatic summarization @xcite . we conduct a case study analyzing paraphrases generated by a representative state-of-the-art llm, gpt-4 @xcite , focusing on the abilities of the model to create meaning-preserving and diverse paraphrases using systematic instructions related to the 25 paraphrasing categories of @xcite . our analysis further looks into the complexity of individual transformations and how gpt-4 copes with them with varying degrees of in-context learning @xcite . furthermore, we study how humans perceive the produced paraphrases in terms of semantic similarity and linguistic diversity. the contributions of the paper are the following: (1) our study connects a descriptive theory of paraphrasing with generative language models and human perception of sentence-level semantic similarity. (2) we conduct a limited case study, in- full name abbreviation synonym substitution synonym antonym substitution antonym converse substitution converse change of voice voice change of person person pronoun/co-referent substitution pron./co-ref. repetition/ellipsis repetition function word variations func. word actor/action substitution actor/action verb/'semantic-role noun' substitution verb/sem. noun manipulator/device substitution manip./device general/specific substitution gen./spec. metaphor substitution metaphor part/whole substitution part/whole verb/noun conversion verb/noun verb/adjective conversion verb/adj. verb/adverb conversion verb/adv. noun/adjective conversion noun/adj. verb-preposition/noun substitution vp./noun change of tense tense change of aspect aspect change of modality modality semantic implication sem. impl. approximate numerical equivalences num. eq. external knowledge ext. knowl. ## background paraphrasing denotes variability in expressed meaning. vague definitions such as this one are typical ways of framing the concept of paraphrasing in nlp research @xcite . however, previous research in (computational) linguistics has presented various, more fine-grained typologies that outline the linguistic transformations defining paraphrasing. through the lens of existing paraphrase theories @xcite @xcite , @xcite empirically validate paraphrase examples from two corpora: the multiple-translation corpus @xcite and the microsoft research paraphrase corpus (mrpc) @xcite . they outline 25 concrete operations with systematic linguistic instructions of transformations that produce sentences with near-equivalent meaning. the perspective to these operations is mostly lexical, focusing on the specific lexical changes that can be made at the sentence or phrase level to create paraphrases @xcite . however, several of the operations trigger changes that would traditionally fall within the domain of syntactic theory. one such operation would be ellipsis. we list all the transformations defined in @xcite in correctly applying these transformations in automatic paraphrase generation requires the model to process fundamental linguistic concepts and accurately recognize the phrase-level transformations triggered by the defined lexical operations. furthermore, not every transformation is appropriate for every context. therefore, the model must thoroughly process the definition and have intricate semantic reasoning abilities to construct sentence pairs that are appropriately suited for the intended transformation. to this end, we analyze the capabilities of llms in producing paraphrastic sentence pairs given systematic linguistic instructions. the transformations span from simple local changes, such as synonym substitution (to build/to construct) or change of aspect (studying/studies), to more complex alterations, such as converse substitution (buy/sell). along with systematic, descriptive definitions, @xcite provide 1-3 examples for each paraphrase transformation. synonym substitution, for example, is defined as follows: 1 replacing a word/phrase by a synonymous word/phrase, in the appropriate context, results in a paraphrase of the original sentence/phrase. this category covers the special case of genitives, where the clitic 's is replaced by other genitive indicators like of, of the, and so forth. this category also covers near-synonymy, that is, it allows for changes in evaluation, connotation, and so on, of words or phrases between paraphrases. example: these definitions followed by a small number of examples can be utilized as such in prompts for few-shot in-context learning, where an llm is instructed to generate sentence pairs incorporating the specific transformations. as few-shot learning has been shown to be an effective approach for applying llms in various tasks @xcite , we focus on leveraging the framework of @xcite for evaluating few-shot learning with gpt-4 across a wide range of linguistic operations related to paraphrasing. in a contemporary work, @xcite analyze various paraphrase types generated by gpt-3.5 by employing more abstract linguistic definitions of paraphrase phenomena as defined by @xcite and @xcite . these phenomena comprise abstract linguistic properties, such as changes based on morpholexicon, structure, and semantics. each of these classes is further divided into subclasses and types, where one type (e.g., same-polarity substitution) can include multiple concrete transformations (e.g., synonymy, general/specific substitution, or exact/approximate alternations) @xcite . @xcite select 10 of such types for their analysis. many of the selected types focus on local substitutions, such as inflectional changes, punctuation changes, and spelling changes, while only a few focus on global changes that require intricate contextual understanding. as opposed to this, we use the typology of @xcite , which provides an empirically validated list of concrete linguistic transformations for generating paraphrases, along with their linguistic definitions and examples, covering a wider range of local and contextual transformations. these concrete definitions enable a precise assessment of which specific linguistic features are well-represented by the chosen llm and which areas the model still lacks sufficient knowledge in. ## related work previous work related to diverse paraphrasing has studied the generation of specific linguistic features, for instance on lexical (e.g., @xcite or syntactic level @xcite @xcite , or controlling for various granularities @xcite . additionally, previous research has presented various taxonomies of paraphrase types for better understanding of the diverse paraphrase phenomena. @xcite propose a typology of 24 paraphrase types spanning three levels of granularity, while @xcite define rephrasing modifications extracted from the revision history of wikipedia. less fine-grained categorizations can include for instance differences in specificity or tone @xcite . @xcite propose a list of 25 empirically validated paraphrase transformations with a systematic definition and examples of each transformation. detection and generation of diverse paraphrases leveraging a corpus of various paraphrase types @xcite has been proposed @xcite . in a concurrent work, @xcite leverage the linguistic phenomena defined in @xcite to generate specific types of paraphrases. meier et al. (2024) also gather human annotations to analyze the accuracy of gpt-3.5 across the different paraphrase types and to evaluate how human annotators rank the generated paraphrases. their findings are in line with ours, suggesting that llms struggle with performing more complex paraphrase transformations. conversely to the framework of paraphrase operations that we use, the phenomena outlined in @xcite can often manifest themselves in various surface-form alternations (i.e., one phenomenon can include multiple operations) as they attempt to capture the general phenomena rather than providing specific mechanisms for paraphrasing. furthermore, we focus on analyzing the performance of llms on various specific paraphrase transformations given their detailed linguistic definitions, and connect the theoretical perspectives of paraphrasing with generative language models and human understanding of semantic similarity. another line of related work has focused on benchmarking various pretrained language models, such as bert @xcite , across a diverse range of downstream tasks, e.g., glue @xcite , senteval @xcite , and sick @xcite , or a limited range of linguistic phenomena @xcite @xcite . diverging from this line of work, we focus on the capabilities of one state-of-the-art llm and connect human perception of semantic equivalence to the theory and practice of diverse paraphrasing. in particular, we propose a method and conduct a pilot study to analyze how llms manage semantic abstractions in the context of systematically defined paraphrase transformations. ## conclusions in this paper, we design a methodology for testing llms to analyze whether they can follow theoretically motivated instructions in the case of paraphrase generation. we utilize explicit linguistic prompts to guide complex transformations and evaluate the results based on human assessment. using this framework, we conduct a focused case study on the capabilities of gpt-4 in accurately generating paraphrases. this study is based on 25 paraphrase transformations provided in bhagat and hovy (2013), whose definitions of the transformations serve as prompts for few-shot learning. we have customized a web-interface for collecting manual annotations for the generated sentences in order to assess how accurately the model produces paraphrases that follow the specified transformations. our findings indicate that gpt-4 can effectively follow detailed linguistic instructions to generate paraphrastic sentence pairs through simple, local transformations. however, it often prioritizes simple lexical or syntactic substitutions for paraphrasing instead of following specified transformation guidelines. this is especially true when the transformations trigger more complex alternations, indicating limitations in controllability and its ability to process complex linguistic instructions. furthermore, increasing the number of examples for fewshot in-context learning does not seem to improve the model's ability to accurately produce paraphrase pairs involving complex operations. this suggests that the model may still lack sufficient proficiency in these linguistic structures. future work could include a more comprehensive evaluation of how additional few-shot examples, encompassing a broader range of operations, influence performance. the presented methodology opens many alternative directions for further research. the use of systematic linguistic instructions in text generation tasks is still very much under-explored. theoretically controlled prompts may help to further understand the abilities of llms to generalize and follow explicit rules and guidelines. such prompts can also be used to compare and benchmark different models about their abstraction capabilities, and the analysis of the results can also be combined with interpretability studies of the network itself in case model weights are openly available.