CEV - LM : Controlled Edit Vector Language Model for Shaping Natural Language Generations (2024)

Abstract: As large-scale language models become the standard for text generation, there is a greater need to tailor the generations to be more or less concise, targeted, and informative, depending on the audience/application. Existing control approaches primarily adjust the semantic (e.g., emotion, topics), structural (e.g., syntax tree, parts-of-speech), and lexical (e.g., keyword/phrase inclusion) properties of text, but are insufficient to accomplish complex objectives such as pacing which control the complexity and readability of the text. In this paper, we introduce CEV-LM - a lightweight, semi-autoregressive language model that utilizes constrained edit vectors to control three complementary metrics (speed, volume, and circuitousness) that quantify the shape of text (e.g., pacing of content). We study an extensive set of state-of-the-art CTG models and find that CEV-LM provides significantly more targeted and precise control of these three metrics while preserving semantic content, using less training data, and containing fewer parameters.

## introduction as large-scale pre-trained language models allow the generation of more diverse and fluent text, controllable text generation (ctg) is crucial to meet the needs of different applications and audiences. for instance, complex ideas can be presented concisely to an expert, but non-technical audiences may need more context and a slower-paced introduction to grasp the same idea. existing ctg approaches empirically evaluate three types of control conditions: semantic (e.g., emotion, topics), structural (e.g., syntax tree, parts-of-speech), and lexical controls (e.g., keyword/phrase inclusion) @xcite . while this taxonomy covers a broad range of features, it does not target more complex figure 1 : generated examples of change in speed, volume, and circuitousness, metrics that define the shape of text, and stylized illustrations. the points represent the word embeddings of windows of text, {x 1 , ..., x n }. the original text has a lower value of the metric, and our generation (cev-lm) demonstrates a higher value. in summary, our contributions are as follows: firstly, we present the cev-lm framework to provide a lightweight, "tuning-knob" to control speed, volume, and circuitousness. we adopt a semiautoregressive paradigm to exploit both the generation quality of autoregressive models and the controllability of deep generative models. second, we propose a controlled edit vector approach where we preselect examples from a constrained similarity neighborhood to match our criteria and apply a controlled edit vector in the latent space to tune the desired attribute. lastly, we study a robust set of benchmarks used in ctg and demonstrate that cev-lm provides significantly more control and preserves both relevance and fluency ( §6) across both high and low-resource settings while using fewer training samples and parameters. ## related work in this section, we briefly introduce existing literature on controllable text generation ( §2.1) and the prototype-then-edit architecture ( §2.2). ## nonstandard control conditions we pick three non-standard control metrics to evaluate our approach: speed, volume, and circuitousness @xcite . speed is a measure of how quickly content moves in a given text, calculated as the distance traveled by consecutive windows of text. specifically, speed is equal to ## controllable edit vectors the prototype-then-edit architecture @xcite features three main components: a neural editor p edit (x|x ′ , z), an inverse neural editor q(z|x ′ , x), and an edit prior p(z). the inverse neural editor and neural editor combine to form the encoder and decoder of a variational autoencoder (kingma and welling, 2013), respectively. the neural editor is implemented as an autoregressive, sequence-to-sequence model with attention, where given x ′ as input and z, which is concatenated to the input of the decoder at each step, the model generates x. the edit prior is defined as @xmath0, the strength of the edit, is drawn from u(0, 10) and z dir , the direction of the edit, is sampled from a uniform distribution on the unit sphere. note that both z dir and z norm are vectors. the inverse neural editor is given the edit pair (x, x ′ ) and must infer the edit vector z. the difference between x and x ′ is represented as to exhibit control over our target attributes, we alter the prototype-then-edit model in two ways: neighborhood creation and edit vector perturbation. the former constrains the inferred edit vector to demonstrate the desired change in attribute within some tolerance ϵ. the latter encourages a perturbation to the edit vector in the desired direction to compensate for ϵ. constrained neighborhood creation: the likelihood of a sentence is formulated as p(x) = a sum over all prototypes x ′ is expensive, so we only sum over the x ′ that are lexically similar to x -a lexical similarity neighborhood, n (x). further, we create an additional constraint on the target attribute to ensure that inferred edit vectors from the inverse neural editor correspond to a specified change in that attribute. more formally, we define the neighborhood with a tolerance ϵ as controlled edit vector peturbation: we hypothesize that by altering the magnitude of z norm and the direction z dir , we can control the strength and behavior of the edit vector. expressly, we can condition the formulation of the inverse neural editor on the target attribute by defining q(z norm |x ′ , x) = n (∆, 1) • u(f norm , f norm + ϵ), where n is the normal distribution and u is the uniform distribution. ## experimental settings we train variants of cev-lm on the yelp restaurant reviews corpus @xcite . the corpus has over 5.84 million training and 2.08 million test reviews (english) foot_1 in the original similarity neighborhood (i.e., jaccard distance less than 0.5 foot_2 ). the dataset provides a broad variety of writing styles (e.g., formal vs. informal, positive vs. negative sentiment) and topics (e.g., hotels, food, service, etc.) to test our approach @xcite @xcite . ## results in this section, we evaluate cev-lm on the strength of control (section 6.1) as well as the relevance to the original text (section 6.2 and section 6.3). in section 6.4, we discuss tuning the tolerance hyperparameter and in section 6.4, the effect of the data distribution on training. we provide qualitative results in section 6.6. ## conclusion in this work, we present cev-lm, an inexpensive, semi-autoregressive language model that uses constrained edit vectors for controllable text generation. we compose an extensive set of controllable text generation benchmarks, and through quantitative and qualitative evaluations, we show that our approach leads to significantly substantially more control over nonstandard control conditions (e.g., speed, volume, circuitousness) while preserving semantic meaning. steerable natural language generation remains an open challenge, and we plan to continue improving our work in various directions, such as using a weighted mixture of cev-lm models to capture all potential target deltas and replacing pieces of our architecture with larger language models. ultimately, we hope to apply these models to subjective traits like memorability and persuasiveness, which are compositions of many smaller constraints (e.g., conciseness, readability, etc.).