Composable Text Controls in Latent Space with ODE s (2023)

Abstract: Real-world text applications often involve composing a wide range of text control operations, such as editing the text w.r.t. an attribute, manipulating keywords and structure, and generating new text of desired properties. Prior work typically learns/finetunes a language model (LM) to perform individual or specific subsets of operations. Recent research has studied combining operations in a plug-and-play manner, often with costly search or optimization in the complex sequence space. This paper proposes a new efficient approach for composable text operations in the compact latent space of text. The low-dimensionality and differentiability of the text latent vector allow us to develop an efficient sampler based on ordinary differential equations (ODEs) given arbitrary plug-in operators (e.g., attribute classifiers). By connecting pretrained LMs (e.g., GPT2) to the latent space through efficient adaption, we then decode the sampled vectors into desired text sequences. The flexible approach permits diverse control operators (sentiment, tense, formality, keywords, etc.) acquired using any relevant data from different domains. Experiments show that composing those operators within our approach manages to generate or edit high-quality text, substantially improving over previous methods in terms of generation quality and efficiency.

## introduction many text problems involve a diverse set of text control operations, such as editing different attributes (e.g., sentiment, formality) of the text, inserting or changing the keywords, generating new text of diverse properties, and so forth. in particular, different composition of those operations are often required in various real-world applications (figure 1 ). conventional approaches typically build a conditional model (e.g., by finetuning pretrained language models) for each specific combination of operations @xcite @xcite , which is unscalable given the combinatorially many possible compositions and the lack of supervised data. most recent research thus has started to explore plug-and-play solutions. given a pretrained language model (lm), those approaches plug in arbitrary constraints to guide the production of desired text sequences @xcite @xcite @xcite . the approaches, however, typically rely on search or optimization in the complex text sequence space. the discrete nature of text makes the search/optimization extremely difficult. though some recent work introduces continuous approximations to the discrete tokens @xcite @xcite , the high dimensionality and complexity of the sequence space still renders it inefficient to find the accurate highquality text. in this paper, we develop latentops, a new efficient approach that performs composable control operations in the compact and continuous latent space of text. latentops permits plugging in arbitrary operators (e.g., attribute classifiers) applied on text latent vectors, to form an energy-based distribution on the low-dimensional latent space. we then develop an efficient sampler based on ordinary differential equations (odes) @xcite @xcite to draw latent vector samples that bear the desired attributes. a key challenge after getting the latent vector is to decode it into the target text sequence. to this end, we connect the latent space to pretrained lm decoders (e.g., gpt-2) by efficiently adapting a small subset of the lm parameters in a variational auto-encoding (vae) manner @xcite . previous attempts of editing text in latent space have often been limited to single attribute and small-scale models, due to the incompatibility of the latent space with the existing transformer-based pretrained lms @xcite @xcite @xcite . latentops overcomes the difficulties and enables a single large lm to perform arbitrary composable text controls. we conduct experiments on three challenging settings, including sequential editing of text w.r.t. a series of attributes, editing compositional attributes simultaneously, and generating new text given various attributes. results show that composing operators within our method manages to generate or edit high-quality text, substantially improving over respective baselines in terms of quality and efficiency. ## composable text latent operations we develop our approach latentops that quickly adapts a given pretrained lm (e.g., gpt-2) to enable composable text latent operations. the approach consists of two components, namely a vae based on the pretrained lm that connects the text space with a compact continuous latent space, and ebms on the latent space that permits arbitrary attribute composition and efficient sampling. more specifically, the vae decoder p(x|z) offers a way to map any given latent vector z into the corresponding text sequence. therefore, text control (e.g., editing a text or generating a new one) boils down to finding the desired vector z that bears the desired attributes and characteristics. to this end, one could plug in any relevant attribute operators (e.g., classifiers), resulting in a latent-space ebm that characterizes the distribution of z with the desired attributes. we could then draw the z samples of interest, performed efficiently with an ode solver. figure 2 gives an illustration of the approach. latentops thus avoids the difficult optimization or sampling in the complex text sequence space as compared to the previous plug-and-play methods (e.g., @xcite @xcite . our approach is also compatible with the powerful pretrained lms, requiring only minimal adaptation to equip the lms with a latent space, rather than costly retraining from scratch as in the recent diffusion lm @xcite . in the following, we first present the latent-space ebm formulation ( §3.1) for composable operations, and derive the efficient ode sampler ( §3.2); we discuss the parameter-efficient adaptation of pretrained lms for the latent space ( §3.3); we then discuss the implementation details ( §3.4). ## experiments we conduct extensive experiments of composable text controls to show the flexibility and efficiency of latentops, including generating new text of compositional attributes ( §4.1) and editing existing text in terms of desired attributes sequentially or simultaneously ( §4.2). all code will be released upon acceptance. ## related work recent works on text generation can be divided into two categories. one generates desirable texts by directly modifying the text sequence space. the other operates on the latent space to obtain a representation that can be decoded into sequence with desired attributes. more detailed discussions can be found in section §c. ## conclusions we have developed a new efficient approach that performs composable control operations in the compact latent space of text, named latentops. the proposed method permits combining arbitrary operators applied on a latent vector, resulting in an energy-based distribution on the low-dimensional continuous latent space. we develop an efficient and robust sampler based on odes that effectively samples from the distribution guided by gradients. we connect the latent space to popular pretrained lm by efficient adaptation without finetuning the whole model. we showcase its compositionality, flexibility and firm performance on several distinct tasks. in future work, we can explore the control of more complicated texts.