Incorporating Stylistic Lexical Preferences in Generative Language Models (2020)

Abstract: While recent advances in language modeling has resulted in powerful generation models, their generation style remains implicitly dependent on the training data and can not emulate a specific target style. Leveraging the generative capabilities of a transformer-based language models, we present an approach to induce certain target-author attributes by incorporating continuous multi-dimensional lexical preferences of an author into generative language models. We introduce rewarding strategies in a reinforcement learning framework that encourages the use of words across multiple categorical dimensions, to varying extents. Our experiments demonstrate that the proposed approach can generate text that distinctively aligns with a given target author’s lexical style. We conduct quantitative and qualitative comparisons with competitive and relevant baselines to illustrate the benefits of the proposed approach.

## introduction with recent advances in unconstrained language generation @xcite , an emerging direction is to adapt such pre-trained language models to follow certain stylistic constraints @xcite . these approaches rely on the inherent properties of the training corpus to tailor generation to target characteristics; for example, implicitly learning author-stylized text generation by training on author-specific corpus @xcite and learning to generate formal text @xcite . however, it is desirable to have explicit control over certain stylistic aspects in such generation, for e.g., emulating lexical choices of an author in a generation, capturing syntactic constructs, inducing sentential preferences (active vs. passive) in generation. to this end, we propose an approach to adapt a pre-trained transformer-based language model @xcite , specifically gpt-2 @xcite , to generate text that aligns with given lexical elements of style by providing explicit rewards in a reinforcement learning framework. reinforcement learning (rl) has been successfully applied to several natural language generation tasks like summarization @xcite and paraphrase generation @xcite . rl overcomes the 'exposure bias' @xcite in cross-entropy based training of language models and allows for optimization with respect to non-differentiable objectives. however, existing explorations around the use of rl in generation tasks have been limited to rnn-based models due to issues surrounding stabilization of rl training on transformer models @xcite . @xcite further conclude that a transformer requires reordering of the normalization layer from output to the input streams along with a gated mechanism instead of residual connections to stabilize its training. building on this, we leverage the shifted position of the normalization layers in gpt-2 to train an rl framework with gpt-2 for aligning generated text to target lexical characteristics. recent work by @xcite explored rl frameworks with gpt-2 to generate text with different styles. however, they treat their target characteristics (sentiment and descriptiveness) as a binary variable (viz. +ve/-ve). it is non-trivial to extend their work to generate lexically-aligned text, since each of the target dimensions is a continuous value. our task further requires simultaneously aligning along multiple lexical dimensions calling for a rewarding strategy that accounts for multiple dimensions. to this end, our key contributions are: (1) an rl framework that introduces lexical style elements in a transformer-based language generation model; (2) a rewarding scheme that incorporates continuous multi-dimensional lexical elements; ## author's lexical style an author's writing style is a combination of several factors that include, but are not limited to, their lexical preferences, syntactic and sentential choices (e.g., active vs. passive voice, use of detached adjectival clause), discourse structure, narrative style, etc. to perfectly reproduce a given author's style, the language generation model should operate in accordance to all these factors. however, we limit ourselves to replicating an author's lexical style, which refers to their writing choices at word-level. brooke and hirst (2013a; 2013b) enumerate lexical style elements into subjective, objective, literary, colloquial, abstract and concrete categories. an author's choices of words in these categories define their lexical style. for example, rudyard kipling, known for classics of children's literature, had a higher tendency to use more concrete words (like, gongs, rockets, torch) unlike abraham lincoln, who being a political writer, used more abstract words (like freedom, patriotism) @xcite . since an author's style is an amalgam of preferences along these dimensions, our goal is to ensure simultaneous alignment to these multi-dimensional lexical preferences of an author. to quantify a target author's lexical preferences, following brooke and hirst (2013b), we compute normalized pointwise mutual information index (pmi) of each vocabulary word with every seed word of 6 categories using their co-occurrences in the emobank @xcite corpus yielding a raw style score for each category, for each word in the vocabulary. an author's affinity to a particular style is calculated by the fraction of positive style words in their corpus, yielding a 6-dimensional vector with each @xmath0, 1]. unlike (brooke and hirst, 2013b), we do not consider formality-informality pair due to the imbalance in the number of seed lexicons provided for formality and informality. with a suitable seed lexicon, our approach however can be extended to these two characteristics as well. ## proposed approach a language model (lm) g models generation of a sequence x as a task of sampling tokens x 0 to x m . here, a token x i is sampled from a probability distribution conditioned on the previously generated tokens episode unrolling: an agent (g in our case) in an rl framework, learns a policy π to perform a set of actions a i (i.e. generating tokens) resulting in a change of its states. the policy's action a i at a state s i-1 : {c, x 0 , . . . , x i-1 } results in the generation of a token x i which takes the model to state s i : {c, x 0 . . . x i }. we refer to the sequence of tokens e : {a 0 , a 1 ..., a t } generated by the lm as it arrives at the terminal state s t as an episode. instead of relying on a linguistic terminal property (such as end of sentence), we utilize length of the generated sequence as the terminal property of a state. this ensures that the lexical statistics across episodes are consistent while computing the rewards. for each context c, we unroll n episodes, x 1 to x n , enabling the policy to explore the space better. unlike the traditional multinomial sampling, we use the nucleus sampling @xcite that restricts the sampling to the 'nucleus' of the distribution for generating episodes. by dissuading the choices from the long tail of the distribution, nucleus sampling allows the framework to exploit the policy's learning (so far) and hit a balance between exploration and exploitation. rewarding strategy: brooke and hirst (2013b) quantify lexical characteristics using paragraphlevel statistics. extending this, we reward the model with r at the final action a t of the episode and give 0 reward to the intermediate actions a i , @xmath1. the overall reward r i for an action at a time step i, where the terminal time step is t and immediate reward received at step i is r i , is given by discounting the future rewards (sutton and barto, 2018) with a factor γ, setting γ = 1, we distribute the reward uniformly over the entire sequence; all the actions in a particular sequence receive same award irrespective of their position. since the style is considered at the sequence level, the position of token is irrelevant as long as fluency is maintained. ## experiments we used the 2, 857 books of 142 authors in gutenberg corpus @xcite and divided each author's corpus into train and test sets. we concatenate 50 paragraphs from each author's train corpus and use this for fine-tuning with language modelling loss for each author. to evaluate our model on unseen data, we set aside a subset of 5 paragraphs from each author's test corpus to be used as test context. having contexts from all authors removes any bias from author-specific contexts. we compute the average lexical vectors l avg for all authors and retain top 10 authors with maximum deviation from l avg for our experiments: charles darwin, albert einstein, michael faraday, john maynard keynes, abraham lincoln, john locke, john stuart mill, beatrix potter, bertrand russell, and herbert spencer. we use the 117m parameters version of the gpt-2 @xcite trained on webtext corpus and 50, 257 token invertible byte pair encoding to preserve capitalization and punctuation @xcite . the model is a 12-layers 12-head transformer with embedding size of 768. finetuning 1 gpt-2 on entire gutenberg corpus yields gpt-2 (baseline). finetuning further for one epoch on the target author's corpus with causal language modelling (clm) loss yields gpt-2 + ft. for rl finetuning, we use a batch size 1, 10 episodes for each context, context length 200, episode length 100 and 0.05 as . we explored self-critical sequence training (scst) @xcite and proximal policy optimization (ppo) @xcite for our rl setup and chose our episode unrolled scst because of its stability. figure 2 shows the mean reward curves averaged over 3 randomly drawn authors; scst does not help in improvement of rewards, perhaps due to the lesser exploration carried out in the vanilla setup leading to little-to-noimprovement in the lexical style of the generation. ppo rewards goes down a bit and stays almost constant, perhaps due to the failure of critic. approximating continuous lexical score is challenging unlike binary positive or negative style -the ones for which ppo has been successful @xcite , hence the critic finds it difficult to approximate the value functions. our episode unrolling based scst explores enough to improve rewards quickly while frequent cross-entropy objective training ensures that the improvements do not come at cost of fluency. note that the rewards saturate after a few steps as the nucleus of the distribution gets shifted towards target lexical style. we calculate the lexical vector l seq for each generated paragraph and compute the error against the target author's lexical vector l tar . for dimensionspecific error, we take absolute difference between target and generated value for each author and average across all 10 authors. for overall error, we calculate the rmse between l tar and l seq . we report the perplexity of the model on the contexts in the test set to measure its deviation from its general generation capabilities. we also quantify the alignment in relative ordering of target dimen-sions using the l 1 norm between l seq and l tar . this evaluates the generation of models on their ability to achieve the target author's relative ordering. in in the qualitative example in context (originally written by d. h. lawrence): till now he had a tight, hot place in his chest, with suppressing the pain, and holding himself in. there seemed no air when he breathed. but he walked almost lightly. to his orderly he was at first cold and just and indifferent: he did not fuss over trifles. so that his servant knew practically nothing about him, except just what orders he would give, and how he wanted them obeyed. that was quite simple. then the change gradually came. he had served the captain for more than a year, and knew his duty. this he performed easily, as if it were natural to him. the officer and his commands he took for granted, as he took the sun and the rain, and he served as a matter of course. it did not implicate him personally. i have said that the sole effect of my somewhat childish experiment-that of looking down within the tarn had been to deepen the first gpt-2 ( @xcite : singular truth of the pilgrimage of the infant. i asked how i was to give my obedience, and, having been told that i must respond by force with violence, i found i had nothing else, less a price and hunger, but the unimaginable change of love and the uniformity of his love that he manifested, in his actions and his conduct... gpt-2 (baseline): singular comfort i had felt to-day. the idea of it came back to me, with all its allurement in the old days, over and over again; that it had been a serviceable accomplishment, but a training is apt to all but overwhelm the rest. that, too, was the fact of the plunge, the sudden crisis of this large one. with the active man and the active woman, with the mistress and the husband, with the shop-lenderer and the governess albert einstein (gpt-2 + ft): singular feeling in me-or to strengthen the second. i don't exactly know the two; but both of them, even in the early days of my illness, occurred in the true light. i felt sorry, indeed, for my sailor, for his predicament, and for some of the miserable conditions of the household. perhaps, in so doing, i had really helped myself. it was not at all like a stock with its cartridges. the quartermaster and his company, the earl of albert einstein (ours) singular feature of the observations of the experimental method of physics. i shall leave the first of these observations (i don't wish to show that the experiments which have been referred to are successful) to the reader who, during a similar period of the subject, considers the relations of natural and experimental phenomena to the general principles of inductive theory. abraham lincoln (gpt-2+ft): singular feeling in me-and to restore the deepness of the conviction that i was a woman; but, in a personal way, i had achieved what was possible only by looking down. the emotion still hovered; and i recollect at that instant a certain feeling-i say feeling-as i struck the buckle. the feeling made me feel that i must be rather an offensive person to be of any use or inconvenience to me. the thought had not, of course, touched abraham lincoln (ours): singular aspect of this pleasant and yet intimate connection which that early talk had formed with the man i now occupy, and, instead of being self-contained and thoroughly historical, it had quite come to be represented by a carefully and strongly expressed recollection, as an aggregation of our relations and the signs and symptoms of an attachment. indeed, the meeting with meade's sense of belonging, as an influence on his future, had been rendered infinitely more interesting by the clear knowledge of a more definite direct relation to myself ## conclusion and future work we proposed an approach to incorporate lexical choices of a target author in the generations of a transformer-based lms. our quantitative and qualitative evaluations illustrate that our proposed method is successful in aligning lexical characteristics of generation with target author. we believe that our work can also lead to rewriting of the input content tailored to certain characteristics, if we can design additional rewards to retain content. we have not performed a complete human evaluation due to the high-level of required exper-tise among the annotators for this task, as pointed by @xcite . designing the feedback mechanism for such a human evaluation is nontrivial and has been left as a part of the future work along with designing rewarding schemes to capture other author-specific characteristics (e.g., syntactic choices, discourse structure). despite the lack of such evaluation, these results are promising and offer a plausible line of research for replication of an author's style.