C a M - G en: C ausally Aware Metric-Guided Text Generation (2022)

Abstract: Content is created for a well-defined purpose, often described by a metric or signal represented in the form of structured information. The relationship between the goal (metrics) of target content and the content itself is non-trivial. While large-scale language models show promising text generation capabilities, guiding the generated text with external metrics is challenging. These metrics and content tend to have inherent relationships and not all of them may be of consequence. We introduce CaM-Gen: Causally aware Generative Networks guided by user-defined target metrics incorporating the causal relationships between the metric and content features. We leverage causal inference techniques to identify causally significant aspects of a text that lead to the target metric and then explicitly guide generative models towards these by a feedback mechanism. We propose this mechanism for variational autoencoder and Transformer-based generative models. The proposed models beat baselines in terms of the target metric control while maintaining fluency and language quality of the generated text. To the best of our knowledge, this is one of the early attempts at controlled generation incorporating a metric guide using causal inference.

## introduction most content is created for a well-defined goal. for example, a blog writer often publishes articles to gain popularity and trigger conversations, and a columnist may write an opinionated piece to gather feedback. in marketing applications, these goals are business objectives that need to be optimized using the content shared with the customers. the validation of whether the goal was met or not is done by tracking metrics that capture the reader behavior. in social media, metrics include number of comments, likes, or shares whereas for a publishing house they are the number of views and * work done while at adobe research readers. these engagement metrics (hereafter, metrics) are proxy for target goals. based on historical content, textual content characteristics that successfully achieve the desired metrics can be assessed @xcite . guiding text generation models by these signals is important for meeting the required goals. while recent neural language models have shown tremendous success towards fluent text generation @xcite , achieving controlled, goal-specific generation is challenging. there has been work on text generation controlling for style, topic, or size @xcite . these methods are able to leverage content characteristics that are common between the definition of goal (i.e., control) and the text. however, for metrics that are not explicit in the text, controlled generation is non-trivial to codify. the challenge is introduced due to the fact that for external metrics, there is a need to first identify the relationship between the content characteristics and the metric and then to explicitly introduce a guide/constraint enabling the generator to learn the desired content properties. contrary to style, these choices might be difficult for a layman to manually identify and input to the generative models. textual content is an amalgam of various linguistic features -lexical, pertaining to word choices; semantics, concerned with the meaning; syntactic, relating to parts of speech tags; and surfacelevel features, comprising punctuation, word count, sentence count, etc. to avoid misinformation (or clickbait-y) generation, automated tools should be able to alter the syntactic and surface-level characteristics of text to meet the desired outcome. explicitly identifying features of interest that result in intended outcome can enable finer control. in this paper, we first discuss method to identify a subset of these features that have direct and significant impact on the outcome metric, derived from causality literature @xcite . a causally signifi-cant relationship helps encode the 'if this, then that' logic; adding such a guide for the generator can help ensure on-metric generation. in this paper, we propose causal guidance mechanism for two modeling frameworks that are used for metric-guided generation -conditional variational autoencoders @xcite and transformerbased language models @xcite . for conditional variational autoencoders (cvae), we modify the vae graph to introduce causal guidance. in transformer-based language models, we introduce causal guidance by adding causal losses for explicit feedback on causal features. our key contributions are introducing causal guidance frameworks for metric-guided, controlled text generation in cvae and transformer-based generative models. we experiment with a new dataset of news articles related to covid-19 along with the nyt-comments dataset, 1 showing improved performance against baseline methods. to the best of our knowledge, this is one of the first attempts towards controlled generation on engagement metrics and inclusion of causal guidance for controlled generation in generative models. ## related work the literature on text generation spans various generative models, including variational autoencoder (vaes), generative adversarial networks (gans), and sequential models. vaes have been used for unconditional @xcite , as well as constrained text generation @xcite . @xcite generate a sentence sequence y conditioned on the input sentence for machine translation, thus mimicking a sequence-to-sequence model. @xcite control sentiment and tense in text generation using discriminators with vaes. @xcite introduce an additional reconstruction network in cvaes for controlling linguistic features in dialog generation. as we show in our experiments, this does not adapt well to controlled generation where the relationship with the target goal is not as explicit in text. we identify these nuanced relationships between the text and the underlying goal and enable explicit control over the text features influencing the target outcome by modifying the vae graph. while vaes enable controlled generation, they do not generate fluent language with limited data. large transformer-based language models @xcite have shown efficacy in generating fluent language, allowing for fine-tuning for specific tasks on a smaller dataset while maintaining good language quality. @xcite introduce style control, such as domain (books, wikipedia, etc.), by conditioning the generated distribution on the style token y, i.e. p(x|y) = n @xmath0, y). the language model learns the conditional probability p(x i |@xmath1, y) by training on sequences of raw text prepended with the style control. this approach provides only weak control, especially if the variation in textual features for the same target metric is large. @xcite enable finer control over generation space by introducing the control y in various internal layers of the transformer network. @xcite control for a combination of lexical styles to reproduce author's styles using a rl framework for transformer-based language models. while style is well reflected in the choice of vocabulary and language distribution, the difference in the language distribution is not as apparent for an external metric as control. we observe that the external metric is more influenced by various syntactic and surfacelevel text features, as opposed to the underlying vocabulary. we achieve finer control over these by a causally aware generative language model. ## causal features identification to incorporate finer control over generation of text to achieve a specific target metric, we first identify features that contribute to the respective outcome. here, the outcome metric is the target value we wish to control. we consider various syntactic (e.g. noun/adjective count) and surface-level textual features (e.g. word/sentence/paragraph count) and measure their effect on the metric. consider two text choices -s1: "the dog sprinted ahead so fast, the girl had much hard time keeping up with it.", s2: "the dog sprinted fast ahead. the girl panted trying to keep up."; both meaningful and reasonable generations. say, textual content with less words per sentence and more sentences is better liked. in this case, word count would have negative effect on outcome metric and sentence count would have a positive effect. thus, the model should generate shorter sentences, resulting in s2. although this example uses semantically equivalent text pieces for illustration, we do not have such parallel instances for generation task discussed in the paper. in absence of parallel data, it is non-trivial to isolate the effect of a specific text feature on the outcome metric. thus, we turn to causal estimation methods to identify this effect without controlled parallel data. the hypothetical change in an input feature of text in the observed data is defined as an intervention, and the input feature in question is termed as the treatment variable (t). for a binary treatment, the effect of treatment on the outcome (i.e., y) in the i th text sample is defined as y 1 (x i ) -y 0 (x i ). here, y 0 represents outcome in absence of treatment and y 1 represents outcome when treatment is applied and x i are the other covariates (text features). the average treatment effect (ate) is the expected effect of providing the treatment (i.e. including a specific feature) and is given by e[y 1 (x i ) -y 0 (x i )]. this can not be directly calculated as we do not know what the outcome is if a certain part of text is changed in a certain way, i.e., y 0 (x i ) and y 1 (x i ) is not known for the same i. moreover, in observed data, the treatment assignment is not independent of baseline covariates. we account for this by employing a propensity-based scoring, which serves to balance treatment assignment in treated and untreated groups @xcite . the propensity score is defined as the probability of treatment assignment conditional on baseline covariates, i.e. π(x i ) = p(t @xmath2. we employ multi-layer neural networks to approximate propensity scores @xcite . the propensity scoring model is trained using the assigned treatment t i corresponding to the observed covariates x i with cross entropy loss. the average treatment effect (ate) can be estimated by inverse propensity treatment weighing (iptw) @xcite , where each outcome is weighed by inverse probability of receiving the corresponding treatment. thus, for a doubly robust estimate, we augment iptw with potential outcome model @xcite . the potential outcome models estimate outcomes if treatment is applied (@xmath3, given the other covariates. we model potential outcome using two neural networks (for @xmath4, 1), trained to minimize mean squared error in predicted and actual outcome in observed articles with @xmath5, respectively. the expected outcome in presence of the treatment feature is then a function of the observed outcome with treatment for the treated group and the predicted outcome with treatment for the untreated group, given article features, weighted by a function of the propensity scores. the average effect of the treatment feature on the outcome is estimated as the mean of the difference of expected outcome with and without treatment. this provides an estimate of which text features have the most impact on the outcome (target) metric. 2 the ate of continuous treatment features can be estimated in a similar fashion, assuming a normal treatment distribution @xcite . ## cam-gen we present a causally aware text generation method in vae and transformer-based models. in section 4.1, we begin by discussing the metric-guided ## results we compare causal and non-causal variants of the proposed cvae and transformer-based models. in the transformer variants, we evaluate the performance with metric added as a guide in embedding, attention, and normalization layers, trained with l g (eq. 7). next, we introduce the metric loss to add feedback for adherence to target metric, training the model with l g + l metric (eq. 8). the final proposed causal model is trained with l g + l metric + l causal (eq. 9). for cvae, non-causal and causal models are trained with l vnc and l vc (eq. 5, 6) respectively. we fine-tune a gpt-2 @xcite model with metric token added to the prompt for control, similar to @xcite , and use it as a baseline. we also use the method proposed by @xcite as the baseline cvae model. as seen in causal cvae exhibits better metric control than the non-causal and baseline cvae but performs poorer than the causal transformer model. this could also be an artifact of language quality since the underlying classifiers are trained on fluent language. across transformer variations, addition of metric loss and causal guidance improves metric control, validating our hypothesis. it is interesting to note that the perplexity drops substantially on adding the metric loss in transformer-based model. this could raise the question on how additional losses (constraints) could result in more fluent generation. we emphasize that, in baseline and all other variants, the constraint is on the target metric. thus, both baseline gpt-2 and modified transformer (with only l g ) attempt to align their generation space to this target. an inadequate alignment of generation space to the desired control is likely to result in noisy generations. in that sense, metric/causal do not add more constraints, rather add feedback to meet the specified constraint we generate text with high, medium and low target participation count (pcount) as target and record average value of various treatment features (fig. 5 ). here, the word and sentence counts are normalized and pos features are fraction of words with certain pos tag over total number of words in the generated text. we test the adoption of 'causally significant' features in the causal model by analyzing feature distributions of text generated by causal model and baseline transformer model across classes (high/medium/low). for instance, word count has a negative ate on pcount (fig. 4a ). thus, we would expect a text with higher word count to have lesser pcount. as seen in fig. 5a , our causal model with 'high' target pcount generated articles with lower word count on average than the causal model with 'low' target (red and blue bars in first group in fig. 5a respectively). similar trends are observed across other 'causally significant' treatment features. in contrast, the text generated by baseline model (fig. 5b ) either do not show significant variation in these features across text generated with high, medium and low target or the difference is inconsistent, reflecting the lack of control over aspects of text in baseline models where generation is only guided by target metric. as these features, by definition, significantly im-pact the outcome; this analysis adds further confidence in stronger adherence to the target metric in our proposed causal approach over the baseline. ## conclusion we present a framework for causally aware metricguided generation in vae and transformer-based models. we successfully identify causally significant text features using causal analysis and incorporate them into the generative model. we show that integrating causal guidance in guided generation enables better control over the target metric, while maintaining language quality. our proposed causally guided transformer model shows improved performance across datasets. moreover, we show that the generated text adheres to these causal features, in line with their observed effect in historic data. this exploration opens up avenues for leveraging causality for controlled generation. ethics statement. we recognize and acknowledge that our work carries a possibility of misuse for fake news generation, the same as any text generation system. we strongly recommend coupling any such technology with a fake news detection and review system before deployment. we do not believe that our method exacerbates fake news generation as it aims to optimize syntactic and surfacelevel features, and not topical or semantic features. on the contrary, having a causal guidance towards these specific factors may guide models to focus on these features and deter them from other nondesirable optimization of content. the data and approaches for generating text that optimizes for clicks exist already. our proposed approach adds a nuanced control on the linguistic features to optimize for generating desirable content, rather than unconstrained optimization for clicks. tiancheng zhao, ran zhao, and maxine eskenazi. 2017. learning discourse-level diversity for neural dialog models using conditional variational autoencoders. in proceedings of the 55th annual meeting of the association for computational linguistics (volume 1: long papers), pages 654-664, vancouver, canada. association for computational linguistics.