Controllable Text Generation for All Ages: Evaluating a Plug-and-Play Approach to Age-Adapted Dialogue (2022)

Abstract: To be trusted and perceived as natural and coherent, conversational systems must adapt to the language of their users. While personalized dialogue is a promising direction, controlling generation for fine-grained language features remains a challenge in this approach. A recent line of research showed the effectiveness of leveraging pre-trained language models toward adapting to a text’s topic or sentiment. In this study, we build on these approaches and focus on a higher-level dimension of language variation: speakers’ age. We frame the task as a dialogue response generation, and test methods based on bag-of-words (BoW) and neural discriminators (Disc) to condition the output of GPT-2 and DialoGPT without altering the parameters of the language models. We show that Disc models achieve a higher degree of detectable control than BoW models based on automatic evaluation. In contrast, humans can partially detect age differences in BoW but not Disc responses. Since BoW responses are deemed better than Disc ones by humans, simple controllable methods thus appear to be a better tradeoff between adaptation and language quality. Our work confirms the challenges of adapting to higher-level dimensions of language variation. Moreover, it highlights the need to evaluate natural language generation thoroughly.

## introduction developing dialogue systems that can hold humanlike conversations has been a long-standing goal in artificial intelligence (ai) research. this includes the ability to mimic speakers' speaking styles and language traits, which is shown to be of crucial importance for systems to be trusted and perceived as natural and coherent @xcite . current approaches in conversational models typically aim to improve dialogues by leveraging persona-specific traits-a speaker's age, gender, geographic location, etc. this is achieved by training systems with either implicit @xcite or explicit @xcite @xcite representations of a speaker. these approaches are generally shown to produce multi-turn conversations that are deemed of better quality by humans, but they pay little attention to understanding what factors determine human judgements. recently, @xcite showed that linguistic aspects such as specificity, relatedness, and repetition play an important role, and that explicitly controlling for them during generation increases human engagement in a conversation. however, fine-tuning these large models to control the generation turns out to be a challenging task, which is further limited by the scarcity of annotated conversational datasets. a recent growing interest in controllable text generation has been fostered by approaches leveraging large pre-trained language models (plms; see sec. 2.2). in particular, one direction is to operate at the decoding stage while leaving the underlying plm unaltered @xcite , which was shown to be successful in generating texts that adapt to a specific topic, length or sentiment. though these approaches are not typically aimed at modelling conversations, they have been shown to be also suitable to generate controlled responses to dialogue utterances @xcite . building on this line of research, in this study we explore adaptation by large plms to a specific, yet unexplored dimension: language variation due to speakers' age. the relationship between a person's age and their use of language is a thoroughly studied subject in linguistics and psychology, and various differences between younger and older speakers have been reported at the level of both content and style (see @xcite . while a few studies showed that speakers' age can be predicted both in discourse and dialogue (see sec. 2.1), no work to date has explored whether, and to what extent, age-related detectable features can be leveraged by controllable text generation models. in this study, we explore this issue for the first time. though previous work showed that some degree of adaptation can be achieved to a text's sentiment or topic, we argue that age-related traits are different since they involve subtle, fine-grained features lying at a more abstract level compared to other language dimensions. therefore, we hypothesize that this task is more challenging and possibly requires complex adaptation strategies. following the approach by @xcite , we experiment with dialogue data and frame the controlled generation problem as the task of generating a response to a dialogue utterance. we opt for this setup since it allows us to genuinely investigate language adaptation while leaving aside the extra challenges of modelling full, multi-turn dialogues. though generally short, single dialogue utterances are shown to contain a fair amount of age-related language signal @xcite . we employ the plug-and-play language model (pplm) method by @xcite and condition the generation of two large plms, gpt-2 @xcite and dialogpt @xcite , by means of various age-specific attribute models. with this approach, generation is steered leaving the underlying plm unaltered. we test two attribute models based on bag-of-words (bow) methods or more complex neural discriminators, and perform extensive evaluation of the generated outputs. through automatic evaluation, we show that (1) some degree of detectable age adaptation is achieved by all tested models, with (2) discriminator methods outperforming simpler bow strategies. at the same time, (3) bow models turn out to produce more fluent and less repetitive responses compared to the more complex models. these results are partially disconfirmed when moving to human evaluation. indeed, (1) humans can detect age-related differences in the generated language only to a very limited extent, and (2) this is restricted to responses by bow but not discriminator models. as for the quality of the generated language, (3) outputs by bow are deemed more fluent and human-like compared to discriminator ones, though this does not systematically correspond to a perceived better output. based on these results, bow-based controllable strategies appear to be a better tradeoff between adaptation and language quality compared to more complex methods. overall, our results confirm the challenges of adapting to higher-level dimensions of language variation, such as those due to speakers' age. moreover, we highlight the need of complementing automatic analyses with finegrained human evaluation. data and code to reproduce our experiments can be found here: https://github.com/lennertjansen/ pplm-age-adapt-dialogue . ## problem formulation in general terms, the problem we tackle is the following: given a dialogue utterance (prompt), we generate a dialogue response (output). ## method in our experimental pipeline, we condition the generation of two large pre-trained language models by means of two age-specific attribute models. in particular, we generate responses to a number of dialogue utterances used to prompt text generation. we then evaluate the extent to which the generated responses contain age-related features that can be detected by automatic metrics. ## results age adaptation tables 2a and 2b report the results by the younger-adapted and older-adapted models, respectively. as can be seen, discriminatorbased models (disc) achieve a higher degree of age control as detected by automatic means compared to bag-of-words (bow). this is particularly the case for the older setting, where both gpt2-based and dialogpt-based disc models outperform their bow counterparts by more than 30 accuracy points, with bow models being far below chance level. as for the younger setting, bow models perform comparably better by slightly underperforming (di-alogpt) or outperforming (gpt2) their disc coun- striking differences in performance can be observed between gpt2-and dialogpt-based models. while the latter clearly outperform the former in the younger setting (+13-18 acc. points), an opposite pattern is observed in the older setting, with gpt2-based models gaining 18-22 points over their dialogpt-based counterparts. this divergent pattern is interesting, and could be due to a youngerlanguage bias of dialogpt (fine-tuned on reddit threads, where the majority of users are in the age range 20-29), which would limit adaptation toward the older group. 6 on average, bow models are more effective in gpt-2 than dialogpt (56.7 vs 52.3), while disc results are on par (71.1 vs 71.3). taken together, these results show that the pplm approach is effective in controlling for age-related language features that can be detected by a trained 6 this is supported by the mean probabilities assigned by the bert-based classifier to the younger class on dialogpt and gpt-2 outputs: 0.76 for dialogpt, 0.62 for gpt-2. classifier, and that adaptation is stronger when using a neural discriminator attribute model. language quality moving to measures of language quality, we observe that the base gpt-2 is either the best or the second-best with respect to both perplexity and number of distinct n-grams. as for age-adapted models, bow ones are generally shown to outperform disc models. gpt2based bow models, in particular, appear to be the best overall age-adapted models: they compare to gpt-2 in terms of both fluency and diversity, which confirms that conditioning generation through wordlists does not negatively impact on the quality of the generated language. in contrast, disc models perform comparably worse on these metrics, which suggests a much bigger impact. since automatic metrics are known to have their own limitations (see, e.g., the case of perplexity in capturing language fluency; @xcite , in the next section we complement the results by the automatic metrics via extensive human analysis. ## analysis we run 3 crowdsourcing studies with human participants aimed at (1) exploring whether age differences detected by a classifier correspond to human intuitions on age-related language features; (2) assessing the quality of each model's generated language; (3) testing which age-adapted model produces the best outputs. based on their overall better age control and language quality, we choose to focus on gpt2-based models: gpt2 (hence, base), bow-younger, bow-older, discyounger, and disc-older. data collection is performed on appen (appen.com). participants are paid 0.08@xmath0/hour considering a conservative rate of 2 judgements/minute). in total, the full data collection costed around 2.5k$. the instructions given to the participants in the three studies we describe below are available in appendix d. participants were restricted to be from english-speaking countries and we used test-questions for quality control: only participants who correctly answered at least 70% of test-questions were considered trustworthy. ## conclusion we focused on age-related language variation and tested whether current approaches to controllable text generation can capture it in a dialogue response. we showed that models achieve substantial adaptation based on automatic metrics, while age-related differences can be detected only to a limited extent by humans. at the same time, simple controllable methods based on bow appear to be a good tradeoff between control and quality. from a broader perspective, our case-study on age adaptation reveals that controlling for subtle, fine-grained language features remains an open challenge. moreover, we show that human evaluation is crucial to assess the degree of achieved control since it provides different insights compared to automatic metrics @xcite .