Refocusing on Relevance: Personalization in NLG (2021)

Abstract: Many NLG tasks such as summarization, dialogue response, or open domain question answering, focus primarily on a source text in order to generate a target response. This standard approach falls short, however, when a user’s intent or context of work is not easily recoverable based solely on that source text– a scenario that we argue is more of the rule than the exception. In this work, we argue that NLG systems in general should place a much higher level of emphasis on making use of additional context, and suggest that relevance (as used in Information Retrieval) be thought of as a crucial tool for designing user-oriented text-generating tasks. We further discuss possible harms and hazards around such personalization, and argue that value-sensitive design represents a crucial path forward through these challenges.

## introduction the evaluation of natural language generation tasks (such as automatic summarization, machine translation, and dialogue generation) is commonly framed as one of comparing an automated system's generated output to some reference output, foot_0 with the goal of achieving as close an alignment as possible. implicit in this experimental framing is the notion that for any given system input, there must exist a single, "correct" output. while this may arguably be a necessary simplifying assumption in terms of experimental evaluation, 2 this "one-size-fits-all" philosophy has also constrained the ways in which nlg systems are designed, trained, and deployed. for example, standard approaches to automated document summarization and machine translation rely only on the original source text, and do not typically take into consideration contextual factors involving the user or their situation of use. the problem is even larger when we consider nlg systems whose outputs are less firmly grounded to some specific input text, such as dialogue and question-answering systems. this has a number of negative effects on nlg system performance and utility. one such effect is system output that does not meet a given user's needs: for example, a summarization system that chooses to focus on different aspects of the source text than those in which the user was interested, or a machine translation system whose output is of an inappropriate register of formality. other effects are more subtle; for example, consider that, by designing systems to produce a single universal output, we amplify the effects of label and sample bias in training data @xcite , since when there is only one "right" answer, a statistical model will generally tend towards whatever it has seen most frequently during training. one example of this is the well-documented behavior of machine translation systems defaulting to "male" for certain categories of phrase in gender-marked languages @xcite ; another can be seen in the tendency for neural language models to overpredict frequent words @xcite . recent work by @xcite made a compelling argument for an increased emphasis on user-and task-level personalization in nlp applications, particularly those involved in classification or prediction. in this work, we build on their foundation and turn our attention specifically to tasks involving natural language generation. while the paramount importance of user-level personalization in nlg applications has long been known in our field @xcite , recent years have seen reduced emphasis on this aspect of system design and evaluation.we challenge the universalist simplifying assumption that underlies much of how such systems are built and evaluated today, and discuss how concepts from information retrievalspecifically different notions of relevance -may be fruitful tools for considering personalization in the context of common nlg tasks. we discuss several case studies in which an nlg system would benefit (or indeed, require) additional context about the user and their task, and discuss some possible challenges and harms that could be introduced by textual personalization in nlg. finally, we discuss how design methodologies that center the needs, goals, and rights of users may provide a crucial path forward to developing more robust and personalized nlg systems; specifically, we point to value-sensitive design @xcite ) as one such option. ## potential harms from textual personalization potential harms from personalization can be grouped into several categories. the first stems from the fact that personalized information systems must necessarily have access to and make use of personal information about their users, which brings with it the risk of that private data being improperly disclosed or otherwise misused @xcite . a second, related category of potential harm is that of such an information system being used by its operators to leverage its capabilities against the interests of its users; possible scenarios include targeted ad-vertisements for predatory educational or financial products being shown to vulnerable individuals, or a system that shows or hides certain job listings based on the gender of its user (or, in truth, based on the system's limited model thereof). an additional, more subtle family of risk has to do with the indirect consequences of widespread deployment of personalized information systems. in social media, and search engines, the choices made by the model developer when incorporating the users' data are based on the confirmation bias theory proposed by @xcite , describing the tendency to favor information that confirms one's prior beliefs. 8 the assumption is that presenting a user with evidence that supports their belief would increase their engagement, resulting in higher margins for these companies. strategizing around this goal was shown to contribute to the formation of 'filter bubbles'/'echo chambers' decreasing the users' exposure to diversity of perspectives @xcite as well as radicalization @xcite . this intellectual isolation can erode the healthy functioning of democracies, which thrives on exposure to multiple points of view @xcite @xcite . to illustrate this, golebiewski and boyd (2018) describe the phenomenon of searching on "fragmented concepts," where politically-involved queries return only results that support the query's narrative instead of exposing the user to both sides. golebiewski and boyd argue that intentionally avoiding returning the results containing the other side stems from the fear of having the search engine portrayed as politically biased. users are often unaware of the active role that personalized software is taking in this process; @xcite describes an interview with a participant who "admitted that her google searches rarely revealed alternative points of view. however, she did not consider how her returns were tied to her own search practices or google's algorithmic ordering of information." tripodi's interviews reveal how users ascribe the lack of alternative views to their nonexistence, rather than search manipulation. finally, another possible concern in the process of personalizing a system is that the system's assumptions (and thus its personalized model of the user) may be incorrect, resulting in output that is irrelevant at best, and misleading at worst. for instance, a recipe system supporting a user in the task of making enchiladas might instruct its user to "fold in the cheese"; depending on the user's level of experience as a cook, this may be an unfamiliar usage of the word "fold," and it may be more appropriate to instruct them to "gently incorporate the cheese with a scooping-and-folding motion" @xcite . similar issues could arise in other contexts, with more serious results; imagine a system providing personalized medical advice, for example. ## value-sensitive design to design a system that can address the shortcomings described in section 3, we can follow research in value-sensitive design (vsd, friedman and hendry, 2019) that is aimed at identifying the stakeholders and prioritize the values to instill in the system under the assumption that "in designing tools we are designing ways of being." how might this affect the direct or indirect stakeholders of the system? in this section, we focus on the goals of the user of the system. there is of course no single answer to what this looks like, but in this section we attempt to dissect the problem in order to evaluate the possible outcomes of personalization through the lens offered through vsd. one value achieved by personalization is that of accessibility of information. a summary written for a child may be realized differently than one for an adult. by learning about the user, the system can adjust the syntax/register/style it produces, as well as the content, to make it more comprehensible to the user @xcite . however, this requires that privacy also be taken into account, as personalized systems make use of data about their user. since these values may be at odds with each other, understanding their possible trade-offs is necessary. by cautiously balancing the most relevant data the system requires for personalization, as well as the privacy concern of its users, system designers may optimize the amount and types of data that the user may be required to turn in. another indirect outcome of accessibility of information can be shown in discourse generation, where, for example, the quality of the interaction can elicit user responses that can more easily (or more quickly, or more accurately) direct the system towards the user's goals of the conversation. in other words, an accurate and thorough (language) understanding of the user's response may help build a shared ground and ease the transfer of communicating ideas, intents, or requests which subsequently accomplish the user's goal in a conversation. safety can be another important value to be accountable for as developing models that interact with humans. @xcite elaborate how a dialogue agent can cause real world harm by proactively introducing harmful language or content, foot_7 or posing as an expert, which can mislead and create harm in the physical world (providing medical advice for covid-19, for instance). of course, source-to-target transformation tasks are susceptible to similar problems, as well. another opportunity such systems bear is that of enabling a proactive personalization that can provide a sense of agency to a user. this can be done through allowing them to define their specific needs and interests, and by that providing a sense of control over the system as described by @xcite . a personalized system can find ways to reason/explain on the type of data presented to its users, promoting transparency. for instance to paraphrase a sentence for a user, a system can indicate what were the relevant dimensions it maintained and why (see the source-to-target example in section 2.1). moreover, if a user knows how their data is used that may contribute to developing trust. @xcite identified two types of trust; the first, whether a user trusts an individual prediction sufficiently to take some action based on it, and the second, whether the user trusts a model to behave in reasonable ways if deployed "in the wild." we add to that a third sense of trust, can we trust the developer, that their knowledge integrated into these systems is serving the needs of the user faithfully and respectfully. projecting aspects of value-sensitive design into text personalization tasks is the first step towards making a human-centered communication system. ## architectures for personalization the previous section described the conceptual guidelines we propose, while this section is focused on practical aspects on how to implement a language generation system that allows for textual personalization to a situation or a user. at the very basic level, personalization of text can be done through keywords, much as a search engine attempts to return relevant text to a user in response to a query. however, keyword retrieval may not always recover the user's intent and as a result, the returns may not always be relevant. for instance the concept of "neural network" is highly associated with a particular sense, but has another sense that is actively used in the field of neuroscience, making some matches irrelevant. in this work we claim that the advances of natural language generation should be incorporated to personalization in ways that could benefit the user beyond good search engine returns. since many nlg tasks are based on language models, we propose architectures that can generate personalized text through the personalization of language models. @xcite present in their work a language generating system that enables generating text based on pre-trained language models, that can both apply pointwise as well as distributional constraints, while deviating a little as possible from the original distribution. the essence of this work in the context of personalization is the ability to generate text in a controlled fashion. the aspect of control is crucial for the development of future systems. one could adapt a language model to a particular user through conditioning on all outputs to speak about topics relevant to the user (a pointwise constraint) and have 50% of them to be related to topic discussed in the given query (a distributional constraint). 10 a possible concern, at this point, when employing this model has to do with the ability within the same model to adapt different distributional constraints quickly. another direction that could overcome this limitation presented by @xcite , where the model is conditioned on a prefix intended to provide context to steer the generated text. one way to incorporate this model is to condition the textual generation on different users, however, compromising centralized models bears higher costs to overall users' privacy @xcite ; instead, a per-user model can be trained on user's personal data, perhaps remaining entirely on their device. moreover, ideally, implementing personalized agents can be done through federated learning @xcite , where the model updates are made locally and globally through maintaining the user's privacy. 11 as an example of applying the prefix scenario (by @xcite ) to personalization, providing a prefix of "how does the email my mom sent relate to my sister?", together with the email in question as the source, would then generate a personalised textual output. while the focus of this work is mainly on text, it is plausible to assume that the greater objective of such work is to improve and enhance these communication models through the process extending the shared knowledge both parties maintain for the purpose of problem solving. therefore, personalization may likely be more effective through integrating modalities beyond text and should be a cross disciplinary effort. when considering architectural perspectives for textual personalization, the realization of the information can vary greatly both across different individuals and within the same individual. in their work, following a survey on communication preferences in adults, @xcite concluded the due to high variation, it is generally recommended to support several modalities of communication, showing that there is no one size fits all approach. in a survey on multimodality in educational setting, @xcite describes the importance of the impairment-specific approaches to accommodate individuals with specific disabilities and contributing to an inclusive environment; furthermore, takayoshi and selfe describes how "the more channels students (...) have to select from when composing and exchanging meaning, the more resources they have at their disposal for being successful communicators." the optimal modality may also depend on the situation, and it is reasonable to consider an event in which a user may opt for more than one modality within the same task; in the example of a driver asking siri "how can one recognize mount saint helens?", while an image would be a highly informative modality, it would provide a less optimal output at that precise point in time. instead, in order to avoid removing the driver's sight from the road, a verbal description would be a less risky alternative. at times one modality may complement another as shown by the work of wang et al. (2016) and zhu et al. (2018 visual appearance of clothes and matching styles are crucial in understanding the user's intention. returning to an earlier example, multimodality can help overcome a system's limitations. recall the example of a recipe system providing its beginnerlevel user with the (unhelpful) instruction to "fold in the cheese;" if the instruction was accompanied by a video demonstrating of the process, this could help the user not only accomplish their immediate goal but also to expand their capabilities for next time by learning new vocabulary. one major obstacle to the development of personalized nlg systems is that such systems often depend on access to sensitive and personal data from users, making large-scale data resources difficult (or impossible) to obtain and share across a community. while this is a major challenge, we can draw inspiration from related fields that must work with such data and have developed best practices around how to do so, such as the biomedical domain. one possible model is that of the mimic-iii @xcite dataset, which is made up of thoroughly deidentified electronic medical records from thousands of patients from beth israel deaconess medical center in boston, ma. this dataset is freely available to community members to use in research, but access is closely managed. before gaining access to the dataset, users are required to take an online human-subjects research training module, which also contains content on the protocols for working with the dataset. users of mimic-iii must also sign a data use agreement under which they (and, importantly, their institutions) legally agree to restrictions on the use and redistribution of the data. this solution has proven adequate for the mimic-iii dataset; however, there exist datasets where even heavily controlled redistribution is not an option. in such cases, data enclaves can be effective tools for community research on large, private datasets @xcite . one example of such an enclave is the national covid cohort collaborative (n3c) system @xcite , which is run by the national institutes of health and stores harmonized demographic and clinical data about millions of patients from participating health care organizations. after obtaining approval by their local institutional review board (irb) and executing a formal data use agreement, users who have been granted access are able to analyze and interact with the dataset in a secure cloud computing environment. the environment is designed such that data may not be removed or exported, and the platform includes capabilities for machine learning, statistical analysis, and data visualization. a similar enclave was used during the 2021 cl-psych shared task @xcite in an nlp context, in order to allow the research community to interact with sensitive data relating to mental health in an irb-controlled manner. a related approach has been used in information retrieval in situations where even limited and controlled access to the original data is not possible. under the "evaluation-as-a-service" paradigm @xcite , developers virtualize their systems, and send them to a secure computing environment where they are trained and evaluated on an entirely private dataset, with the results being shared among the community. while more limiting than a data enclave approach, experience in ir has shown that this model is a feasible one for shared-task evaluation @xcite . such a system could be constructed for the design and evaluation of nlg applications involving sensitive data about individuals. it may also be possible to augment more traditional data sets to simulate personalized behavior without actually requiring sensitive data. for example, in the context of a paraphrasing task, one might supplement the original passage to be paraphrased with a set of features or questions that the algorithm should use to ground its behavior; the same input passage could be repeated with different grounding questions, and the evaluation design could take into account the degree to which the system's output was responsive to the additional data. similarly, in a summarization context, one could augment an input document with different sets of questions representing different users' needs, and an evaluation could take into consideration how well the output summary addressed the stated questions. open domain questions-answering datasets and tasks could be similarly constructed. in dialogue systems, beyond the persona-based approach previously discussed, there is a long tradition of explicit user modeling @xcite @xcite with many approaches that could be drawn on in order to generate simulated needs or goals for such a system to meet. another useful avenue to explore is that of richer evaluation methodologies. rather than restricting our analysis to the system's output, truly assessing whether an nlg system is producing relevant personalized output necessitates extrinsic, task-based evaluations involving users. such methods have long been used in the evaluation of automated summarization systems @xcite @xcite as well as nlg systems @xcite @xcite , though we note that recent years have seen somewhat less of this sort of ecologically valid evaluation, and much more focus on statistical evaluation; the work of @xcite and @xcite represent examples of very welcome exceptions to this trend. ## future work in this opinion paper we argue for the benefit of personalizing nlg tasks. we hope that through this work and others' we will continue to make steps towards personalized text, from developing relevant focused datasets through methods that would make text more accessible and of practical use for real users.