How Well Do You Know Your Audience? Toward Socially-aware Question Generation (2022)

Abstract: When writing, a person may need to anticipate questions from their audience, but different social groups may ask very different types of questions. If someone is writing about a problem they want to resolve, what kind of follow-up question will a domain expert ask, and could the writer better address the expert’s information needs by rewriting their original post? In this paper, we explore the task of socially-aware question generation. We collect a data set of questions and posts from social media, including background information about the question-askers’ social groups. We find that different social groups, such as experts and novices, consistently ask different types of questions. We train several text-generation models that incorporate social information, and we find that a discrete social-representation model outperforms the text-only model when different social groups ask highly different questions from one another. Our work provides a framework for developing text generation models that can help writers anticipate the information expectations of highly different social groups.

## introduction writers are often expected to be aware of their audience @xcite and to minimize the effort required for others to understand them, especially if they cannot receive immediate feedback. however, nlp tools for writing assistance are not often made aware of the social composition of the audience @xcite and the information needs that different people may have. preemptive writing feedback may therefore fail to help writers address the expectations of different people in their audience. this is especially important when the writer requests feedback from a specific group of people: in one post on a forum related to personal finance, a writer asks for help from financial "gurus" for advice about accepting a job offer. a system that can preempt the hypothetical audience's information needs would enable the writer to revise their original post and avoid possible information gaps @xcite . some online forums have already implemented crude solutions for this problem with automated reminders for writers to include basic information (e.g., location) in their post. providing writers with preemptive questions can help especially in domains where different social groups have diverse information expectations. in the earlier example about personal finance, the advice-seeker could adapt their original post with answers to hypothetical "expert-level" questions (e.g. "have you saved enough money for retirement?"), adding extra information that would enable experts to provide advice more quickly. we cannot predict everyone's information needs, but some social groups with similar backgrounds (e.g., domain experts) will likely have consistent patterns in information expectations @xcite . in this work, we evaluate several socially-aware question generation models with the goal of providing customized clarification questions to writers. our work contributes answers to the following questions: we extend an existing generation model to incorporate social information about the question-askers ( § 5.1). in automated evaluation, a token-based socially-aware model outperforms the baseline for questions that are "divisive" and questions that are specific to a social group, particularly with respect to location as a social group ( § 5.2.3, § 5.2.4). importantly, the research presented in this paper shows that there are significant differences across groups with respect to questions they ask, and that we can develop models that are more attuned to these differences. note that the goal of our work is not to improve the overall accuracy of a question generation system, but rather to develop methods that are sensitive to the needs of specific groups, thus paving the way toward technology that is available and useful for all. ## related work question generation question generation (qg) is unique among text generation tasks because it tries to address what a person does not know, rather than what they already know and want to write. qg systems are expected to create fluent and relevant questions based on prior text, in order to provide qa systems with augmented data @xcite and students with question prompts to help their learning @xcite . in addition to typical supervised learning approaches @xcite , reinforcement learning has proven useful, where questions are assigned a higher "reward" if they are more likely to have interesting answers @xcite and more relevant to the context @xcite . furthermore, work such as @xcite has proposed controllable generation techniques to encourage less generic questions, e.g. with higher difficulty. such controllable-generation systems often leverage human-generated questions from a variety of domains, including wikipedia @xcite , stack overflow @xcite , and twitter @xcite . to our knowledge, prior work in question generation did not leverage the prior expectations of the question-askers. while sometimes providing controls for difficulty, no datasets currently include information about the inferred background of the question-askers. it seems natural that a person's prior knowledge would shape the information that they seek in response to a particular situation, yet analysis of the impact of social information on question generation remains absent. this study tests the role of social information in question generation using a dataset of posts from online forums, which feature complicated scenarios that can result in different information expectations between social groups. language model personalization personalized language modeling often seeks to improve the performance of common language tasks, such as generation, using prior knowledge about the text's author @xcite . personalization can improve task performance and make language processing more human-aware @xcite , which ensures that a more diverse population is included in language models @xcite . to represent the text writer, personalized systems often integrate a writer's identity @xcite or a writer's social network information @xcite into existing language models. a more generalizable approach converts the text-writer to a latent social representation such as an embedding @xcite , to be combined with the language representation in a neural network model where the social and text representations are learned jointly @xcite . we draw inspiration from the contextualized view of personalization from @xcite , and we represent the question-askers based on their prior behavior with respect to the specific context of a given post. ## data in this study, we consider the task of generating clarification questions on information-sharing posts in online forums. we choose to study subreddits that have a high proportion of text-only posts, diverse topics, and where community members often ask information-seeking questions: advice (lifestyle improvement), amitheasshole (social norms in complicated we conduct extensive filtering to include questions that are relevant and that seek extra information from the original post. the details are available in appendix a. we summarize the overall data in ## defining social groups in this work, we assess the relevance of the question-asker's background in the task of question generation, by defining social groups and assessing their differences in question-asking.