Engage the Public: Poll Question Generation for Social Media Posts (2021)

Abstract: This paper presents a novel task to generate poll questions for social media posts. It offers an easy way to hear the voice from the public and learn from their feelings to important social topics. While most related work tackles formal languages (e.g., exam papers), we generate poll questions for short and colloquial social media messages exhibiting severe data sparsity. To deal with that, we propose to encode user comments and discover latent topics therein as contexts. They are then incorporated into a sequence-to-sequence (S2S) architecture for question generation and its extension with dual decoders to additionally yield poll choices (answers). For experiments, we collect a large-scale Chinese dataset from Sina Weibo containing over 20K polls. The results show that our model outperforms the popular S2S models without exploiting topics from comments and the dual decoder design can further benefit the prediction of both questions and answers. Human evaluations further exhibit our superiority in yielding high-quality polls helpful to draw user engagements.

## introduction social media is a crucial outlet for people to exchange ideas, share viewpoints, and keep connected with the world. it allows us to hear the public voice for decision making and better understanding our society. nevertheless, for the silent majority, they tend to read others' messages instead of voicing their own opinions with words, possibly because of the introvert personality, busy schedule, and others. how shall we better engage them into the discussions and learn from their thoughts? in this work, we present a novel application to automatically generate a poll question for a social media post. it will encourage public users, especially those reluctant to comment with words, to figure 1 : example polls from sina weibo. p i , q i , and a i (@xmath0, 2) refer to the i-th source post, its poll question, and the corresponding poll choices (answers). different choices are separated by the ";". italic words in "()" are the english translation of the original chinese texts on their left. in the source posts, we fold the words irrelevant to polls in "..." for easy reading. to date, most progress made in question generation is built upon the success of encoder-decoder frameworks @xcite . despite of the extensive efforts made in this line @xcite @xcite , most previous work focus on the processing of formally-written texts, such as exam questions 1 weibo.com in reading comprehension tests. the existing methods are therefore suboptimal to handle social media languages with short nature and informal styles, which might present challenges to make sense of the source posts and decide what to ask. for example, from the limited words in p 1 , it is hard to capture the meanings of "b站" (b site) and "爱奇 艺" (iqiyi) as video apps, which is nevertheless crucial to predict q 1 . moreover, the question itself, being in social media fashion, is likely to contain fresh words, such as "c位" (center position) in q 2 , which may further hinder the models' capability to predict the poll questions in social media style. to tackle these challenges, we first enrich the short contexts of source posts with other users' comments; a neural topic model is employed to discover topic words therein and help identify the key points made in source posts. it is based on the assumption that the salient words in a source post are likely to be echoed in its comments @xcite , potentially useful to learn the map from posts to poll questions. for example, the core words in q 1 -"app" and "视频" (video)co-occur frequently in the comments with "b站" (b site) and "爱奇艺" (iqiyi), which may help the model to link their meanings together. the topic representations are then incorporated into a sequence-to-sequence (s2s) architecture to decode poll questions word by word. furthermore, we extend the basic s2s to a version with dual decoders to generate questions and answers in a multi-task learning setting and further exploit their correlations. for example, modeling answers in a 2 might help indicate that p 2 centers around "赵粤" (akira) and "希林娜依高" (curley g), two celebrities. to the best of our knowledge, this work is the first to study poll questions on social media, where their interactions among answer choices, source posts, and reader users' comments are comprehensively explored. as a pilot study over social media polls, we also contribute the very first dataset containing around 20k weibo polls associated with their source posts and user comments. 2 we believe our dataset, being the first of its kind, will largely benefit the research on social media polls and how they help promote the public engagements. on our dataset, we first compare the model performance on poll question generation in terms of automatic evaluation and human evaluation. the automatic evaluation results show that the latent topics learned from the first few pieces of user comments is already helpful -they result in our models' significantly better performance than the s2s baselines and their trendy extensions proposed for other tasks. for example, our full model achieves 38.24 rouge-1 while s2s with roberta @xcite yields 34.08. human evaluation further demonstrates our models' capability to generate poll questions relevant to the source post, fluent in language, and particularly engaging to draw user attentions for discussions. we then quantify models' sensitivities to the length of varying source posts and poll questions, where the scores of our model are consistently better. next, we find our model exhibits an increasing trend in predicting poll questions that will engage more comments in the future, which suggests the potential helpfulness of comments to indicate engaging questions. at last, the performance of dual decoder designs are discussed and it is shown that joint prediction of questions and their answers can benefit both tasks. ## poll question generation framework this section introduces our framework with two variants: one based on a basic s2s (single decoder) and the other is its extension with dual decoders to predict poll questions and answer choices in a multitask learning setting. the model architecture of the dual decoder model is shown in figure 3 . ## experimental setup data preprocessing. first, we removed meta data (e.g., author's locations and emoji labels) and replaced links, mentions (@username), and digits with generic tags "url", "ment", and "digit". then, for some poll questions echoed in the source posts, we took them away for fair experiments. next, an open-source toolkit jieba is employed for chinese word segmentation. 8 afterwards, we filtered out stop words and for the remaining, we maintained two vocabularies with the most frequent 50k words for sequences (input and output) and another 100k words for bow. finally, comments are capped at the first 100 words to examine poll question generation with the early comments and their potential to draw future user engagements. in evaluations, we split our data into 80% for training, 10% for validation and 10% for test. baselines and comparisons. for baselines, we first consider the basic s2s @xcite (i.e., base); also compared are the s2s with pretrained models from the bert family -tiny er-ine @xcite (i.e., erine), bert (devlin et al., 2019) (i.e., bert), and roberta @xcite (i.e., roberta), which were implemented with the paddle hub platform foot_7 . for all s2s with pre-trained models, their pre-trained parameters were further fine-tuned on our training data. then, we consider the following s2s extensions with copy mechanism (i.e., copy) @xcite , topic modeling from posts (i.e., topic) @xcite , and bidirectional attentions over posts and comments (i.e., cmt (biatt)) @xcite . all of them were proposed for keyphrase generation tasks and set up following their original papers. for our models, we consider two variants -cmt (ntm) in the single decoder archetecture and its dual decoder version dual dec. 10 model settings. all the hyperparameters are tuned on the validation set via grid search. for ntm, it is pre-trained for 50 epochs before joint training and afterwards different modules take turns to update parameters. we adopt two-layers bidirectional gru to build source post encoder and one-layer unidirectional gru question and answer decoders. the hidden size of each gru is 300. for a word embedding, the size is set to 150 and randomly initialized. in training, we apply adam optimizer with initial learning rate as 1e-3, gradient clipping as 1.0, and early-stopping strategy adopted. the weights to trade off losses in multitask learning is set to γ @xmath1. 8). evaluation metrics. we adopt both automatic measures and human ratings for evaluations. for the former, we examine two popular metrics for language generation tasks -rouge @xcite and bleu @xcite . for the latter, human annotators rates with 4 point likert scale (i.e., {0, 1, 2, 3}) and over three criteria are considered: the relevance to the source posts (relevance), how fluent the generated language reads (fluency), the attractiveness degree of the questions in drawing people's engagements (engagingness). ## experimental results in this section, we first show the main comparison results on poll question generation involving both automatic evaluations and human ratings (in §5.1). then, model sensitivity to varying lengths of source posts and poll questions are discussed in §5.2, followed by the analyses of models' capability to handle poll questions exhibiting varying degrees of user engagements ( §5.3). next, §5.4 discusses the performance of dual decoders that jointly generate questions and answers. a case study is presented at last (in §5.5) to interpret the sample outputs. ## related work our work is in the line with question generation, where most prior efforts focus on how to ask good exam questions given an article and the pre-defined answers. some adopt manually-crafted rules or features @xcite @xcite , largely relying on the labor-intensive process for rule design or feature engineering. to simplify the training, automatic feature learning hence becomes popular. for example, chali and hasan (2015) first employs a bayesian model to learn topic features and then leverages them to yield questions. these pipeline methods require the expertise involvement to manually customize the model inference algorithms, while our neural network design allows end-to-end training of topic modeling and question generation. recently, s2s-based question generation architecture has demonstrated promising results @xcite . to better encode the input, researchers adopt successful training design from other tasks, such as self-attention mechanism @xcite , language model pre-training @xcite , variational inference @xcite , and reinforcement learning @xcite . heuristic features, e.g., the answers' positions in the article @xcite ; base 你会看吗 (would you watch) roberta 你 平 时 喜 欢 哪 个 视 频 频 频 (which videooooo do you usually like) topic 你平时常用哪个视频 (which video do you usually use) cmt (ntm) 你平时在哪个视频网站 (which video site are you on) dual dec 你平时用哪个视频 app (which video app do you usually use) >bili 哔哩 (bilibili); 爱奇艺 (iqiyi); 腾 讯视频 (tencent video); 芒果tv (mango tv); 优酷 (youku); 其他评论区补充 (comment with other choices) base 你觉得谁的表现更强 (who do you think is better) roberta 你觉得谁更好 (who do you think is better) topic 你觉得谁出道了 (who do you think debuted) cmt (ntm) 你觉得谁更适合c位 (who do you think is more suitable for the center position) ## conclusion we have presented a novel task to generate social media poll questions. user comments encoded with a neural topic model are leveraged in a s2s framework; dual decoder architecture is further adopted to explore the interactions between questions and answers. extensive experiments on a large-scale dataset newly collected from weibo have demonstrated the effectiveness of our proposed model.