Masked Language Models Know Which are Popular: A Simple Ranking Strategy for Commonsense Question Answering (2022)

Abstract: We propose a simple ranking strategy to solve a generative commonsense question answering (QA) problem. Compared with multiple-choice QA, it is challenging because the answers to a question are not unique and they are supposed to be popular and diverse. Our strategy exploits the dataset itself and negative samples that we collect from WordNet to train a ranker that picks out the most popular answers for commonsense questions. The effectiveness of our strategy is verified on different pre-trained masked language models (MLMs) in a pipeline framework, where an MLM reranks the generated answers. Further, we explore an end-to-end framework where MLMs are utilized to guide the generation of generative language models (GLMs). Taking advantage of reinforcement learning, we apply policy gradient to train a GLM with the rewards fed back by an MLM. Empirical results on ProtoQA dataset demonstrate that MLMs can acquire the ability to distinguish the popular answers and improve the typical answer generation of GLMs as well.

## introduction commonsense reasoning has been making progress over recent years @xcite @xcite @xcite , arising from the advent and wide application of pre-trained language models (plms). most current commonsense reasoning studies focus on multiple-choice question answering (qa), such as commonsenseqa @xcite and social iqa @xcite , for which a welldesigned model is required to determine which of the candidate choices can best answer the question. however, such multiple-choice qa models may not be helpful in practical scenarios where candidate * corresponding authors answers are not provided (e.g, answering a question asked in a search engine or during a conversation). towards this problem, boratko et al. (2020) present a novel question/answer dataset protoqa for generative qa, in which several plausible answers are generated as a ranked list, rather than selected from candidates. for example, as shown in figure 1 , given a question "name something that people often remember for a long time, even when they get old", a qa model is expected to generate commonsensical and typical answers which cover commonest clusters as many as possible. in this case, a combination of "first love", " friends", and "name" would receive the highest score when three answers are allowed. in this setting, generative language models (glms) are apt to generate plausible answers @xcite . however, in our preliminary experiments, we observe that glms such as gpt-2 @xcite , t5 @xcite , and bart @xcite have difficulty distinguishing the most typical answers from the rare ones. meanwhile, @xcite find that masked language models (mlms) such as bert @xcite and roberta @xcite which utilize bidirectional contexts are more capable of learning commonsense knowledge than unidirectional lms (unilms) such as gpt-2. based on this observa-tion, we pose a question: whether mlms can be utilized to promote the typicality of answers generated by glms? figure 2 : the pipeline framework and the end-to-end framework. in pipeline, the ranker reranks the generated answers to yield the final results. in end-to-end, the trained agent (glm) directly yields the final results; during training phase, the agent passes the generated answers to the environment and receives the feedback rewards to update its parameters. to this end, we propose a simple ranking strategy for mlms to model the typicality of answers. in this strategy, an mlm is trained with the dataset purely without extra knowledge. to further increase the discrimination of the mlm, it is trained with the original answers as positive samples and negative samples gleaned from wordnet @xcite . after training, it serves as a ranker to find out the most popular answers among the generations by a fine-tuned glm. on top of that, we attempt to take advantage of mlms' discrimination to improve the generation probability of typical answers by glms. inspired by reinforcement learning @xcite , we construct a network with two plms: an agent (glm) and an environment (mlm). we apply policy gradient @xcite to train the agent in three steps. first, the agent samples one answer for each question. second, the environment, which is trained ahead with our ranking strategy, calculates a reward for every generated answer. third, the agent updates its parameters according to both ground truth answers and the rewards from the environment. during inference, the final answers are generated by the trained agent without post-processing. the pipeline and the endto-end frameworks are illustrated in figure 2 . we design a series of experiments on protoqa to comprehensively examine our proposed ranking strategy. we develop our trials from unilm gpt-2 to t5 and bart, which are sequence-tosequence glms. for mlms, we investigate bert, roberta, and debertav3 @xcite . the effectiveness of our strategy is evidenced by the experimental results: a leap (over 11 points) in the pipeline framework and a modest improvement (around 3 points) in the end-to-end framework. our research reveals that mlms can learn to tell which answers are more popular, with little knowledge or even without knowledge. moreover, they can guide the training of glms by providing higher rewards for the popular answers, and consequently the trained glms gain higher generation probabilities of typical answers. ## preliminary we introduce the concept of reinforcement learning and the essential of policy gradient in this section. reinforcement learning. along with supervised learning and unsupervised learning, reinforcement learning (rl) is one of the three basic machine learning methods. it is composed of five elements: agent, environment, state, action, and reward. the agent takes actions within the environment and its states changes accordingly. reversely, the environment feedbacks a reward to the agent. policy gradient. in rl, the actor does not know whether an action is correct or not, it can only judge the quality of the action by the rewards. if an action gets more rewards, then the actor increases the probability of its occurrence; if fewer rewards, the probability decreases. given a neural network with parameters θ and its state-action sequence τ , the expected reward of this network is the sum of the product of the likelihood of each sequence p θ (τ ) and its corresponding reward r(τ ). the objective function is max θ rθ . in our task, an answer may not be absolutely right or wrong, so a reward r(•) is introduced to indicate the typicality of the answer. in our end-toend framework, the agent, environment, and action are a glm, an mlm, and answers generated by the glm, respectively. the glm is trained to receive a maximum expected reward. ## methodology in this section, we elaborate on our simple ranking strategy for mlms (i.e. the ranker or environment). then, we detail the implementation of policy gradient applied for the training of glms in the end-to-end framework. ## results in this section, we report results in both pipeline and end-to-end framework. ablation study, further analysis of the two frameworks, and case study are located in section 5.2, 5.3, 5.4, and 5.5. results on dev set are listed in appendix d and e. in addition, we test our method on multiple-choice dataset commonsenseqa @xcite in section 5.6. ## conclusion in this work, we propose a simple ranking strategy for masked language models (mlms) to find out typical answers from the generation of generative language models (glms). it is a knowledge-free strategy when training without the negatives. in addition, we apply policy gradient to the training of glms to improve the generation of glms in endto-end mode. comprehensive experiments demonstrate the effectiveness of our proposed strategy and the discriminative ability of mlms.