Building a Buzzer-quiz Answering System (2023)

Abstract: A buzzer quiz is a genre of quiz in which multiple players simultaneously listen to a quiz being read aloud and respond it by buzzing in as soon as they can predict the answer. Because incorrect answers often result in penalties, a buzzer-quiz answering system must not only predict the answer from only part of a question but also estimate the predicted answer’s accuracy. In this paper, we introduce two types of buzzer-quiz answering systems: (1) a system that directly generates an answer from part of a question by using an autoregressive language model; and (2) a system that first reconstructs the entire question by using an autoregressive language model and then determines the answer according to the reconstructed question. We then propose a method to estimate the accuracy of the answers for each system by using the internal scores of each model.

## introduction we use the term "buzzer quiz" to refer to a genre of quiz in which questioner reads quiz questions aloud and players answer by buzzing in as soon as they can predict the answer. a well-known example of a similar format to what we call a buzzer quiz here is the u.s. tv program jeopady!, in which contestants must buzz in with a lock-out device before trying to answer a question. however, in jeopady!, answers are only allowed after all the questions have been read aloud, whereas we assume a format in which answers are allowed while the questions are being read out. because of the importance of buzzing in quickly, players normally answer incomplete questions in buzzer quiz. quizzes have been studied as open-domain question answering (qa) tasks because they do not limit the scope of knowledge. however, the major datasets for open-domain qa tasks, like natural questions @xcite and triviaqa @xcite cointain complete questions. consequently, systems built using those datasets q (75% completeness): pete rose and this player are tied with ten 200-hit seasons each. this japanese outfielder played most of his career with the mariners, and currently plays for the marlins. confidence score: 0.991 a: ichiro suzuki correct q (25% completeness): pete rose and this player are tied with ten 200-hit seasons each. this japanese outfielder played most of his career with the mariners, and currently plays for the marlins. confidence score: 0.125 a: ty cobb incorrect we believe that the capability to respond to buzzer quizzes by answering incomplete questions could help replicate the human capacity to smoothly generate responses in a conversation by sequentially predicting the content of the dialogue. in this study, we first constructed a buzzer-quiz answering system that produces appropriate answers for incomplete questions, and we propose the methods for calculating the confidence scores for two different models. specifically, we constructed two systems: the gpt-only system, which directly generates answers in response to a question by using gpt @xcite ; and the gpt+dpr system, which generates answers through a retrieverreader approach using dense passage retrieval (dpr) @xcite , after completing the question via gpt. for the former system, we calculate a confidence score by using token output probabilities during answer generation, while for the latter system, we use scores that are used in the output of the model. ## proposed method we propose two types of buzzer-quiz answering systems based on open-domain qa systems. we also propose methods to estimate the accuracy of the answers in each system by using the internal scores in each model. ## experiments we conducted two experiments: an evaluation of the proposed buzzer-quiz answering system's accuracy, and an investigation of the effectiveness of the confidence scores for each model. we define question completeness as x% when a question is truncated after the first x% of the text in terms of the character count. for the accuracy verification, we applied the gpt-only and the gpt+dpr models to questions with completeness levels of 25%, 50%, 75%, and 100%. for investigation of the confidence scores' effectiveness, we evaluated the confidence scores for each model by examining the relationship between the confidence scores and the accuracy at each level of question completeness. ## conclusion in this study, we constructed two models for answering buzzer quiz questions, which have not been considered in previous research: gpt-only and gpt+dpr. then, we evaluated the accuracy for various levels of question completeness. furthermore, we investigated the relationship between the model's internal scores, which were treated as confidence scores, and the accuracy; as a result, the validity of using the internal scores of the models as confidence scores was confirmed. in the future, we consider the use of more powerful models like fid @xcite or @xcite to improve the correct answer rate for quizzes. we also would like to validate the differences in performance between our systems and humans.