What Question Answering can Learn from Trivia Nerds (2020)

Abstract: In addition to the traditional task of machines answering questions, question answering (QA) research creates interesting, challenging questions that help systems how to answer questions and reveal the best systems. We argue that creating a QA dataset—and the ubiquitous leaderboard that goes with it—closely resembles running a trivia tournament: you write questions, have agents (either humans or machines) answer the questions, and declare a winner. However, the research community has ignored the hard-learned lessons from decades of the trivia community creating vibrant, fair, and effective question answering competitions. After detailing problems with existing QA datasets, we outline the key lessons—removing ambiguity, discriminating skill, and adjudicating disputes—that can transfer to QA research and how they might be implemented.

## introduction this paper takes an unconventional analysis to answer "where we've been and where we're going" in question answering (qa). instead of approaching the question only as computer scientists, we apply the best practices of trivia tournaments to qa datasets. the qa community is obsessed with evaluation. schools, companies, and newspapers hail new sotas and topping leaderboards, giving rise to troubling claims @xcite that an "ai model tops humans" @xcite because it 'won' some leaderboard, putting "millions of jobs at risk" @xcite . but what is a leaderboard? a leaderboard is a statistic about qa accuracy that induces a ranking over participants. newsflash: this is the same as a trivia tournament. the trivia community has been doing this for decades @xcite ; section 2 details this overlap between the qualities of a first-class qa dataset (and its requisite leaderboard). the experts running these tournaments are imperfect, but they've learned from their past mistakes (see appendix a for a brief historical perspective) and cre-ated a community that reliably identifies those best at question answering. beyond the format of the competition, trivia norms ensure individual questions are clear, unambiguous, and reward knowledge (section 3). we are not saying that academic qa should surrender to trivia questions or the community-far from it! the trivia community does not understand the real world information seeking needs of users or what questions challenge computers. however, they have well-tested protocols to declare that someone is better at answering questions than another. this collection of tradecraft and principles can nonetheless help the qa community. beyond these general concepts that qa can learn from, section 4 reviews how the "gold standard" of trivia formats, quizbowl can improve traditional qa. we then briefly discuss how research that uses fun, fair, and good trivia questions can benefit from the expertise, pedantry, and passion of the trivia community (section 5). first, the questions. either you write them yourself or you pay someone to write questions by a particular date (sometimes people on the internet). then, you advertise. you talk about your questions: who is writing them, what subjects are covered, and why people should try to answer them. next, you have the tournament. you keep your questions secure until test time, collect answers from all participants, and declare a winner. afterward, people use the questions to train for future tournaments. these have natural analogs to crowd sourcing questions, writing the paper, advertising, and running a leaderboard. trivia nerds cannot help you form hypotheses or write your paper, but they can tell you how to run a fun, well-calibrated, and discriminative tournament. such tournaments are designed to effectively find a winner, which matches the scientific goal of knowing which model best answers questions. our goal is not to encourage the qa community to adopt the quirks and gimmicks of trivia games. instead, it's to encourage experiments and datasets that consistently and efficiently find the systems that best answer questions. ## the craft of question writing trivia enthusiasts agree that questions need to be well written (despite other disagreements). asking "good questions" requires sophisticated pragmatic ## why quizbowl is the gold standard we now focus our thus far wide-ranging qa discussion to a specific format: quizbowl, which has many of the desirable properties outlined above. we have no delusion that mainstream qa will universally adopt this format (indeed, a monoculture would be bad). however, given the community's emphasis on fair evaluation, computer qa can borrow aspects from the gold standard of human qa. we have shown example of quizbowl questions, but we have not explained how the format works; see @xcite for more. you might be scared off by how long the questions are. however, in real quizbowl trivia tournaments, they are not finished because the questions are interruptible. interruptible a moderator reads a question. once someone knows the answer, they use a signaling device to "buzz in". if the player who buzzed is right, they get points. otherwise, they lose points and the question continues for the other team. not all trivia games with buzzers have this property, however. for example, take jeopardy!, the subject of watson's tour de force @xcite . while jeopardy! also uses signaling devices, these only work once the question has been read in its entirety; ken jennings, one of the top jeopardy! players (and also a quizbowler) explains it on a planet money interview (malone, 2019): jennings: the buzzer is not live until alex finishes reading the question. and if you buzz in before your buzzer goes live, you actually lock yourself out for a fraction of a second. so the big mistake on the show is people who are all adrenalized and are buzzing too quickly, too eagerly. malone: ok. to some degree, jeopardy! is kind of a video game, and a crappy video game where it's, like, light goes on, press button-that's it. jennings: (laughter) yeah. jeopardy!'s buzzers are a gimmick to ensure good television; however, quizbowl buzzers discriminate knowledge (section 2.3). similarly, while triviaqa @xcite is written by knowledgeable writers, the questions are not pyramidal. pyramidal recall that effective datasets discriminate the best from the rest-the higher the proportion of effective questions ρ, the better. quizbowl's ρ is nearly 1.0 because discrimination happens within a question: after every word, an answerer must decide if they know enough to answer. quizbowl questions are arranged so that questions are maximally pyramidal: questions begin with hard clues-ones that require deep understandingto more accessible clues that are well known. well-edited quizbowl questions are created in phases. first, the author selects the answer and assembles (pyramidal) clues. a subject editor then removes ambiguity, adjusts acceptable answers, and tweaks clues to optimize discrimination. finally, a packetizer ensures the overall set is diverse, has uniform difficulty, and is without repeats. unnatural trivia questions are fake: the asker already knows the answer. but they're no more fake than a course's final exam, which-like leaderboards-are designed to test knowledge. experts know when questions are ambiguous (section 3.1); while "what play has a character whose father is dead" could be hamlet, antigone, or proof, a good writer's knowledge avoids the ambiguity. when authors omit these cues, the question is derided as a "hose" @xcite , which robs the tournament of fun (section 2.1). one of the benefits of contrived formats is a focus on specific phenomena. @xcite exclude questions an existing mrqa system could answer to focus on challenging quantitative reasoning. one of the trivia experts consulted in @xcite crafted a question that tripped up neural qa by embedding the phrase "this author opens crime and punishment" into a question; the top system confidently answers fyodor dostoyevski. however, that phrase was in a longer question "the narrator in cogwheels by this author opens crime and punishment to find it has become the brothers karamazov". again, this shows the inventiveness and linguistic dexterity of the trivia community. a counterargument is that real-life questionse.g., on yahoo! questions @xcite , quora @xcite or web search @xcite -ignore the craft of question writing. real humans react to unclear questions with confusion or divergent answers, explicitly answering with how they interpreted the original question ("i assume you meant. . . "). given real world applications will have to deal with the inherent noise and ambiguity of unclear questions, our systems must cope with it. however, addressing the real world cannot happen by glossing over its complexity. complicated quizbowl is more complex than other datasets. unlike other datasets where you just need to decide what to answer, in quizbowl you also need to choose when to answer the question. 12 while this improves the dataset's discrimination, it can hurt popularity because you cannot copy/paste code from other qa tasks. the cumbersome pyramidal structure complicates some questions (e.g., what is log base four of sixty-four). ## a call to action you may disagree with the superiority of quizbowl as a qa framework (de gustibus non est disputandum). in this final section, we hope to distill our advice into a call to action regardless of your question format or source. here are our recommendations if you want to have an effective leaderboard. talk to trivia nerds you should talk to trivia nerds because they have useful information (not just about the election of 1876). trivia is not just the accumulation of information but also connecting disparate facts @xcite . these skills are exactly those we want computers to develop. trivia nerds are writing questions anyway; we can save money and time if we pool resources. 13 computer scientists benefit if the trivia community writes questions that aren't trivial for computers to solve (e.g., avoiding quotes and named entities). the trivia community benefits from tools that make their job easier: show related questions, link to wikipedia, or predict where humans will answer. likewise, the broader public has unique knowledge and skills. in contrast to low-paid crowdworkers, public platforms for question answering and citizen science @xcite are brimming with free expertise if you can engage the relevant communities. for example, the quora query "is there a nuclear control room on nuclear aircraft carriers?" is purportedly answered by someone who worked in such a room @xcite . as machine learning algorithms improve, the "good enough" crowdsourcing that got us this far may not be enough for continued progress. eat your own dog food as you develop new question answering tasks, you should feel comfortable playing the task as a human. importantly, this is not just to replicate what crowdworkers are doing (also important) but to remove hidden assumptions, institute fair metrics, and define the task well. for this to feel real, you will need to keep score; have all of your coauthors participate and compare scores. again, we emphasize that human and computer skills are not identical, but this is a benefit: humans' natural aversion to unfairness will help you create a better task, while computers will blindly optimize an objective function @xcite . as you go through the process of playing on your question-answer dataset, you can see where you might have fallen short on the goals we outline in section 3. won't somebody look at the data? after qa datasets are released, there should also be deeper, more frequent discussion of actual questions within the nlp community. part of every post-mortem of trivia tournaments is a detailed discussion of the questions, where good questions are praised and bad questions are excoriated. this is not meant to shame the writers but rather to help build and reinforce cultural norms: questions should be well-13 many question answering datasets benefit from the efforts of the trivia community. ethically using the data, however, requires acknowledging their contributions and using their input to create datasets (jo and gebru, 2020, consent and inclusivity criterion). similarly, discussing and comparing the actual predictions made by the competing systems should be part of any competition culture-without it, it is hard to tell what a couple of points on some leaderboard mean. to make this possible, we recommend that leaderboards include an easy way for anyone to download a system's development predictions for qualitative analyses.