Similarity or deeper understanding? Analyzing the TED - Q dataset of evoked questions (2020)

Abstract: We take a close look at a recent dataset of TED-talks annotated with the questions they implicitly evoke, TED-Q (Westera et al., 2020). We test to what extent the relation between a discourse and the questions it evokes is merely one of similarity or association, as opposed to deeper semantic/pragmatic interpretation. We do so by turning the TED-Q dataset into a binary classification task, constructing an analogous task from explicit questions we extract from the BookCorpus (Zhu et al., 2015), and fitting a BERT-based classifier alongside models based on different notions of similarity. The BERT-based classifier, achieving close to human performance, outperforms all similarity-based models, suggesting that there is more to identifying true evoked questions than plain similarity.

## introduction asking explicit questions is essential for resolving information gaps, maintaining common ground, and fostering a shared view of the discourse topics. understanding implicit questions is crucial for the same reasons, and moreover underlies our perception of discourse coherence (e.g., @xcite ): a discourse is coherent if each utterance addresses a question implicitly evoked by what came before. despite their theoretical significance and their potential relevance to many natural language processing (nlp) tasks, the questions implicitly evoked by a discourse have not received much attention in nlp, where the typical questions sought after are, rather, comprehension-testing questions -although this has begun to change, see section 2. recently a crowdsourced dataset became available aimed at making implicit questions explicit: the ted-q dataset @xcite . we test to what extent the relation between a discourse and the questions it evokes is merely one if similarity or association, as opposed to deeper semantic/pragmatic interpretation. we do so by constructing a classification task from ted-q where the goal is to predict, given a preceding discourse, whether the given question was evoked at that point or not. we compare human performance, a bert-based classifier and four models based on different notions of similarity. in order to, moreover, investigate how the implicit questions of ted-q compare to explicit questions from existing (unannotated) corpora, which are far easier to come by, we construct an analogous task from written dialogue in the existing bookcorpus @xcite . 1 ## background questions have long been an important topic in nlp, but mostly in their role of comprehension-testing questions, i.e., questions that are answered by the discourse, rather than unanswered questions that it evokes, as in the current work. recently this has begun to change. for example, in terms of available datasets, the quac ('question answering in context') dataset @xcite consists of 100k questions asked in unscripted dialogue about a given topic; @xcite present a dataset of 75k clarification questions collected from stackexchange; @xcite attempts expert annotation of implicit and explicit questions in their role of structuring discourse (so-called questions under discussion, see below); @xcite crowdsource question-answer pairs as a more natural way of annotating discourse structure (qadiscourse); and @xcite present inquisitive, a dataset of 19k questions that are evoked by news texts, crowdsourced using a method very similar to the ted-q dataset on which we will rely (see section 3 for an overview of ted-q). on the modeling side, exceptions to the field's focus on comprehension questions are @xcite 's model of clarification questions, @xcite 's informative question generation based on quac, and the approach in @xcite towards ranking questions evoked by the preceding discourse, a task which they construct from annotations in the style of @xcite . in the more theoretical linguistics literature, the notion of an evoked question relates closely to the notion of question under discussion or qud @xcite @xcite . in qud-based approaches, discourse is assumed to consists of a number of moves, where each move addresses a specific implicit or explicit qud, and the quds are related and together form a structured whole, e.g., a tree with super-questions and sub-questions, where one discourse move addresses a question left open (or evoked) by another. despite its centrality in theoretical semantics and pragmatics for describing discourse structure, the notion of qud has not yet gained traction in computational linguistics and nlp. this is in part because of data scarcity: whereas discourse structure annotations in the relational approach have long been available (e.g., @xcite ; webber et al. (2019) ), qud annotation has only recently started @xcite . because evoked questions are potential quds for an upcoming discourse move @xcite , we hope that the current work on evoked questions can help empirically ground theoretical work on quds. the type of research question that we address -whether a given phenomenon, in our case evoked questions, reflects patterns that are more sophisticated than mere similarity -has been considered in relation to many other phenomena in the literature, where similarity-based baselines commonly serve either model comparison or, as in the current paper, dataset evaluation. ## constructing the tasks the ted-q dataset consists of 6 ted talks, annotated throughout with the questions evoked by the talk up to a certain point. 111 crowd-sourced annotators read excerpts of the talks and entered free-form questions that were evoked by the excerpt up to that point @xcite . this resulted in 2421 evoked questions, with on average 5 questions after every sentence. after entering a question the continuation of the excerpt was revealed, and annotators were asked to rate on a 5-point scale to what extent the question had been answered by the continuation (answered, used for analysis below). we take each evoked question from ted-q with its preceding context as a positive item, and combine the same context with a random nearby evoked question as a negative item. we chose our context size to be 3 sentences, guaranteeing that the context would include at least the context seen by the crowd worker of ted-q who posed the question. 2 for the negative items we sampled evoked questions from between 3 and 4 sentences away in either direction, a range on which we converged through manual inspection, aimed at maximizing the challenge (nearby questions will be more similar) without making the task entirely impossible. this resulted in 4824 items, half of which positive. we construct an analogous task from the bookcorpus, which consists of around 11000 books of fiction in plain text. we extracted 2 million multi-turn dialogues by extracting quotations, assuming that quotations within 200 characters of each other are part of a single dialogue, and that quotations on the same line constitute a single dialogue turn. for any turn that begins with a question and is preceded by a non-question, we took the question with its preceding context as a positive item, and the same context with a random nearby question as a negative item, using the same context size and negative question range as for ted-q. this resulted in 3.8 million items, half of which are positive. we divide each task into a train and test set (80%-20%) for the final evaluation. contexts can overlap considerably (e.g., for ted-q, every sentence has evoked questions, and every context has three sentences). we avoid overlap between train and test set by removing any context from the test set that has a neighboring (within 3 sentences) context in the training set. to minimize data loss we sample the train and test partitions not by single items, but by sampling contiguous subsets of items of 20 subsequent contexts each. we use matthew's correlation coefficient (mcc) @xcite as our main metric (see @xcite for arguments in favor of mcc in binary classification). ## models to test whether ted-q's evoked questions reflect patterns that are more sophisticated than mere similarity, we fit two kinds of models: random decision forests based on various notions of similarity, and the neural network transformer bert @xcite . 3 we use four similarity notions: (i) lem-maoverlap, the proportion of question lemmata that are also found in the context (using the spacy lemmatizer; www.spacy.io ); (ii) gleu @xcite , a slightly more syntax-aware measure of surface form similarity based on matching n-grams (we use the sentences in the context as separate references); (iii) meancos, mean cosine similarity between words in the question and words in the context (crossing all pairs), using glove vectors @xcite ; and (iv) bertscore @xcite , which measures the match between tokens of the question and the context based on their contextualized semantic similarity according to pre-trained bert (we use its f 1 -score, again using the sentences in the context as separate references). the first two concern word choice and some syntax; the latter two are more semantic/pragmatic. lastly, allsims combines all four notions in a single random forest. for each model we ran a hyperparameter grid search with 5-fold cross-validation on the training set. for the random forest we varied the number of estimators (100, 200, 300), maximum tree depth (1, 2, 4, 8, 16) and minimal items per leaf @xcite @xcite @xcite . observing only little difference (also between models), we chose 300 estimators with maximum depth 4, and minimally 8 items per leaf for all random forests. for bert we varied the warm-up ratio (0.03 (default), 0.1), learning rate (4e-4, 4e-5 (default), 2e-5, 1e-5, 4e-6), and weight decay (0 (default), 1e-7, 1e-6, 1e-5), finding the best and most stable results for warm-up ratio 0.1, learning rate 2e-5 and zero weight decay. ## results and discussion main results analyzing bert to analyze bert's behavior, two of us independently completed a portion of the task, resulting in 195 items with human annotations, reaching mcc scores which are similar to bert's: 0.55 and 0.60. annotation reliability by unweighted cohen's κ @xcite reached 0.66, or a substantial level of agreement according to landis and @xcite . 4 on these 195 items, bert makes 32 errors (21 false positives; 11 false negatives), 12 of which were also incorrectly annotated by both annotators (either because they fit but they were actually non-evoked in ted-q, or because, despite being actually evoked in ted-q, they were somewhat vague). of the 21 false positives, 10 were correctly annotated by both annotators. these 10 are mostly questions that easily fit in multiple places (e.g. "what would be an example of this?") or questions that are related to the context but not quite in the right way (e.g. in a context mentioning eclipses but not their effects "do eclipses have any other effects?"). of the 11 false negatives, 4 were correctly annotated by both annotators; 3 out of these 4 are again quite general and would fit in many contexts (e.g. 'what exactly is being talked about?'). see the supplementary material for more examples (1-15). we also looked into bert's errors by using another source of human judgments: ted-q's existing answered ratings (see section 2). we find no significant difference in answered between correctly and incorrectly classified items (t(872)=-0.29, p=.76), but a difference between false positives and false negatives (for bert, cossim, bertscore and allsims): questions of false positives are significantly more answered (bert: t(194)=-2.13, p=.03; cossim: t(372)=-2.30, p=.01; bertscore: t(295)=-2.77, p=.005; allsims t(235)=-2.68, p=.007). we take this to reflect that questions that are subsequently answered tend to fit the discourse better than questions that remain unanswered, to such an extent that they can seem to be evoked even by neighboring contexts (hence false positives). comparing the similarity models as for the similarity-based models, what is arguably the simplest notion of similarity performs best: lemmaoverlap. the slightly more syntactic notion, gleu, is a lot worse, reflecting that although questions and the preceding context (which are mostly assertions) may share some content words, they tend to have quite a different structure -this is due for instance to subjectverb inversion in questions, but also due to the fact that the relation between question and context is not one of answerhood (contrary to, e.g., question answering tasks). 5 the more semantic notion of lexical similarity, meancos, is surprisingly bad: apparently surface form matching (lemmaoverlap) is more reliable than semantic matching (say, association). 6 comparing lemmaoverlap and bertscore, our best similarity-based models, shows similar precision on the positive items while the former wins on recall (0.78 vs. 0.52). (for some examples see (16-20) in the supplementary material.) the foregoing findings may reflect one (or both) of two aspects of the ted-q dataset. one is the more general linguistic tendency for interlocutors to literally re-use each other's words where possible rather than variations and synonyms (pickering and garrod ( 2004 )), which likely plays a role in ted-q as well. the second is the possibility that reusing words from the context where possible is a cognitively easy way of completing the ted-q crowdsource task: simply repeat some words from the context and put a question word in front of it (note that the ted-q elicitation task prevented workers from directly copy-pasting a selection from the context). however, the comparison to explicit questions, discussed next, where lemmaoverlap is likewise the best notion of similarity, provides some evidence against it being a mere crowdsourcing artifact. comparison to explicit questions in bookcorpus lastly, ## conclusion and outlook the ted-q dataset appears to give us a decent window on the questions implicitly evoked by a text: the task we constructed cannot be solved by means of similarity alone, while the bert-based classifier comes close to expert human performance (though human performance was evaluated for only a small subset of the data). moreover, we observed some theoretically interesting patterns in the data through error analysis and by means of a comparison to explicit questions from the bookcorpus. for instance, the most successful similarity notion was plain lemmaoverlap, reflecting that evoked questions tend to reuse material from the discourse -where the fact that we see the same for the bookcorpus data speaks against this being a mere crowdsourcing artifact. the data we analyzed is richer than we could cover in this paper. for instance, bert and lem-maoverlap are much better on wh-questions (mcc 0.61/0.53, respectively) than on polar questions (0.30 and 0.17, respectively), suggesting a different kind of relationship to the context, which we hope to investigate in the future. it would also be good to collect a more comprehensive human benchmark on the tasks, expand our error analysis and consider a wider range of theoretically inspired models. we call that our flower petal starshade. if we make the edges of those petals exactly right, if we control their shape, we can control diffraction, and now we have a great shadow. it's about 10 billion times dimmer than it was before, and we can see the planets beam out just like that. question: what exactly is being talked about? question: what more would they like to be doing? false negatives by bert also coded incorrectly by both annotators. masters are not experts because they take a subject to its conceptual end. they're masters because they realize that there isn't one. now it occurred to me, as i thought about this, why the archery coach told me at the end of that practice, out of earshot of his archers, that he and his colleagues never feel they can do enough for their team, never feel there are enough visualization techniques and posture drills to help them overcome those constant near wins. question: how does that apply in practical terms here? no one loses their inner demons by taking to the road. addiction is real, the elements are real, freight trains maim and kill, and anyone who has lived on the streets can attest to the exhaustive list of laws that criminalize homeless existence. who here knows that in many cities across the united states it is now illegal to sit on the sidewalk, to wrap oneself in a blanket, question: where do they go? the world is changing in some really profound ways, and i worry that investors aren't paying enough attention to some of the biggest drivers of change, question: what is the change? the world is changing in some really profound ways, and i worry that investors aren't paying enough attention to some of the biggest drivers of change, especially when it comes to sustainability. and by sustainability, i mean the really juicy things, like environmental and social issues and corporate governance. i think it's reckless to ignore these things, because doing so