Question Type Prediction in Natural Debate (2024)

Abstract: In spontaneous natural debate, questions play a variety of crucial roles: they allow speakers to introduce new topics, seek other speakers’ opinions or indeed confront them. A three-class question typology has previously been demonstrated to effectively capture details pertaining to the nature of questions and the different functions associated with them in a debate setting. We adopt this classification and investigate the performance of several machine learning approaches on this task by incorporating various sets of lexical, dialogical and argumentative features. We find that BERT demonstrates the best performance on the task, followed by a Random Forest model enriched with pragmatic features.

## introduction questions are at the core of human communication and can be used in a variety of ways, from simply eliciting information from the hearer to communicating a speaker's standpoint. they are also crucial in argumentation, where they make up around 5% of all speech acts and are rarely left ignored @xcite . however, question type prediction faces a number of challenges. first, the lexical surface does not correlate with the function of the question (e.g., 'who would do that?' can either be a request for information on a set of entities or a rhetorical question communicating that no one would do that). secondly, context is assumed to be crucial for their interpretation, however, it is not exactly clear what features in the context are indeed relevant. and lastly, a large majority of computational work assumes a bipartite distinction into information-seeking and rhetorical questions, a classification that does not capture the variety of functions that questions fulfil in debate. there is, in fact, a third category of questions referred to as assertive questions that has been theoretically motivated @xcite and empirically tested @xcite . such questions are characterised by the speaker's intention to express their opinion while still seeking information (as opposed to information-seeking questions the main purpose of which is to elicit a response, and the rhetorical ones which do not necessitate one). with this paper, we examine the impact of different (1) context configurations and (2) combinations of carefully selected lexical, pragmatic and argumentative features for the question type prediction. we explore deep learning approaches that tend to effectively capture lexical features as well as statistical models that while still capable of representing lexical information, also benefit from categorical feature inputs. our results indicate the introduction of the third question type drastically increases the complexity of the task when compared to the binary classification. we find that bert achieves the highest scores when the input is enriched with lexical features of either the preceding material or response. furthermore, we report our second-best results with a random forest model that makes use of a rich pragmatic feature set. ## previous work there is a significant body of work on questions in computational linguistics in the context of questionanswering systems, i.e., question classification based on the type of information expected as an answer, which focuses predominantly on factoid questions. earlier approaches include extensive use of lexical and syntactic features combined with traditional statistical approaches @xcite @xcite @xcite , while recently more work utilizes extensive capabilities of the deep learning models @xcite @xcite @xcite questions in natural communication, however, are often used to elicit more than just factual information from interlocutors and instead serve a variety of communicative purposes. thus, to interpret the function of questions in discourse, researchers often adhere to a bipartite distinction: @xcite focus on identification of conversational versus factual questions, @xcite ; @xcite ; @xcite distinguish rhetorical questions from non-rhetorical ones; @xcite identifies information and non-information-seeking questions; @xcite categorize questions into unpalatable and not unpalatable ones from the perspective of abusive language detection. they adopt a range of approaches such as the use of lexical features like n-grams, pos tags, speaker roles and word embeddings as well as the modelling of the context surrounding questions. however, recent research in pragmatics suggests that question typology is a bit more complex than previously assumed. in their work, hautli-janisz et al. (2022a) and @xcite discuss a more fine-grained question typology that attempts to better capture the conversational functions that are fulfilled by questions in debate. in this work, we follow their distinction into information-seeking, rhetorical and assertive questions. information-seeking questions also called pure questions (pqs), those are used to elicit information from an interlocutor. for instance, in example 1, moderator fiona bruce seeks the views of the panel members on the matter of the voter id in the uk. 1 (1) fiona bruce: will voter ids protect the integrity of elections or just undermine the uk democracy? rhetorical questions with rqs, speakers express their own opinion or standpoint, illustrated in example 2 by liz saville-roberts's intention to communicate her dissatisfaction with the current state of the prison system in the uk. 2 the speaker poses the question without expecting to hear a response which is signalled by the fact that she continues talking. the black population in wales is over-represented by five times within the prison population of wales, 1 http://corpora.aifdb.org/qt30, json id: 21308 2 http://corpora.aifdb.org/qt30 , json id: 18464 assertive questions aqs serve the double purpose of communicating information and asking for confirmation/rejection from an interlocutor. in example 3, gillian keegan expresses her frustration regarding the police having to inspect every package due to the new brexit regulations, while at the same time expecting other panel members to agree with her opinion on the matter. 3 (3) gillian keegan: the police probably have the legal right to open every packet and inspect every sausage. isn't that unreasonable? ## data the data underlying our investigation is taken from qt30 (hautli-janisz et al., 2022b), the largest ever dataset of broadcast political debate. the corpus comprises the transcriptions of 30 episodes of the uk's talk show 'question time' (qt) between june 2020 and november 2021 and is manually annotated with inference anchoring theory (iat) @xcite , a framework that captures how argumentation unfolds and is reacted to in dialogue which allows us to extract questions of the three types as well as pragmatic features associated with them for the analysis. the questions dataset used in the current work contains 2 867 questions, with the split into training and test given in ## conclusion in this paper, we introduce a task of question type prediction using a more fine-grained typology than the typically adopted bipartite distinction and find that the introduction of the third class increases the complexity of the task drastically. while questions can be tricky to interpret as speaker intention, which is notoriously hard to capture, is often one of the main indicators of the question type, the task complexity is further increased by a high class imbalance in naturally occurring data. we tackle the task by adopting several approaches conventionally used for the binary question type prediction such as the use of lexical features and the incorporation of question-adjacent context as well as by using novel for this task pragmatic features including propositional relations and epistemic markers in responses to questions. we find that bert exhibits the best performance with an rf model trained on a combination of pragmatic features and unigrams taking second place. however, neither of these results is truly satisfactory, suggesting that current machine learning approaches are not yet powerful enough to reason about the nature of a question if we adopt a finer granularity into three types. a confusion matrices for the best-performing models in the context setting figure 2: bert in qu-resp configuration b confusion matrix for the best-performing model in the pragmatic and lexical feature setting