Socratic Question Generation: A Novel Dataset, Models, and Evaluation (2023)

Abstract: Socratic questioning is a form of reflective inquiry often employed in education to encourage critical thinking in students, and to elicit awareness of beliefs and perspectives in a subject during therapeutic counseling. Specific types of Socratic questions are employed for enabling reasoning and alternate views against the context of individual personal opinions on a topic. Socratic contexts are different from traditional question generation contexts where “answer-seeking” questions are generated against a given formal passage on a topic, narrative stories or conversations. We present SocratiQ, the first large dataset of 110K (question, context) pairs for enabling studies on Socratic Question Generation (SoQG). We provide an in-depth study on the various types of Socratic questions and present models for generating Socratic questions against a given context through prompt tuning. Our automated and human evaluation results demonstrate that our SoQG models can produce realistic, type-sensitive, human-like Socratic questions enabling potential applications in counseling and coaching.

## introduction researchers in education and psychology have recognized the role of cognitive biases in shaping a person's perspective towards learning and understanding @xcite @xcite . indeed, both pedagogical and counseling environments involve recognizing and alleviating any flawed cognitive biases in students/subjects through appropriate interventions by trained professionals @xcite . socratic questioning is one such intervention technique pervasive in education and psychotherapy @xcite @xcite . socratic questioning involves the use of specific types of probing questions that guide people into thought/passage: i believe that eating meat is ethically wrong. since we can easily substitute meat with vegan food without much nutritional complications, we have no logical reasons to continue eating living animals. we should stop killing these animals. ## dataset collection to construct a dataset of contentious viewpoints and the questions challenging these viewpoints, we consider the social news, content, and discussions website reddit, in particular, the subreddit r/changemyview, or cmv. the cmv platform is an active, targetted community described as "a place to post an opinion you accept may be flawed, in an effort to understand other perspectives on the issue" with people specially encouraged to "enter with a mindset for conversation, not debate". upon closer examination of the posts, we found that cmv posts pertain to various controversial topics (e.g. politics, media, culture, etc.) and, due to its very purpose of "change my view", subsequent comments often include questions aimed to evoke introspection and reflection (further discussed in section 2.1). we obtained the raw data from cmv using the "pushshift reddit" api provided by @xcite . after removing moderator comments that are not relevant to the topic, 3 we identify questions from comments following a given post using regular expressions comprising of question cues (e.g. who/what/where/when/why/how), and lexical indicators such as '?'. next, each identified question sequence is paired with the most relevant paragraph sequence (context) in the previous posts based on the similarity of the two sequences. sentence bert encodings @xcite were used for similarity computation. after manually inspecting several examples and their similarity values, we retain pairs having a similarity value above the threshold of 0.55. in total, we produce a dataset of 110, 050 english foot_3 (question, context) pairs from the cmv content generated during january 2013 and december 2021. on average, the number of sentences in each context is 4(±2) question type description exemplar clarification question probing the ambiguities of a thought. what do mean by ...? probing assumptions question probing the assumptions behind a thought. why do you assume ...? probing reasons and evidences question probing the justifications or concrete evidences that could have supported a thought. how did you know that ...? probing implications and consequences question probing the impacts or implications of a thought. if ..., what is likely to happen as a result? probing alternative viewpoints and perspectives question probing other possible viewpoints. what else should we consider about ...? others * question unrelated to the question types above (e.g. rhetorical, irrelevant, and/or illogical questions, etc.) who wouldn't want to be rich? ## socratic question generation to establish baseline performance on socratiq, we follow the current practice in nlp by fine-tuning state-of-the-art large pretrained language models (plms) with socratiq for socratic question generation (soqg ) (brown et al., 2020; peters 8 topic analysis results are included in appendix a et al., 2019; @xcite . we use commonly employed transformer-based language models, namely, generative pre-trained transformer or gpt @xcite , the text-to-text transfer transformer or t5 @xcite ), and prophetnet @xcite . all three models yield relatively high performance on various nlp tasks @xcite including qg on other datasets @xcite . qg models: following standard answeragnostic qg models @xcite , for our first set of baselines (gpt, t5, prophetnet), we directly fine-tuned the plms with socratiq paragraph contexts as inputs and questions as outputs. we devised a second set of models (gpt-p, t5-p, prophetnet-p) by employing prompt-based learning in keeping with recent developments in controllable text generation @xcite @xcite . in this second set of models, we use the socratic question-types as prompts in addition to the paragraph contexts as inputs for learning qg (figure 3 ). evaluation: we use standard n-gram based metrics used for measuring question generation performance @xcite , namely bleu (papineni et al., 2002a), meteor @xcite , and rouge @xcite . previous works have highlighted the problems of using n-gram based measures for qg @xcite . for soqg in particular, these measures are limited in handling equally valid paraphrases of the available reference questions. to address these limitations, we adopt recently-designed learnt metrics, namely, bert_score and bleurt @xcite @xcite . metrics based on the bert-based models @xcite were shown to provide robust evaluation for gpt @xmath0... kind of things. <de-@xmath1, why are they so bad at them when organized as a government? t5 input reasons_evidence: the government are people ... kind of things. t5 target if people are fantastic ... bad at them when organized as a government? prophetnet @xmath2... kind of things. prophetnet target if people are fantastic ... bad at them when organized as a government? figure 3: training input for each qg model. text in red shows is the prompt based on question type annotation. <@xmath3. text generation tasks such as summarization and translation. configuration r p f1 re 0.681 0.706 0.617 l train 0.797 0.929 0.828 (re d ) + l train 0.779 0.779 0.778 (l train + u re ) 0.708 0.691 0.627 selftrain 0.826 0.883 0.846 ganbert 0.784 0.930 0.802 l train + u v * 0.933 0.887 0.905 @xcite ). 9 for qg experiments, we use the implementations provided by the transformers library for gpt-2 10 and t5 11 and prophetnet. 12 on our experimental machine, all models take between 12 -19 hours for training depending on the task, specific model and the dataset used. 13 ## conclusions we created a novel dataset socratiq to support research on automatic socratic question generation. we applied latest research in prompt-based conditional text generation to fine-tune existing large language models from gpt, t5, and prophetnet to learn soqg. through our study and the release of this novel dataset, we take a first step towards enabling future research on models for soqg as well as impactful applications in areas such as counseling and education @xcite .