Weakly Supervised Context-based Interview Question Generation (2022)

Abstract: We explore the task of automated generation of technical interview questions from a given textbook. Such questions are different from those for reading comprehension studied in question generation literature. We curate a context based interview questions data set for Machine Learning and Deep Learning from two popular textbooks. We first explore the possibility of using a large generative language model (GPT-3) for this task in a zero shot setting. We then evaluate the performance of smaller generative models such as BART fine-tuned on weakly supervised data obtained using GPT-3 and hand-crafted templates. We deploy an automatic question importance assignment technique to figure out suitability of a question in a technical interview. It improves the evaluation results in many dimensions. We dissect the performance of these models for this task and also scrutinize the suitability of questions generated by them for use in technical interviews.

## introduction asking good questions is crucial for assessing candidates in technical interviews. but this requires human experts with technical knowledge and experience. therefore, the capability to automatically generate technical questions to assess knowledge and understanding for a specific subject can significantly reduce expert effort in conducting interviews and in scaling up the interview process. in this paper, we focus on automated generation of interview questions from textbook contexts. there has a been a lot of interest in recent years on question generation (qg) @xcite @xcite , and many benchmark data sets exist @xcite . dhole and manning (2020); @xcite ; @xcite focus on rule-based question generation, wherein dhole and manning (2020) transform declarative sentences into question-answer pairs using syntactic rules, universal dependencies, shallow semantic parsing and lexical resources. these generate precise questions but often fail due to their heavy reliance on manually crafted feature sets. recent papers @xcite @xcite use deep neural networks for qg. @xcite focus on generating factoid questions from knowledge bases such as freebase. these use answers as clues to generate questions. @xcite ; @xcite ; @xcite propose answer-agnostic qg. @xcite predict answer-like candidates for the given passage and then generate questions from these. @xcite ; @xcite use transformer-based generative models such as bart @xcite and t5 @xcite . in summary, all existing models and benchmark datasets address factoid question generation for reading comprehension (rc). generating technical interview questions from a context is harder than generating rc questions. such questions focus on technical concepts and their relationships and test for knowledge and understanding of these. the answers are long-form (2-5 sentences) and must be contained in the context ('what is regularization?' is not answerable from the context 'this issue can be addressed using l2 regularization'). questions must be semantically complete ('what is the form of the optimal solution?' is an incomplete question) and of appropriate specificity ('what is machine learning' is too generic for assessing expertise in ml). finally, questions should have a diverse mix of intent and task complexity (compared to just remembering). as a result, rc-question generation models are not appropriate for interview question generation, and rc-question generation data sets are not useful for evaluating interview question generation models. to address this gap, we first create a dataset of textbook contexts and corresponding technical in-terview questions from two popular textbooks on machine learning (ml) and deep learning (dl). we use this for evaluation of interview question generation algorithms. generation of large volumes of gold-standard training data for this task is difficult and expensive. so, we first evaluate pre-trained large language models (llm) such as gpt-3 for this task in a zero-shot setting. we then explore fine-tuning a relatively smaller lm (bart) on weakly supervised training data. @xcite have recently explored gpt-3 for creating training data for many nlp tasks including factoid rc question generation. we explore the use of gpt-3 and template-based algorithms for silverstandard interview question generation from textbook contexts. we improve the question generation quality by developing a post-processing unit which filter out questions automatically by checking its suitability in a technical interview. we evaluate the models before and after filtering and see that it improves the performance in many evaluation dimensions. using detailed analysis of interview questions generated using these models, we highlight the challenges of this task, and also the key aspects that future models will need to address. ## technical interview question dataset there is no public dataset on technical questions from textbook contexts for evaluating our task. available question datasets for reading comprehension such as squad @xcite are not ideal. so we create a gold-standard dataset by manually generating questions on machine learning (ml) and deep learning (dl) from two publicly available books, @xcite for ml and @xcite for dl. we selected 3 chapters each from the ml and dl books. we distributed these chapters to 10 annotators (internal to our organisation) with subject knowledge as well as interview experience. we consider each section in a chapter as one context, and the annotators generate all possible long-form questions from each such context. the annotators are reminded of interview question templates (sec.3.2) but are not restricted to these. however, they are instructed to ensure answerability from context. each chapter was first annotated independently by two annotators. then, annotation differences, i.e., questions generated by one but not the other, are resolved via discussion. a question is added in the dataset only if both annotators are convinced after discussion, thus making the interannotator agreement 1 for the final dataset. we obtained 161 questions from 3 chapters of the dl book, and 187 questions from the 3 chapters of the ml book. we discuss quality aspects of the dataset in sec.4. we will make this dataset public. ## question generation approaches here we describe the different question generation and post-processing technique we use in the paper. ## experiments and analysis we present experimental evaluation and in-depth quality analysis of questions generated for test contexts by zero-shot and fine-tuned models using gold-standard technical interview questions as a reference. detailed hyper-parameter settings are included in the appendix. the results of our analysis before applying the importance-based question filtering are summarized in ## limitations in this work, we are trying to address what is the best way to evaluate the task of question generation for the technical interview. first limitation of this work is that we could not create a large annotated dataset as reference data as we need people with subject expertise for annotation. due to the same reason, we were forced to train the generative models like bart on silver-standard dataset generated using gpt-3. as gpt-3 is not freely available, the amount of silver-standard data is also limited in volume. the evaluation dimensions we propose for this task involve human effort and obviously this is not scalable. ## ethical impact statement generating interview questions using llms may have some ethical concern when these questions are used in actual interviews. in this work, we propose how to evaluate such questions in multiple dimensions. people should be careful before using such questions in actual interviews. ## conclusion in this paper, we explore the problem of technical question generation for interviews from textbook contexts in an weakly supervised approach. we curate a dataset for evaluation for two domains, machine learning and deep learning, using two popular books. we analyzed zero-shot question generation using gpt-3 and fine-tuning bart on silver-standard training data for the same. we also suggested a post processing filtering unit which further improves the quality of generated questions. our manual analysis brings out the complementary strengths and weaknesses of these approaches for this task. more importantly, our detailed error analysis highlights the challenges of the task and shows a pathway to better models by identifying the major types of errors to focus on. finally we devised automatic evaluation techniques which positively correlate with our manual evaluation. yao zhao, xiaochuan ni, yuanyuan ding, and qifa ke. 2018. paragraph-level neural question generation with maxout pointer and gated self-attention networks. in emnlp.