A Weak Supervision Approach for Predicting Difficulty of Technical Interview Questions (2022)

Abstract: Predicting difficulty of questions is crucial for technical interviews. However, such questions are long-form and more open-ended than factoid and multiple choice questions explored so far for question difficulty prediction. Existing models also require large volumes of candidate response data for training. We study weak-supervision and use unsupervised algorithms for both question generation and difficulty prediction. We create a dataset of interview questions with difficulty scores for deep learning and use it to evaluate SOTA models for question difficulty prediction trained using weak supervision. Our analysis brings out the task’s difficulty as well as the promise of weak supervision for it.

## introduction for effective technical interviewing, it is important to know the question difficulty -the probability of a student from a cohort, e.g. senior undergraduate cs students, correctly answering the question. we address the problem of predicting the difficulty of interview questions for candidate cohorts. predicting difficulty from the question statement, answer choices and related documents has been studied for multiple choice or factoid questions for reading comprehension and exams @xcite @xcite @xcite @xcite @xcite . all publicly available datasets @xcite @xcite also contain multiple choice or factoid questions. the nature of technical assessment questions in interviews is different. these look to assess knowledge and understanding rather than memorization of facts and are more open-ended. answers are long-form, typically spanning 2-5 sentences. existing approaches, particularly recent deep models @xcite @xcite , require large volumes of candidate response data to train the models. this is a challenge when creating a question bank for a new domain or a subject, since field tests need to be performed with real students. in contrast, we explore training question difficulty prediction models using weak supervision based on subject textbooks and bloom's taxonomy. this removes dependence on candidate responses and answer assessment. we explore various strategies of creating weaklysupervised training data. weak supervision has been explored extensively for many nlp tasks @xcite @xcite . for question difficulty, the training data requires not just difficulty scores but interview questions as well. we explore pre-trained large language models (gpt3) and template-based algorithms for generating training questions. we then assign difficulty to these questions using an unsupervised algorithm that uses subject textbooks and bloom's taxonomy @xcite . while bloom's taxonomy has been used extensively in computer educational testing @xcite and for analysis of difficulty for short answer questions @xcite , but not in predictive models. for evaluation, we create a dataset of interview questions with difficulty scores from an authoritative textbook on deep learning. we use this to evaluate the performance of state-of-the-art qde models @xcite when trained using weak-supervision. our analysis highlights both the challenges of the task as well as the promise of weak-supervision for it. our contributions in this paper are as follows. ## dataset we created a dataset for evaluating interview question difficulty prediction. we made this dataset publicly available foot_0 . we focus on deep learning and use the book "deep learning" by courville et. al. available freely online. first, annotators familiar with technical interviewing and deep learning generate interview questions from different chapters of this book. a chapter was given to 2 annotators who reached agreement over validity of generated questions for use in interviews. next, we needed to annotate these questions for difficulty on a scale of 1-10. we define higher (lower) difficulty as indicating lower (higher) probability of getting the correct answer from a candidate who has studied this book, and does not have any other exposure to this subject. attempts to directly annotate difficulty of individual questions led to very low inter-annotator agreement. instead, we annotated relative difficulty for a pair of questions with 3 possible labels: (a) q1 more difficult, (b) q2 more difficult and (c) equally diffi-cult/easy. we introduced a difficulty explanation label for individual questions in a pair. possible values were (i) lot of pre-req, (ii) little pre-req, (iii) lot of mathematics, (iv) little mathematics, (v) wellhighlighted answer, (vi) hard-to-find answer, (vii) about fundamental concept(s), (viii) about niche concept(s), and (ix) other. annotators were advised to decide the pair-wise label considering the explanations for the two questions. the final dataset has 150 unique questions from 16/20 chapters of the book. the questions are well distributed over cognitive tasks (sec.3.2) and templates (sec.3.1). 360 question pairs were selected for annotation after running our unsupervised difficulty prediction algorithm (sec.3.2) to ensure nontriviality of the pair-wise decision. there were 30 unique annotators and each pair was annotated by 5 annotators. after the first round, inter-annotator agreement was 0.23 fleiss kappa (fair), and 60/360 questions had a tie. these were broken by 2 addi-tional annotators. the final distribution over labels is 100 q1, 130 q2 and 130 equal. ## weak supervision in this section, we address weak supervision (ws) approaches for question difficulty prediction. ws has been extensively explored for various nlp tasks. one specific challenge is that the training dataset needs not only difficulty scores for questions, but also the questions. generation of questions is also expertise intensive and gold-standard questions are small in volume. therefore, ws needs to generate both questions and difficulty scores. ## experiments and analysis in this section, we report our experiments on the interview question difficulty dataset. we test the usefulness of the following aspects for weaksupervision (ws): (a) difficulty scores predicted by our unsupervised algorithm, (b) algorithmgenerated questions, and (c) questions from a related subject. figure 2: performance vs threshold θ for question generation algorithms for trqde[dl] all experiments were run on an a100 20gb server. we used adam with batch-size 16, learning rate 1e-5, dropout rate 0.5 and 50 epochs. we evaluated two state-of-the-art models for difficulty prediction for factoid / mcq questions. r2de @xcite regresses on questions to predict difficulty. we consider the ques_only version, since we do not use answers. it uses tf-idf representation of the questions. we report performance for linear regression as the regression model, which was the best. trqde @xcite uses transformers to represent the question, with a final regression layer. it fine-tunes the transformer mlm layer using the question corpus, and then further fine-tunes it for the regression task. we report performance for distilbert, which worked better than bert. here too, we used the q only setting. for the unsupervised difficulty prediction algorithm uqdp, we use @xmath0.8 for combining context and task difficulty. the primary ws training data covered all 20 chapters of the deep learning book, and had 2536 questions (gpt3:1647, template:889). the secondary ws training data covered 20 chapters from "pattern recognition and machine learning" (bishop), also available online. this had 2218 questions (gpt3:1268, template: 950). since the test set has relative difficulty labels, equal is predicted when the difference between a model's predicted difficulty scores for the two questions in a pair is less than or equal to a threshold θ. the main results are shown in tab.1. first, ws using uqdp generated difficulty scores for algorithm generated questions improves performance beyond that achieved by using uqdp alone for mlm-fine-tuned versions of trqde. this shows the usefulness of both aspects (a) and (b). however, r2de and the trqde with just regression-layer fine-tuning cannot beat uqdp. next, we analyze aspect (c). note that uqdp scores difficulty of the dl questions in the training data using the dl book and those of the ml questions using the ml book. still, including ml questions to train the regression layer does not help, even after including the ml book to fine-tune the mlm layer. the most likely explanation is that the test questions and difficulties are from dl. including ml questions changes the train distribution, even though the subjects are quite related. fine-tuning the mlm-layer fits the altered training distribution more closely, leading to poorer results in test. in fig. 1 , we show how micro f1 varies across threshold θ for the 3 best models. this reveals a more nuanced picture. while peak performance of dl-only training is higher, including ml questions in training results has more stable gains across θ values. however, including the ml book for mlm fine-tuning results in worse performance than both. we investigate aspect (b) further in fig. 2 by plotting performance vs θ when training using different question generation algorithms. we see that performance is the best when using both template and gpt3 generated questions. but, interestingly, templates have better performance individually than gpt3 across θ values. this is very likely because template questions, though smaller in volume and lacking diversity, better mimic the human interview questions seen in test. tab.1 showed that uqdp itself has competitive performance on the test set, outperforming r2de and trqde w/o mlm fine-tuning. we investigate aspect (a) further in tab.2 by performing ablation over different uqdp features. we see that context similarity makes the most significant contribution but adding task similarity improves performance slightly. the contributions of verb and wh similarity are limited compared to template similarity. ## error analysis we now report results of error analysis for our best performing model trqde[dl](dl). the question pairs in the data belong to 3 groups. (a) same-task-different-context: the questions belong to the same bt level, but are from different book contexts and are about unrelated concepts. (b) same-context-different task: the questions are from the same context or about related concepts, but belong to different bt levels. (c) different-contextdifferent-task: the questions are about unrelated concepts / different context and belong to different bt levels. in our labeled data, 36%, 18% and 46% are from groups a, b and c respectively. the third group is the most challenging for relative difficulty labeling. this is so even for human annotators. the fleiss kappa scores for inter-annotator agreement are 0.23, 0.27 and 0.21 respectively. note that while the tasks (question templates) are labeled by annotators when annotating question, while concepts of a question are obtained by eliminating stopwords, wh-words and prepositions using nltk libraries. we analyze difficulty prediction errors for each group separately. the errors are of two types. ol-1 (ordinal loss 1) errors occur when the predicted and true relative difficulty differ by 1, i.e. the predicted (or true) label is equally difficult/easy and the true (or predicted) label is q1 more dif-ficult or q2 more difficult. ol-2 (ordinal loss 2) errors occur when the predicted and true labels are the two extremes, i.e. the predicted (or true) label is q1 more difficult and the true (or predicted) label is q2 more difficult. overall, ∼ 32.5% of the predictions of trqde[dl](dl) correspond to ol-1 errors and ∼ 13% to ol-2. fig. 3 shows a group-wise drilldown. first, we observe that the total error is highest for group c, as expected, as is ol-2 error, demonstrating that it is the hardest group. between groups a and b, total error is slightly higher for b, indicating that predicting task-difficulty is a bigger challenge than context difficulty. deeper analysis provided further insights into the prediction errors of trqde. one of these stems from an underlying assumption for uqde that context difficulty is higher for later parts of a book. however, concepts introduced earlier are often revisited in later chapters in the context of related concepts. uqde assigns context difficulty incorrectly in such cases and corrupts training data for trqde. for example, graphical models and their types are first introduced when structured probability models are introduced in 3.14 under probability and information theory, and discussed again when discussing graphs for model structure in 16.2 structured probabilistic models for deep learning. as a result, the question 'what are the different categories of graphical models?' from 16.2 incorrectly gets assigned a higher difficulty level. other than this, we observe that while trqde learns from uqde-assigned levels in general, sometimes it makes an incorrect prediction for test question pairs where uqde makes the correct prediction. ## conclusions in summary, we have motivated the task of difficulty prediction for technical interview questions and curated a dataset for evaluation. we have shown that weak-supervision using algorithmgenerated questions and an unsupervised difficulty scoring algorithm is a promising direction for finetuning related state-of-the-art models for this task. the simple unsupervised algorithm itself shows competitive performance and hints at aspects that new models for this challenging problem will need to consider.