DT - QDC : A Dataset for Question Comprehension in Online Test (2020)

Abstract: With the transformation of education from the traditional classroom environment to online education and assessment, it is more and more important to accurately assess the difficulty of questions than ever. As teachers may not be able to follow the student’s performance and learning behavior closely, a well-defined method to measure the difficulty of questions to guide learning is necessary. In this paper, we explore the concept of question difficulty and provide our new Chinese DT-QDC dataset. This is currently the largest and only Chinese question dataset, and it also has enriched attributes and difficulty labels. Additional attributes such as keywords, chapter, and question type would allow models to understand questions more precisely. We proposed the MTMS-BERT and ORMS-BERT, which can improve the judgment of difficulty from different views. The proposed methods outperforms different baselines by 7.79% on F1-score and 15.92% on MAE, 28.26% on MSE on the new DT-QDC dataset, laying the foundation for the question difficulty comprehension task.

## introduction intelligent education systems are heavily studied and investigated since it can generate great value both academically and commercially, especially when the covid-19 pandemics is prevailing. a welldeveloped educational assistance system can facilitate students to grasp the learning progress and customize personalized learning approaches. it improves the whole learning experience, enhances students' initiative, and results in better performance in assessments. one key component of an intelligent education system is to assess the level of understanding of students, and asking questions is the most intuitive way of accomplishing that. however, the most online quiz system treats questions equally without factoring in the difference in difficulty. some researcher @xcite ) argued that we should not treat all questions equivalently, because questions exhibit significant variations in difficulty and discrimination @xcite . @xcite once introduced the modeling of the problem difficulty in the knowledge tracing task, but they only used the guess-and-slip parameters related to the question, without paying attention to the nature of the question itself. for every new question, they must use the corresponding user answer data to estimate its difficulty. in this paper, we go one step further to investigate methods that perform question difficulty comprehension using various attributes of the question. question understanding is one of the key component in machine reading comprehension, which is viewed as the sign of machine understand nature language @xcite . although question comprehension is not studied as a special task, it is positively related to the difficulty of the question. generally speaking, the difficulty is a more abstract and personalized concept, and it is difficult to quantify to define. to evaluate the students online, we can measure the difficulty of a question to be answered from the perspective of classification. by evaluating the difficulty of each question, the performance of downstream applications such as deep knowledge tracing and question answering can be improved. we release a dataset, chinese driving test question difficulty comprehension (dt-qdc), which is formed with a large volume of user records from the driving license examination website. the dataset contains 14,933 questions with 10 attributes. figure 1 shows two examples of questions in the dataset, which attribution includes question explanation, keywords and test information, etc. we propose the ordinal regression multi-source bert (orms-bert) model to solve the difficulty comprehension problem. multi-source bert @xcite text representation and relation modeling enable us to better understand question difficulty. a novel category encoding technique is applied to transform multi-class classification tasks into multiple binary classification tasks. our model outperforms different baselines by 6.77% on f1 score and 15.92% on mae, 28.26% on mse. to summarize our contributions: ## related work in order to simplify the online questions comprehension task, we mainly consider the question difficulty definition, fine-grained division and modeling. related work includes the following two-fold: ## question difficulty comprehension model given an input question @xmath0, w 2 , ..., w m ), which is asked with @xmath1, y 2 , ..., y n ) as its options , @xmath2, z 2 , ..., z l ) as its explanation, @xmath3, t 2 , ..., t h ) as its keywords, t as its question type, c as its chapter, and a as its answer, our task is to predict its difficulty level d. the architecture of our orms-bert model is depicted in figure 4 . the encoder takes the text and the discrete data as inputs. two separate berts @xcite ) are employed to encode the question and options into contextualized representations. and three separate linear layers are used to learn the embedding of the question type, answer, and chapter. besides these standard elements, we also use the attention mechanism with average sequence pooling to model question-option and option-option relationships. moreover, we also apply ordinal regression loss to better model the nature of the difficulty. ## conclusion we proposed the task of question difficulty comprehension and constructed a new dataset dt-qdc with real-world user answer records and multi-attributes questions to target it. in addition, we provide a strong model called orms-bert, and then compared its performance with several baselines. in the future, we will explore question difficulty comprehension tasks in the field of online education. we expect to combine the question difficulty comprehension task with the question answering task and knowledge tracing task to model the student performance more scientifically, so as to better assess the student's knowledge level and track the student's learning progress. our dataset can also be used to explore related tasks such as question answering, question generation, option generation, question normalization, question rewriting, etc. our dataset is available online, and we expect it is beneficial for future research in this field.