Comparison of Machine Learning Methods for Multi-label Classification of Nursing Education and Licensure Exam Questions (2020)

Abstract: In this paper, we evaluate several machine learning methods for multi-label classification of text questions. Every nursing student in the United States must pass the National Council Licensure Examination (NCLEX) to begin professional practice. NCLEX defines a number of competencies on which students are evaluated. By labeling test questions with NCLEX competencies, we can score students according to their performance in each competency. This information helps instructors measure how prepared students are for the NCLEX, as well as which competencies they may need help with. A key challenge is that questions may be related to more than one competency. Labeling questions with NCLEX competencies, therefore, equates to a multi-label, text classification problem where each competency is a label. Here we present an evaluation of several methods to support this use case along with a proposed approach. While our work is grounded in the nursing education domain, the methods described here can be used for any multi-label, text classification use case.

## introduction all nurses within the united states must pass the national council licensure examination (nclex ® ) to begin professional practice. a nursing curriculum will typically cover a wide range of topics related to the theory and practice of nursing. however, the nclex measures students against a specific set of competencies comprising the activities that entry-level nurses are most commonly expected to perform. these activities are identified by the national council of state boards of nursing (ncsbn) through analysis of nursing practice. figure 1 shows a subset of nclex competencies called "activity statements" with descriptions. activity statements are grouped into primary topics and sub-topics, as shown in the image. nursing education content may be related to one or more competency; they are not mutually exclusive. passage of the nclex has significance not only for students but also for learning institutions. nursing school accreditation is partially based on how well their student body performs on the nclex. if their performance drops below a certain threshold for too many consecutive years, the school risks losing its accreditation. it is, therefore, paramount for instructors to gauge student preparedness for the nclex and course correct where necessary. one way to do this is by repeatedly testing students with simulated exams. this approach may reveal that gaps exist, however, it does not necessarily identify what competencies are deficient or what content may address those deficiencies. by labeling both questions and educational content with the competencies that they relate to, instructors can more precisely identify where students are struggling and what content may help with remediation. this approach enables coursework to be tailored for individual students based on their performance in a manner that maximizes likelihood of passing the nclex. for instance, if a student incorrectly answers questions related to "provide pulmonary hygiene" (activity statement shown in figure 1 ), the instructor may assign the student additional content (e.g., simulations and practice problems) related to that competency. this general approach is called formative testing. prepu is a wolters kluwer product for nursing education that features several types of content including books, simulations, videos, audio, and quizzes. to support formative testing in prepu, quiz questions are tagged with the nclex competency related to them. when students take quizzes, their scores can be aggregated according to nclex competency. figure 2 shows an example of the interface displaying this information. the image shows the student achieves a score of 66.7% for the competency "prioritize the delivery of client care". this is calculated based on the student answering 4 out of 6 questions correctly that were labeled with that competency. there is also a tab that shows class performance so that instructors can see if there is a pattern of multiple students struggling with a particular competency. instructors can use this information to make changes to the curriculum to address problem areas. corrective actions may include assigning students additional content or practice materials related to the competencies they are struggling with. to aggregate scores according to nclex competencies as shown in figure 2 , each question needs to be labeled according to which competencies it relates to. prior to our work, editors would manually label questions using the editorial platform shown in figure 3 . a drop-down menu shows a selection of nclex competencies. the editor must scroll through this list, identify which are appropriate, and select them to add them to the question. this process was costly and time-consuming. one challenge is that each question can belong to more than one competency. further, different editors may have differing opinions as to which competencies a question relates to. reconciling these differences and maintaining consistency across editors and content is a huge challenge. to streamline the labeling of nursing education questions, we integrated a machine learning model for automated tagging into the current workflow. as editors review each question, the model makes suggestions about which nclex competencies are related to that question. rather than scrolling through a long list of options, editors can rapidly click to accept or reject suggestions (though they still have the ability to scroll through all possibilities if they believe none of the suggestions are applicable). this approach has greatly streamlined the process of labeling questions and added additional consistency in the application of labels. the following sections detail the data involved, the modeling techniques evaluated, and the chosen solution. ## the data in this paper, we focus on nclex competencies related to what are called "activity statements". activity statements are presented in a hierarchical structure with two levels as shown in figure 1 . we consider only leaf nodes to simplify the problem. given this consideration, there are 138 activity statements or labels in total. less than 100 questions that were assigned more than one activity statement label. however, there was a desire to accommodate multiple activity statements per question for future labeling efforts. therefore, we maintained an approach using multilabel classification techniques. ## related research the task of tagging questions with relevant activity statements can be considered a multi-label document classification task where each question is a document. there are several well-known methods for this type of task. many of them represent a document as a vector of numbers. we can use similarity and/or distance metrics between document vectors to perform several operations such as clustering and classification. a key set of decisions is how to represent documents as vectors, and what distance metrics to use for comparing them. the following sections describe a number of approaches for document vectorization as well as methods for multi-label classification. ## evaluating vectorization methods and similarity metrics for clustering and classification we began our analysis by evaluating how different vectorization techniques and similarity metrics perform at differentiating questions related to one label (i.e., activity statement) from another. the ability to differentiate questions in this manner directly affects the performance of classification and clustering algorithms. the results helped establish a baseline of how much overlap there was between questions in different label groups. it also informed decisions on which vectorization methods and similarity metrics to use with what algorithms for evaluation. to perform this analysis, we leveraged techniques often used in clustering. the nursing education questions were grouped into clusters based on the activity statements they were associated with. this resulted in 138 clusters, one for each of the activity statements. questions associated with more than one activity statement were included in the groups for each. we experimented with several vectorization methods (techniques for transforming the questions into numeric vectors) as well as similarity metrics for comparing vectors. we converged on using cosine similarity to compare vectors because of its ability to deal with both sparse and dense vectors when normalized. the vectorizations evaluated included the following: for each vectorization method, we computed the silhouette score across our manually constructed clusters. the silhouette score measures the similarity of questions within a cluster (cohesion) versus the dissimilarity of questions in one cluster as compared to those in other clusters (separation). higher silhouette scores indicate better cohesion within clusters and separation between clusters. in clustering, this measure can help inform the number of clusters to use. for our analysis, we were more interested in what vectorization methods achieved better separation of questions assigned to different nclex labels. the vectorizations achieving the best silhouette scores could be expected to perform better in classification tasks. we therefore controlled the number of clusters to the number of activity statements, i.e. 138. figure 5 shows the silhouette scores for every pair of activity statement clusters using tf-idf vectorization of questions. tf-idf resulted in the highest silhouette score of -.02. however, the scores for all vectorization methods were relatively low. this result indicated two things 1) there is a great degree of similarity between questions assigned to different activity statement labels, and 2) no vectorization method performed much better than the others. this result indicated that algorithms may need further grouping and sampling of questions to better differentiate them during classification. we hypothesized that ignoring the current labels and clustering questions may achieve better separation for classification algorithms. to evaluate this hypothesis, we performed a standard clustering of questions using the various vectorization methods. normally we would optimize the number of clusters based on the silhouette score or other related metrics. however, in the interest of time, we used a fixed number of 512 clusters. this number was estimated from the number of questions and their distribution across activity statements. the resulting silhouette scores improved by .02 on average but did not reflect a significant change. on average, each cluster contained questions from five different activity statement label groups. this result motivated some of the modeling experiments described in the following section. ## modeling we use micro-averaged area under the receiver operating characteristic (auc-roc) to compare multiple algorithms for the described use case. this metric is able to address class imbalance and is used canonically for benchmarking models @xcite . note that the metrics reported here only reference the auc score of first label predicted. in production, the top five labels with the highest confidence values are shown to users and results in an accuracy of 95% in predicting all relevant labels. this is discussed further in section 5.6. nonetheless, the initial auc score of the first prediction was a good benchmark for comparing models. the following sections provide details on each algorithm evaluated. ## impact analysis we are currently logging editor activity and calculating metrics to perform a thorough impact analysis. initial estimates show that time spent on labeling questions with nclex tags went from a few minutes pre-machine learning to less than one minute after our solution was deployed. there are tens of thousands of questions in wolters kluwer products like prepu and coursepoint and more content being generated every year. this impact is therefore significant, measuring several hours and potentially up to $100,000 or more savings annually. editors have responded very positively and regularly use machine learning label suggestions in their current workflow. that said, it will take some time for them to accept a completely automated process. perhaps more importantly, subject matter experts have assessed that the consistency and quality of labels assigned to questions increase with the model suggestions. nursing content editors often apply labels based on their personal understanding of content, which is sometimes subjective. there may also be biases in selecting "convenient" labels when having to choose from a lengthy list in a complicated workflow. the predictive model provides consistent label suggestions which in turn results in more consistent labels being assigned. ## future work the class imbalance of this task motivates the potential use of active machine learning. some labels have only been assigned to a handful of questions. for these labels, we may work with editors to find more exemplar questions or create new ones. these new questions can then be merged with training data and the model retrained to ameliorate ef-fects of class imbalance. in active learning, this process is typically repeated in an iterative process to target problem areas for a model. by selectively labeling new questions and down sampling over represented labels, we can fine tune data for retraining models to improve overall accuracy. active machine learning has specifically been used for multi-label text classification problems @xcite . another area for further study is the evaluation of more recent, deep, transformer models. because there is a great deal of semantic similarity between questions, these models may not fare better than more traditional vectorization and classification techniques. we intend to evaluate this hypothesis in future work. there are many different tag sets and taxonomies that can be used to label nursing education content. tagging both content and questions supports more advanced features such as dynamic remediation and adaptive learning. for instance, when a student answers a question incorrectly, learning software can automatically provide links to learning materials that are related to that topics addressed in that question. we are actively investigating how tagging and organizing content can support various use cases for adaptive learning.