Interventions Recommendation: Professionals’ Observations Analysis in Special Needs Education (2021)

Abstract: We present a new task in educational NLP, recommend the best interventions to help special needs education professionals to work with students with different disabilities. We use the professionals’ observations of the students together with the students diagnosis and other chosen interventions to predict the best interventions for Chilean special needs students.

## introduction one consequence of the adoption of digital technologies in schools is the wide availability of data records associated with their enrolled students. these records have been successfully employed in many ways to help students improve their school performance. for example, @xcite present different projects that use student records, such as their grades in different subjects, to help them in different ways, for example recommending activities or books that can help them to improve their grades. these investigations focus on, for example, correction of grammatical errors, automated writing evaluation, automated content evaluation, vocabulary analysis, among others. these tasks use free text pieces written by students as inputs to develop different kinds of outputs, allowing students to improve their grammar together with reading and writing skills @xcite . another important task is to analyze free text observations and reports made by teachers and other professionals (e.g., physiologists, speech therapists) that have worked with the students to decide certain strategies or interventions that can help students improve their school performance and social skills. this task is especially important in "special needs education" (sne), where students with different disabilities, such as motor disorders, down's syndrome, specific language impairment, among others, may need particular interventions, such as access curricular adaptation, involve the family in the educational process, interdisciplinary support, among others @xcite . as the decisions made by professionals working in sne usually are more critical than the decisions made in traditional education, teachers and professionals are not always sure about their decisions @xcite . for this reason, different tools have been developed to help sne professionals diagnosing special needs students and performing the chosen interventions with these students @xcite . in this work we study the problem of recommending the best interventions to students in special needs education considering their diagnosis, professionals' observations and even other applied interventions, formulating this problem as a multilabel text classification task with additional data. we explore different input sets and text transformations that allow a machine learning system to make intervention predictions for each student. to achieve this, we collected data from differ-ent chilean special needs schools to build a new the special needs education corpus (snec). this dataset was built with records from around 3,000 students with different diagnoses, spanish-written observations by professionals and the best interventions for each of them. each student has one of 12 possible diagnoses and has one or more recommended interventions, chosen from a set of 14 possible interventions. the professionals that give their observations of students are in most cases doctors, psychologists and special education teachers, but speech therapists can also observe if the student is required to work with these professionals. the content of the observations are free text instances and vary according to the professional who issues them. we have therefore designed different experiments to analyze whether these observations should be differentiated or use a simplification such as joining the observations and working with a single joined document and see if it is enough to get correct interventions prediction. as each student might need more than one intervention, we formulate this problem as a multi-label text classification task with additional attributes, such as the diagnosis and other chosen interventions. we use different sparse bag of n-grams text representations and dense neural text transformers models, such as bert @xcite , to analyze which text representation approach is best suited to our problem. the focus of this paper is therefore to introduce a new task in the educational data mining field, specifically in sne field using natural language processing (nlp) techniques, together with a our new snec corpus with spanish-written professionals' observations of students with different disabilities and results of several experiments using different approaches and text transformations to predict the best interventions for each one of these students using machine learning techniques. the remainder of this paper is structured as follows. in section 2 we present related work to our practical problem, recommending interventions in special needs education, and the technical problem, use different text representations for observations written by professionals and different input sets to make intervention predictions. in section 3 we propose our task and show the dataset origin and details of each attribute. in section 4 we present our experiment settings and the evaluation metrics used to analyze the results of the experiments. the details and results of the experiments are shown and discussed in section 5. finally, in section 6 we present our conclusions and future work. ## related work this work can be viewed from multiple perspectives: from a domain point of view it lies in the fields of educational data mining (edm) and sne and from a technical point view we have a text classification problem. in this section we present different tools used both in traditional educational data mining and particularly in sne and then we also present text classification methods that are relevant to this work. educational data mining is a specific field that uses computational approaches to analyze educational data in order to study educational questions, as defined in @xcite . our problem shares features with many tasks defined in the above paper, such as providing feedback for supporting instructors (where the objective is to provide feedback to support course teachers in decision making) and recommendations for the students (where the objective is to make recommendations directly to the students analyzing their personalized activities). in special needs education there are also different tools that help teachers and other professionals to work with students. @xcite show different tools and projects trying to solve particular problems present only in sne, such as diagnosing sne students using different a.i. learning methods. this work also presents different instruments that help professionals carry out the interventions chosen for each student. our study also analyzes the technical problem of the interventions recommendation -the text classification problem when we use the free text instances, such as the professionals' observations. a relatively recent architecture that has yielded state-of-the-art results in many nlp tasks is the transformer @xcite , which can be used for training large language models from unalabeled corpora (e.g., bert @xcite ). these pre-trained models can be later fine-tuned to almost any target task at hand with successful results. although most of these large pre-trained models are only available for the english language, several language-specific versions have been made available in recent years, such as beto @xcite , the spanish version of bert. ## problem formulation in this section we present the task studied in this work together with its context and our new corpus. our task lies in the field of educational data mining, specifically in the area of special needs education. it consists of automatically recommending interventions for students with different disabilities and behaviors, and its main goal is to help sne teachers and professionals decide the best interventions for their students. in chile, special needs education professionals write different documents that contain observations and relevant information of students they have worked with at the beginning of the school year. in these documents the diagnosis of the sne students is established and in case students have more than one diagnosis they are diagnosed with 'multiple cognitive deficit' and more details of the diagnoses are also settled down in the documents. professionals also give details about the students behavior and relevant observed information together with interventions that can help students with their problems. with these document professionals then decide the best interventions for each student by analyzing the information of the student's different documents and diagnosis. at the end of the year these professionals write a single document for each student, writing down the interventions that were useful for the student and also interventions that were not applied, but professionals think would have been useful had it been applied. interventions that did not work are also written, but for this research we use just the interventions that did work. we built a new special needs education corpus (snec) from the records from around 3,000 students with different disabilities from chilean sne schools enrolled between 2018 and 2019. each student record consists in his or her diagnosis, the observations of the professionals that have worked with each student at the beginning of the year and the interventions that professionals chose as the best for this student (applied or not) at the end of the scholar year. the students' diagnosis were obtained from a web application that helps teachers store basic student information, such as his or her name and diagnosis, and students' digitized documents. this web application is used to track the students' progress, both in school performance and evolution of their diagnosis. for this work we summarized the possible diagnosis of a student, grouping the same diagnoses but different severity into a single general diagnosis. this way, students have a diagnosis from 12 possible diagnoses. the number of students per diagnosis are shown in the professionals can be doctors, speech therapists, psychologists or special needs teachers. each student can have 0 or more observations of each professional type, but must have at least one observation. a student can also have more than one document containing observations from the same professional, being able to have more than one observation per professional observation type. for this work if the student has more than one observation per observation type, these are joined and used as a single observation. the number of students that have each observation type are shown in the student data can thus be represented as shown in figure 1 . the student is diagnosed with one of 12 different diagnoses and has different evaluations containing free text observations from a doctor, a psychologist, and a special education teacher observations, also including speech therapist observations if the student requires it. professionals then choose 1 or more interventions from 14 available. based on this, we formulate the task of predicting interventions for a given student as a multi-label classification problem, in which all the free text observations and the student's diagnosis are used as input features. ## methods as we are dealing with a multi-label classification task, we employ the stratification method of @xcite to generate training, validation and testing partitions dividing our data in a ratio of 3:1:1 respectively. for this research we do not use the testing partition as we plan to build more complex architectures in future research. this way, we use 1836 students data for training and 612 for validation. we then define general settings and evaluation metrics for all our designed experiments. our experiments are applied to each intervention, considering a single intervention as output to analyze single intervention behaviors. as behavior of the predictions and possible correlations could be different in well-balanced and unbalanced interventions, we group interventions into two sets: unbalanced interventions (present in less than 15% of the students or in more than 85% of the students) and balanced interventions (not unbalanced interventions). we consider a dummy majority vote classifier as the baseline. as this study explores the possible correlation between the diagnosis, professional observations or interventions of students and tries different text transformations for the professionals' observations, we use a 'simple' logistic regression classifier with the same settings for all the experiments, only changing the input data (data selection and transformations) for each intervention. the diagnosis of each student is represented using a one-hot encoding (ohe) representation, adding 12 different binary attributes representing each one of them. interventions are also represented as binary attributes, both when used as input attributes and as output. in the "simple" text representation experiments we use a sparse bag of n-grams representation of the professional observations, considering from 1 to 3 words as tokens, using unigrams, bigrams and trigrams this way. we also use a simple approach for cases where a student had more than one observation of the same observation type. in these cases we just join the observations and use it as a single observation. we also design two versions for experiments in which we use the bag of n-grams representation of the observations, first considering the joined different types of observations of each student as a single text input and another version adding a special token to each word of the observations, using a different token depending on the professional that wrote the observation. furthermore, we design two versions for experiments using dense neural representations of the observations. as adding a special token to words will cause pre-trained dense transformers generate inaccurate representations, we use each observation type as a different text input and use their representations as different features as input for our classifier. for this study we use the following metrics: accuracy, cohen's kappa score @xcite , f1 score average of positive and negative class and area under the receiver operating characteristic curve (roc auc) (flach, 2016) score of each experiment for each intervention. in this work we report the macro-average of all the interventions. ## experiments for our experiments we train logistic regression models considering different sub-sets of the feature space and also using different representation approaches for our textual features. we also have two groups of experiments: the first one consisted in using sparse n-gram representation of our text inputs, using the bag of n-grams mentioned above, while for experiments of the second group we use dense neural representations of the observations. ## conclusions and future work in this paper we have presented a new task, recommend interventions to professionals to work with special needs students using machine learning techniques. we also released snec, a new corpus 2 http://crscardellino.github.io/sbwce/ with more than 3,000 students records containing their diagnosis, free text written observations and the best interventions for each of them. finally, we showed the results of experiments using different text representations. we observed that the professionals' observations of a student can be used to predict the best interventions for him or her. besides, the diagnosis and other chosen interventions showed to be valuable in predicting some interventions, but not enough to improve on the predictions made using observations alone. we also observed that contextualized neural representations of the observations, such as bert or sentence embeddings, are useful to predict balanced interventions, but not for the unbalanced ones. it is important to remark, though, that, in this study the weights obtained from these pre-trained models were left frozen. the literature suggests that fine-tuning these weights to the task at hand can lead to significant improvements @xcite . as a general conclusion we can say that no single approach to represent observations consistently outperformed the others for all the interventions. we believe that the models trained in this study can be useful to develop new tools to help teachers in deciding which interventions are the best for each student knowing his or her diagnosis, professionals' observations and other possible interventions. our next steps are to analyze our results further and find out the limitations of our approach.