Explaining Neural NLP Models for the Joint Analysis of Open-and-Closed-Ended Survey Answers (2022)

Abstract: Large-scale surveys are a widely used instrument to collect data from a target audience. Beyond the single individual, an appropriate analysis of the answers can reveal trends and patterns and thus generate new insights and knowledge for researchers. Current analysis practices employ shallow machine learning methods or rely on (biased) human judgment. This work investigates the usage of state-of-the-art NLP models such as BERT to automatically extract information from both open- and closed-ended questions. We also leverage explainability methods at different levels of granularity to further derive knowledge from the analysis model. Experiments on EMS—a survey-based study researching influencing factors affecting a student’s career goals—show that the proposed approach can identify such factors both at the input- and higher concept-level.

## introduction surveys and questionnaires are prevalent tools to inquire about an audience and collect ideas, opinions, and thoughts. common examples are requesting user feedback concerning a specific product or service, regular reports for scientific studies that involve human subjects, and census questionnaires directed to a certain demographic population. carrying out an appropriate and thorough analysis of the collected answers is of major relevance for researchers both in the industry and academia. however, the generated data are often a combination of open-ended and closed-ended questions. while the former gathers a participant's thoughts in text form, the latter consists in selecting one (or more) of the options specified by the survey designer. utilizing both types remains a popular choice as closed-ended questions are very suitable to derive statistical conclusions but may lack details which are in turn provided by open-ended answers. currently, the two dominant analysis practices comprise traditional closed-vocabulary and open-vocabulary methods @xcite . whereas the former introduces human biases and is resource-intensive, the latter overcomes these challenges with the help of natural language processing (nlp) techniques. nonetheless, both approaches fail to consider contextual information and do not leverage currently available nlp architectures to deal with more complex patterns. in this work, we bridge the gap in research and investigate the usage of deep-learning-based methods from nlp and explainability techniques to extract knowledge and interpret correlations from surveys presenting both structured and unstructured components. our contribution can be summarized as follows: ## results results are presented in two distinct sections. firstly, we present the numerical results for the prediction task in the case of both the regression and the classification heads for the whole architecture. the performance here is evaluated through macro f1 score for all eight individual topic predictions. secondly, we show explanations for these model predictions through explainability frameworks shap and conceptshap. ## discussion and comparison we employed several architectures to solve the the problem of career choice prediction to improve over prevailing closed and open-vocabulary methods. while for some survey responses correlations were strenuous, we found general success in predicting variables relating to entrepreneurial aspirations. we see an overall increase in performance by combining textual and numerical input data. while numerical data is generally more predictive in our experiments, the 119 numerical variables are also a lot more nuanced than the free-text answers q22 and inspire. despite this, prediction from text alone still manages to perform relatively well across different tasks. the negative impact on performance of including the inspire variable in models is likely due to the limited amount of text in the answers to the question. to back up our model findings with explanations, we applied shap and conceptshap as post-hoc approaches. the first confirmed what we observed in terms of model performance and provided us with a good understanding of the global and local relevance of each component: numerical features, text features, and embeddings. the second, instead, led to the identification of relevant concepts -clarity of plans, career characteristics, and plan timeline-in line with the human judgment of previous works. ## conclusion and future work this work investigated the usage of state-of-theart nlp and xai techniques for analyzing usergenerated survey data. instead of manually examining individual answers, our methodology heavily relies on analyzing and interpreting a predictor model trained to extract correlations and patterns from the whole data set. we proposed a multi-modal architecture consisting of a distil-bert transformer architecture and fc layers. the former is used to extract information from openended textual answers while the latter process the numerical features representing closed-ended answers. the model achieves satisfactory accuracy in predicting students' career goals and aspirations. we leveraged shap and conceptshap to generate both instance-level and concept-level explanations. these methods were applied at different levels of granularity to assemble a holistic understanding of the model's reasoning. experiments on the ems survey show promising results in predicting the students' entrepreneurial ambition. moreover, local explanations provide us insights about the most relevant questions overall as well as relevant factors w.r.t. a single student. the automatic high-level concept analysis also led to insightful findings which were very similar to what was found in previous research including human judgment. we release our code to the public to facilitate further research and development foot_0 .