SETS um: Summarization and Visualization of Student Evaluations of Teaching (2022)

Abstract: Student Evaluations of Teaching (SETs) are widely used in colleges and universities. Typically SET results are summarized for instructors in a static PDF report. The report often includes summary statistics for quantitative ratings and an unsorted list of open-ended student comments. The lack of organization and summarization of the raw comments hinders those interpreting the reports from fully utilizing informative feedback, making accurate inferences, and designing appropriate instructional improvements. In this work, we introduce a novel system, SETSUM, that leverages sentiment analysis, aspect extraction, summarization, and visualization techniques to provide organized illustrations of SET findings to instructors and other reviewers. Ten university professors from diverse departments serve as evaluators of the system and all agree that SETSUM help them interpret SET results more efficiently; and 6 out of 10 instructors prefer our system over the standard static PDF report (while the remaining 4 would like to have both). This demonstrates that our work holds the potential of reforming the SET reporting conventions in the future.

## introduction colleges and universities rely on student evaluations of teaching (sets) to assess students' perceptions about their courses @xcite . these evaluations about the course consist of both quantitative ratings using likert-type scales and open-ended comments that describe student experiences. in many universities, sets are a standard component of evaluations of teaching and have multiple functions. first, they help individual faculty members examine their own teaching performance in a diagnostic way so they can work to improve their approach in subsequent offerings of the course. second, sets allow institution leaders to review and describe the educational usually, at the end of each semester, set results are summarized into a pdf report for instructors or other reviewers. as shown in a sample standard set report in fig. 5 of appendix, quantitative ratings are summarized using basic statistics, such as mean and median, while students' comments from open-ended questions are simply listed as raw textwithout adequate organization and analyses. when a college course is particularly large (e.g., with more than 100 students), the final set report can be longer than 10 pages, which is time-consuming to read and analyze @xcite . in addition, instructors' or other reviewers' own cognitive biases may lead to inaccurate inferences and analyses, e.g., people tend to pay more attention to negative than positive comments @xcite . therefore, the goal of our work is to provide a new dynamic presentation of set results to facilitate more efficient and less biased interpretations compared to the standard pdf report. after obtaining institutional set data of four semesters from the university of north carolina (unc) at chapel hill, for demonstration we select two quantitative and two open-ended questions from the total number of questions (sec. 3). we develop a system, setsum, to summarize and visualize the results of these four questions. for quantitative ratings (sec. 4.1), we visualize two statistics: response rate and sentiment distribution. for open-ended comments (sec. 4.2), we develop a sentiment analysis model to predict whether each comment sentence is positive or negative. we use an aspect extraction approach to help instructors quickly know the popularly discussed topics by students, e.g., assignment, and the corresponding topic sentiments. finally, we propose an unsupervised extractive summarization method that extracts top sentences with high centrality, low redundancy, and balanced sentiments as a summary of each aspect. automatic evaluations (sec. 5.1) demonstrate that our sentiment prediction and aspect extraction modules achieve good accuracy, and our summarization method produces more diverse and less biased summaries than simply picking top central sentences. more critically, the effectiveness of setsum should be judged by its main usersinstructors. thus, we begin by conducting human evaluations (sec. 5.2) with 10 professors from 8 different academic departments at unc. note that setsum is continuously under development, and our human evaluations were conducted on our very first version: setsum v1.0. after evaluating the two set presentation approaches, instructors are asked to complete a survey comparing the usefulness of setsum to the standard set report. according to their responses, most of the new features introduced on setsum are perceived as useful to very useful by most instructors (on average, 8.8 out of 10), compared to the standard report. all 10 instructors agree that setsum helps them interpret their ratings and comments more efficiently; while 4 out of 10 think the new system also supports less biased interpretations. finally, 6 of 10 favor setsum more than the standard approach; the remaining 4 think both reports could be helpful. overall, for our first evaluation, instructors hold a positive attitude towards setsum and offer valuable and constructive suggestions to us. lastly, in sec. 7, we discuss if machine-involved representations of sets may introduce new errors or bias and if so, what improvement needs to be made before the "demonstration" can transition to an "application". our system aims to provide accurate, efficient, and visualized set results to instructors or other reviewers. it does not directly make any value judgments or evaluations about the instructor's skills, the course design, or the amount of student learning during the term. to the best of our knowledge, despite its widespread use, we are among the few researchers to develop a pilot system that presents studentreported evaluations of teaching by using natural language processing (nlp) techniques. in addition, we are the first to apply the system for a set instrument and evaluate it using actual set data from a large public university. though more development work is in progress, our results demonstrate that our approach is promising to reform the set report conventions in the future. our setsum v1.1 website requires credentials to login, please contact us for an access to the website. we provide a youtube video to walk you through setsum v1.1. our code is hosted at setsum github repo. ## background & related work as mentioned, sets are widely used in higher education @xcite . set studies have shown that they can capture students' opinions about instruction @xcite , enhance course design, can be used as a tool for assessing teaching performance @xcite , and reflect institutional accountability about teaching @xcite . many instructors view sets as valuable feedback to improve their teaching quality @xcite . many studies focus on instrument design (i.e., which questions to ask), reliability and validity of set results (i.e., are the scores consistent across contexts; are scores related to other key constructs), and potential confounding variables that affect sets (e.g., do scores differ by discipline, instructor race/ethnicity and gender, student grade) @xcite . typical set instruments include quantitative likert-scale ratings. they are supplemented by open-ended comments @xcite . therefore, compared to quantitative ratings, open-ended comments are often underanalyzed or ignored completely due to labor required to provide an adequate summary @xcite , raising the need for contemporary methods in automated text analysis. recent works start to analyze student comments via text mining and machine learning methods such as sentiment analysis @xcite @xcite @xcite , and identify topics, themes, or suggestions from student comments @xcite @xcite @xcite . the common goal of these works is to answer some research questions (e.g., what are sentiment differences across courses and students). in contrast, we provide a demonstration tool of set results to help instructors gain insights on their own and to allow others have access to organized summaries. the most relevant works to ours are sufat @xcite and palaute @xcite -two analytic tools for student comments. they both support sentiment analysis and lda topic models @xcite , while we use more advanced roberta-based sentiment analysis and weakly-supervised aspect extraction models. su-fat requires users to install the tool and load set files locally, while our online website directly reads the data from the set instrument. more importantly, none of them conducts human evaluations, which makes it unclear if their tools are useful from the actual users' perspectives. therefore, we develop the first demonstration system that uses an actual university set instrument and is evaluated by university instructors who are interpreting their own evaluations. ## set data we use student evaluations of teaching (sets) data of over four academic terms (fall 2017, spring 2018, fall 2018, spring 2019) collected at unc chapel hill. we utilize "semester + course number" as the specific identity of each course. we assume each course has just one instructor. 1 in total, there are about 5.6k courses and 298k sets. each set is an evaluation questionnaire assigned to a student for a specific course they enrolled in, including both quantitative and open-ended questions. unc's set instrument includes a series of evaluation questions assessing different aspects of the course and instructor. for demonstration, we select four representative questions -two quantitative and two open-ended items. for quantitative items, we choose overall, this course was excellent (course rate) and overall, this instructor was an effective teacher (instructor rate), showing students' overall ratings on the course and instructor performance. both items are based on a 5-point likert scale (1 = strongly disagree to 5 = strongly agree). for open-ended items, we choose comments on overall assessment of this course (course comments) and comments on overall assessment of this instructor (instructor comments). our system can be easily extended to the full set of set questions. because completing the set form is not mandatory, the average response rates of the two quantitative items we choose are 46% and 43% respectively, and even lower response rates are observed for the two open-ended items: 17% and 16%, respectively. ## system description after logging in, instructors can select to display their set results of which semester and which course. the dashboard shows two main sections: rating analysis and comment analysis. see screenshots of our demo in fig. 6 (setsum v1.0) and fig. 7 (setsum v1.1) in appendix. ## conclusion in this work, we propose setsum, a system to summarize and visualize results from student evaluations of teaching. we integrate nlp, statistical, visualization, and web service techniques. we are among the few researchers to build a tool for instructor use and are the first to evaluate the tool by university professors. our results demonstrate that our system is promising at improving the set report paradigm and helps instructors gain insights from their sets more efficiently. in the future, we will keep improving the sentiment analysis and aspect extraction models to provide more accurate summarization of set results. the instructor evaluation offered key recommendations for the next iterations of the system. we will incorporate more functions to our system, including allowing instructors to compare different courses and track their own teaching history of their courses as well as developing a separate administrator dashboard to identify themes across academic courses, departments, and programs. 77 as mentioned earlier, sets have multiple functions such as (1) faculty members examining their teaching performance in a diagnostic way, (2) allowing institution leaders to review and describe the quality of course offerings, (3) part of an instructor's larger portfolio to demonstrate their teaching history during high-stakes settings, and (4) summaries being released to students to guide them with course selections. given this wide range of uses for the set summaries, our work's purpose is to take initial steps towards developing thoughtful, accurate, and well-designed representations that can be provided to draw accurate inferences about teaching quality, course design, and student learning. however, it is also critical to examine all aspects of sets through an ethical lens. errors in nlp-based analysis could lead to misinterpretation and inaccurate judgments in high-stakes settings. therefore in the following, we discuss how each module of our setsum website affects the interpretation of set results. for quantitative items, we provide visualizations of two statistics that are directly computed from set data. therefore, no errors or biases should be introduced compared to the standard pdf report. in fact, some instructors who participated in our evaluations say that the response rate feature for each individual question helps them understand the results with less bias. for open-ended items, to obtain their sentiment distributions, we develop sentiment analysis models to obtain sentence-level sentiments. though we obtain good sentiment prediction performance ( though instructors express positive attitudes towards our system and 4 instructors think it help them understand sets with less bias, we believe that additional thorough evaluations need to be conducted in the future. outside of sets, our work recognizes the different ways, reporters, and methods that could be used to assess teaching effectiveness, including but not limited to peer reports, analysis of classroom sound, student learning, and an instructor's own teaching portfolio. finally, at this time our system is designed to be used and reviewed by instructors or other reviewers, and it does not directly make any broad judgments or decisions (e.g., whether the instructor is qualified for promotion). the primary end-users of the system should be instructors who wish to analyze their set findings more thoroughly and acquire the main takeaways more efficiently. other reviewers and administrators can use the system to view the set findings in a broader scope, such as reading the report summary per department or per division. overall, the goal of setsum is to help instructors and other reviewers to understand more of students' needs and make improvements to future course design. @xcite with learning rate 1e-5 and batch size 16. aspect extraction. we used nltk to conduct sentence and word segmentation. we initialized the mate model using glove embeddings @xcite . during the training, the word embeddings, seed word matrices, and seed weight vectors were fixed and we trained the model for 10 epochs using adam optimizer @xcite with learning rate 10 -1 and batch size 50. we also experimented the multi-label classification approach by finetuning a roberta-base model @xcite for 10 epochs using adamw optimizer @xcite with learning rate 2e-5 and batch size 16. website. we developed the website using the react framework for the front-end interface. for the back-end, we set up a database with firebase and created a restful api with firebase cloud functions. our website is deployed to netlify.com for an online demonstration.