Detecting Objectifying Language in Online Professor Reviews (2020)

Abstract: Student reviews often make reference to professors’ physical appearances. Until recently RateMyProfessors.com, the website of this study’s focus, used a design feature to encourage a “hot or not” rating of college professors. In the wake of recent #MeToo and #TimesUp movements, social awareness of the inappropriateness of these reviews has grown; however, objectifying comments remain and continue to be posted in this online context. We describe two supervised text classifiers for detecting objectifying commentary in professor reviews. We then ensemble these classifiers and use the resulting model to track objectifying commentary at scale. We measure correlations between objectifying commentary, changes to the review website interface, and teacher gender across a ten-year period.

## introduction natural language processing techniques have long been used to study subjectivity and sentiment in media and product reviews. in this study, we employ these technologies to study objectifying language in reviews of professors using archival data from ratemyprofessors.com (rmp). detecting such language is difficult because it is somewhat rare, making up a small part of a small proportion of reviews @xcite , and references to physical appearance show enormous linguistic variation (discussed in section 2.2), making them difficult to detect accurately using simple text features. this study provides insights into bias in professor reviews and their interaction with the design of the web user interface. we propose two modelsa chunk tagger and a document classifier-used to build an ensemble to detect objectifying reviews at scale. this approach could be applied to many other domains where noisy user-generated reviews may contain harassment or exhibit harm. we focus on the rmp website because it has been active for over twenty years, giving us ample data to study trends across time. the website has long been associated with students commenting on their professors' appearances @xcite and has been the subject of many prior studies on bias in course reviews. recent changes to the website interface allow us to consider how text reviews may have been influenced by its design feature for rating professor "hotness". ## data for this study, anonymous rmp reviews of professors were scraped on two occasions. 1 the first scrape, in july 2018, paired textual data with the professor's "hotness" rating, defined by the number of times a student rated the professor as "hot" minus the number times they rated them as "not hot" @xcite . in the web interface, the names of professors receiving positive attractiveness scores are marked with a chili pepper emoji (see appendix, figure 5 ). the second scrape, in august 2019, targeted a broader set of regions and schools. test data was drawn from this latter data set, which was also used for trend analysis. by this latter date, the chili pepper emoji had been removed from the website in response to public criticism @xcite , so it was no longer possible to extract hotness scores. in addition to text, both scrapes also collected the names of professors, student-reported quality and difficulty scores (averaged by professor, on a five-point scale), subject area, and the name of the school. see ## methods we implement classification techniques with unique strengths for capturing the qualities and contexts of objectifying comments. the first, a chunk tagger, represents a bottom-up strategy, whereas the second, a document classifier, uses top-down processing and a richer feature set. ## analysis building on previous rmp research studying bias in student reviews, we continue this inquiry focusing on how attractiveness commentary is distributed based on teacher gender, and quality and classifier prec. rec. f1 acc. κ ensemble 1 .72 .44 .55 .93 .50 ensemble 2 .72 1.00 .84 .99 .83 ## discussion while our work has focused on the text contents of reviews, our analysis of objectifying comments follows previous findings about biases of the original chili pepper rating, correlating with teacher gender, quality, and difficulty ratings. this is the first study to find a correlation between attractiveness commentary and the website interface. more research is needed to understand the observed steady eight-year decline. as this was an observational study rather than a controlled experiment, there are many uncontrolled variables. for instance, we cannot compare attractiveness commentary by size of professor's class or attributes of the reviewer. we tried to estimate these factors with proxies such as university size, geographic area, and tuition amounts, but these only provide rough estimates and did not have significant effect on the presence of attractiveness commentary. mc-neil (2020) reflects on how users' perceptions of anonymity have changed, from posting to online bulletin boards in the late 1990s, to present-day "sharing" on corporate-owned, heavily surveilled social network sites like facebook. this turn from anonymity to self-awareness is observed by @xcite in their study of twitter users. these users describe their own selfcensoring behaviors by imagining their audiences to include not only friends but also parents and employers. the decline in attractiveness commentary on rmp may reflect broader internet trends, corresponding with internet users being more conscious of their perceived audience and realizing that true online anonymity is impossible. ## conclusion we find that a small change to the rmp website, removal of the chili pepper rating, is associated with a lower likelihood of comments on professor attractiveness. our experiments show that an ensemble of classifiers can accurately detect objectifying language in online professor reviews and can allow us to analyze trends in a large unlabeled dataset. one area where classifiers disagreed was in the "fuzzy samples" such as accents and godliness discussed in section 2.3. @xcite describe similar challenges in classifying microaggressions and label themes within their dataset to better define these utterances. based on our classifier's success in pulling out objectifying comments from large datasets, we can identify enough examples to consider labeling categories such as accent criticism and comments about unattractiveness. finally, one could apply an active learning approach @xcite to label and train on examples where the classifiers disagreed. with further exploration it is hoped these techniques could be applied to detecting other forms of abusive language in online reviews. insofar as the removal of the chili pepper feature correlated with a significant decrease in attractiveness commentary, we suggest that web interface design may positively influence online discourse. as the scope of the gig economy continues to expand and more workers find themselves evaluated by anonymous online reviews, we hope these findings will inspire future research around potential biases in online reviews based on gender, appearance, and the design of the online interface used.