G rease V ision: Rewriting the Rules of the Interface (2022)

Abstract: Digital harms can manifest across any interface. Key problems in addressing these harms include the high individuality of harms and the fast-changing nature of digital systems. We put forth GreaseVision, a collaborative human-in-the-loop learning framework that enables end-users to analyze their screenomes to annotate harms as well as render overlay interventions. We evaluate HITL intervention development with a set of completed tasks in a cognitive walkthrough, and test scalability with one-shot element removal and fine-tuning hate speech classification models. The contribution of the framework and tool allow individual end-users to study their usage history and create personalized interventions. Our contribution also enables researchers to study the distribution of multi-modal harms and interventions at scale.

## appendix 6.1 greaseterminator in response to the continued widespread presence of interface-based harms in digital systems, datta et al. @xcite developed greaseterminator, a visual overlay modification method. this approach enables researchers to develop, deploy and study interventions against interface-based harms in apps. this is based on the observation that it used to be difficult in the past for researchers to study the efficacy of different intervention designs against harms within mobile apps (most previous approaches focused on desktop browsers). greaseterminator provides a set of 'hooks' that serve as templates for researchers to develop interventions, which are then deployed and tested with study participants. greaseterminator interventions usually come in the form of machine learning models that build on the provided hooks, automatically detect harms within the smartphone user interface at run-time, and choose appropriate interventions (e.g. a visual overlay to hide harmful content, or content warnings). the greaseterminator architecture is shown in figure 6 (a) in contrast to the greasevision architecture.