G rease V ision: Rewriting the Rules of the Interface (2022)

Abstract: Digital harms can manifest across any interface. Key problems in addressing these harms include the high individuality of harms and the fast-changing nature of digital systems. We put forth GreaseVision, a collaborative human-in-the-loop learning framework that enables end-users to analyze their screenomes to annotate harms as well as render overlay interventions. We evaluate HITL intervention development with a set of completed tasks in a cognitive walkthrough, and test scalability with one-shot element removal and fine-tuning hate speech classification models. The contribution of the framework and tool allow individual end-users to study their usage history and create personalized interventions. Our contribution also enables researchers to study the distribution of multi-modal harms and interventions at scale.

## introduction the design of good user interfaces can be challenging. in a fast changing world, however, with sometimes highly individual needs, traditional one-fitsall software development faces difficulty in keeping up with the pace of change and the breadth of user requirements. at the same time, the digital world is rife with a range of harms, from dark patterns to hate speech to violence. this paper takes a step back to improve the user experience in the digital world. to achieve this, we put forward a new design philosophy for the development of software interfaces that serves its users: greasevision. contributions: our work aims to contribute a novel interface modification framework, which we call greasevision. at a structural-level, our framework enables end-users to develop personalized interface modifications, either individually or collaboratively. this is supported by the use of screenome visualization, human-in-the-loop learning, and an overlay/hooks-enabled low-code development platform. within the defined scopes, we enable the aggregation of distributionally-wide end-user digital harms (self-reflection for end-users, or analyzing the harms dataset of text, images and elements for researchers), to further enable the modification of user interfaces across a wide range of software systems, supported by the usage of visual overlays, autonomously developed by users, and enhanced by scalable machine learning techniques. we provide complete and reproducible implementation details to enable researchers to not only study harms and interventions, but other interface modification use cases as well. structure: we introduce the challenge of end-user interface modification in sections 1 and 2 to curb digital harms. we share our proposed method -greasevision -in section 3. we evaluate our method in section 4, and share final thoughts and conclusions in section 5. ## related works & problem we summarize key related work here; for detailed related works, we refer the reader to appendix: section 6.2. our motivating problem is the high individuality of digital harms across a distribution of users. the harms landscape is quickly changing with ever-changing digital systems, ranging from heavily-biased content (e.g. disinformation, hate speech), self-harm (e.g. eating disorders, self-cutting, suicide), cyber crime (e.g. cyberbullying, harassment, promotion of and recruitment for extreme causes (e.g. terrorist organizations), to demographic-specific exploitation (e.g. child-inappropriate content, social engineering attacks) @xcite @xcite @xcite . though interface modification frameworks exist, the distribution of the interface modifications (interventions) are constrained to the development efforts of an intervention developer, the availability of interventions are skewed towards desktop browsers (much sparser on mobile), and the efforts (a) user authentication: secure gateway to the user's screenomes, personal devices, and intervention development suite. (b) interface & interventions selection: listings of all registered devices/emulators on server, as well as interventions contributed by the users or community using the tool in figure 3. (c) interface access: accessing a linux desktop from another (linux) desktop browser. (d) interface access: accessing an android emulator from another android host device. due to the non-uniform distribution of users, the diverging distribution of harms would require a wide distribution of interventions. we hypothesize we can generate this matching distribution of interventions by enabling end-users to render personalized interventions by themselves (i.e. removing intervention developers from the ecosystem). to test this hypothesis, we attempt to bind the harms landscape to the interventions landscape by developing a collaborative human-in-the-loop system where end-users can inspect their browsing history and generate corresponding interventions. we pursue a visual overlay modifications approach, extending on the work of greaseterminator @xcite . the framework renders overlay graphics over an underlay screen based on detected gui elements, images or text (as opposed to implementing program code changes natively), hence changes the interface rather than the functionality of the software. to provide end-users with input for self-reflection @xcite and source data for generating interventions, users can be shown their screenome @xcite , a record of a user's digital experiences represented as a sequence of screen images that they view and interact with over time to connect the input (screenome) and output (in-tervention), human-in-the-loop (hitl) learning can be used for users to annotate their screenomes for harmful text, images or gui elements, and these annotations can be used to develop interventions. @xcite offers a detailed review of hitl. specifically, the procedure to generate interventions using visual overlays will require the one-shot detection of masks (e.g. gui elements) and few-shot learning and/or model fine-tuning of image/text classification models. ## evaluation we evaluate the usability of (req 1) the hitl component (usability for a single user with respect to inputs/outputs; or "does our system help generate interventions?"), and (req 2) the collaborative component (improvement to usability for a single user when multiple users are involved; or "does our system scale with user numbers?") with cognitive walkthroughs and scalability tests respectively. ## conclusion to enable end-user autonomy over interface design, and the generation and proliferation of a distribution of harms and interventions to analyze and reflect upon, we contribute the novel interface modification framework greasevision. end-users can reflect and annotate with their digital browsing experiences, and collaboratively craft interface interventions with our hitl and visual overlay mechanisms. with respect to requirements 1 and 2, we find that our greasevision framework allows for scalable yet personalized end-user development of interventions against element, image and textbased digital harms. we hope greasevision will enable researchers and end-users to study harms and interventions, and other interface modification use cases.