Ethics Sheets for AI Tasks (2022)

Abstract: Several high-profile events, such as the mass testing of emotion recognition systems on vulnerable sub-populations and using question answering systems to make moral judgments, have highlighted how technology will often lead to more adverse outcomes for those that are already marginalized. At issue here are not just individual systems and datasets, but also the AI tasks themselves. In this position paper, I make a case for thinking about ethical considerations not just at the level of individual models and datasets, but also at the level of AI tasks. I will present a new form of such an effort, Ethics Sheets for AI Tasks, dedicated to fleshing out the assumptions and ethical considerations hidden in how a task is commonly framed and in the choices we make regarding the data, method, and evaluation. I will also present a template for ethics sheets with 50 ethical considerations, using the task of emotion recognition as a running example. Ethics sheets are a mechanism to engage with and document ethical considerations before building datasets and systems. Similar to survey articles, a small number of carefully created ethics sheets can serve numerous researchers and developers.

## the case: importance of ethics considerations at the level of ai tasks good design helps everyone. it is well established, for example, that designing for accessibility helps society at large. 1 as artificial intelligence (ai), machine learning (ml), and natural language processing (nlp) systems become more ubiquitous, their broad societal impacts are receiving more scrutiny than ever before. however, several high-profile instances such as face-recognition systems that perform poorly for people with dark skin tones @xcite , machine translation systems that are biased against some genders @xcite , question answering systems that produce moral judgments @xcite , and mass testing of emotion recognition systems on certain sub-populations (article19, 2021; wakefield, 2021), have highlighted how technology is often at odds with the very people it is meant to help, and how it will often lead to more adverse outcomes for those already marginalized. this raises uncomfortable questions for us ai researchers, developers, and leaders of technology companies: what role do we play in the harms perpetrated by technology? what are the assumptions in our research? what are the implications of our choices? are we striking at the barriers to opportunity or are we amplifying societal inequities? the answers are often complex and multifaceted. while many ai systems have clear benefits, we are increasingly seeing examples such as those discussed above where real-world ai systems are causing harm. academic research (which often feeds into real-world systems), is also seeing growing amounts of criticisms: criticisms of physiognomy, racism, bias, discrimination, perpetuating stereotypes, ignoring indigenous world views, and more. see @xcite and @xcite for recent examples. there have also been criticisms of thoughtlessness (e.g., is automating this task, this way, really going to help people?) and a seemingly callous disregard for the variability and complexity of human behavior @xcite @xcite . this position paper makes the following contributions: (1) it describes recent efforts by the ai community to encourage responsible research, the limitations of those efforts, and the need for thinking about ethical considerations at the level of ai tasks. (2) presents a detailed proposal for a new kind of document, ethics sheets for ai tasks, ded-icated to fleshing out the assumptions and ethical considerations hidden in how a task is commonly framed and in the choices we make regarding the data, method, and evaluation. (3) provides a template for ethics sheets (that includes fifty ethical considerations), with the task of automatic emotion recognition (aer) as a running example. nlp tasks, such as aer from text, machine translation, and summarization, are particularly rife with ethical considerations because they deal with language and people. ethics sheets can help in recognizing and communicating the social and psychological complexities of language use; thereby, driving the desired design choices in nlp systems. more broadly, all ai tasks that deal with people and their artifacts (such as text, images, and video) can benefit from carefully thought out ethics sheets. every year, tens of thousands of people are joining the ranks of ai researchers and developers. ethics sheets can serve them and others as useful introductory documents for ai tasks, guiding research/system design, facilitating the creation of datasheets and model cards, and acting as springboards for new ideas in responsible research. ## proposal: ethics sheets for ai tasks if one wants to do work on an ai task, then right at the beginning it is useful to have access to: a document that substantively engages with the ethical issues relevant to that task; going beyond individual systems and datasets, drawing on a body of relevant work. similarly, if one conceptualizes a new ai task, then it is useful to simultaneously create such a source of information. therefore, i propose that we researchers and developers write such articles, which i will refer to as ethics sheets for ai tasks. in some ways, ethics sheets are similar to survey articles for areas of research, except here the focus is on ethical considerations for an ai task. simply put: an ethics sheet for an ai task is a semi-standardized article that aggregates and organizes a wide variety of ethical considerations relevant for that task. it: the sheet should flesh out various ethical considerations that apply at the level of task. it should also flesh out ethical consideration of common theories, methodologies, resources, and practices used in building ai systems for the task. ethics sheets may sometimes suggest that certain applications in specific contexts are appropriate or inappropriate, but largely they are meant to discuss the various considerations to be taken into account when the developer is deciding whether to build or use a particular system, how to build it, and how to assess its societal impact. it is meant to help the developer identify what is more appropriate for their given deployment context. a good ethics sheet will question some of the assumptions that often go unsaid. it will encourage more thoughtfulness: thinking about these questions is important if we want to break away from the current paradigm of building things that are divisive (that work well for some and poorly for others) and instead move towards building systems that treat human diversity and variability as a feature (not a bug); systems that truly dismantle barriers to opportunity, and bring diverse groups of people together. thus, questions such as those shown above can be useful in determining what is included in ethics sheets. target audience: the target audience for an ethics sheet includes the various stakeholders of the ai task. the stakeholders may or may not have the time and background to understand the technical intricacies of an ai task. however, they build on, use, and make laws about what we create. further, people are impacted by ai systems. they should be able to understand its decisions that impact them, understand its broad patterns of behaviour, contest the predictions, and find recourse. ethics sheets can help to that end. it is our responsibility to describe our creations in accessible terms, so that others can make informed decisions about them. thus the target audience includes: ethics sheets complement datasheets and model cards: while the latter are post-production documents produced by system/data builders, ethics sheets are meant to be accessed before building systems. similar to traditional survey articles, a small number of carefully created ethics sheets can serve numerous researchers and developers creating systems and data for ai tasks. see the faq in the appendix (after references) for a discussion on some practicalities involved with who should create ethics sheets, when they should be created, for which tasks, etc. i discuss below some key characteristics and benefits of ethics sheets, followed by a template and a pointer to an example ethics sheet in the next section. ## a template and an example sheet i present below a template that can serve as a handy starting point in the creation of new ethics sheets, and that further clarifies what can be included in an ethics sheet. in the template below i will use automatic emotion recognition (aer) as the running example. aer is a particularly interesting, widely applicable, and complex example of ai tasks with notable benefits and risks. thus an ethics sheet for aer can be particularly instructive. in her seminal book, affective computing, dr. rosalind picard described automatic emotion recognition (aer) as: "giving emotional abilities to computers". it is a sweeping interdisciplinary area of study exploring many foundational research questions and many applications @xcite . however, some of the recent commercial and governmental uses of aer have garnered considerable criticism, including: infringing on one's privacy, exploiting vulnerable sub-populations, and even allegations of downright pseudo-science (wakefield, 2021; article19, 2021; woensel and nevil, 2019). even putting aside high-profile controversies, emotion recognition impacts people and thus entails ethical considerations (big and small). mohammad (2022) presents an ethics sheet for automatic emotion recognition and sentiment analysis. it is a critical reflection of this broad field of study with the aim of facilitating more responsible emotion research and appropriate use of the technology. i will use some details from that sheet below to clarify the elements of the generic template. ## concluding thoughts in this position paper, i discussed how ethical considerations apply not just at the level of individual models and datasets, but also at the level of ai tasks. i presented a new form of documenting ethical considerations, which i call ethics sheets for ai tasks. it is a document dedicated to fleshing out the assumptions and ethical considerations hidden in how a task is commonly framed and in the choices we make regarding the data, method, and evaluation. i listed various benefits of such ethics sheets and discussed caveats such as how a single ethics sheet does not speak for the whole community. i also provided a template sheet and an example, proof-of-concept, ethics sheet for automatic emotion recognition. ethics sheets have the potential for engaging various stakeholders of ai tasks towards responsible research and development. i hope that this work spurs the wider community to ask and document: what ethical considerations apply to my task? note: see faq in the appendix for practical considerations involved in who should create ethics sheets, when, for what tasks, etc. jane wakefield. 2021. ai emotion-detection software tested on uyghurs. lieve van woensel and nissy nevil. 2019. what if your emotions were tracked to spy on you?