Participatory Design for Positive Impact: Behind the Scenes of Three NLP Projects (2025)

Abstract: Researchers in Natural Language Processing (NLP) are increasingly adopting participatory design (PD) principles to better achieve positive outcomes for stakeholders. This paper evaluates two PD perspectives proposed by Delgado et al. (2023) and Caselli et al. (2021) as interpretive and planning tools for NLP research. We reflect on our experiences adopting PD practices in three NLP projects that aim to create positive impact for different communities, and that span different domains and stages of NLP research. We assess how our projects align with PD goals and use these perspectives to identify the benefits and challenges of PD in NLP research. Our findings suggest that, while Caselli et al. (2021) and Delgado et al. (2023) provide valuable guidance, their application in research can be hindered by existing NLP practices, funding structures, and limited access to stakeholders. We propose that researchers adapt their PD praxis to the circumstances of specific projects and communities, using them as flexible guides rather than rigid prescriptions.

## introduction participatory design (pd) is an approach to design and development that actively involves stakeholders, such as end users, customers, and citizens in the design process to ensure the resulting products meet their needs and reflect their values @xcite . pd engages with stakeholders using various methods, including surveys, focus groups, interviews, and workshops. while the human-computer interaction (hci) community has employed pd for decades, it remains less common in the natural language processing (nlp) community, likely due to the field's fast-paced development and focus on leaderboards. however, there is a growing recognition within nlp research of the need to move beyond purely technical * contribution conducted at edinburgh napier university. the adoption of pd in nlp has been suggested as a means to alleviate contemporary issues with nlp technology, namely bias, fairness, and usability @xcite . by involving diverse stakeholders in the design process, pd supports the development of nlp systems that are more inclusive and representative of various user groups, thus fostering equitable outcomes. additionally, incorporating stakeholder input early and iteratively can benefit developers by improving the suitability of nlp applications for real world use cases. @xcite suggested nine principles to guide the adoption of pd in nlp systems. these focus on community-based practice, fictional design scenarios, and enhanced reflexivity throughout data collection, annotation, deployment, and evaluation. more recently, @xcite proposed a framework for evaluating pd of ai systems. this focuses on various dimensions of participation that range from consultation to ownership in terms of design goals, scope, and methods. an overview of @xcite s framework are provided in tables 1 and 2 . in this paper, we use these principles and framework as lenses to reflect on three research projects which did not all necessarily aim to be maximally participatory from the outset. the projects present intricate challenges, varying risk levels and degrees • the shift from language as data to language as people: language data are produced by human speakers • communities should be involved in the different stages of the nlp pipeline 5. community involvement is not scraping the intention of this exercise is not to objectively report results for each project, which are available in other publications @xcite . here, the aim is to explore the relationship between the principles and framework and the practicalities of using pd in nlp research by focusing on our experience as researchers. incorporating our own inherent subjectivity into this reporting also furthers pd praxis by providing a concrete illustration of the process of researcher reflexivity that is central to pd. this paper makes the following contributions: (1) providing methodological insights into three distinct nlp projects that incorporate pd in their design; (2) evaluating these projects with respect to @xcite s principles to further understanding of how these can be applied in practice; and (3) supporting researchers interested in adopting these approaches in their own work by providing insights into researchers' experiences conducting participatory nlp research. our analysis indicates that the framework and principles should not be used as prescriptive templates. instead, they are useful for consultation as researchers plan and evaluate their individual research projects. however, researchers should consider other context-specific factors when incorporating participatory design in their research, such as the nature of the communities they are working with, funders' requirements, project timescales, and the expertise and knowledge available to support their projects. while pd can be viewed both as a research methodology and a design approach @xcite , its implementation varies and may draw upon different methods to create hybrid spaces that bring together users & designers to challenge assumptions and for mutual learning & co-creation. practitioners highlight the importance of local context and knowledge @xcite ; 'design as change', whereby possible futures, new tools, changed infrastructures, and interactions are envisioned @xcite ; and dissensus, with pluralities of conflicting positions inherent to the design process @xcite . the shift to pd means going beyond extractive forms of research participation, where people are sources of data, labels or evaluation @xcite , and as such represents an alternative approach to traditional user-centric design which does not allow for users to become full collaborators or own the direction of a project. instead, a range of stakeholders should have an opportunity to shape the research process and outputs. this can include, a number of approaches, such as partici-patory research (i.e. citizen or open science (e.g. @xcite ; participation in model deployment and use @xcite ; and participation in evaluation or feedback for nlp systems (e.g. @xcite . @xcite and @xcite reviewed existing research to develop a framework and principles for pd in nlp. in turn, this paper uses these as a lens to reflect on three projects, offering practical insights into the application of pd in nlp. framework @xcite reviewed work that claims to engage in pd, considering the research goals, scope and methods, mapping these to the spectrum of participation modes: consultinclude -collaborate -own. their analysis found that most of the 59 publications engaged in consultation rather than ownership. they concluded that computer scientists need to do more than simply 'add diversity and stir' to achieve true pd. principles @xcite developed nine principles for pd in the development of nlp systems. the principles integrate general pd principles and nlp practice, with an emphasis on researcher reflexivity. summaries of the framework and principles are available in tables 1 and 2 . ## three pd nlp projects in this section, we provide an overview of the projects as context for the analysis using the framework and principles discussed in §4 ## analysis we now apply @xcite framework @xcite principles to structure our reflection on these projects. italics are used to highlight where specific concepts from these sources are used. with respect to @xcite , we map the goal, scope, and form of participation of each project on the consultation to ownership spectrum. with regard to @xcite , we identify the extent to which each project's planning and execution exemplify the participatory design principles (1-3), principles for nlp tools (4-6), and principles of researcher reflexivity (7-9). see ## conclusion we do not highlight the challenges to deter potential participatory researchers, but to equip them with the knowledge needed to plan effective research. we find @xcite framework to be useful for planning participatory research and understanding the landscape of pd in nlp. @xcite principles provide valuable support for ongoing reflexivity. both are helpful for improving our understanding of pd in practice. however, as delgado et al. emphasise, pd research should not be evaluated based on its position on the framework. the goals, scope and form of pd research should be determined based on the specific research context. our practical advice to potential pd researchers takes a similarly pragmatic approach. giving consideration to how pd could be incorporated into projects can benefit both researchers and stakeholders. in practice, this means taking time early in the research process to understand who might have an interest in the topic or be impacted by future applications of the research. before finalising detailed research objectives or plans, we recommend engaging with those people, even informally, to explore opportunities for them to contribute to shaping the goals, scope, or form of the research. this should also take into account what is feasible within the social, political, and financial context of the research. as pd often requires flexible timelines and resources, this should be factored into funding proposals to ensure that the shared goals can be realized. critical engagement with a wide range of research methods is also key. to this end, we encourage nlp researchers to be open to selecting and adapting methods to suit the specific context. peer review and writing practices that normalize reporting feedback from researchers and participants about their experience of the research can help build our understanding of which methods are suited to different settings. compared to the usual research cycle, where a problem is explored from conception to publication (and then considered complete and dropped), pd is time-consuming. it requires a broad range of skills from the research team, and requires ongoing engagement from participating stakeholders. it is an uncertain process, requiring researchers to let go of ownership and see where the co-design process leads. however, the benefits of undertaking pd are outputs that are closely aligned with stakeholders' needs and that reflect their priorities and language. this means that the research is more likely to result in positive impacts for the communities it aims to serve.