Risks and NLP Design: A Case Study on Procedural Document QA (2023)

Abstract: As NLP systems are increasingly deployed at scale, concerns about their potential negative impacts have attracted the attention of the research community, yet discussions of risk have mostly been at an abstract level and focused on generic AI or NLP applications. We argue that clearer assessments of risks and harms to users—and concrete strategies to mitigate them—will be possible when we specialize the analysis to more concrete applications and their plausible users. As an illustration, this paper is grounded in cooking recipe procedural document question answering (ProcDocQA), where there are well-defined risks to users such as injuries or allergic reactions. Our case study shows that an existing language model, applied in “zero-shot” mode, quantitatively answers real-world questions about recipes as well or better than the humans who have answered the questions on the web. Using a novel questionnaire informed by theoretical work on AI risk, we conduct a risk-oriented error analysis that could then inform the design of a future system to be deployed with lower risk of harm and better performance.

## introduction much of the current discussion about ai-in both the research community and the broader publicfocuses on the tension between deployment of systems whose behavior is nearly indistinguishable from humans @xcite and understanding the potential consequences of such deployment, including fairness, reliability, and other social and ethical implications @xcite @xcite . a common theme is the lack of rigorous assessment or guidelines for deploying models to end users @xcite , with work in mitigating harms operating broadly over large, diverse settings @xcite @xcite . we propose specializing the analysis of potential harms to more concrete applications, with identifiable user communities, to help close the gap between theoretical research on harms and research on real systems and users. we focus on question answering grounded in procedural documents (proc-docqa), instructions written for a user to follow in order to complete a real-world task without supervision or assistance (e.g., cooking recipes), with discrete states of success. procdocqa can further be characterized along dimensions of risk of harm to the user and environment and the ex-pertise required for a user to complete the procedure successfully. by articulating clear user goals (i.e., executing an instruction successfully), designers can more systematically assess the interplay of risks and system performance. we introduce the first version of a risk-aware design questionnaire (radq) providing questions whose answers will be actionable for nlp designers of user-facing applications and conduct a case study in cooking recipes to illustrate how system designs evolve with the discovery of new risks. the case study shows how, despite zero-shot gpt-3 text-davinci-003 (brown et al., 2020) achieving performance that is quantitatively on par with human-written answers, a deeper inspection of multiple answers per question reveals errors that will require application-specific resolutions. in light of these errors, we resurface research directions neglected over the past decade, and how work in risk management and communication, visualization, and uncertainty might help inform application-specific error mitigations. ## procdocqa question answering is a mature nlp task with a diverse set of research datasets spanning many text and information domains, but risks and harms of question answering systems are underexplored, with work primarily in open-domain web question answering @xcite , user experience of a system (e.g., @xcite , or privacy and security of users in an information retrieval stage of the system (e.g., @xcite . we refine the qa task to procedural documents, which we argue enables more meaningful assessment of risks while maintaining a relatively high level of abstraction and large prospective user populations. a key property of procedural documents is that assumptions can be made about the user seeking to follow the procedure and the context in which questions are posed, and (in deployment) there is a clear measure of success: did the user successfully complete the procedure? assumptions about the user allow us to characterize genres and procedures within procdocqa along dimensions of risk of harm to the user and environment, concrete harms to specific entities that are more easily conceptualized than broad abstract harms to populations or society (as in @xcite @xcite , and expertise, skill required to successfully complete a procedure (figure 1 ). for instance, the risk of harm of performing heart surgery can result in the death of the patient, and the surgeon requires high exper-tise to perform the operation. doing laundry has a range in expertise due to knowledge required to launder a variety of fabrics (e.g., jeans vs. a suit jacket), but there is low risk of harm (e.g., damaged clothing). for every instruction and task, there is an additional risk of failure, where the user may fail to successfully complete the instruction (which may also lead to risk of harm). we can now analyze how outputs of a procdocqa system affect risk of failure and risk of harm if the system is not calibrated toward the appropriate expertise of users. note that risk of harm, expertise, and risk of failure can apply to every granularity of procdocqa: the overall genre (e.g., cooking), specific tasks (e.g. baking cookies), and individual instructions (e.g., chop onions). ## case study: procdocqa for recipes we present a case study on cooking recipes, a genre of procedural documents with tasks that span a large range of risk of harm and expertise required for its tasks, but narrow the scope to homestyle recipes, which require less expertise and have lower risk of harm than professional-style recipes. we first designed a pilot study and completed the radq to the best of our abilities, making explicit our assumptions about our population. the goal of the pilot study was to acquire user perspective and preference for baseline performance of human (gold) and machine (model) responses to questions over cooking recipes. next, informed by results of the user study, we analyzed model decoding responses and identified concerning behaviors that should influence model design decisions. in §4, we return to the radq and propose ways in which the model design could be updated to be more risk of harm and risk of failure aware. ## radq post-study update informed by our user perspective study and multioutput error analysis, we update our radq responses from §3.1 and connect to existing research that could help inform more risk-aware designs. q2 though we were initially skeptical when we observed explicit recommendations of specific brands in answers, users preferred them even with the knowledge that the recommendation comes from an automated system (appendix b.1). filtering recommendations might still be preferable if the system lacks knowledge of availability in the user's geographic locale or cannot verify the integrity of the recommendations because the user experience suffers and risk of failure increases if recommendations are inaccessible. work in qa answer verification (e.g., @xcite and fact verification (e.g., @xcite where answers require citations could help filter such recommendations. further work in balancing user preferences with theoretical harms of ai bias is needed to support development of practical, safe systems while maximizing user experience. ## ethics statement user studies were conducted after review by our institution's irb, and participants were paid a fair wage in accordance with our local government. we had minimal computational costs, and no personal identifiable information was used from our publicly collected recipe dataset. 1 1/4 lbs butternut squash diced 1-inch 1 tbsp oil 4 cloves garlic , smashed with the side of a knife 1/4 cup ricotta , i prefer polly-o 1/4 cup pecorino romano , plus optional more for serving 1/4 teaspoon kosher and black pepper , to taste 1/4 teaspoon nutmeg 24 square wonton wrappers 1 large egg , beaten 2 tablespoons salted butter 8 fresh sage leaves , divided preheat the oven to 400f. place butternut, 4 sage leaves and garlic on a sheet pan and toss with 1 tablespoon oil. season with 1/4 teaspoon salt and pepper, to taste. roast until tender, about 35 minutes. transfer to a bowl and mash with a fork until very smooth (a blender would work too). mix in ricotta and pecorino, season with nutmeg, 1/4 teaspoon salt and black pepper. place the wonton wrapper on a work surface, brush the edge lightly with egg wash and add 1 tablespoon filling onto the center. fold over into a triangle and press the edges to seal. cover with a damp cloth while you make the rest. chop remaining sage leaves. place butter and sage in a medium saucepan and melt over low heat. keep warm over very low heat. bring a large pot of salted water to a boil. add half of the ravioli (they are very delicate) and cook until the rise to the surface, about 2 minutes. use a slotted spoon to remove and add to the pan with the butter. repeat with the remaining ravioli. gently toss raviolis with the butter until warm, 1 to 2 minutes. top with black pepper and serve with additional pecorino romano, if desired. question: hello!!! can you use frozen butternut squash that's already cubed? (just to save time so i don't have to peel, cube myself?) if so, how would you recommend going about it? answer: 1 tablespoon olive oil 2 skinless and boneless chicken breast fillets , halved horizontally to make four fillets salt , to season 14 ounces | 400 grams sliced mushrooms 2 teaspoons butter 1 large french shallot , finely chopped (normal shallot for u.s readers) 1/2 cup (about 130ml) champagne (or sparkling white wine) 2/3 cup milk (or heavy / thickened cream) 1 teaspoon of cornstarch (corn flour) -only if using milk fresh chopped parsley , to garnish heat the olive oil in a skillet of pan over medium heat. season each chicken filet with a pinch of salt. sear chicken on both sides, for about 3-5 minutes each side (depending on thickness), until golden all over. transfer chicken to a plate. fry the mushrooms in the butter and fry for a further 3-5 minutes, or until just beginning to soften. transfer to the same plate as the chicken. cover and keep warm. add the shallot into the pan and cook for 4 minutes, while occasionally stirring. pour in the champagne; stir well, while scraping any food bits from the bottom of the pan for added flavour. season with salt and pepper and allow to simmer for about 10 minutes. add milk (or cream) and simmer for 2-3 minutes. if using milk: mix the cornstarch with a tablespoon of water; pour into the centre of the pan and mix throughout the sauce until slightly thickened. add the chicken and mushrooms into the sauce along with any cooking juices. coat in the creamy sauce. sprinkle with parsley and serve! question: do you have any suggestions for a mushroom substitute? that's a recipe deal breaker for me, otherwise this looks great. answer: you would need 4 bags of pre-cut cabbage for this recipe.