Establishing Trustworthiness: Rethinking Tasks and Model Evaluation (2023)

Abstract: Language understanding is a multi-faceted cognitive capability, which the Natural Language Processing (NLP) community has striven to model computationally for decades. Traditionally, facets of linguistic intelligence have been compartmentalized into tasks with specialized model architectures and corresponding evaluation protocols. With the advent of large language models (LLMs) the community has witnessed a dramatic shift towards general purpose, task-agnostic approaches powered by generative models. As a consequence, the traditional compartmentalized notion of language tasks is breaking down, followed by an increasing challenge for evaluation and analysis. At the same time, LLMs are being deployed in more real-world scenarios, including previously unforeseen zero-shot setups, increasing the need for trustworthy and reliable systems. Therefore, we argue that it is time to rethink what constitutes tasks and model evaluation in NLP, and pursue a more holistic view on language, placing trustworthiness at the center. Towards this goal, we review existing compartmentalized approaches for understanding the origins of a model’s functional capacity, and provide recommendations for more multi-faceted evaluation protocols.

## introduction understanding natural language requires a multitude of cognitive capabilities which act holistically to form meaning. modeling this ability computationally is extremely difficult, thereby necessitating a compartmentalization of the problem into isolated tasks which are solvable with available methods and resources @xcite . undoubtedly as of late 2022, we are witnessing a paradigm shift: powerful llms, in the form of instruction-tuned, prompt-based generative models such as chatgpt and gpt-4 @xcite @xcite @xcite , have found widespread adoption reaching far beyond the nlp community. part of this success story is the casting of heterogeneous nlp tasks into sequence-to-sequence tasks @xcite @xcite ; which in turn enables extreme multi-task learning, and cross-task transfer learning. this is in stark contrast to the traditional compartmentalized nlp paradigm (visualized in figure 1 ), wherein a human-motivated language task with an input expression and an output expectation is clearly formalized into a dataset with machinereadable inputs and outputs. both feature design and model development are highly task-specificoften manually curated. paired with evaluation protocols for comparing model predictions with human expectations via formalized metrics or qualitative judgement, this general methodology has been widely adopted and trusted. foot_0 however, with contemporary llms this compartmentalization is breaking down-having severe impacts on all stages of the cycle. therefore, a persistent and critical question regains importance: how can trust be established between the human and the model? as early as 44 years ago, hays (1979) offers an attempt and provides a definition of trustworthiness (cf. quote). today, the topic of trustworthiness is an ongoing discussion deserving special attention @xcite @xcite . we argue that to establish trust, it is time to rethink how we deal with tasks and their evaluation. why now? it is getting increasingly hard to predict a priori when we can expect models trained on web-scale data to work well. were we to live in a hypothetical world with full knowledge of origin and functional capacity, then each task instance could be routed to the right model(s) to not only tap into the llms' full potential, but to also enable trust in their predictions. today, the absence of this knowledge is directly linked to our lack of trust in deploying models in real-world scenarios. in this position paper, we synthesize contemporary work distributed throughout different subfields of nlp and ml into a conceptual framework for trust, guided by hays (1979)'s definition and centered around knowledge facets as a guiding principle for all aspects of the model development and evaluation cycle. we outline high-level desiderata ( §2), and suggest directions on how to gain trust, by providing starting points of facets ( §3) aimed to stipulate uptake and discussion. in §4 we discuss how trustworthiness relates to user trust. ## desiderata for trustworthy llms llms today pose a conundrum: they are seemingly universally applicable, having high functional capacity, however, the larger the model, the less we appear to know about the origins of its capabilities. how did we get here, which aspects contribute to trustworthiness, and what did we lose on the way? in the following, we aim to provide a brief history of central trust desiderata (d1-4), discussing how our knowledge of functional capacity and its origins has changed over time. ## trustworthiness and user trust so far we have discussed different avenues for improving our knowledge about llm's functional capacity and origin, paving the way for establishing trustworthiness. from a user perspective it is essential to not only understand knowledge facets but also how they empirically impact user trust in a collaborative environment. this is especially important in high-risk scenarios such as in the medical and legal domain. one could argue, if llms such as chatgpt are already widely adopted, do we already trust llms (too much)? to better understand user trust we need interdisciplinary research and user experience studies on human-ai collaboration. specifically, we need to know what users do with the model output across multiple interactions (e.g., verify, fact check, revise, accept). for example, @xcite investigate the connection between explanations (d2) and user trust in the context of question answering systems. in their study users are presented with explanations in different modalities and either accept (trust) or reject (don't trust) candidate answers. similarly, smith-renner et al. (2020) discuss how generated explanations can promote over-reliance or undermine user trust. a closely related question is how the faithfulness of explanations affect user trust @xcite . for a comprehensive overview on user trust we refer to the recent survey by @xcite . while such controlled studies using human feedback are cost and time intensive, the minimum viable alternative for establishing trust may simply be the publication of a model's input-output history. in contrast to standalone metrics and cherry-picked qualitative examples, access to prior predictions enables post-hoc knowledge of model behaviour (d2), even without direct access to the model. this democratizes the ability to verify functional capacity and helps end users seeking to understand how well a model works for their task. in summary, evaluating user trust is an integral part of trustworthiness and goes hand in hand with careful qualitative analyses and faceted quantitative evaluation. towards this goal, we believe llm development needs to be more human-centric. ## conclusions in this position paper, we emphasize that the democratization of llms calls for the need to rethink tasks and model evaluation, placing trustworthiness at its center. we adopt a working definition of trustworthiness and establish desiderata required to improve our knowledge of llms ( §2), followed by suggestions on how trust can be gained by outlining directions guided by what we call knowledge facets ( §3). finally, we draw a connection between trustworthiness as knowledge facets and user trust as means to evaluate their impact on human-ai collaboration ( §4).