Modeling Disclosive Transparency in NLP Application Descriptions (2021)

Abstract: Broader disclosive transparency—truth and clarity in communication regarding the function of AI systems—is widely considered desirable. Unfortunately, it is a nebulous concept, difficult to both define and quantify. This is problematic, as previous work has demonstrated possible trade-offs and negative consequences to disclosive transparency, such as a confusion effect, where “too much information” clouds a reader’s understanding of what a system description means. Disclosive transparency’s subjective nature has rendered deep study into these problems and their remedies difficult. To improve this state of affairs, We introduce neural language model-based probabilistic metrics to directly model disclosive transparency, and demonstrate that they correlate with user and expert opinions of system transparency, making them a valid objective proxy. Finally, we demonstrate the use of these metrics in a pilot study quantifying the relationships between transparency, confusion, and user perceptions in a corpus of real NLP system descriptions.

## introduction among the draft ethics guidelines for trustworthy ai released by the european union in 2019 were calls for greater transparency around deployed systems using artificial intelligence, advising that an "ai system's capabilities and limitations should be communicated to practitioners or end-users in a manner appropriate to the use case at hand." (high-level expert group on ai, 2019) this is a highprofile example of the vagueness endemic to guidance on ethical disclosure and ai-what constitutes an appropriate manner of communication? there is a growing awareness of the importance of communicating ai system function and performance clearly and understandably. this is both a matter of public interest, for mitigating harms caused by, for example, racial biases in the performance of human classifiers @xcite , but also in the interest of system providers, as users who feel like they do not understand how machine learning models work and what kind of information they rely on tend to be more resistant to using them @xcite . however, while fairness @xcite and privacy as general principles @xcite have been well-studied in the context of ai, transparency does not receive as much attention. the term transparency is overloaded @xcite , with many different definitions in the literature @xcite . it has a broad meaning in the public consciousness, from which some studies adopt a "you'll know it when you see it" definition @xcite . as opposed to the notions of transparency as explainability @xcite and as invisibility @xcite , transparency as disclosure @xcite , the extent to which the producers of an ai system or service provide detailed messaging about a system to stakeholders such as buyers, users, or the general public. this third sense, disclosive transparency, is particularly subjective. @xcite identifies a confusion dynamic where providing "too much information" about a decision system worsens user understanding and trust. this dynamic could have serious ramifications-if disclosive transparency really decreases user trust, stakeholders are further incentivized to not disclose. unfortunately, studies on this relationship are small-scale and subjective. this is why measurable, repeatable, and objective measures of disclosive transparency are sorely needed. in this work, we decompose disclosive transparency into two components: replicability and style-appropriateness, develop three objective neural language model measures of them, and apply them in a pilot study of real systems. 1 2 decomposing disclosive transparency our goal is to develop an objective measure of disclosive transparency that is repeatable, explainable, intuitive, and well-correlated with subjective opinions. however, at face value there is not an obvious way to assign a numerical value to the "level of transparency" in a description. to resolve some vagueness, we decompose disclosive transparency into two components: the replicability, or degree to which the requisite content to reproduce the system being described is present in a description, and the style-appropriateness, or the degree to which it is written in a manner understandable to a member of the general public. these components are more specific than "transparency" alone, but still quite subjective. a key insight underpins the possibility of developing objective measures of replicability and styleappropriateness: disclosure is a communication task. in describing a system, an explainer, typically an authority providing an ai service to the public, attempts to encode information about the design and function of their system in a summary. we find the communication-based definition of meaning provided in bender and koller (2020) useful for motivation. in their framework, the meaning, m of an utterance is a pair (e,i) of the surface form e (such as text or speech audio) and communicative intent, i which is external to language. in the case of disclosive transparency, this i is the particulars of the system being described. any assessment of the disclosive transparency of a description e is fundamentally an assessment of its underlying i-the degree to which i contains the information necessary to reconstruct the system being described. furthermore, some short descriptions come paired with longer ones. anyone who has clicked "i agree" without reading a terms of service, but has taken the time to look over a short, to-thepoint statement on how data is used, knows that the lengthy legalese-laden descriptions that truly define systems we interact with can be sufficiently summarized succinctly. however, the succinct summarization is intended to carry the detailed meaning of the actual contract a user is agreeing to. in other words, for the short description e ′ of an agreement, there exists a 'source' document e that is much more detailed, which share a common i. an analogue in system descriptions is what enables our metrics. corpus of technical cs text bert training corpus technical language model general language model brief system desc. (e') detailed description recovery model corpus of (brief description, detailed description) pairs log-likelihood ratio detailed description of system. (e) information recovery ratio (e g , e) style-appropriateness replicability bertscore affinity rating ## objective transparency metrics we now move to developing usable metrics that quantify both the replicability and styleappropriateness of real system descriptions. the system demonstration tracks at academic ai conferences represent a good source of real-world system descriptions. ideally, the purpose of a system demonstration paper at an academic ai venue is to provide a sufficiently detailed system description such that a peer could (given sufficient resources and data) replicate it. each of these lengthy documents, e i , is accompanied by an abstract, e ′ i which briefly describes the system. although these abstracts are not intended for consumption by the general public, they do succinctly describe implemented systems, some of which are intended to (at least hypothetically) be used by the public. combined with the fact that these paper/abstract pairs are freely available online, these are an appealing subject of study for modeling disclosive transparency. details on our pilot study using demo track nlp papers is provided in section 4. figure 1 provides a high-level depiction of our objective disclosive transparency metrics, the styleappropriateness 'clarity' metric c(e), and the replicability metrics of 'sentence affinity' r a (e ′ ,e) and simulated 'information recovery ratio' r r (e ′ ,e). we assess these features using language model scores derived from gpt-2 @xcite and bert @xcite . for fine-tuning data on academic language in ai and cs, we produce a training dataset by crawling a random sample of 100k l a t e x files from the arxiv preprint repository with topic label cs.* from 2007-2020. additionally, we further collect all 30k crawlable l a t e x files from the cs.cl label from 2017-2020, to adequately sample recent research in natural language processing and computational linguistics. from these raw .tex files we produce task-specific plaintext training corpora as described in appendix c. we then fine-tune the gpt-2 regular pretrained model from huggingface @xcite on the corpora as explained below. we manually exclude all papers used in our downstream case study from pretraining. ## pilot studies to analyze how well our objective measures track with subjective notions of transparency, and to demonstrate how they might be used to investigate how transparency affects prospective user opinions in the real world, we perform two pilot studies utilizing our metrics and a corpus of real-world ai system descriptions. ## results to study the correlations both between pairs of subjective variables and between our objective metrics and the subjective responses, we use pearson's correlation coefficient (pcc). in subsection 5.1 we study how the objective transparency metrics connect to expert opinion ratings along the three dimensions of transparency identified in subsection 4.2. these are the results from the first pilot study, testing if our objective metrics capture the subjective notions of disclosive transparency as understood by experts. in subsection 5.2 we present the results from the second pilot study, and analyze relationships that appear between our objective transparency metrics, the subjective transparency ratings, and user opinion responses. in particular, we seek to determine if our objective measures can be used as suitable proxy for the subjective expert opinions in modeling user responses to transparency. in this section we have scatter plots containing several overlapping (x,y) coordinates. we opt to represent these points where multiple identical samples are present by varying both marker color and size to convey the relative quantities of repeats. ## related work most previous work on explainability and intelligibility in transparency focuses on "explanations that are contrastive and sequential rather than purely subjective" @xcite . furthermore, prior studies regarding transparency have tended to focus on transparency in post-facto user judgments of automated systems following their use. for example, kizilcec (2016) and wang et al. (2020) both studied how system users viewed the fairness of a system that judged them. in both studies, the authors found that while transparency affects perceived fairness, the strongest predictor of perceived fairness was final outcome favorability. work in explainable ai (xai) intersects with disclosive transparency when asking what kinds of explanations are preferred by users. several studies have directly asked users of experimental xai systems to respond to @xcite @xcite the quality and effects of the explanations. judging how well system function descriptions are motivated by explainable outputs is thus a promising direction. many solutions intended to remedy ethical concerns of machine learning include a disclosive component. @xcite studied how novice endusers can understand the function in context of intelligible systems. similarly to us, they derive explanations from fundamental questions of "what," "why," and "how" that resolve the gulf of evaluation and gulf of execution in the user, as put forward by @xcite . lipton (2018) introduced simulatability, the capacity for a user to comprehend all constituent operations of a model by manually performing them, as fundamental to transparency. this relies on a notion of decomposability @xcite . intelligibility is both an end in itself and a useful tool for ensuring buy-in from stakeholders such as users @xcite and management @xcite . knowles (2017) deal with designing systems that use intelligibility to convey evidence of trustworthiness under uncertain scenario, and frame intelligibility as the degree to which the system's underlying logic can be conveyed to users. ## conclusions our replication-based framework allows us to tackle the problem of characterizing disclosive transparency in nlp system descriptions. we developed style-appropriateness and replicability metrics using neural language models, which capture multiple subjective dimensions of transparency. we ran a pilot study characterizing layperson responses to varying levels of transparency in system descriptions, demonstrating the value of the conceptual framework and the concrete tools proposed. we release all code, data, and annotations online at https://github.com/michaelsaxon/ disclosive-transparency /. three natural directions for future work follow these results. the automated transparency metrics can remove the costly expert annotator constraint, enabling scaled up transparency-user opinion relationship studies. more sophisticated conditional abstract-to-full text inversion models can be directly swapped in with the replicability metric to produce better automated transparency scores. finally, the metrics can be used to guide nlg processes, perhaps to enable transparency-constrained abstractive summarization or style transfer.