Credible without Credit: Domain Experts Assess Generative Language Models (2023)

Abstract: Language models have recently broken into the public consciousness with the release of the wildly popular ChatGPT. Commentators have argued that language models could replace search engines, make college essays obsolete, or even write academic research papers. All of these tasks rely on accuracy of specialized information which can be difficult to assess for non-experts. Using 10 domain experts across science and culture, we provide an initial assessment of the coherence, conciseness, accuracy, and sourcing of two language models across 100 expert-written questions. While we find the results are consistently cohesive and concise, we find that they are mixed in their accuracy. These results raise questions of the role language models should play in general-purpose and expert knowledge seeking.

## "expertise" in language models individuals and companies are increasingly looking to language models as a source of expert question answering. for example, @xcite lays out a vision for search that involves language models providing answers to user-generated questions. unfortunately, the challenge for many language models is that they are trained to generate language, not to have correct answers. as shah and bender write, "to the extent that [language models] sometimes get the right answer to. . . questions @xcite is only because they happen to synthesize relevant strings out of what was in their training data. no reasoning is involved" @xcite . this has led @xcite to characterize chatgpt as a "bullshit generator"plausible, but not accurate @xcite . while language models might incidentally produce accurate answers to simple and uncontested queries (e.g., "what is the capital of france?"), we might be understandably skeptical that it will produce correct answers to more nuanced questions. generated language reflects its training data and-to the extent the training data is publicly known-it is more reflective of the web than expert speech @xcite . by using experts evaluating material in their domain of choice, we provide an initial assessment of expertise provided by these models. ultimately what constitutes sufficient accuracy for broader use depends on the use case. ## methodology we evaluate two recently-released language models: openai's chatgpt and you.com's youchat (google's bard and many other options weren't released at the time of initial submission). ope-nai's chatgpt is the wildly popular evolution of the gpt-3 model @xcite and youchat is built specifically for search. both systems have a free and public option (at the time of writing) which makes them generally accessible. we survey 10 experts across a range of arbitrarily-chosen disciplines from quantum information to ballroom dance (see a complete list in the appendix). we recruited experts from our personal networks aiming to cover a wide-range of different types of knowledge (with the understanding we cannot be exhaustive or representative). the majority hold a doctorate or medical degree. we asked each expert to fill out an online survey with their own description of their area of expertise, two wikipedia pages pertinent to it, and five common questions and five niche questions from their domain (see the survey design elicits subjective expert judgment of system performance. we evaluate coherence, conciseness, and accuracy as important properties in information-seeking @xcite . comparing assessments to wikipedia provides a difficult-to-beat baseline with which many people are already familiar. we also ask whether the language model provides a source for its information. evaluating the source of the information in the response is important not only for the purposes of giving credit, but also as a mechanism for accountability @xcite . after all the questions, we directly ask whether the expert would recommend the tool for general purpose and professional use, and if the style of the content is obviously automated @xcite . we make our data (including the full context for all quotes we use here) publicly available to help support future work. 2 ## results across disciplines we discuss two running examples: one about maria callas, a famous new york-born opera singer with an international career often associated with greece, and one about quantum information. our opera expert asks, "was maria callas american?". chatgpt answers "no, maria callas was not american. maria callas, whose full name was maria anna cecilia sofia kalogeropoulos, was born on december 2, 1923, in new york city, united states. however, she was of greek descent. her parents were greek immigrants, and she later acquired greek citizenship. maria callas is renowned as one of the greatest opera singers of the 20th century and was particularly known for her performances in the bel canto repertoire." as a second example, a quantum expert asks "can quantum computers be realized in the physical world?". youchat answers, "yes, quantum computers can be realized in the physical world. researchers have been able to build prototype systems that demonstrate the principles of quantum computing, and there are a number of research groups and companies working on building more advanced versions. additionally, several companies have announced plans to build full-scale quantum computers in the next few years." ## discussion language models were coherent, but undersourced and not always accurate. they were generally not endorsed for professional use, but were seen as valuable by some experts as a source of knowledge for people out of the domain. providing sourcing citations will be an important step in building confidence. citations are sufficiently inconsistent when they appear to merit verifying important results. our findings are reinforced by the contemporaneous work of liu et al. ( 2023 ) which provides a more systematic audit of four generative search engines (including youchat, but not chatgpt) on a diverse series of queries (including common google searches and questions on reddit) using 34 prescreened mturk annotators. they also find that these search engines are "credible without credit"having high fluency and perceived utility, but insufficient sourcing. they find that about half of the responses are fully supported by citations and three fourths of the citations given didn't actually support the sentence. one of their main findings is a negative correlation between citation recall/precision and fluency/perceived utility. sourcing is so absent in our study that we observe no meaningful correlation with other variables and accuracy has positive correlation with cohesion and conciseness. further work could investigate if these discrepancies are due to differences in the systems evaluated, the kinds of questions asked, or the judgments of experts vs. annotators. this difference aside, their findings resonate with ours that credibility without credit should make us cautious in looking to language models as a source of expertise.