How Far Can We Extract Diverse Perspectives from Large Language Models? (2024)

Abstract: Collecting diverse human opinions is costly and challenging. This leads to a recent trend in exploiting large language models (LLMs) for generating diverse data for potential scalable and efficient solutions. However, the extent to which LLMs can generate diverse perspectives on subjective topics is still unclear. In this study, we explore LLMs’ capacity of generating diverse perspectives and rationales on subjective topics such as social norms and argumentative texts. We introduce the problem of extracting maximum diversity from LLMs. Motivated by how humans form opinions based on values, we propose a criteria-based prompting technique to ground diverse opinions. To see how far we can extract diverse perspectives from LLMs, or called diversity coverage, we employ a step-by-step recall prompting to generate more outputs from the model iteratively. Our methods, applied to various tasks, show that LLMs can indeed produce diverse opinions according to the degree of task subjectivity. We also find that LLMs performance of extracting maximum diversity is on par with human.

## introduction using nlp for tasks that require social reasoning or involve human subjectivity like argumentation @xcite or toxicity detection @xcite often calls for diverse perspectives. instead of providing a single viewpoint, an ideal nlp model should accommodate various perspectives to avoid any bias towards a dominant one. prior works emphasize the importance of modeling multiple viewpoints @xcite . some studies have addressed this challenge by gathering responses from multiple human annotators with diverse backgrounds (rottger et al., 2022; @xcite . however, this approach is costly in time and resources. recent advancements of llms have gained much interest from researchers to exploit their capability of creative generation for data augmentation with less cost and higher diversity @xcite @xcite . llm is known as a compressed parametric knowledge (e.g., blurry jpeg) of the training corpus @xcite , and our work study how people's pluralistic diverse opinions are compressed in the parameter and how far we can reversely extract them from an llm. figure 1 illustrates the significance of understanding the extent of diversity achievable by llms. during training, llms have access to various writings from humans with distinct values. yet, can llms reflect this diversity when generating text, or do they tend to favor majority opinions? how do llm-generated opinions overlap with human viewpoints? if the purple circle in figure 1 hypothetically represents the maximum diversity achievable by humans, we aim to explore methods for llms to approach this diversity. in real-world scenarios, humans may take different stances on a subjective statement (figure 2 ). for instance, those valuing teamwork and goal achievements may agree with the statement, while others valuing creativity and innovation could oppose it. then the key question is: how many and what diverse perspectives do llms model? we introduce a novel problem called maximum diversity extraction from llms. formally, our task involves (1) asking llms to generate as many different stances as possible, (2) providing reasons for each stance, and (3) producing important criteria words that guide their reasoning process. we apply this diversity prompting across four subjective tasks: social norms, argumentation, hate speech labeling, and story continuation. contributions first, we propose perspective diversity as a new focus for generative llms, distinct from lexical, syntactical, and semantic diver-sity mainly explored in previous studies. through various experiments, we assess llms' capacity to achieve maximum perspective diversity. second, we introduce criteria-based diversity prompting to extract and ground diverse perspectives from llms. finally, we suggest a step-by-step recall approach to measure the extent of diversity coverage of llms, comparing the coverage between llmgenerated opinions with human-authored opinions. our study reveals a saturation point in the diversity that llms can achieve, depending on the subjectivity of a task. also, llms generally produce more diverse opinions than an individual human, but two or more humans achieve greater diversity. regarding the quality of llm's generation, llm is able to generate opinions which are semantically similar to human opinions. however, some frequent criteria words by llm are different from what humans consider as important. ## human vs. llms: diverse opinion generation to assess human capabilities of generating diverse opinions, we hire crowd-workers from amazon mechanical turk to generate as many opinions as they can (≥ 3) for each stance (agree, disagree) on 100 statements from social-chem-101 and 67 statements from cmv. these statements are a subset of the dataset used in our recall prompting experiments. each worker writes opinions for five ## conclusion and future work to the best of our knowledge, this is the first work that tackles extracting maximum perspective diversity from llms. to do this, we propose a criteria-based prompting method and probe llms' capacity to generate as many diverse perspectives as possible and explain their reasons for choosing their corresponding stances on subjective statements. through our step-by-step recall prompting, we characterize the subjectivity of various tasks and reach the maximum diversity of llm's generation. llms can generate comparable number of diverse outputs with humans and similar values as humans' responses. as we compare llms' opinion generations with human's, they are quite "precise" as they are semantically similar to human opinions, bur their recall is slightly lower than humans. while the number of criteria clusters does not precisely mean the ideal maximum diversity, it indicates that we could use llms to push further perspective diversity to include more diverse opinions. our work opens up a wider range of possibilities for examining more advanced diversity "quantification" and "maximization" methods. there are also many application possibilities for extending this work in the future. in this study, we have not assessed how much the extracted diverse opinions are similar with the real world's diverse opinions yet. instead, we focus on the diversity coverage. future work could evaluate this by comparing the distribution of extracted opinions with a distribution of people's opinions collected from real world survey or poll data. we recommend further exploration of cultural aspects, persona, or human values on diversity extraction. moreover, our method could be applied for curating diverse data for open-ended tasks, such as generating diverse outputs for instructiontuning tasks or subjective task labeling. findings that it takes two humans to equalize llms' diverse generation capacity suggest that communications between multiple humans or llms can be a future work to introduce more diverse perspectives.