Modular Pluralism: Pluralistic Alignment via Multi- LLM Collaboration (2024)

Abstract: While existing alignment paradigms have been integral in developing large language models (LLMs), LLMs often learn an averaged human preference and struggle to model diverse preferences across cultures, demographics, and communities. We propose Modular Pluralism, a modular framework based on multi-LLM collaboration for pluralistic alignment: it “plugs into” a base LLM a pool of smaller but specialized community LMs, where models collaborate in distinct modes to flexibility support three modes of pluralism: Overton, steerable, and distributional. Modular Pluralism is uniquely compatible with black-box LLMs and offers the modular control of adding new community LMs for previously underrepresented communities. We evaluate Modular Pluralism with six tasks and four datasets featuring questions/instructions with value-laden and perspective-informed responses. Extensive experiments demonstrate that Modular Pluralism advances the three pluralism objectives across six black-box and open-source LLMs. Further analysis reveals that LLMs are generally faithful to the inputs from smaller community LLMs, allowing seamless patching by adding a new community LM to better cover previously underrepresented communities.

## introduction alignment of large language models (llms) aims to adapt models to reflect human values, intentions, and preferences @xcite . however, human preferences are not a monolith: norms, values, and priorities vary greatly informed by community, culture, demographics, ideology, and more @xcite @xcite @xcite . the increasing ubiquity of llms necessitates them to model and reflect pluralistic human values (e.g., pluralistic alignment @xcite ), but existing alignment procedures might actually harm pluralism according to empirical and theoretical studies @xcite @xcite . improvements in data composition @xcite , alignment objective @xcite , and modeling frameworks @xcite might produce more pluralistic models by retraining or re-aligning llms. nevertheless, some of the most popular llm services with the broadest set of users are proprietary and feature black-box llms @xcite , whereas existing methods are not directly applicable in black-box settings. in addition, when one community, culture, or perspective is found to be underrepresented after training/alignment completed, retraining or adapting llms to patch those representation gaps is very expensive. to this end, we propose modular plural-ism, a plug-and-play pluralistic alignment framework with multi-llm collaboration @xcite . in modular pluralism, an llm that only needs black-box access collaborates with a pool of specialized community lms, incorporating values and perspectives across diverse communities through token-level interactions. concretely, we first train community lms-language models specialized to represent a certain community-by finetuning existing lm checkpoints on communityspecific corpora. depending on the type of pluralism (adopted from @xcite , mod-ular pluralism features three modes of multi-llm collaboration (figure 1 ): (1) overton pluralism, where llms should provide a range of reasonable answers in the overton window foot_1 to a user query. in this setting, community lms generate diverse comments and the black-box llm summarizes these arguments into a coherent response. we evaluate modular pluralism with six open-source and proprietary llms of varying sizes on four datasets and six tasks spanning the three types of pluralism. we compare modular plu-ralism against vanilla llms, existing alignment procedures, prompting for pluralism, and a mixture-of-experts method @xcite . extensive experiments demonstrate that modular pluralism improves the coverage of diverse values for overton pluralism by 68.5% on average, offers greater steerablility towards values and demographic attributes when generating responses in 26.6% and 10.4% of cases, and better reflects the distributional nature of moral scenarios and global perspectives by at least 10.9%. further analysis reveals that modular pluralism enables patching underrepresented communities by plugging in a new community lm and could be extended to model cultural pluralism in addition to opinions and perspectives. we will make all code and data publicly available upon publication. ## methodology modular framework in modular plural-ism, we aim to enable the collaboration between llms with black-box access and a pool of smaller but specialized community models for pluralistic alignment (figure 1 ). concretely, we assume access to an llm's output and token probabilities and train a pool of community lms @xmath0, c 2 , • • • , c k }, each finetuned on a community corpora d i from an existing model checkpoint c, formally c @xmath1. these corpora {d i } aim to represent diverse demographics, cultures, and socio-political backgrounds, collected from news, social media, and more @xcite . given a user query q, instead of solely relying on llm, the smaller community lms generate messages/comments first m @xmath2. depending on the type of pluralism objective @xcite , modular pluralism features three modes of decoding-time collaboration @xcite . overton pluralism overton pluralistic models should reflect diverse values and perspectives in response to user queries. to this end, all smaller community lms are employed to generate comments {m 1 , • • • , m k }. these comments are then concatenated together along with the query q, where the llm serves as a multi-document summarization system to synthesize diverse viewpoints into a coherent response: @xmath3, • • • , m k }). specifically, we employ the prompt "please comment on a given situation with the help of the following passages." for the llm to encourage faithful representation of diverse perspectives from community lms. steerable pluralism steerable pluralistic models should be able to faithfully steer towards certain values/attributes when requested to in the user query, respecting the agency of diverse llm user populations. the role of the llm in this case is to select a community lm that best reflects the priorities of the given attribute. concretely, given the diverse messages from community lms {m 1 , • • • , m k } about the query q, the llm select one message based on the attribute @xmath4: @xmath5, • • • , m k } | llm, q, a). we use the prompt "which of the following comments best @xmath6. we expect llms to pick different community lm messages based on different attributes in a and generate a response conditioned on that message: @xmath7, a). ## experimental settings models we employ six open and proprietary llms for model's pluralism evaluation: llama2-13b @xcite , chatgpt @xcite , llama2-7b, llama2-70b, llama3-8b, and gemma-7b @xcite . we mainly focus on llama2-13b and chatgpt in the main paper to cover large and small, black-box and open llms: we present results for other models in appendix a. for each llm, we employ both unaligned base models and their aligned versions. ## results modular pluralism better covers diverse values and perspectives. we present the results for overton w/ nli evaluation in figure 2 . modu-lar pluralism achieves the highest coverage of values on both llms, with an improvement of up to 50.3%. prompting for pluralism does not result in stable improvements: we find that promptingbased approaches often produce a rigid and templated response ("on one hand, . . . on the other, . . . therefore . . . "). in contrast, modular plu-ralism produces a natural and coherent summarization of varying perspectives from community lms and the llm itself (appendix a). we additionally observe that our approach works better with aligned llms than unaligned ones, especially for chatgpt with an improvement of 27.2%. this is attributable to the role of llms in modular pluralism: they act as multi-document summarization systems to synthesize diverse comments from community lms into a coherent response, while aligned llms are better at instruction following at carrying out these tasks. nevertheless, our approach also significantly improves unaligned base llms. human and gpt-4 evaluation find that mod-ular pluralism produces more pluralistic responses. we present the results for overton w/ human and gpt-4 evaluation in figure 3 . we find that modular pluralism consistently achieves higher win rate against all three baselines and two evaluation settings. the five human annotators have a fleiss' kappa of 0.4678, indicating moderate and reasonable agreement. among the three baselines, prompting offers a more competitive approach in both evaluation settings, while mod-ular pluralism's win rate is still 45.8% and 16.5% higher in human and gpt-4 evaluation. together with the nli evaluation, modular plu-ralism is consistently established as more overton pluralistic and could produce better well-rounded responses that cover diverse sides of the problem. modular pluralism offers stronger steerability for value-specific contexts. we present the results for steerable w/ value kaleidoscope in modular pluralism are more faithful to personas of socio-political attributes. we present the results for steerable w/ opinionqa in ## related work aligning llms with human preferences has been an integral part of llms' preliminary success @xcite @xcite @xcite @xcite . early alignment approaches involve training a reward model with human feedback and preferences, then employing an rl algorithm such as ppo @xcite to help learn llms that maximize such rewards @xcite . direct preference optimization (dpo) @xcite was later proposed to directly adapt llms with human preference pairs, without explicitly training or updating a reward model. most recent alignment research features self-alignment @xcite @xcite @xcite , iterative alignment @xcite , as well as self-play approaches @xcite @xcite . in addition to general alignment as a technical problem, an increasing line of work focuses on whose preferences and which values are we aligning with in llm alignment @xcite . while the annotators or reward modeling data might be diverse, the training objective of llm alignment forces llms to minimize the loss and align with an averaged human preference @xcite , while different users could have distinctly different or conflicting preferences informed by culture, demographics, perspectives, and more @xcite . to quantify the concept of pluralism @xcite @xcite , @xcite highlights the importance of pluralistic alignment and sets out three pluralism objectives (overton, steerable, and distributional). to achieve these three objectives, we propose mod-ular pluralism, a modular multi-llm collaboration framework to operationalize and evaluate the three pluralism objectives. we uniquely focus on the setting of patching the pluralism gaps of black-box llms by integrating several smaller lms specialized for community representation, in contrast to previous proposals where white-box llms are required for rlhf tuning @xcite and parameter merging @xcite . ## conclusion we propose modular pluralism, a multi-llm collaboration framework to advance pluralistic alignment. general-purpose llms are augmented with a pool of smaller but specialized community lms, where they interact in distinct modes to achieve various pluralistic alignment objectives. extensive experiments demonstrate that modu-lar pluralism advances pluralistic alignment across numerous models and evaluation datasets. further analysis reveals the benefit of modularity in modular pluralism, that previously underrepresented communities in llms could be seamlessly patched by adding a smaller community lm representative of their culture and values.