A dvisor QA : Towards Helpful and Harmless Advice-seeking Question Answering with Collective Intelligence (2025)

Abstract: As the integration of large language models into daily life is on the rise, there is still a lack of dataset for *advising on subjective and personal dilemmas*. To address this gap, we introduce AdvisorQA, which aims to improve LLMs’ capability to offer advice for deeply subjective concerns, utilizing the LifeProTips Reddit forum. This forum features a dynamic interaction where users post advice-seeking questions, receiving an average of 8.9 advice per query, with 164.2 upvotes from hundreds of users, embodying a *collective intelligence*. Therefore, we’ve completed a dataset encompassing daily life questions, diverse corresponding responses, and majority vote ranking, which we use to train a helpfulness metric. In baseline experiments, models aligned with AdvisorQA dataset demonstrated improved helpfulness through our automatic metric, as well as GPT-4 and human evaluations. Additionally, we expanded the independent evaluation axis to include harmlessness. AdvisorQA marks a significant leap in enhancing QA systems to provide subjective, helpful, and harmless advice, showcasing LLMs’ improved understanding of human subjectivity.

## introduction large language models (llms) @xcite have significantly enhanced objective decision-making in various domains, such as science @xcite , and coding @xcite . this was made possible, in part, by numerous benchmarks that assess the objective helpfulness of llms @xcite @xcite . however, llms' impact on subjective decisionmaking-e.g. determining a better way to figure out one's girlfriend's ring size-has been minimal, despite the need @xcite ; chiu figure 1 : the example of test set thread in advisorqa: it consists of an advice-seeking question and the advising answers sorted by their upvote rankings. llm advice is evaluated by the trained helpfulness metric based on its ranking against human-written answers. @xcite . given the unique challenges introduced by the subjectivity, such as the subjectivity of what constitutes better advice and the necessity of a harmlessness metric, there are few qa datasets available to support research on providing advice on subjective problems @xcite @xcite . to this end, we present advisorqa, a dataset of 10,350 questions seeking advice on subjective and personal issues, each paired with a ranked list of 8.9 answers on average, as shown in figure 1 . both the questions and the answers in advisorqa were written by users in a millions-user subreddit lifeprotips foot_0 , and the ranking of answers is also based on their preferences expressed as votes. advisorqa has two main features that differ from existing objective qa, first, it is highly complex; the questions typically contain a detailed narrative on personal issues to solicit advice with dy-namic language uses. they are not only long-75.2 words on average-but also cover a wide range of issues-daily topics from social conversation to travel tips as shown in figures 3 and 10 . also, due to the subjective and complex nature of the questions, multiple answers, each providing a unique perspective, can all be helpful. this is distinct from existing qa datasets consisting of objective questions, each with a single correct answer. second, helpfulness in subjective topics is determined not only by objective criteria, such as correctness, but rather by personal preferences. to avoid having helpfulness rankings of answers biased to the few annotator's opinions @xcite , we collected the majority preferences from million-scale active users included in the community upvote system. as a result, the answers for each question are ranked by an average of 164.2 votes per thread, which is a form of collective intelligence. we verified that the model trained on the upvote rank improved on gpt-4 and human evaluation, suggesting that using upvotes as a proxy for helpfulness is effective. to account for the sensitive real-world issues in advice-seeking qa, we adopt appropriate metrics along two independent dimensions: helpfulness and harmlessness. for helpfulness, we designed a helpfulness metric based on the plackett-luce (pl) model @xcite , which is used for ranking predictions to incorporate preferences among diverse candidate advice. for harmlessness, we employ the lifetox moderator @xcite , a model to compute harmlessness scores. since it was also trained on the data from the lifeprotips subreddit, it aligns well with our dataset. we experimented with llms to measure their ability to provide subjective advice before and after supervised fine-tuning (sft) and reinforcement learning with human feedback (rlhf). without sft, llama @xcite and @xcite were the most harmless, but the gpt models (openai, 2023) were the most helpful. experiments on the two most harmless models show that sft boosts helpfulness, but reduces harmlessness. the trend is amplified with rlhf using ppo @xcite , but most of the decline in harmlessness can be recovered with dpo @xcite . further analysis reveals that dpo's safe results stemmed from its tendency to follow demonstrations and produce strictly written advice. in contrast, ppo generates more empathic and diverse advice, but can be un-safe depending on reward models. this analysis concludes that each existing rlhf has limitations regarding advice-seeking qa. the main contributions of this paper are summarized as twofold; 1. we present advisorqa, the dataset for subjective and personal questions with appropriate evaluation metrics along the dimensions of helpfulness and harmlessness. ## evaluation metrics in this section, we discuss how to evaluate the advice generated by language models in the advi-sorqa benchmark. given the task's pronounced subjectivity, we measure helpfulness not by similarity to references but through comparative ranking. moreover, as an auxiliary measure, we evaluate the safety of the advice by evaluating its harmlessness. ## experiments this section outlines the baselines for advisorqa. four advices accompany each question in the test set. the helpfulness of the advice generated by llms is determined by its ranking among a total of five pieces of advice. the safety of the llms is assessed based on the harmlessness score assigned to each piece of advice. these two criteria are used to analyze the performance of baseline models and training approaches. ## conclusion we introduce advisorqa, a benchmark for adviceseeking question answering that focuses on questions rooted in personal experiences and the corresponding advice, ranked by collective intelligence. advisorqa serves as a valuable resource for advancing everyday qa systems that provide in-depth, empathetic, and practical advice towards daily dynamic dilemmas. by leveraging upvote ranks to evaluate various subjective opinions and through baseline experiments, we have confirmed the dataset's validity and shed light on the impact and limitations of rlhf trainers in subjective domains. further, we analyze and highlight critical remaining issues to handle subjectivity that future research should consider. these analyses suggest a broad potential to facilitate research in evaluating and training systems for daily neural advisors.