H ello F resh: LLM Evalutions on Streams of Real-World Human Editorial Actions across X Community Notes and W ikipedia edits (2024)

Abstract: Benchmarks have been essential for driving progress in machine learning. A better understanding of LLM capabilities on real world tasks is vital for safe development.Designing adequate LLM benchmarks is challenging: Data from real-world tasks is hard to collect, public availability of static evaluation data results in test data contamination and benchmark overfitting, and periodically generating new evaluation data is tedious and may result in temporally inconsistent results. We introduce HelloFresh, based on continuous streams of real-world data generated by intrinsically motivated human labelers. It covers recent events from X (formerly Twitter) community notes and edits of Wikipedia pages, mitigating the risk of test data contamination and benchmark overfitting.Any X user can propose an X note to add additional context to a misleading post (formerly tweet); if the community classifies it as helpful, it is shown with the post. Similarly, Wikipedia relies on community-based consensus, allowing users to edit articles or revert edits made by other users.Verifying whether an X note is helpful or whether a Wikipedia edit should be accepted are hard tasks that require grounding by querying the web.We backtest state-of-the-art LLMs supplemented with simple web search access and find that HelloFresh yields a temporally consistent ranking.To enable continuous evaluation on Hellofresh, we host a public leaderboard and periodically updated evaluation data at https://tinyurl.com/hello-fresh-LLM.

## introduction with the recent jump in performance and availability of large language models (llms), a better understanding of their abilities is crucial for safe development and deployment. of particular interest are the capabilities of llms to impact the real world in the near future @xcite @xcite . llm benchmarks also support transparency and trust, by showcasing their capabilities and limitations. to correlate with performance in real-world and broadly useful tasks, an ideal benchmark should consist of real-world tasks that humans regularly perform, as opposed to self-contained questionanswering tasks @xcite @xcite @xcite , artificial game environments @xcite , or web-based information retrieval tasks @xcite . it should also include authentic data from real users who are intrinsically motivated to perform a given task -instead of relying on annotators with artificial (monetary) task incentives @xcite , which may, e.g., incentivize quantity over quality. a good benchmark must also be dynamicmeaning that evaluation data must be periodically updated-for the following three reasons. first, static evaluation data that is available on the internet easily leads to unintentional but unpreventable test data contamination @xcite , where evaluation data leaks into the training data. this happens either directly by scraping corresponding websites or indirectly, e.g., by scraping information about tasks in a different language @xcite , or by scraping websites with paraphrasings of given tasks @xcite . second, static benchmarks only contain information about past events. however, for llms to be effective in the real-world they have to generalise to unseen future situations @xcite . third, static benchmarks lead to overfitting over time @xcite . additionally, to facilitate comparisons over time, an ideal dynamic benchmark should be temporally consistent. we introduce the hellofresh benchmark, which uses publicly available data from x (formerly twitfigure 2 : examples of two wikipedia edits. the left edit is classified as 'accepted' by our algorithm, while the right edit is classified as 'rejected' as it was later reverted. in both cases, the article remained unchanged for a certain number of edits after the edit or reversion of the edit, respectively. figure 3 shows a scheme of the algorithm used to filter for 'accepted' and 'rejected' edits. we first describe the dataset curation process in detail and propose two evaluation regimes for llms: (1) a zero-shot classifier, and (2) a websearch agent that initially generates a web search query and then uses the retrieved information itself depends on the time the user has been a wikipedia editor and the number of edits made, among other factors. heavily disputed pages can only be edited by more senior users. see https://en.wikipedia.org/wiki/help:editing for more information. as additional context for the classification problem. all x notes and wikipedia edits we consider were made after the training data cut-off date for all llms we evaluate. since we observe that the performance is sensitive to the exact word choice of classification prompts, we use four llmgenerated rephrasings in addition to the manual prompt written by us. we evaluate gpt4 (openai, 2023), gpt3.5, llama2-70b @xcite , @xcite , and gemini pro (deepmind, 2024) in both the zero-shot and web-search regime. through backtesting, we empirically confirm the temporal consistency of both benchmarks. it is worth noting that in some cases, the zero-shot classifier outperforms the web search agent. a follow-up evaluation suggests that this is due to some models being 'confused' by the large quantity of provided information resulting from the web search. we simulate a deployment scenario by testing different llm prompts on data of the first two months (weeks) and then evaluating the best-performing prompt over the next two months (weeks). we find that the best models achieve 75% and 90% precision on x notes and wikipedia edits, respectively, at a recall of more than 20%. lastly, we observe that all llms have a 30% lower f1 score for x notes that received only between 5 and 36 votes (representing the first quartile). this suggests that the community notes algorithm has a high variance for classifications of notes that received few votes. ## related work in recent years, a broad suite of benchmarks have been developed to evaluate llm capabilities. these span topics such as mathematics (math @xcite , gsm8k @xcite ), coding (humaneval @xcite ), commonsense reasoning (hel-laswag @xcite ) or broader aggregates of language (mmlu @xcite , big-bench @xcite ) and multimodal (mmmu @xcite ) assessments. while these benchmarks have been instrumental in measuring the performance of llms across var-ious domains, they often fail to capture the evolving nature of real-world information, as well as aspects of real-world tasks performed by humans. dynamic benchmarks, such as dynabench @xcite , aim to address this by providing a platform for continuous evaluation and dataset creation. unlike dynabench, hellofresh leverages naturally occurring data streams from social and collaborative platforms, ensuring a direct connection to real-world applicability and inclusion of recent events deemed relevant by the public. the reliance on intrinsically motivated human annotators distinguishes hellofresh from benchmarks that use paid annotators. @xcite and @xcite have explored the dynamics and motivations of crowd work, suggesting that intrinsic motivation can lead to higher-quality contributions. by utilizing data from community members of x notes and wikipedia, hellofresh benefits from annotations that are likely more thoughtful and less prone to the biases introduced by monetary incentives. verifying x notes's helpfulness and the correctness of wikipedia edits inherently involves information retrieval and fact-checking against current, real-world information. prior works @xcite @xcite @xcite have explored the integration of web search capabilities with llms, to enhance their ability to utilize up-to-date information. these approaches align hellofresh, since we evaluate llms not only on their linguistic capabilities but also on their ability to interact with the web to inform responses. the challenge of temporal consistency in benchmarks-ensuring that the distribution of evaluation data remains stable over time while incorporating new information-has been less explored. research on domain adaptation and temporal generalization @xcite , highlight the difficulty of temporal generalization. in summary, hellofresh builds on the foundation of existing work in dynamic benchmarks, intrinsic motivation, web-based information retrieval, and temporal consistency. by leveraging real-world, intrinsically motivated data streams from x notes and wikipedia, it proposes a novel and challenging benchmark for evaluating the real-world applicability and temporal robustness of llms. 12704 3 background 3.0.1 x community notes x community notes (see figure 1 ) is a feature that allows users to contribute additional context to posts they believe are misleading or require more information. when a user comes across a post that they think needs clarification, they can write a note providing additional context, which other users review and rate for helpfulness. if a note is deemed helpful by a diverse and sufficiently large group of users, it is publicly displayed alongside the original post. the core algorithm behind this mechanism is introduced in @xcite , while the most recent implementation can be found at https: //github.com/twitter/communitynotes . this system aims to crowdsource the fact-checking process, leveraging the collective knowledge of x users to combat misinformation. in january 2024, a total of 100007 notes were proposed, of which 7688 were classified as helpful and 3600 as nonhelpful, with the remaining 95712 notes remaining unclassified, e.g., due to a lack of votes. ## implementation of evaluation regimes we implement two regimes based on llms, which we evaluate on both binary classification tasks. the first regime evaluates zero-shot classification accuracy using a single prompt to the language model containing all relevant information. the second regime, which we refer to as the web-search agent, first prompts the language model for a websearch-query, which is then executed using google search; the retrieved websites are first summarised and then appended to the classification prompt (see appendix d for all prompts used). note that this results in different search results for every web-search agent and task, depending on the websearch-query that the llm chose. ## evaluation and results evaluation details. we ran evaluations for both the zero-shot classifier and web-search agent regimes for all models (gpt4, gpt3.5, llama70b, mistral8x7 and gemini pro). please refer to appendix b for the exact models and retrieval dates. the x notes dataset consisted of a total of ∼3000 samples, which we split up by months (october 2023 to january 2024) to evaluate temporal consistency. the wikipedia edits dataset consisted of a total of ∼4000 samples, which we split up by weeks (4 splits -last two weeks of february 2024 and first two weeks of march 2024). we only ran gpt4 on 50% of the data to reduce cost. we evaluated all five classification prompts (manual, rephrases 1 to 4 by models gpt4, gpt3.5, mis-tral8x7, and llama70b) for all llms. we only considered the manually written web-search query and summarization prompts for the web-search agent; otherwise, each sample would have had to be run for 125 different combinations of prompts. we found that blocked responses only very rarely made up more than 10% of a model's responses for a given task. for all results, we compute two-sided we choose the prompt with the highest precision in the first two time periods and evaluate it in the subsequent two. for example, the prompt with the highest precision in october and november (tuning phase) is evaluated for december and january (eval phase). assuming that no llm with a tuning-phase recall of less than 20% is considered for deployment, we observe that gpt4 and gpt3.5 have the highest tuning-phase precision for x notes and wikipedia edits, respectively. the resulting eval-phase precision and recall are indicated by the solid markers (connected to the tuning-phase results by a dashed line).we plot all precision and recall results in figure 10 in the appendix. hellofresh is temporally consistent. figure 4 shows the f1 score of all models over contiguous time periods in the zero-shot classification regime. the results are sensitive to the exact wording of the classification prompt, which we discuss in the next section. to account for the sensitivity, we computed the f1 score by ensembling the classifications of the five evaluated prompts using a majority vote mechanism. while the performance of individual llms varies by task and regime, we find that the ranking of llms is largely consistent for individual tasks and methods. web-search agent often with reduced performance. figure 5 displays the difference in f1 score between the zero-shot and the web-search agent and for the different types of classification prompts. we first observe that results are sensitive to the type of classification prompt used (x-axis), which is either the manually written prompt or a rephrasing of it. this finding aligns with recent research showcasing the importance of prompt tuning. we further find that the f1 score is mostly higher for the web-search agent in the x notes domain (positive difference in f1 score in figure 5 left). in contrast, in the wikipedia domain, the websearch agent achieves both significantly higher and significantly lower f1 scores, depending on the model and classification prompt. specifically, us-ing the above-described voting aggregation, the f1 score of the gpt3.5 web-search agent is 7.3% lower than that of the zero-shot classifier, while that of mistral8x7 is 8.9% lower. a lower f1 score of the web-search agents gpt3.5 and mistral8x7 is unintuitive, as the web-search agent is tasked with the exact same problem as the zero-shot classifier but uses strictly more information, namely the summaries of relevant websites appended to the classification prompt. to better understand this phenomenon, we evaluate the difference in f1 score between the zeroshot classifier and a modified web-search agent where we replace the summaries of relevant web pages with short texts about unrelated topics. compared to the zero-shot classifier, gpt3.5's and mis-tral8x7's performance changes by +0.8% and -3.1% on x notes and by -22.2% and -4.3% on wikipedia edits. all results are shown in figure 9 in the appendix. the decrease in models' performance when the classification task is supplemented with unrelated information suggests that the models cannot handle the quantity of information provided, or in other words, they are 'confused' by the additional information. we suspect that models are similarly confused by the provided summaries of web pages, resulting in the decreased performance of web-search agents for some models. ## conclusion hellofresh is a new living benchmark for evaluating llms. x community notes and wikipedia edits provide a constant source of fresh data that may help alleviate the problems of static evaluations. we demonstrated that, despite the time-evolving nature of the data, the relative ordering of evaluated methods is relatively stable, which is important for consistent evaluation. we hope that this benchmark will serve as both a more dynamic evaluation of llms and as a testbed for equipping llms with better grounding in external sources. future work. in future work, we aim to address four directions. first, we want to extend hellofresh to multiple modalities, including images and potentially videos. such multi-modal fact-checking is already a vital part of x community notes, as over half of the proposed notes on x concern posts containing media. second, we want to extend the benchmark to account for justifications of classification decisions. third, we want to evaluate the capabilities of llms to not only verify proposed notes or edits but to propose x notes or edits to wikipedia pages directly. fourth, we are interested in better understanding the distribution of voters and editors. this can be enabled by investigating correlations between votes and edits and user markers given in the data, such as location or previous platform interactions. lastly, we'd like to evaluate the robustness of the mechanisms underlying notes and wikipedia edits to adversarial collusion.