ALERT : An LLM -powered Benchmark for Automatic Evaluation of Recommendation Explanations (2025)

Abstract: Recommendation explanation systems have become increasingly vital with the widespread adoption of recommender systems. However, existing recommendation explanation evaluation benchmarks suffer from limited item diversity, impractical user profiling requirements, and unreliable and unscalable evaluation protocols. We present ALERT, a model-agnostic recommendation explanation evaluation benchmark. The benchmark comprises three main contributions: 1) a diverse dataset encompassing 15 Amazon e-commerce categories with 2,761 user-item interactions, incorporating implicit preferences through purchase histories;2) two novel LLM-powered automatic evaluators that enable scalable and human-preference aligned evaluation of explanations; and 3) a robust divide-and-aggregate approach that synthesizes multiple LLM judgments, achieving 70% concordance with expert human evaluation and substantially outperforming existing methods.ALERT facilitates comprehensive evaluation of recommendation explanations across diverse domains, advancing the development of more effective explanation systems.

## introduction recommender systems have become essential tools for navigating user preferences and mitigating information overload in the current era of information explosion @xcite @xcite . as these systems evolve, there is a growing demand for convincing and useful explanations that provide insights into the recommendation process, fostering user trust and enhancing overall satisfaction @xcite @xcite . despite the critical role of explanations, their effective evaluation remains a significant challenge, particularly in representing real-world scenarios and developing automated evaluation methods that accurately reflect user preferences. current benchmark datasets @xcite @xcite suffer from two major limitations: lack of diverse categories and user preference data. existing benchmarks @xcite ) mainly focus on a single category, such as movies or books, failing to capture the complexity and variety of real-world recommendation scenarios. the explanatory features vary significantly across product domains; for instance, movie recommendations typically emphasize artistic elements like genres and actors, whereas beauty product recommendations focus on functional attributes such as moisturizing or exfoliating properties. this domain-specific variation presents a significant challenge in evaluating the cross-domain generalization capabilities of explanation models. furthermore, while user preferences are fundamental to modeling and evaluating recommendation explanations, as they inform the underlying rationale for user-item interactions, current user profile collection methodologies are insufficient. several benchmarks @xcite rely on questionnaire-based approaches to collect user profiles and demographic attributes. however, this methodology presents inherent limitations in realworld applications, often yielding incomplete preference profiles and failing to capture the dynamic nature of user preferences. ## task description of recommendation explanations the primary goal of this benchmark is to inspire the development of new methods for generating post-hoc, free-style recommendation explanations that motivate users to purchase recommended items. following prior literature @xcite , we define the task as: "generating a recommendation explanation by considering the user's profile and black-box recommended items, aimed at increasing user satisfaction and purchase inclination" as shown in fig. 1 (a ). formally, for a user u, let i denote the recommended item, with x i representing the item's context (e.g., title, price); let x u represent the user's purchase history which is ## benchmark description our benchmark, alert, consists of two main components: curated benchmark datasets and llmpowered automatic evaluators, as illustrated in fig. 1 (b) that work together to provide a comprehensive evaluation framework for recommendation explanation models. these components are designed to provide evaluation environment that simulate authentic recommendation scenarios while facilitating scalable evaluation that better capture human preferences and decision-making patterns. ## meta-evaluation of alert-gen and alert-disc to rigorously assess the efficacy of our proposed automatic evaluation methods, alert-gen and alert-disc, we conduct a comprehensive metaevaluation framework that examines three critical dimensions: their reliability in detecting suboptimal explanations, their correlation with expert human judgments, and their computational efficiency in terms of time and cost savings. this systematic evaluation approach enables us to validate our methods through both controlled experimental conditions and real-world human preference data, while demonstrating the practical advantages of automated evaluation over traditional manual evaluation procedures. ## leaderboard results this section presents the benchmark performance of diverse recommendation explanation models on the full alert dataset. ## conclusion this paper introduces alert, a novel recommendation model-agnostic explanation benchmark that addresses key limitations in existing evaluation methods. alert contains a diverse dataset from 15 amazon e-commerce categories and introduces two innovative llm-powered automatic evaluation methods for flexible evaluation of free-form explanations without ground-truth references. our metaevaluation experiments demonstrate that alert's automated evaluation approaches: alert-gen and alert-disc achieves high correlation with human judgments, outperforming existing methods. by providing an efficient, scalable, and reliable approach, alert aims to accelerate the development of more transparent and user-friendly recommendation explanation models.