Efficient Pointwise-Pairwise Learning-to-Rank for News Recommendation (2024)

Abstract: News recommendation is a challenging task that involves personalization based on the interaction history and preferences of each user. Recent works have leveraged the power of pretrained language models (PLMs) to directly rank news items by using inference approaches that predominately fall into three categories: pointwise, pairwise, and listwise learning-to-rank. While pointwise methods offer linear inference complexity, they fail to capture crucial comparative information between items that is more effective for ranking tasks. Conversely, pairwise and listwise approaches excel at incorporating these comparisons but suffer from practical limitations: pairwise approaches are either computationally expensive or lack theoretical guarantees and listwise methods often perform poorly in practice. In this paper, we propose a novel framework for PLM-based news recommendation that integrates both pointwise relevance prediction and pairwise comparisons in a scalable manner. We present a rigorous theoretical analysis of our framework, establishing conditions under which our approach guarantees improved performance. Extensive experiments show that our approach outperforms the state-of-the-art methods on the MIND and Adressa news recommendation datasets.

## introduction online news services have become important platforms for a large population of users to stay informed. a massive number of news articles are generated and posted online every day, making it all the more important to personalize news recommendation for the users. the text-rich nature of news articles makes them particularly well-suited for the application of pre-trained language models ⋆ equal contribution. one way to approach news recommendation using plms is through pointwise ranking. this method predicts a relevance score for each candidate based on the user's previously clicked items. while this approach is scalable, the candidate scores are obtained independently without comparing the relative usefulness of a candidate with regards to its competitors. naturally, the task of recommendation requires comparing candidates with each other. this intuition is supported by empirical results that demonstrate pairwise/listwise approaches outperform pointwise ones @xcite . theoretically, listwise approaches are expected to perform the best as ranking is a list-level task @xcite . however, as shown by @xcite , listwise recommendation performs poorly when paired with plms for two main reasons: 1) plms often generate conflicting or useless outputs when provided large quantities of information, and 2) plms have a fixed maximum input prompt length that may be exceeded while attempting to encode all available candidate items. on the other hand, pairwise approaches leverage the comparison between items without the downsides of listwise approaches described above. however, they come with higher computational complexity compared to pointwise approaches. there is a body of relevant work on pairwise approaches for text-based recommendation tasks: @xcite propose pairwise prompting for ranking with plms by performing bubble-sort at inference time. @xcite propose a two-stage approach that first ranks using pointwise scores and subsequently modifies the rankings by computing aggregation scores from pairwise comparisons. however, both methods are computation-ally expensive and scale poorly in practice due to the requirement for o(n 2 ) comparisons. to optimize pairwise approaches, @xcite propose to sparsify the number of pairwise comparisons by random and skip-window sampling of pairs. while this improves scalability, it only compares several random aggregation strategies without theoretical guarantees. in this work we propose a theoreticallyguaranteed method that efficiently performs pairwise ranking. we summarize our main contributions as follows: in this way, we perform only n -1 pairwise comparisons per rtl pass. we emphasize that although our focus is on news recommendation, our proposed algorithm is directly applicable to any text-based task. ## the proposed algorithm: glimpse the task of news recommendation in this paper refers to ranking a set of candidate news items @xmath0, x 2 , x 3 , . . . , x k } given the user's click history. the goal is to rank an item higher than others if it is preferred by the user. this goal implies both a pointwise relevance prediction task and a pairwise preference task: a user click on an item should be treated not only as an absolute judgment for relevancy, but also as a preference judgement. motivated by this, we propose the glimpse (generative tuple-wise prompting for news recommendation) algorithm for optimizing both objectives simultaneously. glimpse is a general method for recommendation problems and works with any generative model. we first describe the multi-task fine-tuning method of a generative model that aligns both the relevance prediction objective and the preference prediction objective into a single text generation task. by performing multi-task fine-tuning, we obtain a single model that can be used as a predictor for both objectives at the same time. to combine the tasks into a complete ranking method, we subsequently propose a novel inference strategy by initialising with pointwise relevance ranking and using unidirectional (right-to-left) adjacent element swaps with pairwise preference comparisons to obtain a final ranking (illustration in figure 1 ). ## theoretical analysis in this section, we provide the theoretical analysis for the proposed rtl rank aggregation strategy. let κ be a permutation of the indices {1, . . . , k} of candidate items {x 1 , . . . , x k }. we denote rank(x i |κ) as the rank of item x i in the permutation κ and write r * (x i |h) ∈ {0, 1} for the ground-truth relevance of item x i with history h. we can then consider additive ranking metrics of the form: for a stochastic algorithm a we are interested in the expectation of the ranking metric over the possible permutations κ, which we can write as because the relevance r * (•) is a binary value (e.g., click or no click), distinct permutations κ can give rise to the same value of the ranking metric. let @xmath1, 1} k denote the binary vector of user relevance of items in ranking κ, such that @xmath2: @xmath3. in this setting, the ranking metric becomes ∆(z | h) = k @xmath4, with z k the k-th element of z. by extension, we can write the expectation over the metric in terms of z as well, e a (∆) = z p(z)∆(z | h), where the summation is over all possible values of z. we can refine the expression for the expected ranking metric by considering our proposed twostage inference approach consisting of an initial ranking of the candidates based on relevance prediction, and a subsequent refinement of that ranking through pairwise preferences. let π ∈ r 2 k denote the probability vector with elements p(z) for all @xmath5, 1} k that reflects the probability of obtaining z from pointwise inference. then, the ranking refinement from pairwise comparisons can be modeled as a markov chain with transition matrix @xmath6, such that the product π ′ = πt is the probability vector of "states" after the transition induced by pairwise reranking. specifically, the elements t i,j of the transition matrix t represent the probability of transitioning from a state z i to a state z j . with a mild abuse of notation we can write t z ′ for the column of t corresponding to an outcome state z ′ . then, the expected value of the ranking metric after our two-stage inference process becomes repeated application of the pairwise refinement (i.e., multiple rtl passes) can be captured through typical markov chain notation, we next analyze the transition matrix to understand the conditions under which the rtl passes are beneficial. ## experiments in this section we conduct experiments to empirically validate our approach. we first describe a comparison with existing works. next, we describe an experiment on various inference strategies, followed by an ablation study to better understand the effect of each component of our approach. additional details about the experimental setup can be found in appendix a.2. ## related work in this section, we focus primarily on news recommendation approaches that leverage llms, as well as the pairwise ranking literature. ## conclusion we proposed glimpse, an algorithm for personalized recommendation that leverages plms alongside a novel inference strategy to efficiently combine pointwise relevance prediction and pairwise comparison. we presented a multi-task finetuning approach to facilitate this inference and conducted extensive experiments on real-world datasets, demonstrating that glimpse outperforms state-of-the-art methods on the mind and adressa datasets. moreover, we provided a rigorous theoretical analysis of our proposed approach and derived conditions under which the rtl reranking pass is beneficial. our work underscores the potential of leveraging language models for news recommendation, emphasizing both performance and real-world efficiency.