N ews R ec L ib: A P y T orch-Lightning Library for Neural News Recommendation (2023)

Abstract: NewsRecLib is an open-source library based on Pytorch-Lightning and Hydra developed for training and evaluating neural news recommendation models. The foremost goals of NewsRecLib are to promote reproducible research and rigorous experimental evaluation by (i) providing a unified and highly configurable framework for exhaustive experimental studies and (ii) enabling a thorough analysis of the performance contribution of different model architecture components and training regimes. NewsRecLib is highly modular, allows specifying experiments in a single configuration file, and includes extensive logging facilities. Moreover, NewsRecLib provides out-of-the-box implementations of several prominent neural models, training methods, standard evaluation benchmarks, and evaluation metrics for news recommendation.

## introduction personalized news recommendation has become ubiquitous for customizing suggestions to users' interests @xcite . in recent years, there has been a surge of effort towards neural content-based recommenders. with increasingly complex neural architectures able to ever more precisely capture users' content-based preferences, neural recommenders quickly replaced traditional recommendation models as the go-to paradigm for news recommendation. despite the abundance of model designs, research on neural news recommenders (nnrs) suffers from two major shortcomings: (i) a surprising amount of non-reproducible research @xcite and (ii) unfair model comparisons @xcite . the former is, on the one hand, due to many nnr implementations not being publicly released @xcite . existing open source repositories, on the other hand, expose a multitude of programming languages, libraries, and implementation differences, hindering reproducibility and extensibility @xcite . moreover, a lack of transparency in terms of evaluation datasets, experimental setup and hyperparameter settings, as well as the adoption of ad-hoc evaluation protocols, further severely impede direct model comparisons. many personalized news recommenders have been evaluated on proprietary datasets (e.g., bing news @xcite , msn news @xcite , news app @xcite ). even the models trained on the more recently introduced open benchmarks (e.g., adressa @xcite , mind @xcite ) cannot be directly compared due to the lack of standard dataset splits and evaluation protocols @xcite @xcite . even more concerning, crucial details regarding the setup of the experiments are regularly omitted from the publications or hard-coded without explanation. it is thus particularly difficult to evaluate the impact of specific components in nnr architecture and training (e.g., news encoder, user modeling, training objectives) on the overall performance of the model @xcite . many models simultaneously change multiple components in both the news and the user encoder, while carrying out only partial ablation studies or evaluating against suboptimal baselines @xcite . in this work, we introduce newsreclib, an open source library for nnrs, to remedy these critical limitations. 2 newsreclib aims to facilitate reproducible research and comprehensive experimental studies, using an end-to-end pipeline powered by a single configuration file that specifies a complete experiment -from dataset selection and preprocessing over model architecture and training to evaluation protocol and metrics. newsreclib is built based on the following guiding principles: modularity and extensibility. with pytorch lightning (falcon and the pytorch lightning team, 2019) as its backbone, newsreclib is designed in a modular fashion, with core individual components being decoupled from one another. this enables mixing and matching different modules, as well as seamlessly integrating new ones. ## newsreclib -the library figure 1 depicts the structure of newsreclib, comprising different functional modules: from data modules for downloading and processing datasets to recommendation modules for training and evaluating a particular nnr. the overall pipeline of an experiment is built automatically from the highlevel experimental flow provided by the user in the form of a single hydra configuration file. ## comparison to related frameworks in the past decade, numerous frameworks for the development and comprehensive evaluation of recommender systems have been proposed to address the problem of reproducibility in the field @xcite @xcite @xcite @xcite @xcite . news recommendation poses different challenges for practitioners in comparison to recommendation in domains such as movies, music, or e-commerce @xcite . however, few of the existing and widely used libraries offer support for news recommenders, and especially for the modern neural news recommendation models. microsoft recommenders @xcite and @xcite provide implementation for five and three nnrs, respectively, as well as utilities for the mind dataset. nonetheless, other datasets, more recent approaches, and in particular fairness-aware models and beyond-accuracy metrics are not supported. streamingrec @xcite is a framework for evaluating streaming-based news recommenders, covering a wide range of algorithms, from trivial baselines (e.g., recently published, most popular) or item-to-item based collaborative filtering or session-based nearest neighbor techniques, to association rule methods and content-based approaches. however, it does not support any of the recent neural models. in these libraries, the sub-modules of a specific recommender are not decoupled from the overall model, which impedes experimentation with and analysis of different model components and training strategies. in contrast to these frameworks, newsreclib focuses solely on the state-of-the-art neural news recommendation models, providing utilities for the most used benchmark datasets, architectures, training techniques, and evaluation metrics tailored to news recommendation. newsreclib unifies disparate implementations of recent neural news recommenders in a single open-source library that is built on top of mature frameworks for deep learning (pytorch lightning), evaluation (torchmetrics), and configuration (hydra). ## experiments we conduct experiments with the pre-implemented recommendation models from newsreclib to investigate their performance when (1) trained with the original architecture (e.g., ne based on word embeddings and contextualization layer) and (2) trained with a plm-based ne @xcite . ## conclusion in this work, we introduced newsreclib, a highly configurable, modular and easily extensible framework for neural news recommendation. our library is specifically designed to foster reproducible research in recommender systems and rigorous evaluation of models -users only need to create a single configuration file for an experiment. we briefly described the underlying principles of newsreclib and the structure of its building blocks. the framework currently provides two standard benchmark datasets, loading and pre-processing functions, 13 neural recommendation models, different training objectives and hyperparameters optimization strategies, numerous evaluation metrics, extensive logging capabilities, and gpu support. we believe that newsreclib is a useful tool for the community that will (i) catalyze reproducible nnr research, (ii) foster fairer comparisons between the models, and (iii) facilitate identification of nnr components that drive their performance.