Original: Learning to Take a Break: Sustainable Optimization of Long-Term User Engagement
Revised: Learning to Take a Break: Sustainable Optimization of Long-Term User Engagement

Original: Anonymous Author(s)

Original: AffiliationAddress

Original: Abstract
Revised: Abstract

Original: Optimizing user engagement is a key goal for modern recommendation systems,but blindly pushing users towards increased consumption risks burn-out, churn, oreven addictive habits.
Revised: Optimizing user engagement is a key goal for modern recommendation systems, but blindly pushing users towards increased consumption risks burn-out, churn, or even addictive habits.

Original: To promote digital well-being, most platforms now offer aservice that periodically prompts users to take a break.
Revised: To promote digital well-being, most platforms now offer a service that periodically prompts users to take a break.

Original: These, however, must beset up manually, and so may be suboptimal for both users and the system.
Revised: These, however, must be set up manually, and so may be suboptimal for both users and the system.

Original: In thispaper, we propose a framework for optimizing long-term engagement by learningindividualized breaking policies.
Revised: In this paper, we propose a framework for optimizing long-term engagement by learning individualized breaking policies.

Original: Using Lotka-Volterra dynamics, we model usersas acting based on two balancing latent states: drive , and interest —which mustbe conserved.
Revised: Using Lotka-Volterra dynamics, we model users as acting based on two balancing latent states: drive , and interest —which must be conserved.

Original: We then give an efficient learning algorithm, provide theoreticalguarantees, and empirically evaluate its performance on semi-synthetic data
Revised: We then give an efficient learning algorithm, provide theoretical guarantees, and empirically evaluate its performance on semi-synthetic data.

Original:  Introduction
Revised:  Introduction

Original: As consumers of content, we have come to rely extensively on algorithmic recommendations.
Revised: As consumers of content, we have come to rely extensively on algorithmic recommendations.

Original: Thishas made the task of recommending—in a relevant, timely, and personalized manner—key to thesuccess of modern media platforms.
Revised: This has made the task of recommending—in a relevant, timely, and personalized manner—key to the success of modern media platforms.

Original: Most commercial systems are built with the primary aim ofoptimizing user engagement , a process in which machine learning plays a central role.
Revised: Most commercial systems are built with the primary aim of optimizing user engagement , a process in which machine learning plays a central role.

Original: But alongsidetheir many successes, recommendation systems have also been scrutinized for heedlessly drivingusers towards excessive and often undesired levels of consumption.
Revised: But alongside their many successes, recommendation systems have also been scrutinized for heedlessly driving users towards excessive and often undesired levels of consumption.

Original: This has raised awareness as tothe need for redesigning recommendation systems in ways that actively promote digital well-being.
Revised: This has raised awareness as to the need for redesigning recommendation systems in ways that actively promote digital well-being.

Original: How can media platforms balance business goals with user well-being?
Revised: How can media platforms balance business goals with user well-being?

Original: One prominent approach,which is now offered by most major platforms, is to periodically prompt users to take breaks(Constine, 2018; Perez, 2018).
Revised: One prominent approach, which is now offered by most major platforms is to periodically prompt users to take breaks (Constine, 2018; Perez, 2018).

Original: The idea behind breaks is that occasional disruptions curb the inertialurge for perpetual consumption, and can therefore aid in reducing ‘mindless scrolling’ (Rauch,2018), or even addiction (Montag et al., 2018; Ding et al., 2016).
Revised: The idea behind breaks is that occasional disruptions curb the inertial urge for perpetual consumption and can therefore aid in reducing ‘mindless scrolling’ (Rauch, 2018), or even addiction (Montag et al., 2018; Ding et al., 2016).

Original: As a general means for promotingwell-being, breaking is psychologically well-grounded (e.g., Danziger et al., 2011; Sievertsenet al., 2016).
Revised: As a general means for promoting well-being, breaking is psychologically well-grounded (e.g., Danziger et al., 2011; Sievertsen et al., 2016).

Original: But for platforms, breaks serve a utilitarian purpose: their goal is to foster long-termengagement by compensating for the myopic nature of conventional recommendation algorithms,which are typically trained to optimize immediate engagement .
Revised: But for platforms, breaks serve a utilitarian purpose: their goal is to foster long-term engagement by compensating for the myopic nature of conventional recommendation algorithms, which are typically trained to optimize immediate engagement .

Original: Since breaking schedules are appliedheuristically on top of existing recommendation policies—and typically need to be set up manually byusers—current solutions unlikely utilize their full potential (Monge Roffarello and De Russis, 2019).
Revised: Since breaking schedules are applied heuristically on top of existing recommendation policies—and typically need to be set up manually by users—current solutions unlikely utilize their full potential (Monge Roffarello and De Russis, 2019).

Original: In this paper, we propose a disciplined learning framework for responsible and sustainable optimization of long-term user engagement by controlling breaks.
Revised: In this paper, we propose a disciplined learning framework for responsible and sustainable optimization of long-term user engagement by controlling breaks.

Original: Our point of departure is that sustainedengagement necessitates sustained user well-being , and here we advocate for breaks as a meansto establish both.
Revised: Our point of departure is that sustained engagement necessitates sustained user well-being , and here we advocate for breaks as a means to establish both.

Original: Focusing on feed-based recommendation, our framework optimizes long-termengagement by learning an optimal breaking policy that prescribes individualized breaking schedules.
Revised: Focusing on feed-based recommendation, our framework optimizes long-term engagement by learning an optimal breaking policy that prescribes individualized breaking schedules.

Original: The challenge in learning to break is that the effects of recommendations on users can slowlyaccumulate over time, deeming as ineffectual policies that rely on clear signs of over-exposure.
Revised: The challenge in learning to break is that the effects of recommendations on users can slowly accumulate over time, deeming as ineffectual policies that rely on clear signs of over-exposure.

Original: To bepreemtive, we argue that breaks must be scheduled in a way that anticipates the future trajectory ofuser behavior, and early on.
Revised: To be preemptive, we argue that breaks must be scheduled in a way that anticipates the future trajectory of user behavior, and early on.

Original: To achieve this, we introduce a novel class of behavioral models basedon Lokta-Volterra (LV) dynamical systems (Lotka, 1910).
Revised: To achieve this, we introduce a novel class of behavioral models based on Lokta-Volterra (LV) dynamical systems (Lotka, 1910).

Original: These depict users as acting based on twobalancing forces: drive to consume and intrinsic interest , with corresponding latent states.
Revised: These depict users as acting based on two balancing forces: drive to consume and intrinsic interest , with corresponding latent states.

Original: Intuitively,high interest increases drive to consume, but prolonged consumption decreases interest; together,these describe how user behavior varies over time and in response to recommendations.
Revised: Intuitively, high interest increases drive to consume, but prolonged consumption decreases interest; together, these describe how user behavior varies over time and in response to recommendations.

Original: Our modelcaptures the notion that interest can exhaust long before over-consumption is observed.
Revised: Our model captures the notion that interest can exhaust long before over-consumption is observed.

Original: This armsour approach with the persceince needed to prevent burn-out by ensuring that interest is sustainablypreserved; thus, whereas current solutions target the symptom—ours aims for the cause.
Revised: This arms our approach with the prescience needed to prevent burn-out by ensuring that interest is sustainably preserved; thus, whereas current solutions target the symptom—ours aims for the cause.

Original: Our proposed learning algorithm consists of two steps: First, we embed user interaction sequencesin ‘LV-space’—the set of all possible trajectories that our behavioral model class can express.
Revised: Our proposed learning algorithm consists of two steps: First, we embed user interaction sequences in ‘LV-space’—the set of all possible trajectories that our behavioral model class can express.

Original: Then, we optimize individualized breaking policies by solving an optimal control problem overthis latent space, in which the control variable is a breaking schedule applied on top of the existingrecommendation scheme.
Revised: Then, we optimize individualized breaking policies by solving an optimal control problem over this latent space, in which the control variable is a breaking schedule applied on top of the existing recommendation scheme.

Original: Here the challenge is that different breaking policies can lead to differentcounterfactual trajectories, of which observational data is only partially informative.
Revised: Here the challenge is that different breaking policies can lead to different counterfactual trajectories, of which observational data is only partially informative.

Original: Since our goalconsiders long-term outcomes, our solution is to optimize directly for counterfactual steady-states .
Revised: Since our goal considers long-term outcomes, our solution is to optimize directly for counterfactual steady-states .

Original: From a behavioral perspective, we view this as aiming to steer towards sustainable habits; from acomputational perspective, under our choice of policy class, this enables tractable learning.
Revised: From a behavioral perspective, we view this as aiming to steer towards sustainable habits; from a computational perspective, under our choice of policy class, this enables tractable learning.

Original: As we show, the optimization landscape of LV equilibria admits a compact representation, whosemain benefit is that it can be fully described by predictions of individualized user engagement rates.
Revised: As we show, the optimization landscape of LV equilibria admits a compact representation, whose main benefit is that it can be fully described by predictions of individualized user engagement rates.

Original: Practically, this is advantageous, as it circumvents the need to take arbitrary and costly explorationsteps, and enables learning using readily available predictive tools (e.g., Gupta et al., 2006).
Revised: Practically, this is advantageous, as it circumvents the need to take arbitrary and costly exploration steps, and enables learning using readily available predictive tools (e.g., Gupta et al., 2006).

Original: Wemake use of a small set of learned predictive models, each trained on a small and minimally-invasiveexperimental dataset, which allow us to tune our policy to suit different conditions.
Revised: We make use of a small set of learned predictive models, each trained on a small and minimally-invasive experimental dataset, which allow us to tune our policy to suit different conditions.

Original: The final learnedpolicy has an intuitive interpretation: it takes as input a small set of predictions for a user, and viacareful interpolation, applies a decision rule that anticipates the effects of breaking on future outcomes(c.f. conventional approaches, which take in predictions and apply the myopic argmax rule).
Revised: The final learned policy has an intuitive interpretation: it takes as input a small set of predictions for a user, and via careful interpolation, applies a decision rule that anticipates the effects of breaking on future outcomes (c.f. conventional approaches, which take in predictions and apply the myopic argmax rule).

Original: Our main theoretical result is a bound on the expected long-term engagement of our learned breakingpolicy, relative to the optimal policy in the class.
Revised: Our main theoretical result is a bound on the expected long-term engagement of our learned breaking policy, relative to the optimal policy in the class.

Original: We show that the gap decomposes into three distinctadditive terms: (i) predictive error, (ii) modeling error (i.e., embedding distortion), and (iii) variancearound the (theoretical) steady state.
Revised: We show that the gap decomposes into three distinct additive terms: (i) predictive error, (ii) modeling error (i.e., embedding distortion), and (iii) variance around the (theoretical) steady state.

Original: These provide an intuitive interpretation of the bound, as well asmeans to understand the effects of different modeling choices.
Revised: These provide an intuitive interpretation of the bound, as well as means to understand the effects of different modeling choices.

Original: Our proof technique relies on carefullyweaving LV equilibrium analysis within conventional concentration bounds for learning.
Revised: Our proof technique relies on carefully weaving LV equilibrium analysis within conventional concentration bounds for learning.

Original: Finally, we provide an empirical evaluation of our approach on semi-synthetic data.
Revised: Finally, we provide an empirical evaluation of our approach on semi-synthetic data.

Original: Using the
Revised: Using the

Original: MoiveLens 1M dataset, we generate discrete time-series data in a way that captures the essence ofour behavioral model, but is different from the actual continuous-time dynamics we optimize over.
Revised: MoiveLens 1M dataset, we generate discrete time-series data in a way that captures the essence of our behavioral model, but is different from the actual continuous-time dynamics we optimize over.

Original: Results show that despite this gap, our approach improves significantly over myopic baselines, andoften closely matches an optimal oracle.
Revised: Results show that despite this gap, our approach improves significantly over myopic baselines, and often closely matches an optimal oracle.

Original: We also study the role of experimental treatments, andanalyze how different users benefit differently from breaks.
Revised: We also study the role of experimental treatments, and analyze how different users benefit differently from breaks.

Original: Taken together, these demonstrate thepotential utility of our approach.
Revised: Taken together, these demonstrate the potential utility of our approach.

Original: Code is available at: https://github . com/lvml-tsrml-2022/lvml.
Revised: Code is available at: https://github . com/edensaig/take-a-break.

Original: Broader perspective.
Revised: Broader perspective.

Original: At a high level, our work argues for viewing recommendation as a task ofsustainable resource management .
Revised: At a high level, our work argues for viewing recommendation as a task of sustainable resource management .

Original: As other cognitive tasks, engaging with digital content requiresthe availability of certain cognitive resources—attentional, executive, or emotional.
Revised: As other cognitive tasks, engaging with digital content requires the availability of certain cognitive resources—attentional, executive, or emotional.

Original: These resourcesare inherently limited , and prolonged engagement depletes them (Kahneman, 1973; Muraven and
Revised: These resources are inherently limited , and prolonged engagement depletes them (Kahneman, 1973; Muraven and

Original: Baumeister, 2000); this, in turn, can reduce the capacity of key cognitive processes (e.g., perception,attention, memory, self-control, and decision-making), and in the extreme—cause ego depletion(Baumeister et al., 1998) or cognitive fatigue (Mullette-Gillman et al., 2015).
Revised: Baumeister, 2000); this, in turn, can reduce the capacity of key cognitive processes (e.g., perception, attention, memory, self-control, and decision-making), and in the extreme—cause ego depletion (Baumeister et al., 1998) or cognitive fatigue (Mullette-Gillman et al., 2015).

Original: As a means to allowresources to replenish, ‘mental breaks’ have been shown to be highly effective (Bergum and Lehr,1962; Hennfng et al., 1989; Gilboa et al., 2008; Ross et al., 2014; Helton and Russell, 2017).
Revised: As a means to allow resources to replenish, ‘mental breaks’ have been shown to be highly effective (Bergum and Lehr, 1962; Hennfng et al., 1989; Gilboa et al., 2008; Ross et al., 2014; Helton and Russell, 2017).

Original: Nevertheless, traditional approaches to recommendation remain agnostic to the idea that recommending takes a cognitive toll: they simply recommend at each point in time the item predicted to be mostengaging (Robertson, 1977).
Revised: Nevertheless, traditional approaches to recommendation remain agnostic to the idea that recommending takes a cognitive toll: they simply recommend at each point in time the item predicted to be most engaging (Robertson, 1977).

Original: As an alternative, our approach explicitly models recommendation asa process which draws on these resources, and therefore—must also conserve them.
Revised: As an alternative, our approach explicitly models recommendation as a process which draws on these resources, and therefore—must also conserve them.

Original: The subclass of ‘Predator-Prey’ LV dynamics which we draw on are used extensively in ecology for modelingthe dynamics of interacting populations, and demonstrate how over-predation can ultimately lead toself-extinction by eliminating the prey population—but also show how enabling resources to naturallyreplenish ensures sustainable relations.
Revised: The subclass of ‘Predator-Prey’ LV dynamics which we draw on are used extensively in ecology for modeling the dynamics of interacting populations, and demonstrate how over-predation can ultimately lead to self-extinction by eliminating the prey population—but also show how enabling resources to naturally replenish ensures sustainable relations.

Original: As such, here we advocate for studying recommendationsystems as human-centric ecosystems , and take one step towards their sustainable design.
Revised: As such, here we advocate for studying recommendation systems as human-centric ecosystems , and take one step towards their sustainable design.

Original: 1.1 Related work
Revised: 1.1 Related work

Original: User dynamics: latent states and feedback.
Revised: User dynamics: latent states and feedback.

Original: A recent body of work aims to capture time-varyingbehavior by modeling users as acting based on dynamic latent states.
Revised: A recent body of work aims to capture time-varying behavior by modeling users as acting based on dynamic latent states.

Original: Broadly, works in this fieldmodel the effects of recommendations as either shifting user preferences via positive-only feedback(Jiang et al., 2019; Kalimeris et al., 2021; Sanna Passino et al., 2021; Dean and Morgenstern, 2022),or reducing willingness to consume via negative-only feedback, e.g. via boredom, satiation, orfatigue (Wang and Lin, 2003; Warlop et al., 2018; Kleinberg and Immorlica, 2018; Cao et al., 2020;
Revised: Broadly, works in this field model the effects of recommendations as either shifting user preferences via positive-only feedback (Jiang et al., 2019; Kalimeris et al., 2021; Sanna Passino et al., 2021; Dean and Morgenstern, 2022), or reducing willingness to consume via negative-only feedback, e.g. via boredom, satiation, or fatigue (Wang and Lin, 2003; Warlop et al., 2018; Kleinberg and Immorlica, 2018; Cao et al., 2020;

Original: Leqi et al., 2021).
Revised: Leqi et al., 2021).

Original: While these restrict behavior to expressing a unidirectional effect, our approachintegrates both types of feedfback and models internal states as competing but balancing forces,which we argue is more realistic.
Revised: While these restrict behavior to expressing a unidirectional effect, our approach integrates both types of feedback and models internal states as competing but balancing forces, which we argue is more realistic.

Original: This draws connections to recent attempts of injecting psychologicalmodeling into recommendation system design (Kleinberg et al., 2022; Dubey et al., 2022; Curmeiet al., 2022).
Revised: This draws connections to recent attempts of injecting psychological modeling into recommendation system design (Kleinberg et al., 2022; Dubey et al., 2022; Curmei et al., 2022).

Original: Works in this field often combine theoretical analysis with simulation studies (Schmitand Riquelme, 2018;
Revised: Works in this field often combine theoretical analysis with simulation studies (Schmit and Riquelme, 2018;

Original: Chaney et al., 2018; Mansoury et al., 2020; Krauth et al., 2020), and here wefollow suit.
Revised: Chaney et al., 2018; Mansoury et al., 2020; Krauth et al., 2020), and here we follow suit.

Original: Lotka-Volterra dynamics: modeling, learning, and control.
Revised: Lotka-Volterra dynamics: modeling, learning, and control.

Original: The study of ecosystem dynamicsand their conservation has a long and rich history, in which LV analysis plays an integral role (see e.g.
Revised: The study of ecosystem dynamics and their conservation has a long and rich history, in which LV analysis plays an integral role (see e.g.

Original: Hofbauer et al.
Revised: Hofbauer et al.

Original: (1998); Takeuchi (1996)).
Revised: (1998); Takeuchi (1996)).

Original: LV systems our used primarily for modeling biologicalecosystems, but are also used in economics (Weibull, 1997; Samuelson, 1998), finance (Farmer, 2002;
Revised: LV systems our used primarily for modeling biological ecosystems, but are also used in economics (Weibull, 1997; Samuelson, 1998), finance (Farmer, 2002;

Original: Scholl et al., 2021), and behavioral modeling (e.g., drug addiction and relapse (Duncan et al., 2019)).
Revised: Scholl et al., 2021), and behavioral modeling (e.g., drug addiction and relapse (Duncan et al., 2019)).

Original: In terms of learning, Gorbach et al.
Revised: In terms of learning, Gorbach et al.

Original: (2017) and Ryder et al.
Revised: (2017) and Ryder et al.

Original: (2018) propose variational techniques fordynamical systems, but do not consider control.
Revised: (2018) propose variational techniques for dynamical systems, but do not consider control.

Original: Our work aims to directly learn optimal policies, forwhich we draw on recent advances in turnpike optimal control (Trélat and Zuazua, 2015).
Revised: Our work aims to directly learn optimal policies, for which we draw on recent advances in turnpike optimal control (Trélat and Zuazua, 2015).

Original:  Learning setting
Revised:  Learning setting

Original: We consider a sequential recommendation setting in which users interact with a stream of recommended items over time.
Revised: We consider a sequential recommendation setting in which users interact with a stream of recommended items over time.

Original: New users u are sampled iid from some unknown distribution D , and timefor each user is measured relative to their time of joining.
Revised: New users u are sampled independently from some unknown distribution D , and time for each user is measured relative to their time of joining.

Original: Interactions occur at discrete time-points incontinuous time, t ∈ R + , and upon user request: when a user u queries the system for additionalcontent at time t , the system responds by presenting a recommended item x ( t ) ∈ X .
Revised: Interactions occur at discrete time-points in continuous time, t ∈ R + , and upon user request: when a user u queries the system for additional content at time t , the system responds by presenting a recommended item x ( t ) ∈ X .

Original: We assumerecommendations are governed by an existing and fixed recommendation policy ψ , which determinesx ( t ) given a request from u at time t .
Revised: We assume recommendations are governed by an existing and fixed recommendation policy ψ , which determines x ( t ) given a request from u at time t .

Original: We allow ψ to be stochastic, and make no additional assumptions on its structure or mechanics.
Revised: We allow ψ to be stochastic, and make no additional assumptions on its structure or mechanics.

Original: User interactions are therefore described by a sequence of pairs{ ( t i , x i ) } i , where x i =
Revised: User interactions are therefore described by a sequence of pairs { ( t i , x i ) } i , where x i =

Original: x
Revised: x

Original: ( t i ) ∼ ψ ( u ; t i ) ∈
Revised: ( t i ) ∼ ψ ( u ; t i ) ∈

Original: X is the item recommended to u at time t i .
Revised: X is the item recommended to u at time t i .

Original: Our key modeling assumption is that subsequent request times, t i +

Revised: Our key modeling assumption is that subsequent request times, t i +

Original: =
Revised: =

Original: t i + ∆ t
Revised: t i + ∆ t

Original: i for ∆ t i > 0 , aredetermined jointly by the user’s ‘state’ at time t i and the recommended item x i (note this means ∆ t ican depend on t i ).
Revised: i for ∆ t i > 0 , are determined jointly by the user’s ‘state’ at time t i and the recommended item x i (note this means ∆ t i can depend on t i ).

Original: We consider user states as latent and in the abstract, but broadly expect ‘good’recommendations to entail frequent requests by inducing small values of ∆ t i .
Revised: We consider user states as latent and in the abstract, but broadly expect ‘good’ recommendations to entail frequent requests by inducing small values of ∆ t i .

Original: Together, user u ’schoice behavior, coupled with the policy ψ , induce a temporal point process (TPP) which governs thegeneration of interaction sequences of duration t as { ( t i , x i ) | t i ∈ [0 , t ] } ∼ TPP ψ ( u ; t ) .
Revised: Together, user u ’s choice behavior, coupled with the policy ψ , induce a temporal point process (TPP) which governs the generation of interaction sequences of duration t as { ( t i , x i ) | t i ∈ [0 , t ] } ∼ TPP ψ ( u ; t ) .

Original: For each user, the system collects data until some (relative) fixed time T 0 , and seeks to optimizeengagement in the subsequent interval [ T 0 , T 0 + T ] , where T is a predetermined time horizon.
Revised: For each user, the system collects data until some (relative) fixed time T 0 , and seeks to optimize engagement in the subsequent interval [ T 0 , T 0 + T ] , where T is a predetermined time horizon.

Original: Wedenote the corresponding input sequences by S 0 u
Revised: We denote the corresponding input sequences by S 0 u

Original: = { ( t i , x i ) | t i ∈ [0 , T 0 ) } , and target sequences135 by S u
Revised: = { ( t i , x i ) | t i ∈ [0 , T 0 ) } , and target sequences by S u

Original: = { ( t i , x i ) | t i ∈
Revised: = { ( t i , x i ) | t i ∈

Original: [ T 0 , T ] } .
Revised: [ T 0 , T ] } .

Original: Defining by 1 T | S u | the engagement rate of u for the chosen time136 horizon T , our goal in learning will be to maximize expected long-term user engagement rate , namely
Revised: Defining by 1 T | S u | the engagement rate of u for the chosen time horizon T , our goal in learning will be to maximize expected long-term user engagement rate , namely

Original: E  (cid:2) 1 T | S  | (cid:3) for ( S 0 u , S u ) ∼ TPP ψ ( u ; T 0 + T ) , in expectation over new users.
Revised: E u ∼ D E (cid:2) 1 T | S u | (cid:3) for ( S 0 u , S u ) ∼ TPP ψ ( u ; T 0 + T ) , in expectation over new users.

Original: Breaking policies.
Revised: Breaking policies.

Original: The unique aspect of our learning problem is that our only means for increasingengagement is by suggesting breaks .
Revised: The unique aspect of our learning problem is that our only means for increasing engagement is by suggesting breaks .

Original: Our task will be to learn a breaking policy π ∈ Π which can override ψ : when user u queries the system at time t i , the policy π ( u ; t i ) ∈ { 0 , 1 } determines141 whether to present the intended item x ( t ) ∼ ψ ( u ; t i ) (for π = 0 ), or suggest a break instead ( π = 1 ).
Revised: Our task will be to learn a breaking policy π ∈ Π which can override ψ : when user u queries the system at time t i , the policy π ( u ; t i ) ∈ { 0 , 1 } determines whether to present the intended item x ( t ) ∼ ψ ( u ; t i ) (for π = 0 ), or suggest a break instead ( π = 1 ).

Original: Denoting the overall composed policy by π ◦ ψ , our aim is to learn a breaking policy π that increasesengagement by complementing an existing ψ .
Revised: Denoting the overall composed policy by π ◦ ψ , our aim is to learn a breaking policy π that increases engagement by complementing an existing ψ .

Original: Hence, our learning objective is:
Revised: Hence, our learning objective is:

Original: [Equation]
Revised: [Equation]

Original: Broadly, we expect breaks to negatively affect short-term engagement (i.e., entail longer ensuing∆ t i ), but have the potential to improve engagement in the long run if scheduled appropriately.
Revised: Broadly, we expect breaks to negatively affect short-term engagement (i.e., entail longer ensuing ∆ t i ), but have the potential to improve engagement in the long run if scheduled appropriately.

Original: Data and exploration.
Revised: Data and exploration.

Original: For learning an optimal breaking policy, we assume the system has accessto a dataset of previously logged interactions S
Revised: For learning an optimal breaking policy, we assume the system has access to a dataset of previously logged interactions S

Original: = { ( S 01 , S 1 ) , . . .
Revised: = { ( S 01 , S 1 ) , . . .

Original: , ( S 0 m , S m ) } for m train-time usersu 1 , . . .
Revised: , ( S 0 m , S m ) } for m train-time users u 1 , . . .

Original: , u m ∼ D , collected under a ‘clean’ recommendation policy ψ .
Revised: , u m ∼ D , collected under a ‘clean’ recommendation policy ψ .

Original: It will be convenient to assumethat the system ‘featurizes’ user input sequences S 0 u via a vector mapping ϕ used in learning.
Revised: It will be convenient to assume that the system ‘featurizes’ user input sequences S 0 u via a vector mapping ϕ used in learning.

Original: Forclarity, we overload notation and represent users as u
Revised: For clarity, we overload notation and represent users as u

Original: = ϕ
Revised: = ϕ

Original: ( S 0 u ) ∈ R d .
Revised: ( S 0 u ) ∈ R d .

Original: This allows us to supportadditional user feedback (e.g., ratings) or information (e.g., demographics) as input to ϕ .
Revised: This allows us to support additional user feedback (e.g., ratings) or information (e.g., demographics) as input to ϕ .

Original: Since Eq.
Revised: Since Eq.

Original: (1) is a policy problem (note breaks affect outcomes), learning requires some formof exploration or experimentation.
Revised: (1) is a policy problem (note breaks affect outcomes), learning requires some form of exploration or experimentation.

Original: Here we aim for experimentation to be simple and minimal.
Revised: Here we aim for experimentation to be simple and minimal.

Original: Specifically, we assume access to a small number of N additional datasets, S ( j ) =
Revised: Specifically, we assume access to a small number of N additional datasets, S ( j ) =

Original: { ( u k , S k ) } m j k =1 ,collected for different users, and under composed policies π j ◦ ψ for various predetermined π j .
Revised: { ( u k , S k ) } m j k =1 , collected for different users, and under composed policies π j ◦ ψ for various predetermined π j .

Original: Thesedatasets can either be given at the onset, or collected as part of the learning process; our only usageof them will be for learning predictors ˆ y =
Revised: These datasets can either be given at the onset, or collected as part of the learning process; our only usage of them will be for learning predictors ˆ y =

Original: f j ( u ) of engagement rate y
Revised: f j ( u ) of engagement rate y

Original: = 1 T | S u | .
Revised: = 1 T | S u | .

Original: Ideally, we wouldlike to make do with only a few S ( j ) of small size and that can be gathered concurrently; our resultsshow that even a single additional dataset can be highly informative, and in some cases sufficient.
Revised: Ideally, we would like to make do with only a few S ( j ) of small size and that can be gathered concurrently; our results show that even a single additional dataset can be highly informative, and in some cases sufficient.

Original: Learning to break.
Revised: Learning to break.

Original: To effectively optimize Eq.
Revised: To effectively optimize Eq.

Original: (1), learning must anticipate the effects of differentbreaking policies π ∈ Π on future sequences S u of unobserved users u .
Revised: (1), learning must anticipate the effects of different breaking policies π ∈ Π on future sequences S u of unobserved users u .

Original: It will therefore be useful todistinguish between the policy π itself, which determines when breaks are applied, and a componentfor estimating individualized counterfactual engagement rates 1 T | ˆ S u ( π ) | induced by π , which willaid in choosing a good policy.
Revised: It will therefore be useful to distinguish between the policy π itself, which determines when breaks are applied, and a component for estimating individualized counterfactual engagement rates 1 T | ˆ S u ( π ) | induced by π , which will aid in choosing a good policy.

Original: Our focus will be on learning simple policies coupled with rich andtask-appropriate models of responsive user behavior.
Revised: Our focus will be on learning simple policies coupled with rich and task-appropriate models of responsive user behavior.

Original: In particular, we set Π to include all stationarypolicies , π ( u ) =
Revised: In particular, we set Π to include all stationary policies , π ( u ) =

Original: π ( p u ) , which for each user u recommends a break with a time-independentpersonalized probability p u = p ( u ) ∈
Revised: π ( p u ) , which for each user u recommends a break with a time-independent personalized probability p u = p ( u ) ∈

Original: Stationary policies are interpretable, amenable toefficient optimization, and straightforward to implement.
Revised: Stationary policies are interpretable, amenable to efficient optimization, and straightforward to implement.

Original: As we show, despite their simplicity, suchpolicies can be quite expressive when coupled with an appropriate behavioral model.
Revised: As we show, despite their simplicity, such policies can be quite expressive when coupled with an appropriate behavioral model.

Original:  Engagement via ‘Predator-Prey’ dynamics
Revised:  Engagement via ‘Predator-Prey’ dynamics

Original: To optimize engagement with breaks, we propose to model user behavior as a dynamical system of
Revised: To optimize engagement with breaks, we propose to model user behavior as a dynamical system of

Original: Lotka-Volterra (LV) Predator-Prey equations.
Revised: Lotka-Volterra (LV) Predator-Prey equations.

Original: The model operates over continuous dynamics, whichhas analytic and optimizational benefits; we later make the connection back to discrete sequences.
Revised: The model operates over continuous dynamics, which has analytic and optimizational benefits; we later make the connection back to discrete sequences.

Original: Behavioral model.
Revised: Behavioral model.

Original: Our model of user behavior views each user as acting based on two types oftime-dependent latent variables: internal drive for consumption, denoted by λ ( t ) ∈ R + ; and intrinsicinterest , denoted by q ( t ) ∈ [0 , 1] .
Revised: Our model of user behavior views each user as acting based on two types of time-dependent latent variables: internal drive for consumption, denoted by λ ( t ) ∈ R + ; and intrinsic interest , denoted by q ( t ) ∈ [0 , 1] .

Original: Intuitively, we expect that engaging experiences will reinforce auser’s desire to further consume; conversely, excessive exposure to content should slowly ‘erode’ herinterest—but left alone, will allow it to replenish (Thoman et al., 2011).
Revised: Intuitively, we expect that engaging experiences will reinforce a user’s desire to further consume; conversely, excessive exposure to content should slowly ‘erode’ her interest—but left alone, will allow it to replenish (Thoman et al., 2011).

Original: Thus, drive and interest actas balancing forces.
Revised: Thus, drive and interest act as balancing forces.

Original: Together, we model the time-dependent relations between λ ( t ) and q ( t ) as:
Revised: Together, we model the time-dependent relations between λ ( t ) and q ( t ) as:

Original: [Equation]
Revised: [Equation]

Original: where θ =
Revised: where θ =

Original: ( α, β, γ, δ ) ≥ 0 parameterize the dynamics.
Revised: ( α, β, γ, δ ) ≥ 0 parameterize the dynamics.

Original: For λ (drive, or ‘predator’), α determines itsnatural decay rate, and β its interest-dependent self-reinforcement rate; for q (interest, or ‘prey’), γspecifies its natural replenishment rate, and δ its rate of depletion from consumption.
Revised: For λ (drive, or ‘predator’), α determines its natural decay rate, and β its interest-dependent self-reinforcement rate; for q (interest, or ‘prey’), γ specifies its natural replenishment rate, and δ its rate of depletion from consumption.

Original: Note the twoequations are coupled: q mediates the reinforcement of λ , and λ mediates the depletion of q .
Revised: Note the two equations are coupled: q mediates the reinforcement of λ , and λ mediates the depletion of q .

Original: 100 Time t
Revised: 100 Time t

Original: C o n s u m p
Revised: C o n s u m p

Original: t
Revised: t

Original: i o n d r i v e λ ( t ) λ ( t ) and q ( t ) - Joint evolution over time  λ ( t ) λ ∗ q ( t )     20 Consumption drive λ ( t ) 0 .0 .0 .0 .0 .0 .
Revised: i o n d r i v e λ ( t ) λ ( t ) and q ( t ) - Joint evolution over time Consumption drive λ ( t ) Intrinsic interest q ( t ) Equilibrium consumption λ ∗    20 Consumption drive λ ( t ) 0 .0 .0 .0 .0 .0 .

Original: I n t e r e s t q ( t ) ( λ,q ) - Phase-space diagram ( λ ( t ) ,q ( t ))
Revised: I n t e r e s t q ( t ) ( λ,q ) - Phase-space diagram ( λ ( t ) ,q ( t ))

Original: ( λ ∗ ,q ∗ ) 0 .0 .0 .0 .0 .
Revised: ( λ ∗ ,q ∗ ) 0 .0 .0 .0 .0 .

Original: I n t e r e s t q ( t )
Revised: I n t e r e s t q ( t )

Original: Dynamical properties.
Revised: Dynamical properties.

Original: The LV model in Eq.
Revised: The LV model in Eq.

Original: (2) describes consumption as a cycle: when interestq ( t ) is high, drive to consume λ ( t ) increases, resulting in positive feedback; conversely, when λ ( t ) ishigh, q ( t ) decreases, which expresses negative feedback.
Revised: (2) describes consumption as a cycle: when interest q ( t ) is high, drive to consume λ ( t ) increases, resulting in positive feedback; conversely, when λ ( t ) is high, q ( t ) decreases, which expresses negative feedback.

Original: In general, λ grows until interest is too lowto sustain consumption, at which point consumption drops sharply, allowing interest to recover—andthe cycle repeats.
Revised: In general, λ grows until interest is too low to sustain consumption, at which point consumption drops sharply, allowing interest to recover—and the cycle repeats.

Original: The cycling behavior exhibits oscillations in λ and q , with one lagging after theother.
Revised: The cycling behavior exhibits oscillations in λ and q , with one lagging after the other.

Original: A typical trajectory is illustrated in Figure 1.
Revised: A typical trajectory is illustrated in Figure 1.

Original: Note how the drop in λ occurs only some timeafter q is depleted; hence, anticipating (and preventing) the collapse of λ requires knowledge (andconservation) of q .
Revised: Note how the drop in λ occurs only some time after q is depleted; hence, anticipating (and preventing) the collapse of λ requires knowledge (and conservation) of q .

Original: Thus, q serves as a resource: necessary for consumption, and of limited supply.
Revised: Thus, q serves as a resource: necessary for consumption, and of limited supply.

Original: Over time, and if no interventions are applied, the magnitude of oscillations decreases, and the systemnaturally approaches a stable equilibrium, denoted ( λ ∗ , q ∗ ) , determined by the system parametersθ =
Revised: Over time, and if no interventions are applied, the magnitude of oscillations decreases, and the system naturally approaches a stable equilibrium, denoted ( λ ∗ , q ∗ ) , determined by the system parameters θ =

Original: ( α, β, γ, δ ) and which attracts all initial conditions λ (0) , q (0) (Takeuchi, 1996).
Revised: ( α, β, γ, δ ) and which attracts all initial conditions λ (0) , q (0) (Takeuchi, 1996).

Original: Equilibriumplays a key role in our approach, as it captures the notion of habits, which we aim to improve.
Revised: Equilibrium plays a key role in our approach, as it captures the notion of habits, which we aim to improve.

Original: Engagement maximization as optimal control.
Revised: Engagement maximization as optimal control.

Original: To optimize engagement (Eq.
Revised: To optimize engagement (Eq.

Original: , we proposeto optimize λ ∗ as an alternative proxy.
Revised: (1)), we propose to optimize λ ∗ as an alternative proxy.

Original: Broadly, our approach will be to associate with each useru a dynamical system parameterized by θ u
Revised: Broadly, our approach will be to associate with each user u a dynamical system parameterized by θ u

Original: =
Revised: =

Original: ( α u , β u , γ u , δ u ) , and recommend breaks which leadto high values of the corresponding λ ∗ u = λ ∗ θ u .
Revised: ( α u , β u , γ u , δ u ) , and recommend breaks which lead to high values of the corresponding λ ∗ u = λ ∗ θ u .

Original: Intuitively, if we think of 1 T | S u | as an ‘empirical’200 rate (determined by the ∆ t i in S u ), then λ ∗ u is its continuous theoretical counterpart, and so ideallywe would like to find a θ u for which λ ∗ u is the limiting behavior of 1 T | S u | (i.e., when ∆ t → 0 and
Revised: Intuitively, if we think of 1 T | S u | as an ‘empirical’ rate (determined by the ∆ t i in S u ), then λ ∗ u is its continuous theoretical counterpart, and so ideally we would like to find a θ u for which λ ∗ u is the limiting behavior of 1 T | S u | (i.e., when ∆ t → 0 and

Original: T → ∞ ).
Revised: T → ∞ ).

Original: In practice, we expect λ ∗ u to be a useful target when the observed 1 T | S u | exhibits habits203 that are ‘close enough’ to the theoretical equilibrium, and in Sec.
Revised: In practice, we expect λ ∗ u to be a useful target when the observed 1 T | S u | exhibits habits that are ‘close enough’ to the theoretical equilibrium, and in Sec.

Original: 4.3 we make this condition precise.
Revised: 4.3 we make this condition precise.

Original: Since our goal is to optimize a breaking policy, we must also be precise about the way breaks affectdynamics in our model.
Revised: Since our goal is to optimize a breaking policy, we must also be precise about the way breaks affect dynamics in our model.

Original: Towards this, we introduce into Eq.
Revised: Towards this, we introduce into Eq.

Original: (2) a control variable , p ( t ) ∈ [0 , 1] ,which acts as a ‘gate’ that controls the mediation strength between λ ( t ) and q ( t ) via:
Revised: (2) a control variable , p ( t ) ∈ [0 , 1] , which acts as a ‘gate’ that controls the mediation strength between λ ( t ) and q ( t ) via:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: When p > 0 , breaking has a dual effect: it decelerates drive λ , and at the same time, lets q recover.
Revised: When p > 0 , breaking has a dual effect: it decelerates drive λ , and at the same time, lets q recover.

Original: Our goal will now be to solve the optimal control problem of finding p ( t ) which maximizes λ ∗ .
Revised: Our goal will now be to solve the optimal control problem of finding p ( t ) which maximizes λ ∗ .

Original: Notethat not every dynamic controlling schedule guarantees convergence to some λ ∗ .
Revised: Note that not every dynamic controlling schedule guarantees convergence to some λ ∗ .

Original: Hence, we focus onfixed controls , p ∈ [0 , 1] , which we prove converge to equilibrium, and having closed form .
Revised: Hence, we focus on fixed controls , p ∈ [0 , 1] , which we prove converge to equilibrium, and having closed form .

Original: Lemma 1.
Revised: Lemma 1.

Original: Let θ
Revised: Let θ

Original: =
Revised: =

Original: ( α, β, γ, δ ) define an LV system as in Eq.
Revised: ( α, β, γ, δ ) define an LV system as in Eq.

Original: Then for any p ∈ [0 , 1] , we have:
Revised: Then for any p ∈ [0 , 1] , we have:

Original: [Equation]

Original: otherwise

Original: [Equation]
Revised: [Equation]

Original: Proof in Appendix A.1.
Revised: Proof in Appendix A.1.

Original: For optimization, Lemma 1 is useful since it depicts λ ∗ as a function of p ,parameterized by θ .
Revised: For optimization, Lemma 1 is useful since it depicts λ ∗ as a function of p , parameterized by θ .

Original: Thus, given some θ u for user u , our objective is to solve the control problem:
Revised: Thus, given some θ u for user u , our objective is to solve the control problem:

Original: [Equation]
Revised: [Equation]

Original: Optimizing engagement now reduces to solving Eq.
Revised: Optimizing engagement now reduces to solving Eq.

Original: (5); given p u , we make the connection back todiscrete time by setting the breaking policy to be π ( p u ) , which recommends breaks at rate p u .
Revised: (5); given p u , we make the connection back to discrete time by setting the breaking policy to be π ( p u ) , which recommends breaks at rate p u .

Original:  Learning optimal breaking policies
Revised:  Learning optimal breaking policies

Original: We now turn to presenting our learning algorithm.
Revised: We now turn to presenting our learning algorithm.

Original: Our approach to optimizing engagement consistsof two steps: (i) associating with each user u a set of LV parameters θ u , and then (ii) finding p u whichmaximizes λ ∗ θ u ( p ) , and plugging into π ( p u ) .
Revised: Our approach to optimizing engagement consists of two steps: (i) associating with each user u a set of LV parameters θ u , and then (ii) finding p u which maximizes λ ∗ θ u ( p ) , and plugging into π ( p u ) .

Original: In practice, we add an intermediate prediction step,which allows us to ‘shortcut’ directly from observations to optimal policies.
Revised: In practice, we add an intermediate prediction step, which allows us to ‘shortcut’ directly from observations to optimal policies.

Original: We conclude with analysisshowing when learned policies π ( p u ) lead to good expected engagement E u ∼ D E TPP (cid:2) 1 T | S u | (cid:3) [-.4.
-]
Revised: We conclude with analysis showing when learned policies π ( p u ) lead to good expected engagement E u ∼ D E TPP (cid:2) 1 T | S u | (cid:3) [-.4.
-].

Original:  Embedding users in LV space
Revised: 4.1 Embedding users in LV space

Original: To find θ u , a seemingly reasonable approach would be to fit an LV trajectory to the initial sequence
Revised: To find θ u , a seemingly reasonable approach would be to fit an LV trajectory to the initial sequence

Original: S 0 u , i.e., by solving min θ (cid:80) i | ∆ t
Revised: S 0 u , i.e., by solving min θ (cid:80) i | ∆ t

Original: i − λ θ ( t i ) | for the observed t i ∈ S 0 u .
Revised: i − λ θ ( t i ) | for the observed t i ∈ S 0 u .

Original: This can be interpreted asembedding S 0 u in ‘LV-space’ by finding the nearest continuous trajectory, for which θ provides acompact representation.
Revised: This can be interpreted as embedding S 0 u in ‘LV-space’ by finding the nearest continuous trajectory, for which θ provides a compact representation.

Original: However, a key issue with this approach is that S 0 u contains past observationsmade under a single policy π ( p ) (e.g., the ‘clean’ policy π (0) ).
Revised: However, a key issue with this approach is that S 0 u contains past observations made under a single policy π ( p ) (e.g., the ‘clean’ policy π (0) ).

Original: To optimize future engagement, thelearned θ u must correctly account for the affects of general p on possbile ensuing sequences S u .
Revised: To optimize future engagement, the learned θ u must correctly account for the affects of general p on possbile ensuing sequences S u .

Original: As an alternative, we propose to find θ u by fitting the entire equilibrium curve of λ ∗ θ ( p ) (Eq.
Revised: As an alternative, we propose to find θ u by fitting the entire equilibrium curve of λ ∗ θ ( p ) (Eq.

Original: Formally, for each p , define the expected empirical engagement rate ¯ λ u ( p ) as:
Revised: Formally, for each p , define the expected empirical engagement rate ¯ λ u ( p ) as:

Original: [Equation]
Revised: [Equation]

Original: (6) i.e., ¯ λ is the rate of counterfactual future trajectories for all possible choices of p .
Revised: (6) i.e., ¯ λ is the rate of counterfactual future trajectories for all possible choices of p .

Original: Ideally, we wouldlike to find θ for which the learned curve λ ∗ θ ( p ) closely aligns with that of ¯ λ u ( p ) across p ∈ [0 , 1] :
Revised: Ideally, we would like to find θ for which the learned curve λ ∗ θ ( p ) closely aligns with that of ¯ λ u ( p ) across p ∈ [0 , 1] :

Original: [Equation]
Revised: [Equation]

Original: for some function norm ∥ · ∥ , and for which λ ∗ ¯ θ u
Revised: for some function norm ∥ · ∥ , and for which λ ∗ ¯ θ u

Original: and ¯ λ u have similar maximizing p (since we aim foroptimizing λ ∗ θ to work well for ¯ λ u ).
Revised: and ¯ λ u have similar maximizing p (since we aim for optimizing λ ∗ θ to work well for ¯ λ u ).

Original: Unfortunately, ¯ λ u is a theoretical construct, and so ¯ θ u cannot beobtained from observations.
Revised: Unfortunately, ¯ λ u is a theoretical construct, and so ¯ θ u cannot be obtained from observations.

Original: Hence, we propose to replace ¯ λ u with a finite set of predictions .
Revised: Hence, we propose to replace ¯ λ u with a finite set of predictions .

Original: The role of prediction.
Revised: The role of prediction.

Original: Recall our input consists of a primary dataset S (0) =
Revised: Recall our input consists of a primary dataset S (0) =

Original: S =
Revised: S =

Original: { ( u i , S i ) } mi =1 , aswell as additional ‘experimental’ datasets S ( j ) collected under different π j 
Revised: { ( u i , S i ) } mi =1 , as well as additional ‘experimental’ datasets S ( j ) collected under different π j = π

Original: = π ( p j ) and of sizes m jfor j = 1 , . . 
Revised: ( p j ) and of sizes m j for j = 1 , . . .

Original:  N .
Revised: , N .

Original: We can use these to learn individualized, policy-specific predictors f j ( u ) =
Revised: We can use these to learn individualized, policy-specific predictors f j ( u ) =

Original: f p j ( u ) ,trained to predict for each user u her engagement rate y
Revised: f p j ( u ) , trained to predict for each user u her engagement rate y

Original: =
Revised: =

Original: 1 T | S u | under π j .
Revised: 1 T | S u | under π j .

Original: For example, if wetrain f j to minimize the squared error (cid:80) k ( f j ( u k ) − y k ) 2 on pairs ( u k , y k ) ∈ S ( j ) , then f j ( u )should be a reasonable estimator of the expected ¯ λ u
Revised: For example, if we train f j to minimize the squared error (cid:80) k ( f j ( u k ) − y k ) 2 on pairs ( u k , y k ) ∈ S ( j ) , then f j ( u ) should be a reasonable estimator of the expected ¯ λ u

Original: ( p j ) .
Revised: ( p j ) .

Original: Hence, for a given u , a finite set of pairs{ ( p j , f j ( u )) } Nj =1 gives points to which we can fit θ to λ ∗ θ .
Revised: Hence, for a given u , a finite set of pairs { ( p j , f j ( u )) } Nj =1 gives points to which we can fit θ to λ ∗ θ .

Original: Our final criterion for choosing ˆ θ u is:
Revised: Our final criterion for choosing ˆ θ u is:

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: given here with the ℓ 2 vector norm, and where f ( u ) =
Revised: given here with the ℓ 2 vector norm, and where f ( u ) =

Original: ( f 1 ( u ) , f 2 ( u ) , . . .
Revised: ( f 1 ( u ) , f 2 ( u ) , . . .

Original: , f N ( u ))
Revised: , f N ( u ))

Original: ∈ R N + .
Revised: ∈ R N + .

Original: As wewill see next, optimizing over p requires only the ratios α/β and γ/δ , in which λ ∗ is quadratic.
Revised: As we will see next, optimizing over p requires only the ratios α/β and γ/δ , in which λ ∗ is quadratic.

Original: Hence, Eq.
Revised: Hence, Eq.

Original: (8) can be efficiently solved using a polynomial Non-Negative Least Squares (NNLS)regression solver (Chen and Plemmons, 2010).
Revised: (8) can be efficiently solved using a polynomial Non-Negative Least Squares (NNLS) regression solver (Chen and Plemmons, 2010).

Original: The role of experimentation.
Revised: The role of experimentation.

Original: Two parameters control the goodness of fit for ˆ θ u : the number ofexperimental datasets, N , and their sizes, m j for j ∈ [ N ] .
Revised: Two parameters control the goodness of fit for ˆ θ u : the number of experimental datasets, N , and their sizes, m j for j ∈ [ N ] .

Original: In general, increasing N provides moredata points for solving Eq.
Revised: In general, increasing N provides more data points for solving Eq.

Original: (8), and increasing each m j reduces noise for that point (i.e., f u ( p ) shouldbe closer to ¯ λ u ).
Revised: (8), and increasing each m j reduces noise for that point (i.e., f u ( p ) should be closer to ¯ λ u ).

Original: But experimentation is costly, and so in reality N and the m j may be small.
Revised: But experimentation is costly, and so in reality N and the m j may be small.

Original: Asmotivation, we next show that under realizability and for accurate predictions, N = 2 suffices.
Revised: As motivation, we next show that under realizability and for accurate predictions, N = 2 suffices.

Original: Proposition 1.
Revised: Proposition 1.

Original: Fix N = 2 , and let p 0 , p 1 ∈ [0 , 1 − α/β ] .
Revised: Fix N = 2 , and let p 0 , p 1 ∈ [0 , 1 − α/β ] .

Original: For a user u , if (i) exists θ u s.t.
Revised: For a user u , if (i) exists θ u s.t.

Original: ¯ λ u = λ ∗ θ u ,and (ii)
Revised: ¯ λ u = λ ∗ θ u , and (ii)

Original: f i ( u ) =
Revised: f i ( u ) =

Original: ¯ λ u ( p i ) for i = 1 , 2 , then ˆ θ u is optimal, i.e., solving Eq.
Revised: ¯ λ u ( p i ) for i = 1 , 2 , then ˆ θ u is optimal, i.e., solving Eq.

Original: (8) gives ˆ θ u = ¯ θ u .
Revised: (8) gives ˆ θ u = ¯ θ u .

Original: Proof in Appendix A.1, and relies on Lemma 1.
Revised: Proof in Appendix A.1, and relies on Lemma 1.

Original: Next, we discuss how to obtain π ( u ) from θ u .
Revised: Next, we discuss how to obtain π ( u ) from θ u .

Original: 6 Probability of break ( p )
Revised: 6 Probability of break ( p )

Original: E qu ili b r i u m c o n s u m p
Revised: E qu ili b r i u m c o n s u m p

Original: t
Revised: t

Original: i o n (cid:16) λ ∗ δ γ (cid:17)
Revised: i o n (cid:16) λ ∗ δ γ (cid:17)

Original: LV equilibrium consumption λ ∗ ( p ) βα
Revised: LV equilibrium consumption λ ∗ ( p ) βα

Original: = 0 .
Revised: = 0 .

Original: 7 βα = 1 .βα = 2 .
Revised: 7 βα = 1 .βα = 2 .

Original:  3 System parameters ( β/α ) 0 .0 .0 .0 .0 .
Revised:  3 System parameters ( β/α ) 0 .0 .0 .0 .0 .

Original: O p t i m a l p o li c y ( p ∗ )
Revised: O p t i m a l p o li c y ( p ∗ )

Original: Optimal stationary policy p ∗ ( α/β ) βα = 0 .
Revised: Optimal stationary policy p ∗ ( α/β ) βα = 0 .

Original: 7 βα = 1 .βα = 2 .p
Revised: 7 βα = 1 .βα = 2 .p

Original: ∗ (cid:16) βα (cid:17) 0 .0 .0 .0 .
Revised: ∗ (cid:16) βα (cid:17) 0 .0 .0 .0 .

Original: 6 Probability of break ( p )
Revised: 6 Probability of break ( p )

Original: C o n s u m p
Revised: C o n s u m p

Original: t
Revised: t

Original: i o n r a t e
Revised: i o n r a t e

Original: Policy optimization from data ¯ λ u ( p ) λ ∗ ¯ θ u ( p ) λ ∗ ˆ θ u ( p ) ( p i ,f p i ( u ))
Revised: Policy optimization from data ¯ λ u ( p ) λ ∗ ¯ θ u ( p ) λ ∗ ˆ θ u ( p ) ( p i ,f p i ( u ))

Original: ˆ p p ∗
Revised: ˆ p p ∗

Original: Recall our goal is to learn an individualized breaking policy π that for each user u applies an appropriate, personalized breaking schedule π ( u ) .
Revised: Recall our goal is to learn an individualized breaking policy π that for each user u applies an appropriate, personalized breaking schedule π ( u ) .

Original: To be able to generalize to unseen u , the conventionalapproach is to learn a parameterized policy π ( u ; η ) , where η is optimized on training data, andapplied to new users at test-time.
Revised: To be able to generalize to unseen u , the conventional approach is to learn a parameterized policy π ( u ; η ) , where η is optimized on training data and applied to new users at test-time.

Original: Our approach, by relying on predictions, circumvents the need tolearn parameterized policies: once the predictors { f j ( u ) } j have been learned, the policy problem261 decomposes over users, and optimal policies π ( u ) are determined independently for each user.
Revised: Our approach, by relying on predictions, circumvents the need to learn parameterized policies: once the predictors { f j ( u ) } j have been learned, the policy problem decomposes over users, and optimal policies π ( u ) are determined independently for each user.

Original: Optimal policies.
Revised: Optimal policies.

Original: Given θ u , our next result establishes a closed form solution for the optimal p u .
Revised: Given θ u , our next result establishes a closed form solution for the optimal p u .

Original: Note that Eq.
Revised: Note that Eq.

Original: (4) shows λ ∗ θ ( p ) is piecwise-polynomial in z = 1 / (1 − p ) .
Revised: (4) shows λ ∗ θ ( p ) is piece-wise polynomial in z = 1 / (1 − p ) .

Original: Solving for z , we get:
Revised: Solving for z , we get:

Original: Lemma 2.
Revised: Lemma 2.

Original: Let θ
Revised: Let θ

Original: =
Revised: =

Original: ( α, β, γ, δ ) define an LV system as in Eq.
Revised: ( α, β, γ, δ ) define an LV system as in Eq.

Original: Then the optimal p ∗ is given by:
Revised: Then the optimal p ∗ is given by:

Original: [Equation]

Original: zero otherwise

Original: [Equation]
Revised: [Equation]

Original: Proof in Appendix A.1.
Revised: Proof in Appendix A.1.

Original: Hence, once θ u is found, our learned policy is defined as:
Revised: Hence, once θ u is found, our learned policy is defined as:

Original: [Equation]
Revised: [Equation]

Original: where ˆ p u = p ∗

Original: [Equation]

Original: ˆ

Original: [Equation] u [Equation] Altogether, to compute ˆ p u , the least squares approach in Eq.
Revised: Altogether, to compute ˆ p u , the least squares approach in Eq.

Original: (8) is applied to obtain the polynomialcoefficients
Revised: (8) is applied to obtain the polynomial coefficients

Original: [Equation]
Revised: [Equation]

Original: , then α u β u is estimated from their ratio, and then plugged into Eq.
Revised: , then α u β u is estimated from their ratio, and then plugged into Eq.

Original: (9) to obtainˆ p u .
Revised: (9) to obtain ˆ p u .

Original: For N = 2 , π ( u ) has a closed-form formulation as a function of predictions (Appendix A.2).
Revised: For N = 2 , π ( u ) has a closed-form formulation as a function of predictions (Appendix A.2).

Original: Additionally, note that by Eq.
Revised: Additionally, note that by Eq.

Original: (10), ˆ p u exhibits a phase transition at α u β u = 12 , below which ˆ p u = 0 .
Revised: (10), ˆ p u exhibits a phase transition at α u β u = 12 , below which ˆ p u = 0 .

Original: Corollary 1.
Revised: Corollary 1.

Original: In LV space, π ( u ) partitions users to those who require breaks, and those who don’t.
Revised: In LV space, π ( u ) partitions users to those who require breaks, and those who don’t.

Original: Corollary 2.
Revised: Corollary 2.

Original: In the realizable case of Prop. 1, π (ˆ p u ) idempodently improves over the myopic π (0) .
Revised: In the realizable case of Prop. 1, π (ˆ p u ) idempotently improves over the myopic π (0) .

Original: Thus, the optimal policy can be interpreted as suggesting breaks only when it deems them necessary.
Revised: Thus, the optimal policy can be interpreted as suggesting breaks only when it deems them necessary.

Original: Figure 2 illustrates λ ∗ ( p ) curves, phase shift, and optimal and learned policies for various user types.
Revised: Figure 2 illustrates λ ∗ ( p ) curves, phase shift, and optimal and learned policies for various user types.

Original: 4.3 Theoretical guarantees
Revised: 4.3 Theoretical guarantees

Original: We are now ready to state our main theoretical result, which bounds the expected long-term engagement obtained by our learned policy, ˆ π .
Revised: We are now ready to state our main theoretical result, which bounds the expected long-term engagement obtained by our learned policy, ˆ π .

Original: Our bound shows that the gap between ˆ π and theoptimal policy, π opt , is goverened by three additive terms, each relating to a different aspect of ourapproach: modeling error ( ε LV ), predictive error ( ε pred ), and deviation from expected behavior ( ε dev ).
Revised: Our bound shows that the gap between ˆ π and the optimal policy, π opt , is goverened by three additive terms, each relating to a different aspect of our approach: modeling error ( ε LV ), predictive error ( ε pred ), and deviation from expected behavior ( ε dev ).

Original: A description and interpretation of each term follows shortly.
Revised: A description and interpretation of each term follows shortly.

Original: For simplicity, we focus on N = 2 .
Revised: For simplicity, we focus on N = 2 .

Original: Theorem 1 (Informal) .
Revised: Theorem 1 (Informal) .

Original: Let p 0 , p 1 ∈ [0 , 1] , and denote by π opt ∈ Π be the optimal stationary policy.
Revised: Let p 0 , p 1 ∈ [0 , 1] , and denote by π opt ∈ Π be the optimal stationary policy.

Original: Then for the learned breaking policy ˆ π , we have:
Revised: Then for the learned breaking policy ˆ π , we have:

Original: [Equation]
Revised: [Equation]

Original: where η TPP is a TPP-specific constant scale factor.
Revised: where η TPP is a TPP-specific constant scale factor.

Original: Formal statement, precise definitions, and proof are given in Appendix A.3.
Revised: Formal statement, precise definitions, and proof are given in Appendix A.3.

Original: The proof consists ofthree main steps: We begin with a clean LV system at T = ∞ , and quantify the downstream effect ofperturbing the optimal policy.
Revised: The proof consists of three main steps: We begin with a clean LV system at T = ∞ , and quantify the downstream effect of perturbing the optimal policy.

Original: Then, we plug in the learned policy, and bound the gap due to predictiveerrors and finite T .
Revised: Then, we plug in the learned policy, and bound the gap due to predictive errors and finite T .

Original: The final step makes the transition from continuous dynamics to the discrete TPP.
Revised: The final step makes the transition from continuous dynamics to the discrete TPP.

Original: We next detail the role of each of the five terms in the bound, and how they can be controlled.
Revised: We next detail the role of each of the five terms in the bound, and how they can be controlled.

Original: • Predictive error : Since targets y = 1 T | S u | are real, ε pred is simply the expected regression errorover users, measured here in RMSE.
Revised: • Predictive error : Since targets y = 1 T | S u | are real, ε pred is simply the expected regression error over users, measured here in RMSE.

Original: As is standard, ε pred can be reduced by increasing the numberof samples m , or by learning more expressive predictive functions f (e.g., larger neural nets).
Revised: As is standard, ε pred can be reduced by increasing the number of samples m , or by learning more expressive predictive functions f (e.g., larger neural nets).

Original: • Modeling error :
Revised: • Modeling error :

Original: LV dynamics permit tractable learning; but as any hypothesis class, this tradesoff with model capacity.
Revised: LV dynamics permit tractable learning; but as any hypothesis class, this trades off with model capacity.

Original: Here, ε LV quantifies the error due limited expressive power.
Revised: Here, ε LV quantifies the error due limited expressive power.

Original: Furtherreducing ε LV can be achieved by consider richer dynamic models—a challenge left for future work.• Deviation from expectation : The learned ˆ p u rely on predicted equilibrium, but are trained onfinite-horizon data.
Revised: Further reducing ε LV can be achieved by consider richer dynamic models—a challenge left for future work. • Deviation from expectation : The learned ˆ p u rely on predicted equilibrium, but are trained on finite-horizon data.

Original: In expectation, ε dev captures how finite sequences deviate from their mean.
Revised: In expectation, ε dev captures how finite sequences deviate from their mean.

Original: As a rule of thumb, we expect larger T to reduce this form of noise, but this cannot be guaranteed.• Sensitivity :
Revised: As a rule of thumb, we expect larger T to reduce this form of noise, but this cannot be guaranteed. • Sensitivity :

Original: For N = 2 , the term | p 1 − p 

Revised: For N = 2 , the term | p 1 − p

Original: | quantifies the added value of exploring beyond thedefault breaking policy of p 0 .
Revised: | quantifies the added value of exploring beyond the default breaking policy of p 0 .

Original: Intuitively, when the points are farther away, fitting the equilibriumcurve is easier.
Revised: Intuitively, when the points are farther away, fitting the equilibrium curve is easier.

Original: Thus, for the likely case of p 0 = 0 , the experimental breaking rate p 1 should bechosen to balance between performance gain and overexposure of experimental subjects to breaks.
Revised: Thus, for the likely case of p 0 = 0 , the experimental breaking rate p 1 should be chosen to balance between performance gain and overexposure of experimental subjects to breaks.

Original:  Experiments
Revised:  Experiments

Original: We conclude with an empirical evaluation of our approach on semi-synthetic data.
Revised: We conclude with an empirical evaluation of our approach on semi-synthetic data.

Original: We use the
Revised: We use the

Original: MovieLens 1M dataset to generate recommendations and simulate user behavior, this enabling us toevaluate counterfactual outcomes under different policies.
Revised: MovieLens 1M dataset to generate recommendations and simulate user behavior, this enabling us to evaluate counterfactual outcomes under different policies.

Original: See Appendix B for additional details.
Revised: See Appendix B for additional details.

Original: 5.1 Experimental setup
Revised: 5.1 Experimental setup

Original: Data.
Revised: Data.

Original: The MovieLens 1M dataset (Harper and Konstan, 2015) includes 1,000,209 ratings providedby 6,040 users and for 3,706 items, which we use to obtain features, determine the dynamics, andemulate ϕ .
Revised: The MovieLens 1M dataset (Harper and Konstan, 2015) includes 1,000,209 ratings provided by 6,040 users and for 3,706 items, which we use to obtain features, determine the dynamics, and emulate ϕ .

Original: We sample and hold out 30% of all ratings r ux via user-stratified sampling, to whichwe apply Collaborative Filtering (CF) to get user features u and item features x that approximateu ⊤ x ≈ r ux ( d = 8 , RMSE = 0 .
Revised: We sample and hold out 30% of all ratings r ux via user-stratified sampling, to which we apply Collaborative Filtering (CF) to get user features u and item features x that approximate u ⊤ x ≈ r ux ( d = 8 , RMSE = 0 .

Original: 917 , r ∈ [1 , 5] ).
Revised: 917 , r ∈ [1 , 5] ).

Original: This mimics a process where ϕ is based on itemsrecommended by ψ and rated by users.
Revised: This mimics a process where ϕ is based on items recommended by ψ and rated by users.

Original: We then take the remaining data points and randomly assign1,000 users to the test set, on which we evaluate policies.
Revised: We then take the remaining data points and randomly assign 1,000 users to the test set, on which we evaluate policies.

Original: The remaining users are assigned to the trainset, which is then further partitioned into the main S and the different S ( j ) per experimental condition.
Revised: The remaining users are assigned to the train set, which is then further partitioned into the main S and the different S ( j ) per experimental condition.

Original: Recommendation policy and user behavior.
Revised: Recommendation policy and user behavior.

Original: Following Kalimeris et al.
Revised: Following Kalimeris et al.

Original: Hron et al.
Revised: Hron et al.

Original: (2022),we set ψ to recommend based on learned softmax scores, softmax( u ⊤ x ) , taken over all non held-outx having ratings for u .
Revised: , we set ψ to recommend based on learned softmax scores, softmax( u ⊤ x ) , taken over all non held-out x having ratings for u .

Original: For user behavior, we generate discrete interaction sequences S u in a way thatrelates to our dynamic model, but is nonetheless quite distinct.
Revised: For user behavior, we generate discrete interaction sequences S u in a way that relates to our dynamic model, but is nonetheless quite distinct.

Original: Specifically, each user u is associatedwith discrete-time latent-state variables, updated via an LV discretization scheme (see Appendix B.5).
Revised: Specifically, each user u is associated with discrete-time latent-state variables, updated via an LV discretization scheme (see Appendix B.5).

Original: These, together with recommended items x i = x
Revised: These, together with recommended items x i = x

Original: ( t i ) , determine consecutive ∆ t i ∈ S u .
Revised: ( t i ) , determine consecutive ∆ t i ∈ S u .

Original: Importantly,we let ∆ t i depend on u ’s rating for x i , namely r u,x i , which we interpret as u ’s utility from consumingx i .
Revised: Importantly, we let ∆ t i depend on u ’s rating for x i , namely r u,x i , which we interpret as u ’s utility from consuming x i .

Original: We parameterize this dependence via ‘immediate’, item-specific consumption gain parametersβ ui that depend on r u,x i , and for simplicity set α u , γ u , δ u to be fixed.
Revised: We parameterize this dependence via ‘immediate’, item-specific consumption gain parameters β ui that depend on r u,x i , and for simplicity set α u , γ u , δ u to be fixed.

Original: Note that since β ui depends on the recommended x
Revised: Note that since β ui depends on the recommended x

Original: i , it varies in time, and hence there is no single βthat underlies the dynamics: even in the limit ( ∆ t → 0 , T → ∞ ), user behavior cannot be described325 by a continuous LV system, which implies ε LV > 0 (see Appendix Figure 5).
Revised: i , it varies in time, and hence there is no single β that underlies the dynamics: even in the limit ( ∆ t → 0 , T → ∞ ), user behavior cannot be described by a continuous LV system, which implies ε LV > 0 (see Appendix Figure 5).

Original: Since the baseline
Revised: Since the baseline

Original: RMSE is high, we set β ui ∝ ˜ r 2 u,x i , where ˜ r u,x i = κr u,x i + (1 − κ ) u ⊤ x , so that κ interpolates327 between predicted ratings ( κ = 0 ) and true ratings ( κ = 1 ).
Revised: RMSE is high, we set β ui ∝ ˜ r 2 u,x i , where ˜ r u,x i = κr u,x i + (1 − κ ) u ⊤ x , so that κ interpolates between predicted ratings ( κ = 0 ) and true ratings ( κ = 1 ).

Original: This allows us to (indirectly) controlε pred .
Revised: This allows us to (indirectly) control ε pred .

Original: For all experiments we use T = 100 , and so expect a roughly fixed ε dev > 0 .
Revised: For all experiments we use T = 100 , and so expect a roughly fixed ε dev > 0 .

Original: Methods.
Revised: Methods.

Original: We compare our approach ( LV ) to several baselines: (i) a default policy which myopically optimizes for immediate engagement (and so does not break); (ii) a ‘safety switch’ policy[Figure] e
Revised: We compare our approach ( LV ) to several baselines: (i) a default policy which myopically optimizes for immediate engagement (and so does not break); (ii) a ‘safety switch’ policy

Original: [Figure]
Revised: [Figure]

Original: a

Original: [Figure]
Revised: [Figure]

Original: a f e

Original: [Figure]
Revised: [Figure]

Original: t

Original: [Figure]
Revised: [Figure]

Original: o f

Original: [Figure]
Revised: [Figure]

Original: V o

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: % % i
Revised: % % i

Original: [Figure]
Revised: [Figure]

Original: p r o v e m e n t ( v s .
Revised: p r o v e m e n t ( v s .

Original: d e f a u l t )
Revised: d e f a u l t )

Original: Engagementfor diﬀerent policies ( κ = 0 . 5 )
Revised: Engagement

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure] [Figure] [Figure]
Revised: diﬀerent policies ( κ

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: schedule
Revised: decision

Original: [Figure]
Revised: [Figure]

Original: LV decision rule [Figure] [Figure] [Figure]
Revised: Engagement

Original: [Figure]
Revised: [Figure]

Original: Engagement for varyingp
Revised: varying

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: ( safety@ τ ) that breaks once consumption surpasses a threshold τ ; (iii) a prediction-based policy ( best-of ) that chooses the best observed p u = arg max p j f j ( u ) (rather than optimizing overp u ∈ [0 , 1] ); and (iv) an oracle benchmark which directly optimizes the (generally unknown) trueunderlying dynamics.
Revised: ( safety@ τ ) that breaks once consumption surpasses a threshold τ ; (iii) a prediction-based policy ( best-of ) that chooses the best observed p u = arg max p j f j ( u ) (rather than optimizing over p u ∈ [0 , 1] ); and (iv) an oracle benchmark which directly optimizes the (generally unknown) true underlying dynamics.

Original: We measure mean long-term engagement rate (LTE) for each approach, andreport averages and standard errors computed over 10 random splits.
Revised: We measure mean long-term engagement rate (LTE) for each approach, and report averages and standard errors computed over 10 random splits.

Original: Performance is measured relativeto the default baseline as it represents no change in policy (typical absolute values are LTE ≈ 10 ).
Revised: Performance is measured relative to the default baseline as it represents no change in policy (typical absolute values are LTE ≈ 10 ).

Original: 5.2 Results and analysis
Revised: 5.2 Results and analysis

Original: Main results.
Revised: Main results.

Original: Figure 3 (left) compares the performance of our method to other policies.
Revised: Figure 3 (left) compares the performance of our method to other policies.

Original: Here weset p 0 = 0 , use N = 3 with p j ∈ { 0 . 05 , 0 . 1 , 0 . 15 } , and consider κ = 0 .
Revised: Here we set p 0 = 0 , use N = 3 with p j ∈ { 0 . 05 , 0 . 1 , 0 . 15 } , and consider κ = 0 .

Original: 5 (note κ affects all policies).
Revised: 5 (note κ affects all policies).

Original: As can be seen, our approach significantly improves over default (+6.5%).
Revised: As can be seen, our approach significantly improves over default (+6.5%).

Original: For safety@ τ ,improvement over the optimal τ = 

Revised: For safety@ τ , improvement over the optimal τ =

Original: (+5.73%; chosen in hindsight) shows the importance of beingpreemptive; for the slightly smaller τ = 14 , breaks are harmful.
Revised: (+5.73%; chosen in hindsight) shows the importance of being preemptive; for the slightly smaller τ = 14 , breaks are harmful.

Original: The gap from best-of (+2.21%)quantifies the gain from the optimization step in Eq.
Revised: The gap from best-of (+2.21%) quantifies the gain from the optimization step in Eq.

Original: (9), and the close performance to oracle(-1.19%) suggests that optimizing the empirical curve (Eq.
Revised: (9), and the close performance to oracle (-1.19%) suggests that optimizing the empirical curve (Eq.

Original: (8)) works well as as proxy.
Revised: (8)) works well as as proxy.

Original: User types.
Revised: User types.

Original: Figure 3 (center) shows for our approach how gain in LTE varies across learnedbreaking policies ˆ p u > 0 .
Revised: Figure 3 (center) shows for our approach how gain in LTE varies across learned breaking policies ˆ p u > 0 .

Original: For increasingly-accurate predictions ( κ ∈ { 0 , 0 .
Revised: For increasingly-accurate predictions ( κ ∈ { 0 , 0 .

Original: 5 ,1 } ), the main plot347 shows performance gains for each group of users, partitioned by their ˆ p u values (binned; plot showsaverage and unit standard deviation per bin).
Revised: , the main plot shows performance gains for each group of users, partitioned by their ˆ p u values (binned; plot shows average and unit standard deviation per bin).

Original: Gains until ˆ p u ≤ 0 .
Revised: Gains until ˆ p u ≤ 0 .

Original: 15 are mild, but for ˆ p u > 0 . 15 , thegeneral trend is positive: users who are deemed to require more frequent breaks, benefit more frombreaking.
Revised: 15 are mild, but for ˆ p u > 0 . 15 , the general trend is positive: users who are deemed to require more frequent breaks, benefit more from breaking.

Original: Gains until ˆ p u ≤ 0 .
Revised: Gains until ˆ p u ≤ 0 .

Original: 3 increase for all κ , but for ˆ p u > 0 . 3 , extrapolation becomes difficult:note the higher variation within each κ , as well as significant differences across κ .
Revised: 3 increase for all κ , but for ˆ p u > 0 . 3 , extrapolation becomes difficult: note the higher variation within each κ , as well as significant differences across κ .

Original: This highlights theimportance of accurate predictions for inferring optimal ˆ p u when the experimental p i are small.
Revised: This highlights the importance of accurate predictions for inferring optimal ˆ p u when the experimental p i are small.

Original: Theinlaied plot shows that, in line with our theory, ˆ p u exhibits an empirical phase shift in the estimated ˆ θ u .
Revised: The inlaied plot shows that, in line with our theory, ˆ p u exhibits an empirical phase shift in the estimated ˆ θ u .

Original: Treatments.
Revised: Treatments.

Original: Figure 3 (right) shows the effects of experimental treatments on performance.
Revised: Figure 3 (right) shows the effects of experimental treatments on performance.

Original: Focusingon N = 2 , we fix p 

Revised: Focusing on N = 2 , we fix p

Original: = 0 , and consider increasingly aggressive experimentation by varyingp 1 ∈ (0 , 0 . 4] .
Revised: = 0 , and consider increasingly aggressive experimentation by varying p 1 ∈ (0 , 0 . 4] .

Original: For our approach, increasing p 1 helps, which is anticipated by our theoretical bound.
Revised: For our approach, increasing p 1 helps, which is anticipated by our theoretical bound.

Original: For the best-of approach, larger p 1 also helps, but exhibits population-level optimum ( p 1 ≈ 0 . 24 ),which is easy to ‘overshoot’.
Revised: For the best-of approach, larger p 1 also helps, but exhibits population-level optimum ( p 1 ≈ 0 . 24 ), which is easy to ‘overshoot’.

Original: Note that when prediction accuracy is low ( κ = 0 ), experimentation isessential: if p 1 is not sufficiently large, then performance can sharply deteriorate.
Revised: Note that when prediction accuracy is low ( κ = 0 ), experimentation is essential: if p 1 is not sufficiently large, then performance can sharply deteriorate.

Original:  Discussion
Revised:  Discussion

Original: Our paper studies the novel problem of learning optimal breaking policies for recommendation.
Revised: Our paper studies the novel problem of learning optimal breaking policies for recommendation.

Original: We posit a tight connection between long-term engagement and user well-being, and argue thatoptimizing the former requires careful consideration of the latter.
Revised: We posit a tight connection between long-term engagement and user well-being, and argue that optimizing the former requires careful consideration of the latter.

Original: Viewing cognitive capacity as alimited but conservable resource, we propose Lotka-Volterra dynamics as a behavioral model thatenables effective, responsible, and sustainable optimization of recommendation ecosystems.
Revised: Viewing cognitive capacity as a limited but conservable resource, we propose Lotka-Volterra dynamics as a behavioral model that enables effective, responsible, and sustainable optimization of recommendation ecosystems.

Original: As any policy task that involves humans, care must be taken regarding potential risks.
Revised: As any policy task that involves humans, care must be taken regarding potential risks.

Original: While our experimentsshow that optimizing engagement also improves well-being, this need not always be the case; infact, merely measuring well-being in reality is challenging.
Revised: While our experiments show that optimizing engagement also improves well-being, this need not always be the case; in fact, merely measuring well-being in reality is challenging.

Original: Breaks, as interventions, are presumably‘safe’, in the sense that at worst they may lead to suboptimal performance (for system) or satisfaction(for users).
Revised: Breaks, as interventions, are presumably ‘safe’, in the sense that at worst they may lead to suboptimal performance (for system) or satisfaction (for users).

Original: But breaks can also be used nefariously, e g., by enabling temporally-varying rewards(Eyal, 2019).
Revised: But breaks can also be used nefariously, e.g., by enabling temporally-varying rewards (Eyal, 2019).

Original: As such, breaking policies should be deployed accountably and transparently.
Revised: As such, breaking policies should be deployed accountably and transparently.

Original: A Deferred proofs 
Revised: A Deferred proofs A.1 Properties of Lotka-Volterra systems

Original: A.1 Properties of Lotka-Volterra systems

Original: Definition A.1 (Static policy equilibrium) .
Revised: Definition A.1 (Static policy equilibrium) .

Original: Let λ ( t ) , q ( t ) denote a Lokta-Volterra model characterizedby parameters θ =
Revised: Let λ ( t ) , q ( t ) denote a Lokta-Volterra model characterized by parameters θ =

Original: ( α, β, γ, δ ) ∈ R 4+ , as defined in Equation 2.
Revised: ( α, β, γ, δ ) ∈ R 4+ , as defined in Equation 2.

Original: Let p ∈
Revised: Let p ∈

Original: [0 , 1] , and denote by π p thestatic policy corresponding to p .
Revised: [0 , 1] , and denote by π p the static policy corresponding to p .

Original: For λ (0) , q (0) > 0 , the static equilibrium of the system is definedas:
Revised: For λ (0) , q (0) > 0 , the static equilibrium of the system is defined as:

Original: [Equation]
Revised: [Equation]

Original: We denote λ ∗ ( p ) =
Revised: We denote λ ∗ ( p ) =

Original: λ ∗ ( p ; θ ) when θ is clear from the context.
Revised: λ ∗ ( p ; θ ) when θ is clear from the context.

Original: We denote λ ∗ ( p ; u ) =
Revised: We denote λ ∗ ( p ; u ) =

Original: λ ∗ ( p ; θ u ) whena user u ∈ U characterized by parameters θ u is given and clear from the context.
Revised: λ ∗ ( p ; θ u ) when a user u ∈ U characterized by parameters θ u is given and clear from the context.

Original: Proposition A.1 (Global stability) .
Revised: Proposition A.1 (Global stability) .

Original: λ ∗ ( p ; θ ) exists and uniquely defined for all θ ∈ R 4+ , p ∈ [0 , 1]and for all initial conditions λ (0) , q (0) > 0 .
Revised: λ ∗ ( p ; θ ) exists and uniquely defined for all θ ∈ R 4+ , p ∈ [0 , 1] and for all initial conditions λ (0) , q (0) > 0 .

Original: Proof.
Revised: Proof.

Original: See (Takeuchi, 1996, Section 3.2).
Revised: See (Takeuchi, 1996, Section 3.2).

Original: Lemma A.1 (Equilibrium of LV behavioral model.
Revised: Lemma A.1 (Equilibrium of LV behavioral model.

Original: Formal proof of Lemma 1) .
Revised: Formal proof of Lemma 1) .

Original: Assume a LoktaVolterra model characterized by θ =
Revised: Assume a LoktaVolterra model characterized by θ =

Original: ( α, β, γ, δ ) ∈ R 4+ , and let p ∈ [0 , 1] denote the proportion ofinteractions in which a forced break is served.
Revised: ( α, β, γ, δ ) ∈ R 4+ , and let p ∈ [0 , 1] denote the proportion of interactions in which a forced break is served.

Original: The static equilibrium of the model under static policyπ p is given by:
Revised: The static equilibrium of the model under static policy π p is given by:

Original: [Equation]
Revised: [Equation]

Original: otherwise
Revised: otherwise

Original: [Equation]
Revised: [Equation]

Original: otherwise
Revised: otherwise

Original: Proof.
Revised: Proof.

Original: The LV dynamical system is given by Equation 3:
Revised: The LV dynamical system is given by Equation 3:

Original: [Equation]
Revised: [Equation]

Original: when p ∈ (cid:104) 0 , 1 − αβ (cid:105) we equate d λ d t = 0 , d q d t = 0 and obtain the result.
Revised: when p ∈ (cid:104) 0 , 1 − αβ (cid:105) we equate d λ d t = 0 , d q d t = 0 and obtain the result.

Original: The solution is guaranteed tobe valid, as both λ ∗ ( p ) > 0 and q ∗ ( p ) ∈ [0 , 1] .
Revised: The solution is guaranteed to be valid, as both λ ∗ ( p ) > 0 and q ∗ ( p ) ∈ [0 , 1] .

Original: Conversely, when p / ∈ (cid:104) 0 , 1 − αβ (cid:105) , there exists ϵ > 0 such that dd t log λ < − ϵ < 0 for all λ > 0 ,q ∈ [0 , 1] .
Revised: Conversely, when p / ∈ (cid:104) 0 , 1 − αβ (cid:105) , there exists ϵ > 0 such that dd t log λ < − ϵ < 0 for all λ > 0 , q ∈ [0 , 1] .

Original: From this we obtain that log λ ( t ) tends towards −∞ , and therefore λ ( t ) tends towards0 , and λ ∗ ( p ) = 0 as required.
Revised: From this we obtain that log λ ( t ) tends towards −∞ , and therefore λ ( t ) tends towards 0 , and λ ∗ ( p ) = 0 as required.

Original: When λ ( t ) is close to zero, the interaction terms vanish in the d q d tequation, and q ( t ) grows logistically towards 1 .
Revised: When λ ( t ) is close to zero, the interaction terms vanish in the d q d t equation, and q ( t ) grows logistically towards 1 .

Original: Proposition A.2 (Equilibrium bounds) .
Revised: Proposition A.2 (Equilibrium bounds) .

Original: For a Lotka-Volterra model, the static equilibrium λ ∗ ( p ) is bounded by:
Revised: For a Lotka-Volterra model, the static equilibrium λ ∗ ( p ) is bounded by:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Denote x = 11 − p .
Revised: Denote x = 11 − p .

Revised: From Lemma A.1, for x ∈

Original: From Lemma A.1, for x ∈ (cid:104) 1 , βα (cid:105) the equilibrium consumption λ ∗ ( x ) is given by:
Revised: [Equation] βα [Equation] the equilibrium consumption λ ∗ ( x ) is given by:

Original: [Equation]
Revised: [Equation]

Original: and is zero otherwise.
Revised: and is zero otherwise.

Original: The equilibrium is a quadratic function of x with roots x ∈ (cid:110) 0 , βα (cid:111) , andtherefore attains its maximum at x = β 2 α .
Revised: The equilibrium is a quadratic function of x with roots x ∈ (cid:110) 0 , βα (cid:111) , and therefore attains its maximum at x = β 2 α .

Original: Plugging back the maximizing x into λ ∗ we obtain the upper bound.
Revised: Plugging back the maximizing x into λ ∗ we obtain the upper bound.

Original: Lower bound is attained as the equilibrium in Lemma A.1 is clipped by 0 frombelow.
Revised: Lower bound is attained as the equilibrium in Lemma A.1 is clipped by 0 from below.

Original: Lemma A.2 (Optimal static policy.
Revised: Lemma A.2 (Optimal static policy.

Original: Formal proof of Lemma 2) .
Revised: Formal proof of Lemma 2) .

Original: The optimal static policy for a
Revised: The optimal static policy for a

Original: Lotka-Volterra system is given by:
Revised: Lotka-Volterra system is given by:

Original: [Equation]
Revised: [Equation]

Original: And the optimal equilibrium engagement rate is given by:
Revised: And the optimal equilibrium engagement rate is given by:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Denote x = 11 − p .
Revised: Denote x = 11 − p .

Original: From Proposition A.2, the global maximum of λ ∗ ( x ) is attained at x = β 2 α .
Revised: From Proposition A.2, the global maximum of λ ∗ ( x ) is attained at x = β 2 α .

Original: Consider two cases: When αβ ≤ 12 , we obtain that x opt =
Revised: Consider two cases: When αβ ≤ 12 , we obtain that x opt =

Original: β 2 α ≥ 1 , and therefore p opt = 1 − 1 x ∈ [0 , 1] .
Revised: β 2 α ≥ 1 , and therefore p opt = 1 − 1 x ∈ [0 , 1] .

Original: From this we obtain that in this case the global maximum is attained on the simplex, and given by theformula from Proposition A.2.
Revised: From this we obtain that in this case the global maximum is attained on the simplex, and given by the formula from Proposition A.2.

Original: Conversely, when αβ > 12 , we obtain p = 1 − 1 x < 0 , and thereforex opt translates to a negative value of p .
Revised: Conversely, when αβ > 12 , we obtain p = 1 − 1 x < 0 , and therefore x opt translates to a negative value of p .

Original: As λ ∗ ( p ) is uni-modal, the optimal policy restricted to thesimplex [0 , 1] in this case is attained on the closest boundary point p = 0 .
Revised: As λ ∗ ( p ) is uni-modal, the optimal policy restricted to the simplex [0 , 1] in this case is attained on the closest boundary point p = 0 .

Original: Figure 2 provides graphical intuition for this proof (left and center subplots).
Revised: Figure 2 provides graphical intuition for this proof (left and center subplots).

Original: Proposition A.3 (Inference of α/β from two-treatment equilibrium data.
Revised: Proposition A.3 (Inference of α/β from two-treatment equilibrium data.

Original: Formal proof of Proposition 1) .
Revised: Formal proof of Proposition 1) .

Original: Let λ ( t ) , q ( t ) be a Lokta-Volterra model, let p 1 , p 2 ∈ [0 , 1] .
Revised: Let λ ( t ) , q ( t ) be a Lokta-Volterra model, let p 1 , p 2 ∈ [0 , 1] .

Original: Denote by λ ∗ ( p 1 ) , λ ∗ ( p 2 ) thestatic equilibrium rates corresponding to static policies π p 1 , π p 2 , and assume λ ∗ ( p 1 ) , λ ∗ ( p 2 ) > 0 .[-
-]
Revised: Denote by λ ∗ ( p 1 ) , λ ∗ ( p 2 ) the static equilibrium rates corresponding to static policies π p 1 , π p 2 , and assume λ ∗ ( p 1 ) , λ ∗ ( p 2 ) > 0 .[-
-]

Original: The parameter ratio αβ is given by the following formula:
Revised: The parameter ratio αβ is given by the following formula:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: From Lemma A.1, the equilibrium consumption λ ∗ ( p ) is given by:
Revised: From Lemma A.1, the equilibrium consumption λ ∗ ( p ) is given by:

Original: [Equation]
Revised: [Equation]

Original: When λ ∗ ( p i ) is observed for different policies p 1 , . .
Revised: When λ ∗ ( p i ) is observed for different policies p 1 , . .

Original: , p m ∈ (cid:104) 0 , 1 − αβ (cid:105) , we obtain a polynomialregression problem for the parameters αβ and αβ γδ , which can be solved e.g using Non-Negative Least
Revised: , p m ∈ (cid:104) 0 , 1 − αβ (cid:105) , we obtain a polynomial regression problem for the parameters αβ and αβ γδ , which can be solved e [Equation] g using Non-Negative Least

Original: Squares.
Revised: Squares.

Original: When m = 2 , we obtain a system of two linear equations.
Revised: When m = [Equation] we obtain a system of two linear equations.

Original: Apply Cramer’s rule to obtain: 
Revised: Apply Cramer’s rule to obtain: γ

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: And therefore αβ is given by:
Revised: And therefore αβ is given by:

Original: [Equation]
Revised: [Equation]

Original: A.2 Model fitting from engagement predictions

Original: Notations.
Revised: Notations.

Original: In this section only, we use the common notation q = 1 − p to denote complementaryprobabilities.
Revised: In this section only, we use the common notation q = 1 − p to denote complementary probabilities.

Original: Definition A.2 (Empirical value of α/β ) .
Revised: Definition A.2 (Empirical value of α/β ) .

Original: For single-channel experiments with forced-break probabilities p 1 , p 2 , denote λ i =
Revised: For single-channel experiments with forced-break probabilities p 1 , p 2 , denote λ i =

Original: λ ∗ ( p i ) , f i =
Revised: λ ∗ ( p i ) , f i =

Original: f p i ( u ) , q i = 1 − p i .
Revised: f p i ( u ) , q i = 1 − p i .

Original: The empirical value of the αβ parameteris given by the following formula:
Revised: The empirical value of the αβ parameter is given by the following formula:

Original: [Equation]
Revised: [Equation]

Original: Proposition A.4 ( α/β estimation error from prediction errors) .
Revised: Proposition A.4 ( α/β estimation error from prediction errors) .

Original: Given a single-channel Lokta-Volterrasystem with parameter αβ ≥ 1 .
Revised: Given a single-channel Lokta-Volterra system with parameter αβ ≥ 1 .

Original: Let p 1 , p 2 ∈ (cid:104) 1 , αβ (cid:105) , denote λ ∗ i = λ
Revised: Let p 1 , p 2 ∈ (cid:104) 1 , αβ (cid:105) , denote λ ∗ i = λ

Original: ∗ ( p i ) ∈
Revised: ∗ ( p i ) ∈

Original: R + , and let f i
Revised: R + , and let f i

Original: = λ ∗ i + ε ibe the predicted engagement rates corresponding to p 1 , p 2 .
Revised: = λ ∗ i + ε i be the predicted engagement rates corresponding to p 1 , p 2 .

Original: When | ε 1 | , | ε 2 | ≤ ε ≤ γδ | p 1 − p 2 |, theestimation error is bounded by:
Revised: When | ε 1 | , | ε 2 | ≤ ε ≤ γδ | p 1 − p 2 |, the estimation error is bounded by:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: denote q i =
Revised: denote q i =

Original: − p i .
Revised: − p i .

Original: The value of αβ is given by Proposition A.3:
Revised: The value of αβ is given by Proposition A.3:

Original: [Equation]
Revised: [Equation]

Original: And the estimator for αβ is obtained by replacing the true value with their predictions:
Revised: And the estimator for αβ is obtained by replacing the true value with their predictions:

Original: [Equation]
Revised: [Equation]

Original: The estimation error is given by:
Revised: The estimation error is given by:

Original: [Equation]
Revised: [Equation]

Original: We now proceed to bound each factor:
Revised: We now proceed to bound each factor:

Original: • For (i), the term ( q 1 q 2 ) 2 is bounded by 1 since q 1 , q 2 ∈ [0 , 1] .
Revised: • For (i), the term ( q 1 q 2 ) 2 is bounded by 1 since q 1 , q 2 ∈ [0 , 1] .

Original:  For (ii), the term (cid:12)(cid:12)(cid:12) q 1 − q 2 q 21 λ ∗ 1 − q 22 λ ∗(cid:12)(cid:12)(cid:12) is equal to (cid:0) γδ (cid:1) − 1 by Eq.
Revised: • For (ii), the term [Equation] − q 2 q 21 λ ∗ 1 − q 22 λ ∗(cid:12)(cid:12)(cid:12) is equal to (cid:0) γδ (cid:1) − 1 by Eq.

Original: (11).• For (iii), from Proposition A.2 we obtain the bound 0 ≤ λ ∗ i ≤ βγ 4 αδ , and therefore the term| ε 2 λ ∗ 1 − ε 1 λ ∗ 2 | is bounded by(cid:16) βγ 4 αδ (cid:17) ε = βγ 2 αδ ε . • For (iv), the term
Revised: • For (iii), from Proposition A.2 we obtain the bound 0 ≤ λ ∗ i ≤ βγ 4 αδ , and therefore the term | ε 2 λ ∗ 1 − ε 1 λ ∗ 2 | is bounded by

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: Note
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: p− p 2 (cid:12) (cid:12)(cid:12) ≤ 2 ε | p 1 − p 2 | .
Revised: − p 2 (cid:12) (cid:12)(cid:12) ≤ 2 ε | p 1 − p 2 | .

Original: When ε is small enough, and specifically when the boundε ≤ γ δ | p 1 − p 2 |holds, we obtain:(cid:12)(cid:12)(cid:12)(cid:12)
Revised: When ε is small enough, and specifically when the bound ε ≤ γ δ | p 1 − p 2 |holds, we obtain:

Original: [Equation]
Revised: [Equation]

Original: (cid:12)(cid:12)(cid:12)(cid:12)
Revised: (cid:12)(cid:12)(cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Original: (cid:12)(cid:12)(cid:12)(cid:12)
Revised: (cid:12)(cid:12)(cid:12)(cid:12)

Original: [Equation]

Original: [Equation]

Original: (cid:12)(cid:12)(cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Original: δ γ and therefore:(iv) ≤
Revised: δ γ and therefore:

Original: [Equation]
Revised: [Equation]

Original: Aggregating results (i)-(iv) above, we obtain the overall bound:(cid:12)(cid:12)(cid:12)(cid:12)
Revised: Aggregating results (i)-(iv) above, we obtain the overall bound:

Original: [Equation]
Revised: [Equation]

Original: (cid:12)(cid:12)(cid:12)(cid:12)

Original: [Equation]

Original: [Equation]

Original: (cid:124) (cid:123)(cid:122) (cid:125)

Original: [Equation]
Revised: [Equation]

Original: (cid:12)(cid:12)(cid:12)(cid:12)
Revised: (cid:12)(cid:12)(cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Original: (cid:12)(cid:12)(cid:12)(cid:12) (cid:124) (cid:123)(cid:122) (cid:125) [Equation] (cid:124) (cid:123)(cid:122) (cid:125) [Equation]
Revised: ≤

Original: (cid:12)(cid:12)(cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Original: 21 λ

Original: [Equation]

Original: 1 −

Original: [Equation]

Original: 22 λ [Equation] [Equation] 21 ε
Revised: αγ

Original: [Equation]

Original: − q 22 ε 2 ) (cid:12)(cid:12)(cid:12)(cid:12) (cid:124) (cid:123)(cid:122) (cid:125) ≤ 2 | p 1 − p 2 | δγ ≤ ε | p 1 − p 2 | βδ αγ

Original: Proposition A.5 (Cost of α/β estimation error) .
Revised: Proposition A.5 (Cost of α/β estimation error) .

Original: Let αβ be the engagement ratio parameter of aone-channel Lotka-Volterra system, and let ˆ (cid:16) αβ (cid:17) be an estimate of these parameters.
Revised: Let αβ be the engagement ratio parameter of a one-channel Lotka-Volterra system, and let ˆ (cid:16) αβ (cid:17) be an estimate of these parameters.

Original: Let λ ∗ opt be theengagement rate of the optimal static policy, and denote λ ∗ ( x )
Revised: Let λ ∗ opt be the engagement rate of the optimal static policy, and denote λ ∗ ( x )

Original: =
Revised: =

Original: λ ∗ (ˆ p ( x )) .
Revised: λ ∗ (ˆ p ( x )) .

Original: When (cid:12)(cid:12)(cid:12)(cid:12) αβ − ˆ (cid:16) αβ 
Revised: When (cid:12)(cid:12)(cid:12)(cid:12) αβ − ˆ (cid:16) αβ (

Original: (cid:17)(cid:12)(cid:12)(cid:12)(cid:12) ≤min (cid:110)
Revised: cid:17)(cid:12)(cid:12)(cid:12)(cid:12) ≤ min

Original: α 2 β ,(cid:111)
Revised: [Equation] (cid:111)

Original: The price of estimation error is bounded by:
Revised: The price of estimation error is bounded by:

Original: [Equation]
Revised: [Equation]

Original: (cid:12)(cid:12)

Original: [Equation]

Original: (cid:41)

Original: Proof.
Revised: Proof.

Original: Denote r = αβ , x
Revised: Denote r = αβ , x

Original: = ˆ (cid:16) αβ (cid:17) , and assume without loss of generality that γδ = 1 and r ≤ 1 .
Revised: = ˆ (cid:16) αβ (cid:17) , and assume without loss of generality that γδ = 1 and r ≤ 1 .

Original: Theoptimal equilibrium engagement rate is given by:
Revised: The optimal equilibrium engagement rate is given by:

Original: [Equation]
Revised: [Equation]

Original: The chosen policy ˆ p ( x ) is given by:
Revised: The chosen policy ˆ p ( x ) is given by:

Original: [Equation]
Revised: [Equation]

Original: otherwise
Revised: otherwise

Original: [Figure]
Revised: [Figure]

Original: −
Revised: −

Original: [Figure]
Revised: [Figure]

Original: −
Revised: −

Original: [Figure]
Revised: [Figure]

Original: − [Figure] .∆
Revised: − 0 .0 .0 .0 .0 .∆

Original: αβ
Revised: αβ

Original: =
Revised: =

Original: ˆ (cid:16) αβ (cid:17) − αβ 0 .0 .0 .

Revised: ˆ (cid:16) αβ (cid:17) − αβ 0 .0 .0 .

Original: .∆
Revised: .∆

Original: λ ∗ =
Revised: λ ∗ =

Original: λ ∗ ( p ∗ ) − λ ∗ ( ˆ p )
Revised: λ ∗ ( p ∗ ) − λ ∗ ( ˆ p )

Original: Cost of estimation error for diﬀerent values of αβ , and the corresponding upper bounds
Revised: Cost of estimation error for diﬀerent values of αβ , and the corresponding upper bounds

Original: [Figure]
Revised: [Figure]

Original: Assume without loss of generality that x ∈ (cid:2) 0 ,(cid:3) , as values of x outside the interval can be clippedto its edges without affecting the result.
Revised: Assume without loss of generality that x ∈ (cid:2) 0 ,(cid:3) , as values of x outside the interval can be clipped to its edges without affecting the result.

Original: The equilibrium engagement rate of the selected policy isgiven by:
Revised: The equilibrium engagement rate of the selected policy is given by:

Original: [Equation]
Revised: [Equation]

Original: (cid:1)

Original: [Equation]

Original: (cid:0)

Original: [Equation]

Original: ,(cid:3)

Original: Denote ∆( x ) =
Revised: Denote ∆( x ) =

Original: λ ∗ opt − λ ∗ ( x ) .
Revised: λ ∗ opt − λ ∗ ( x ) .

Original: We obtain:
Revised: We obtain:

Original: [Equation]
Revised: [Equation]

Original: (cid:0) 0 ,(cid:3) , x ∈ (cid:2) 0 , r
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: (cid:3)
Revised: r

Original: [Equation]
Revised: [Equation]

Original: r ∈ (cid:0) 0 ,(cid:3) , x ∈ (cid:0) r 2 ,(cid:3) (1 − r ) r ∈ (cid:0) 12 ,(cid:3) , x ∈ (cid:2) 0 , r(cid:3) (1 − r )
Revised: (cid:0) 12 ,(cid:3) , x ∈ (cid:2) 0 , r(cid:3) [Equation] [Equation] [Equation]

Revised: (cid:0)

Original: [Equation]
Revised: [Equation]

Original: Observe that 14 r ≥ 1 − r for all r ∈ (0 , 1] , and therefore we obtain for all x, r :[Equation]
Revised: Observe that 14 r ≥ 1 − r for all r ∈ (0 , 1] , and therefore we obtain for all x, r :

Original: [Equation]
Revised: [Equation]

Original: From the convexity of ∆( x ) in the region around x = r we obtain:
Revised: From the convexity of ∆( x ) in the region around x = r we obtain:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Finally, combining the two bounds yields the final result.
Revised: Finally, combining the two bounds yields the final result.

Original: A geometric interpretation of this claim isillustrated in Figure 4.
Revised: A geometric interpretation of this claim is illustrated in Figure 4.

Original: A.3 Optimal stationary policy from engagement predictions
Revised: A.3 Optimal stationary policy from engagement predictions

Original: Definition A.3 (Expected observable rate) .
Revised: Definition A.3 (Expected observable rate) .

Original: Let u ∈ U , p ∈ [0 , 1] , and T > 0 .
Revised: Let u ∈ U , p ∈ [0 , 1] , and T > 0 .

Original: Let p ∈
Revised: Let p ∈

Original: [0 , 1] , denote the corresponding static policy by π p .
Revised: [0 , 1] , denote the corresponding static policy by π p .

Original: The expected observable rate ¯ λ u ( p ; T ) is defined as:
Revised: The expected observable rate ¯ λ u ( p ; T ) is defined as:

Original: [Equation]
Revised: [Equation]

Original: where expectation is taken over the stochastic decisions of π p .
Revised: where expectation is taken over the stochastic decisions of π p .

Original: Definition A.4 (Lokta-Volterra approximation of TPP ) .
Revised: Definition A.4 (Lokta-Volterra approximation of TPP ) .

Original: Let u ∈ U , and T > 0 .
Revised: Let u ∈ U , and T > 0 .

Original: Denote by p ∗ the maximizer of expected observable rate:
Revised: Denote by p ∗ the maximizer of expected observable rate:

Original: [Equation]
Revised: [Equation]

Original: The LV approximation of TPP( u ; T ) is defined as:
Revised: The LV approximation of TPP( u ; T ) is defined as:

Original: [Equation]
Revised: [Equation]

Original: such that arg max p λ ∗ ( p ; θ ) = p ∗ .
Revised: such that arg max p λ ∗ ( p ; θ ) = p ∗ .

Original: The corresponding approximation error is defined as:
Revised: The corresponding approximation error is defined as:

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: Notations.
Revised: Notations.

Original: When u is clear from the context, we denote θ ∗ =
Revised: When u is clear from the context, we denote θ ∗ =

Original: θ ∗ u , ε LV = ε LV ,u .
Revised: θ ∗ u , ε LV = ε LV ,u .

Original: We use α ∗ , β ∗ , . . 
Revised: We use α ∗ , β ∗ , . . .

Original: .to refer to the corresponding parts of the Lokta-Volterra parameters vector θ ∗ .
Revised: to refer to the corresponding parts of the Lokta-Volterra parameters vector θ ∗ .

Original: We are now ready to state and prove the main theorem for this section:
Revised: We are now ready to state and prove the main theorem for this section:

Original: Theorem A.1 (Regret bound for learned static policy.
Revised: Theorem A.1 (Regret bound for learned static policy.

Original: Formal version of Theorem 1) .
Revised: Formal version of Theorem 1) .

Original: Let p 1 , p 2 ∈[0 , 1] denote two static forced-break policies, and denote by U the set of users, and assume theyremain engaged under the stationary policies π ( p 1 ) and π ( p 2 ) .
Revised: Let p 1 , p 2 ∈ [0 , 1] denote two static forced-break policies, and denote by U the set of users, and assume they remain engaged under the stationary policies π ( p 1 ) and π ( p 2 ) .

Original: Assume S u
Revised: Assume S u

Original: ( p ; T ) ∼ TPP
Revised: ( p ; T ) ∼ TPP

Original: π p ◦ ψ ( u ; T ) ,and let µ = 
Revised: π p ◦ ψ ( u ; T ) , and let µ = (cid:16)

Original: [Equation]
Revised: [Equation]

Original: Let f p 1 , f p 2 : U → R + be functions predicting 1 T | S u ( p 1 ; T ) | , 1 T | S u ( p 2 ; T ) | , respectively.
Revised: Let f p 1 , f p 2 : U → R + be functions predicting 1 T | S u ( p 1 ; T ) | , 1 T | S u ( p 2 ; T ) | , respectively.

Original: Denote585 the learned policy by ˆ p , and the optimal policy by p ∗ .
Revised: Denote the learned policy by ˆ p , and the optimal policy by p ∗ .

Original: If (i) the expected RMSE of f p 1 , f p 2 is bounded by ε pred , (ii) the average absolute deviation of1 T | TPP( u ; T ) | is bounded by ε dev , and (iii) the expected LV approximation error of the system isbounded by ε LV , then the learned policy ˆ p has bounded regret:
Revised: If (i) the expected RMSE of f p 1 , f p 2 is bounded by ε pred , (ii) the average absolute deviation of T | TPP( u ; T ) | is bounded by ε dev , and (iii) the expected LV approximation error of the system is bounded by ε LV , then the learned policy ˆ p has bounded regret:

Original: [Equation]
Revised: [Equation]

Original: where expectation is taken over stochastic choices of policies, and η TPP = g ( µ, ν ) ∈ poly( µ, ν ) .
Revised: where expectation is taken over stochastic choices of policies, and η TPP = g ( µ, ν ) ∈ poly( µ, ν ) .

Original: Proof.
Revised: Proof.

Original: By assumption (i), the functions
Revised: By assumption (i) [Equation] the functions

Original: f p 1 , f p 2 have bounded expected RMSE:
Revised: f p 1 , f p 2 have bounded expected RMSE:

Original: [Equation]
Revised: [Equation]

Original: Applying Jensen’s inequality with the convex function φ ( x ) = x 2 yields:
Revised: Applying Jensen’s inequality with the convex function φ ( x ) = x 2 yields:

Original: [Equation]
Revised: [Equation]

Original: Combining with Eq.
Revised: Combining with Eq.

Original: (15) and taking the square root, we obtain an upper bound on the expectedabsolute error:
Revised: (15) and taking the square root [Equation] we obtain an upper bound on the expected absolute error:

Original: [Equation]
Revised: [Equation]

Original: Let ∆ f = | f p i ( u ) − λ ∗ ( p i ) | apply the triangle inequality to obtain:[Equation] i
Revised: Let ∆ f = | f p i ( u ) − λ ∗ ( p i ) | apply the triangle inequality to obtain:

Original: [Equation]
Revised: [Equation]

Original: (cid:12)(cid:12)

Original: Denote ε f = ε pred + ε dev + ε LV .
Revised: Denote ε f = ε pred + ε dev + ε LV .

Original: Applying the triangle inequality and using the bounds in Eq.
Revised: Applying the triangle inequality and using the bounds in Eq.

Original: (16)together with assumptions (ii), (iii), we obtain:
Revised: (16) together with assumptions (ii), (iii), we obtain:

Original: [Equation]
Revised: [Equation]

Original: Denote θ ∗ u
Revised: Denote θ ∗ u

Original: = ( α, β, γ, δ ) .
Revised: = ( α, β, γ, δ ) .

Original: The empirical value ˆ (cid:16) αβ (cid:17) of (cid:16) αβ (cid:17) is given by Definition A.2.
Revised: The empirical value ˆ (cid:16) αβ (cid:17) of (cid:16) αβ (cid:17) is given by Definition A.2.

Original: Denote theestimation error by ∆ αβ
Revised: Denote the estimation error by ∆ αβ

Original: = (cid:12)(cid:12)(cid:12)
Revised: = (cid:12)(cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Original: ˆ [Equation] (cid:12) .
Revised: (cid:12)(cid:12)(cid:12) (cid:12) .

Original: By Proposition A.4, the following pointwise upper bound on ∆ αβ applies when ∆ f ≤ γδ | p 1 − p 2 |: 
Revised: By Proposition A.4, the following pointwise upper bound on ∆ αβ applies when ∆ f ≤ γδ | p 1 − p 2 |: [Equation] αβ

Original: [Equation]
Revised: [Equation]

Revised: |

Revised: [Equation]

Revised: − p

Revised: [Equation]

Revised: | βδ

Revised: [Equation]

Original: Plugging in the bound on the expected value of ∆ f into Eq.
Revised: Plugging in the bound on the expected value of ∆ f into Eq.

Original: (18), we obtain in expectation:
Revised: (18), we obtain in expectation:

Original: [Equation]
Revised: [Equation]

Original: Next, we apply Proposition A.5.
Revised: Next, we apply Proposition A.5.

Original: Denote ∆ λ ∗ = λ ∗ ( p ∗ ) − λ ∗ (ˆ p ) , and define the following probabilityevent: [Equation] and
Revised: Denote ∆ λ ∗ = λ ∗ ( p ∗ ) − λ ∗ (ˆ p ) , and define the following probability event:

Original: [Equation]
Revised: [Equation]

Original: Note that the bound in Proposition A.5 is represented as a minimum between two functions, onelinear in ε and one constant.
Revised: Note that the bound in Proposition A.5 is represented as a minimum between two functions, one linear in ε and one constant.

Original: To leverage this property, apply the law of total expectation:
Revised: To leverage this property, apply the law of total expectation:

Original: [Equation]
Revised: [Equation]

Original: Under A , the first term in Eq.
Revised: Under A , the first term in Eq.

Original: (20) can be bounded by the linear term in Proposition A.5.
Revised: (20) can be bounded by the linear term in Proposition A.5.

Original: Taking606 P [ A ] ≤ 1 and combining with equation Eq.
Revised: Taking P [ A ] ≤ 1 and combining with equation Eq.

Original: [Equation]
Revised: [Equation]

Original: The expectation factor in the second term of Eq.
Revised: The expectation factor in the second term of Eq.

Original: (20) can be bounded by the constant term in
Revised: (20) can be bounded by the constant term in

Original: Proposition A.5:
Revised: Proposition A.5:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Decompose the probability factor P (cid:2)
Revised: Decompose the probability factor P (cid:2)

Original: ¯
Revised: [Equation]

Original: A (cid:3) using the law of total probability:
Revised: using the law of total probability:

Original: P (cid:2) ¯

Original: A
Revised: P

Original: [Equation]
Revised: [Equation]

Original: Apply Markov’s inequality P [ | X | ≥ a ] ≤ E [ | X | ] a on the probabilities to obtain:[Equation]
Revised: Apply Markov’s inequality P [ | X | ≥ a ] ≤ E [ | X | ] a on the probabilities to obtain:

Original: [Equation]
Revised: [Equation]

Original: Plugging back equations Eq.
Revised: Plugging back equations Eq.

Original: (21), Eq.
Revised: (21), Eq.

Original: (22),Eq.
Revised: (22),Eq.

Original: (23),Eq.
Revised: (23),Eq.

Original: (24) into equation Eq.
Revised: (24) into equation Eq.

Original: (20), we obtain boundsfor each term:
Revised: (20), we obtain bounds for each term:

Original: [Equation]
Revised: [Equation]

Original: by

Original: [Equation]

Original: [Equation] by Eq. (23),Eq. (24) (25) we obtain:
Revised: we obtain:

Original: [Equation]
Revised: [Equation]

Original: To obtain the regret bound on the empirical rates, we apply assumptions (ii), (iii) once again to boundthe expected difference between λ ∗ ( p ) and 1 T | S u ( p ; T ) | , and apply the triangle inequality:
Revised: To obtain the regret bound on the empirical rates, we apply assumptions (ii), (iii) once again to bound the expected difference between λ ∗ ( p ) and 1 T | S u ( p ; T ) | , and apply the triangle inequality:

Original: [Equation]
Revised: [Equation]

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: | p 1 − p 2 | > 1 , as βα ≥ 1 since all the users are assumed to remain engaged in thelong term, and | p 1 − p 2 | ≤ 1 as p 1 , p 2 ∈ [0 , 1] .
Revised: | p 1 − p 2 | > 1 , as βα ≥ 1 since all the users are assumed to remain engaged in the long term, and | p 1 − p 2 | ≤ 1 as p 1 , p 2 ∈ [0 , 1] .

Original: Therefore, the function η TPP = g ( µ, ν ) =(cid:16)
Revised: Therefore, the function η TPP = g ( µ, ν ) = (cid:16) ν

Original: [Equation]
Revised: [Equation]

Original: (cid:17) satisfies:
Revised: satisfies:

Original: [Equation]
Revised: [Equation]

Original: B Experimental detailsB.1 Data
Revised: B Experimental details B.1 Data

Original: We base our experimental environment on the MovieLens 1M dataset, which is a standard benchmarkdataset used widely in recommendation system research (Harper and Konstan, 2015).
Revised: We base our experimental environment on the MovieLens 1M dataset, which is a standard benchmark dataset used widely in recommendation system research (Harper and Konstan, 2015).

Original: The datasetincludes 1,000,209 ratings provided by 6,040 users and for 3,706 movies.
Revised: The dataset includes 1,000,209 ratings provided by 6,040 users and for 3,706 movies.

Original: Rating are in the range{ 1 , . . .
Revised: Rating are in the range { 1 , . . .

Original: , 5 } , and all users in the dataset have at least 20 reported ratings.
Revised: , 5 } , and all users in the dataset have at least 20 reported ratings.

Original: The dataset is publiclyavailable at: https://grouplens .
Revised: The dataset is publicly available at: https://grouplens .

Original: org/datasets/movielens/1m/.
Revised: org/datasets/movielens/1m/.

Original: Data partitioning.
Revised: Data partitioning.

Original: To learn latent user and item features, 30% of all ratings were drawn at random.
Revised: To learn latent user and item features, 30% of all ratings were drawn at random.

Original: Stratified sampling was applied to ensure that all users and items were covered, and so that each usershave roughly the same proportion of ratings used for this step.
Revised: Stratified sampling was applied to ensure that all users and items were covered, and so that each users have roughly the same proportion of ratings used for this step.

Original: These ratings were only used only forlearning a CF model, and were discarded afterwards.
Revised: These ratings were only used only for learning a CF model, and were discarded afterwards.

Original: The remaining 70% data points were used fortraining and testing.
Revised: The remaining 70% data points were used for training and testing.

Original: For these, we first randomly sampled 1,000 users to form the test set.
Revised: For these, we first randomly sampled 1,000 users to form the test set.

Original: Then, theremaining users were partitioned into the main train set S , which included 70% ( ≈ 3,528) of theseusers, and the experimental treatment sets S ( j ) , each including 10% ( ≈ 504) users for N = 3 .
Revised: Then, the remaining users were partitioned into the main train set S , which included 70% ( ≈ 3,528) of these users, and the experimental treatment sets S ( j ) , each including 10% ( ≈ 504) users for N = 3 .

Original: Thisprocedure was repeated 10 times, and we report average results and standard errors.
Revised: This procedure was repeated 10 times, and we report average results and standard errors.

Original: B.2 Implementation details
Revised: B.2 Implementation details

Original: • Hardware : All experiments were run on a single laptop, with 16GB of RAM, M1 Pro processor,and with no GPU support.• Runtime :
Revised: • Hardware : All experiments were run on a single laptop, with 16GB of RAM, M1 Pro processor, and with no GPU support. • Runtime :

Original: A single run consisting the entire pipeline (data loading and partitioning, collaborativefiltering, training classifiers, simulating dynamics, learning policies, measuring and comparingperformance) takes roughly 20 minutes.
Revised: A single run consisting the entire pipeline (data loading and partitioning, collaborative filtering, training classifiers, simulating dynamics, learning policies, measuring and comparing performance) takes roughly 20 minutes.

Original: The main bottleneck is the discrete LV simulation, takingroughly 70% of runtime to compute, mostly due to bookkeeping necessary for the non-stationarybaselines.
Revised: The main bottleneck is the discrete LV simulation, taking roughly 70% of runtime to compute, mostly due to bookkeeping necessary for the non-stationary baselines.

Original: Simulation code was optimized using the NUMBA jit compiler, which improves runtime.• Optimization packages :– Collaborative filtering (CF) :
Revised: Simulation code was optimized using the NUMBA jit compiler, which improves runtime. • Optimization packages : – Collaborative filtering (CF) :

Original: We use the SURPRISE package (Hug, 2020), which includes animplementation of the SVD algorithm for CF.
Revised: We use the SURPRISE package (Hug, 2020), which includes an implementation of the SVD algorithm for CF.

Original: All parameters were set to default values.
Revised: All parameters were set to default values.

Original: – Regression : We use the SCIKIT - LEARN implementation of linear regression for predictinglong-term engagement from user features (i.e the prediction models f j ( u ) in Eq.
Revised: – Regression : We use the SCIKIT - LEARN implementation of linear regression for predicting long-term engagement from user features (i.e the prediction models f j ( u ) in Eq.

Original: Allparameters were set to default values.
Revised: All parameters were set to default values.

Original: – Non-Negative Least Squares (NNLS) :
Revised: – Non-Negative Least Squares (NNLS) :

Original: We use the SCIPY .
Revised: We use the SCIPY .

Original: OPTIMIZE implementation of NNLS.The algorithm was used with
Revised: OPTIMIZE implementation of NNLS. The algorithm was used with

Original: its default parameters.
Revised: its default parameters.

Original: • Code : Code for reproducing all of our figures and experiments is available in the followinganonymous repository: https://github . com/lvml-tsrml-2022/lvml.
Revised: • Code : Code for reproducing all of our figures and experiments is available in the following repository: https://github . com/edensaig/take-a-break.

Original: B.3 Other baselines 
Revised: B.3 Other baselines • Safety :

Original: • Safety : In each step of the TPP simulation, look k step back, and calculate the empirical rate˜ λ i =
Revised: In each step of the TPP simulation, look k step back, and calculate the empirical rate ˜ λ i =

Original: k t i − t i − k .
Revised: k t i − t i − k .

Original: If this rate exceeds the threshold ˜ λ i > τ , the policy enters a ‘cool-down’ policystate, serving only forced breaks until the next time period.
Revised: If this rate exceeds the threshold ˜ λ i > τ , the policy enters a ‘cool-down’ policy state, serving only forced breaks until the next time period.

Original: In our experiments, we used thresholds τ ∈ { 14 , 16 } , k = 10 look-behind steps, and defined the cool-down period as 0 .
Revised: In our experiments, we used thresholds τ ∈ { 14 , 16 } , k = 10 look-behind steps, and defined the cool-down period as 0 .

Original: 5 time units.
Revised: 5 time units.

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: L a t e n t c o n s u m p
Revised: L a t e n t c o n s u m p

Original: t
Revised: t

Original: i o n d r i v e λ ( t )
Revised: i o n d r i v e λ ( t )

Original: Discrete simulation ( B = 1 )
Revised: Discrete simulation ( B = 1 )

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: Continuous ODE

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure] Theoretical λ ∗ • Oracle : To estimate the effect of perfect predictions, we implement an oracle predictor f oracle p ( u )which has access to the latent user parameters.
Revised: • Oracle : To estimate the effect of perfect predictions, we implement an oracle predictor f oracle p ( u ) which has access to the latent user parameters.

Original: For a given u and for each p , the predictor outputs the infinite-horizon LV equilibrium for u , namely f oracle p ( u ) =
Revised: For a given u and for each p , the predictor outputs the infinite-horizon LV equilibrium for u , namely f oracle p ( u ) =

Original: λ ∗ ( p ; ˜ θ u ) 
Revised: λ ∗ ( p ; ˜ θ u ) .

Original:  define ˜ θ 
Revised: We define ˜ θ u

Original: u = ( α u , ˜ β u , γ u , δ u ) , where α u , γ u , δ u are the unobserved parameters for the given user, and ˜ β u is the expected value of β ux induced by the distribution over recommended items x induced by the recommendation policy ψ .
Revised: = ( α u , ˜ β u , γ u , δ u ) , where α u , γ u , δ u are the unobserved parameters for the given user, and ˜ β u is the expected value of β ux induced by the distribution over recommended items x induced by the recommendation policy ψ .

Original: We view ˜ θ u as a useful proxy for the otherwise unattainable ¯ θ u 
Revised: We view ˜ θ u as a useful proxy for the otherwise unattainable ¯ θ u .

Original: B.4 Hyperparameters
Revised: B.4 Hyperparameters

Original: [Figure] • Collaborative filtering : We used d = 8 latent factors and enabled bias terms, which ensuredperformance is close to the benchmark of RMSE = 0 .
Revised: • Collaborative filtering : We used d = 8 latent factors and enabled bias terms, which ensured performance is close to the benchmark of RMSE = 0 .

Original: 873 reported in the SURPRISE documentation.
Revised: 873 reported in the SURPRISE documentation.

Original: We used the vanilla SVD solver, with all hyper-parameters set to their default values.
Revised: We used the vanilla SVD solver, with all hyper-parameters set to their default values.

Original: • Recommendation policy : Softmax temperature was set to 0.5.
Revised: • Recommendation policy : Softmax temperature was set to 0.5.

Original: • Prediction : We trained regressors f ( u ) on input feature vectors
Revised: • Prediction : We trained regressors f ( u ) on input feature vectors

Original: u ∈ R d
Revised: u ∈ R d

Original: +2 consisting of: (i) SVDlatent user factors, (ii) SVD user bias terms, (iii) an additional feature consisting of the averagepredicted ratings for unseen items (exponentiated and normalized), which we found to slightly improve predictive performance.
Revised: +2 consisting of: (i) SVD latent user factors, (ii) SVD user bias terms, (iii) an additional feature consisting of the average predicted ratings for unseen items (exponentiated and normalized), which we found to slightly improve predictive performance.

Original: We chose to focus on linear models since the treatment datasetsare relatively small (each |S ( j ) | ≈ 500 ), and since other model classes (including boosted treesand MLPs) did not perform significantly better.
Revised: We chose to focus on linear models since the treatment datasets are relatively small (each |S ( j ) | ≈ 500 ), and since other model classes (including boosted trees and MLPs) did not perform significantly better.

Original: • Discrete TPP : Interaction sequences for each user were generated according to an LV discretization scheme, described in detail in the next section.
Revised: • Discrete TPP : Interaction sequences for each user were generated according to an LV discretization scheme, described in detail in the next section.

Original: Latent sates were initialized randomly with relativeuniform noise around the theoretical LV equilibrium point ( λ 0 , q 0 ) =
Revised: Latent sates were initialized randomly with relative uniform noise around the theoretical LV equilibrium point ( λ 0 , q 0 ) =

Original: ((1 + ξ λ ) λ ∗ , (1 + ξ q ) q ∗ ) ,where ξ λ , ξ q ∼ Uniform( − 0 .
Revised: ((1 + ξ λ ) λ ∗ , (1 + ξ q ) q ∗ ) , where ξ λ , ξ q ∼ Uniform( − 0 .

Original: Latent states were updated each B = 10 recommendationsto stabilize noise (see Figure 5).
Revised: Latent states were updated each B = 10 recommendations to stabilize noise (see Figure 5).

Original: When x is recommended to u at time t , latent states and ∆ t areset according to β u ( t ) , which depends on ratings r ux (true or mixed with predictions u ⊤ x via κ ).
Revised: When x is recommended to u at time t , latent states and ∆ t are set according to β u ( t ) , which depends on ratings r ux (true or mixed with predictions u ⊤ x via κ ).

Original: Specifically, we use β u ( t ) =
Revised: Specifically, we use β u ( t ) =

Original: r 2 ux / 5 ∈ { 0 . 2 , 0 . 8 , 1 . 8 , 3 . 2 , 5 } , which is convex, to accentuate the role of low ratings since they are underrepresented in the data.
Revised: r 2 ux / 5 ∈ { 0 . 2 , 0 . 8 , 1 . 8 , 3 . 2 , 5 } , which is convex, to accentuate the role of low ratings since they are underrepresented in the data.

Original: For B ≥ 1 , we take the effective β u ( t ) to be the average over the B items recommended in that step.
Revised: For B ≥ 1 , we take the effective β u ( t ) to be the average over the B items recommended in that step.

Original: We set α = 1 . 3 , and chose γ = 0 .
Revised: We set α = 1 . 3 , and chose γ = 0 .

Original: 2 , δ = 0 .
Revised: 2 , δ = 0 .

Original: 01 (which together determine scale) so that typical values for engagement rate1 T | S u | are on the order of ≈ 10 for the chosen T = 100 .
Revised: 01 (which together determine scale) so that typical values for engagement rateT | S u | are on the order of ≈ 10 for the chosen T = 100 .

Original: B.5 Discrete TPP
Revised: B.5 Discrete TPP

Original: The TPP we use for simulating user behavior is based on a discretization of the LV system describedin Eq.
Revised: The TPP we use for simulating user behavior is based on a discretization of the LV system described in Eq.

Original: (2), based on the forward Euler method with variable step sizes.
Revised: (2), based on the forward Euler method with variable step sizes.

Original: Each user is associated with discrete latent states
Revised: Each user is associated with discrete latent states

Original: λ i , q i , and parameters α u , γ u , δ u .
Revised: λ i , q i , and parameters α u , γ u , δ u .

Original: Initial statesλ 0 , q 0 are set randomly.
Revised: Initial states λ 0 , q 0 are set randomly.

Original: At each step, and in time t i , the system recommends x i =
Revised: At each step, and in time t i , the system recommends x i =

Original: x
Revised: x

Original: ( t i ) , whichtriggers updates in latent states, and determines the next time of interaction t
Revised: ( t i ) , which triggers updates in latent states, and determines the next time of interaction t

Original: i +1 .
Revised: i +1 .

Original: As noted, theseupdate depend on item-specific parameters β u,x i .
Revised: As noted, these update depend on item-specific parameters β u,x i .

Original: Under stationary policy π ( p ) , the system recommends an item with probability (1 − p ) , and suggestsa break with probability p .
Revised: Under stationary policy π ( p ) , the system recommends an item with probability (1 − p ) , and suggests a break with probability p .

Original: The simulator considers B recommendation opportunities at each step.
Revised: The simulator considers B recommendation opportunities at each step.

Original: For each k ∈ { 1 , . . .
Revised: For each k ∈ { 1 , . . .

Original: , B } , denote by I k ∈ { 0 , 1 } the break indicator, equal to 0 when a break is696 recommended at the k -th slot in the batch.
Revised: , B } , denote by I k ∈ { 0 , 1 } the break indicator, equal to 0 when a break is recommended at the k -th slot in the batch.

Original: Denote by x ∼ ψ the item recommended by the underlying697 policy ψ , and by β ( x ) =
Revised: Denote by x ∼ ψ the item recommended by the underlying policy ψ , and by β ( x ) =

Original: r 2 ux / 5 the corresponding LV hyperparameter as defined above.
Revised: r 2 ux / 5 the corresponding LV hyperparameter as defined above.

Original: For a givenhorizon T , the TPP process generating S u is described by Algorithm 1:
Revised: For a given horizon T , the TPP process generating S u is described by Algorithm 1:

Original: Input: Break probability p ∈ [0 , 1]
Revised: Input: Break probability p ∈ [0 , 1]

Original: Stationary content recommendation probability ψ
Revised: Stationary content recommendation probability ψ

Original: Time horizon T >
Revised: Time horizon T >

Original: Output: Interaction sequence S u ∼ TPP π ( p ) ◦ ψ ( u ; T )
Revised: Output: Interaction sequence S u ∼ TPP π ( p ) ◦ ψ ( u ; T )

Original: Initialize i = 0 , t 0 = 0 , S u = {} ; while t i < T do foreach k ∈ { 1 , . . .
Revised: Initialize i = 0 , t 0 = 0 , S u = {} ; while t i < T do foreach k ∈ { 1 , . . .

Original: , B } do I k ∼ Bernoulli(1 − p ) ; x k ∼ ψ ; β k ←− β ( r ux k ) ; end ∆ t i = λ − 1 i ; λ i +

Revised: , B } do I k ∼ Bernoulli(1 − p ) ; x k ∼ ψ ; β k ←− β ( r ux k ) ; end ∆ t i = λ − 1 i ; λ i +

Original: ←− λ i + (cid:16) − α + (cid:80)
Revised: ←− λ i + (cid:16) − α + (cid:80)

Original: Bk =

Revised: Bk =

Original: I k β k B q i (cid:17) λ i ∆ t i ;
Revised: I k β k B q i (cid:17) λ i ∆ t i ;

Original: q i +1 ←− q i + (cid:16) γ (1 − q i ) − (cid:80)
Revised: q i +1 ←− q i + (cid:16) γ (1 − q i ) − (cid:80)

Original: Bk =

Revised: Bk =

Original: I k δ B λ i (cid:17) q i ∆ t i ; t i + end{++
+}
Revised: I k δ B λ i (cid:17) q i ∆ t i ; t i {++
+}

Revised: ←− t i + λ − 1 i +1 ;

Revised: S u ←− S