Original: P ERSONALIZED R EWARD L EARNING WITH I NTERACTION -G ROUNDED L EARNING (IGL)
Revised: P ERSONALIZED R EWARD L EARNING WITH I NTERACTION -G ROUNDED L EARNING (IGL)

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In an era of countless content offerings, recommender systems alleviate information overload by providing users with personalized content suggestions.
Revised: In an era of countless content offerings, recommender systems alleviate information overload by providing users with personalized content suggestions.

Original: Due to the scarcity of explicit user feedback, modern recommender systems typically optimize for a fixed combination of implicit feedback signals across all users.
Revised: Due to the scarcity of explicit user feedback, modern recommender systems typically optimize for the same fixed combination of implicit feedback signals across all users.

Original: However, this approach disregards a growing body of work  that (i) implicit signals can be used by users in diverse ways, signaling anything from satisfaction to active dislike, and (ii) different users communicate preferences in different ways.
Revised: However, this approach disregards a growing body of work highlighting that (i) implicit signals can be used by users in diverse ways, signaling anything from satisfaction to active dislike, and (ii) different users communicate preferences in different ways.

Original: We propose applying the recent Interaction Grounded Learning (IGL) paradigm to address the challenge of learning representations of diverse user communication modalities.
Revised: We propose applying the recent Interaction Grounded Learning (IGL) paradigm to address the challenge of learning representations of diverse user communication modalities.

Original: Rather than taking a fixed, human-designed reward function, IGL is able to learn personalized reward functions for different users and then optimize directly for the latent user satisfaction.
Revised: Rather than taking a fixed, human-designed reward function, IGL is able to learn personalized reward functions for different users and then optimize directly for the latent user satisfaction.

Original: We demonstrate the success of IGL with experiments using simulations as well as with real-world production traces.
Revised: We demonstrate the success of IGL with experiments using simulations as well as with real-world production traces.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: From shopping to reading the news, modern Internet users have access to an overwhelming amount of content and choices from online services.
Revised: From shopping to reading the news, modern Internet users have access to an overwhelming amount of content and choices from online services.

Original: Recommender systems offer a way to improve user experience and decrease information overload by providing a customized selection of content.
Revised: Recommender systems offer a way to improve user experience and decrease information overload by providing a customized selection of content.

Original: A key challenge for recommender systems is the rarity of explicit user feedback, such as ratings or likes/dislikes (Gr ˇ car et al., 2005).
Revised: A key challenge for recommender systems is the rarity of explicit user feedback, such as ratings or likes/dislikes (Gr ˇ car et al., 2005).

Original: Rather than explicit feedback, practitioners typically use more readily available implicit signals, such as clicks (Hu et al., 2008), webpage dwell time (Yi et al., 2014), or inter-arrival times (Wu et al., 2017) as a proxy signal for user satisfaction.
Revised: Rather than explicit feedback, practitioners typically use more readily available implicit signals, such as clicks (Hu et al., 2008), webpage dwell time (Yi et al., 2014), or inter-arrival times (Wu et al., 2017) as a proxy signal for user satisfaction.

Original: These implicit signals are used as the reward objective in recommender systems, with the popular Click-Through
Revised: These implicit signals are used as the reward objective in recommender systems, with the popular Click-Through

Original: Rate (CTR) metric as the gold standard for the field (Silveira et al., 2019).
Revised: Rate (CTR) metric as the gold standard for the field (Silveira et al., 2019).

Original: However, directly using implicit signals as the reward function presents several issues.
Revised: However, directly using implicit signals as the reward function presents several issues.

Original: Implicit signals do not directly map to user satisfaction.
Revised: Implicit signals do not directly map to user satisfaction.

Original: Although clicks are routinely equated with user satisfaction, there are examples of unsatisfied users interacting with content via clicks.
Revised: Although clicks are routinely equated with user satisfaction, there are examples of unsatisfied users interacting with content via clicks.

Original: Clickbait exploits cognitive biases such as caption bias (Hofmann et al., 2012) or the curiosity gap (Scott, 2021) so that low quality content attracts more clicks.
Revised: Clickbait exploits cognitive biases such as caption bias (Hofmann et al., 2012) or the curiosity gap (Scott, 2021) so that low quality content attracts more clicks.

Original: Direct optimization of the CTR degrades user experience by promoting clickbait items (Wang et al., 2021).
Revised: Direct optimization of the CTR degrades user experience by promoting clickbait items (Wang et al., 2021).

Original: Recent work shows that users will even click on content that they know a priori they will dislike.
Revised: Recent work shows that users will even click on content that they know a priori they will dislike.

Original: In a study of online news reading,
Revised: In a study of online news reading,

Original: Lu et al.
Revised: Lu et al.

Original: (2018a) discovered that 15% of the time, users would click on articles that they strongly disliked.
Revised: (2018a) discovered that 15% of the time, users would click on articles that they strongly disliked.

Original: Similarly, although longer webpage dwell times are associated with satisfied users, a study by Kim et al.
Revised: Similarly, although longer webpage dwell times are associated with satisfied users, a study by Kim et al.

Original: (2014) found that dwell time is also significantly impacted by page topic, readability and content length.
Revised: (2014) found that dwell time is also significantly impacted by page topic, readability and content length.

Original: Different users communicate in different ways.
Revised: Different users communicate in different ways.

Original: Demographic background is known to have an impact on the ways in which users engage with recommender systems.
Revised: Demographic background is known to have an impact on the ways in which users engage with recommender systems.

Original: A study by Beel et al.
Revised: A study by Beel et al.

Original: (2013) shows that older users have CTR more than 3x higher than their younger counterparts.
Revised: (2013) shows that older users have CTR more than 3x higher than their younger counterparts.

Original: Gender also has an impact on interactions, e.g. men are more likely to leave dislikes on YouTube videos than women (Khan, 2017).
Revised: Gender also has an impact on interactions, e.g. men are more likely to leave dislikes on YouTube videos than women (Khan, 2017).

Original: At the same time, a growing body of work shows that recommender systems do not provide consistent performance across demographic subgroups.
Revised: At the same time, a growing body of work shows that recommender systems do not provide consistent performance across demographic subgroups.

Original: For example, multiple studies on ML fairness in recommender systems show that women on average receive less accurate recommendations compared to men (Ekstrand et al., 2018; Mansoury et al., 2020).
Revised: For example, multiple studies on ML fairness in recommender systems show that women on average receive less accurate recommendations compared to men (Ekstrand et al., 2018; Mansoury et al., 2020).

Original: Current systems are also unfair across different age brackets, with statistically significant recommendation utility degradation as the age of the user increases (Neophytou et al., 2022).
Revised: Current systems are also unfair across different age brackets, with statistically significant recommendation utility degradation as the age of the user increases (Neophytou et al., 2022).

Original: The work of Neophytou et al.
Revised: The work of Neophytou et al.

Original: identifies usage features as the most predictive of mean recommender utility, hinting that the inconsistent performance in recommendation algorithms across subgroups arises from the differences in how users interact with the recommender system.
Revised: identifies usage features as the most predictive of mean recommender utility, hinting that the inconsistent performance in recommendation algorithms across subgroups arises from the differences in how users interact with the recommender system.

Original: These challenges motivate the need for personalized reward functions.
Revised: These challenges motivate the need for personalized reward functions.

Original: However, extensively modeling the ways in which implicit signals are used or how demographics impact interaction style is costly and inefficient.
Revised: However, extensively modeling the ways in which implicit signals are used or how demographics impact interaction style is costly and inefficient.

Revised: Current state-of-the-art systems utilize reward functions that are manually engineered combinations of implicit signals, typically refined through laborious trial and error methods.

Original: Furthermore, as recommender systems and their users evolve, so do the ways in which users implicitly communicate preferences.
Revised: Yet as recommender systems and their users evolve, so do the ways in which users implicitly communicate preferences.

Original: Any extensive models  developed now could easily become obsolete within a few years time.
Revised: Any extensive models or hand tuned reward functions developed now could easily become obsolete within a few years time.

Original: To this end, we propose Interaction Grounded Learning (IGL) Xie et al.
Revised: To this end, we propose Interaction Grounded Learning (IGL) Xie et al.

Original: (2021) for personalized reward learning ( IGL-P ).
Revised: (2021) for personalized reward learning ( IGL-P ).

Original: IGL is a learning paradigm where a learner optimizes for unobservable rewards by interacting with the environment and associating observable feedback with the true latent reward.
Revised: IGL is a learning paradigm where a learner optimizes for unobservable rewards by interacting with the environment and associating observable feedback with the true latent reward.

Original: Prior IGL approaches assume the feedback either depends on the reward alone
Revised: Prior IGL approaches assume the feedback either depends on the reward alone

Original: Xie et al.
Revised: Xie et al.

Original: (2021), or on the reward and action Xie et al.
Revised: (2021), or on the reward and action Xie et al.

Original: These methods are unable to disambiguate personalized feedback which depends on the context.
Revised: These methods are unable to disambiguate personalized feedback which depends on the context.

Original: Other approaches such as reinforcement learning and traditional contextual bandits suffer from the choice of reward function.
Revised: Other approaches such as reinforcement learning and traditional contextual bandits suffer from the choice of reward function.

Original: However our proposed personalized IGL, IGL-P , resolves the 2 above challenges while making minimal assumptions about the value of observed user feedback.
Revised: However our proposed personalized IGL, IGL-P , resolves the 2 above challenges while making minimal assumptions about the value of observed user feedback.

Original: Our new approach is able to incorporate both explicit and implicit signals, leverage ambiguous user feedback and adapt to the different ways in which users interact with the system.
Revised: Our new approach is able to incorporate both explicit and implicit signals, leverage ambiguous user feedback and adapt to the different ways in which users interact with the system.

Original: Our Contributions: We present  the first IGL strategy for context-dependent feedback, the first use of inverse kinematics as an IGL objective, and the first IGL strategy for more than two latent states.
Revised: Our Contributions: We present IGL-P , the first IGL strategy for context-dependent feedback, the first use of inverse kinematics as an IGL objective, and the first IGL strategy for more than two latent states.

Revised: Our proposed approach provides an alternative to agent learning methods which require handcrafted reward functions.

Original: Using simulations and real production data, we demonstrate that recommender systems require at least 3 reward states, and that IGL is able to address two of the biggest challenges for modern online recommender systems.
Revised: Using simulations and real production data, we demonstrate that IGL-P is able to learn personalized rewards when applied to the domain of online recommender systems, which require at least 3 reward states.

Original:  P ROBLEM S ETTING 2. C ONTEXTUAL B ANDITS
Revised:  P ROBLEM S ETTING 2. C ONTEXTUAL B ANDITS

Original: The contextual bandit (Auer et al., 2002; Langford & Zhang, 2007) is a statistical model of myopic decision making which is pervasively applied in recommendation systems (Bouneffouf et al., 2020).
Revised: The contextual bandit (Auer et al., 2002; Langford & Zhang, 2007) is a statistical model of myopic decision making which is pervasively applied in recommendation systems (Bouneffouf et al., 2020).

Original: IGL operates via reduction to contextual bandits, hence, we briefly review contextual bandits here.
Revised: IGL operates via reduction to contextual bandits, hence, we briefly review contextual bandits here.

Original: The contextual bandit problem proceeds over T rounds.
Revised: The contextual bandit problem proceeds over T rounds.

Original: At each round t ∈ [ T ] , the learner receives a context x t ∈ X (the context space), selects an action a t ∈ A (the action space), and then observes a reward r t ( a t ) , where r t :
Revised: At each round t ∈ [ T ] , the learner receives a context x t ∈ X (the context space), selects an action a t ∈ A (the action space), and then observes a reward r t ( a t ) , where r t :

Original: A → [0 , 1] is the underlying reward function.
Revised: A → [0 , 1] is the underlying reward function.

Original: We assume that for each round t , conditioned on x t , r t is sampled from a distribution P r t ( · | x t ) .
Revised: We assume that for each round t , conditioned on x t , r t is sampled from a distribution P r t ( · | x t ) .

Original: A contextual bandit algorithm attempts to minimize the  regret
Revised: A contextual bandit algorithm attempts to minimize the cumulative regret

Original: [Equation]
Revised: [Equation]

Original: relative to an optimal policy π ⋆ over a policy class Π .
Revised: relative to an optimal policy π ⋆ over a policy class Π .

Original: In general, both the contexts x 1 , . . .
Revised: In general, both the contexts x 1 , . . .

Original: , x T and the distributions P r 1 , . .
Revised: , x T and the distributions P r 1 , . .

Original: , P r T can be selected in an arbitrary, potentially adaptive fashion based on the history.
Revised: , P r T can be selected in an arbitrary, potentially adaptive fashion based on the history.

Original: In the sequel we will describe IGL in a stochastic environment, but the reduction induces a nonstationary contextual bandit problem, and therefore the existence of adversarial contextual bandit algorithms is relevant.
Revised: In the sequel we will describe IGL in a stochastic environment, but the reduction induces a nonstationary contextual bandit problem, and therefore the existence of adversarial contextual bandit algorithms is relevant.

Original: 2. I NTERACTION G ROUNDED L EARNING
Revised: 2. I NTERACTION G ROUNDED L EARNING

Revised: IGL is a problem setting in which the learner’s goal is to optimally interact with the environment with no explicit reward to ground its policies.

Original: IGL extends the contextual bandit framework by eliding the reward from the learning algorithm and providing feedback instead (Xie et al., 2021).
Revised: IGL extends the contextual bandit framework by eliding the reward from the learning algorithm and providing feedback instead (Xie et al., 2021).

Original: We describe the stochastic setting where ( x t , r t , yt ) ∼
Revised: We describe the stochastic setting where ( x t , r t , y

Original:  D triples are sampled iid from an unknowndistribution; the learner receives the context x t ∈ X , selects an action a t ∈ A , and then observes the feedback y t ( a t ) , where y t : A → [0 , 1] is the underlying feedback function. Note r t ( a t ) is never revealed to the algorithm: nonetheless, the
Revised: t ) ∼ D triples are sampled iid from an unknown

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: regret notion remains the same as Eq. An information-theoretic argument proves assumptions relating the feedback to the underlyingreward are necessary to succeed (Xie et al., 2022).
Revised: the underlying

Original: 2.2. S PECIALIZATION TO R ECOMMENDATION
Revised: 2.2. S PECIALIZATION TO R ECOMMENDATION

Original: For specific application in the recommendation domain, we depart from prior art in IGL (Xie et al., 2021; 2022) in two ways: first, in the assumed relationship between feedback and underlying reward; and second, in the number of latent reward states.
Revised: For specific application in the recommendation domain, we depart from prior art in IGL (Xie et al., 2021; 2022) in two ways: first, in the assumed relationship between feedback and underlying reward; and second, in the number of latent reward states.

Revised: For the remainder of the paper, we represent the users as context x , content items as action a , the user satisfaction with recommended content as r , and the feedback signals in response to the recommended content (user interactions with the system interface) as y .

Original: Feedback Dependence Assumption Xie et al.
Revised: Feedback Dependence Assumption Xie et al.

Original: (2021) assumed full contextual independence of the feedback on the context and chosen action, i.e. y ⊥ x, a | r .
Revised: (2021) assumed full contextual independence of the feedback on the context and chosen action, i.e. y ⊥ x, a | r .

Original: For recommender systems, this implies that all users communicate preferences identically for all content.
Revised: For recommender systems, this implies that all users communicate preferences identically for all content.

Original: In a subsequent paper, Xie et al.
Revised: In a subsequent paper, Xie et al.

Original: (2022) loosen  full conditional independence by considering context conditional independence, i.e. y ⊥ x | a, r .
Revised: (2022) loosen the full conditional independence by considering context conditional independence, i.e. y ⊥ x | a, r .

Original: For our setting, this corresponds to the user feedback varying for combinations of preference and content, but remaining consistent across all users.
Revised: For our setting, this corresponds to the user feedback varying for combinations of preference and content, but remaining consistent across all users.

Original: Neither of these two assumptions are natural in the recommendation setting because different users interact with recommender systems in different ways.(Beel et al., 2013; Shin, 2020).
Revised: Neither of these two assumptions are natural in the recommendation setting because different users interact with recommender systems in different ways.(Beel et al., 2013; Shin, 2020).

Original: In this work, we assume y ⊥ a | x, r , i.e., the feedback y is independent of the displayed content a given the user x and their disposition toward the displayed content r .
Revised: In this work, we assume y ⊥ a | x, r , i.e., the feedback y is independent of the displayed content a given the user x and their disposition toward the displayed content r .

Original: Thus, we assume that users may communicate in different ways, but a given user expresses satisfaction, dissatisfaction and indifference to all content in the same way.
Revised: Thus, we assume that users may communicate in different ways, but a given user expresses satisfaction, dissatisfaction and indifference to all content in the same way.

Original: Number of Latent Reward States
Revised: Number of Latent Reward States

Original: Prior work demonstrates  a binary latent reward assumption, along with an assumption that rewards are rare under a known reference policy, is sufficient for IGL to succeed.
Revised: Prior work demonstrates that a binary latent reward assumption, along with an assumption that rewards are rare under a known reference policy, are sufficient for IGL to succeed.

Original: Specifically, optimizing the contrast between a learned policy and the oblivious uniform policy is able to succeed when feedback is both context and action independent Xie et al.
Revised: Specifically, optimizing the contrast between a learned policy and the oblivious uniform policy is able to succeed when feedback is both context and action independent Xie et al.

Original: (2021); and optimizing the contrast between the learned policy and all constant-action policies succeeds when the feedback is context independent Xie
Revised: (2021); and optimizing the contrast between the learned policy and all constant-action policies succeeds when the feedback is context independent Xie

Original: et al.
Revised: et al.

Original: Although the binary latent reward assumption (e.g., satisfied or dissatisfied) appears reasonable for recommendation scenarios, it fails to account for user indifference versus user dissatisfaction.
Revised: Although the binary latent reward assumption (e.g., satisfied or dissatisfied) appears reasonable for recommendation scenarios, it fails to account for user indifference versus user dissatisfaction.

Original: This observation was first motivated by our production data, where a 2 state IGL policy would sometimes maximize feedback signals with obviously negative semantics.
Revised: This observation was first motivated by our production data, where a 2 state IGL policy would sometimes maximize feedback signals with obviously negative semantics.

Original: Assuming users ignore most content most of the time (Nguyen et al., 2014), negative feedback can be as difficult to elicit as positive feedback, and a 2 state IGL model is unable to distinguish between these extremes.
Revised: Assuming users ignore most content most of the time (Nguyen et al., 2014), negative feedback can be as difficult to elicit as positive feedback, and a 2 state IGL model is unable to distinguish between these extremes.

Original: Hence, we posit a minimal latent state model for recommender systems involves 3 states: (i) r
Revised: Hence, we posit a minimal latent state model for recommender systems involves 3 states: (i) r

Original: = 1 , when users are satisfied with the recommended content, (ii) r = 0 , when users are indifferent or inattentive, and (iii) r = − 1 , when users are dissatisfied.
Revised: = 1 , when users are satisfied with the recommended content, (ii) r = 0 , when users are indifferent or inattentive, and (iii) r = − 1 , when users are dissatisfied.

Original:  D ERIVATIONS
Revised:  D ERIVATIONS

Original: Prior approaches to IGL use contrastive learning objectives (Xie et al., 2021; 2022), but the novel feedback dependence assumption in the prior section impedes this line of attack.
Revised: Prior approaches to IGL use contrastive learning objectives (Xie et al., 2021; 2022), but the novel feedback dependence assumption in the prior section impedes this line of attack.

Original: Essentially, given arbitrary dependence upon x , learning must operate on each example in isolation without requiring comparison across examples.
Revised: Essentially, given arbitrary dependence upon x , learning must operate on each example in isolation without requiring comparison across examples.

Original: This motivates attempting to predict the current action from the current context and the currently observed feedback, i.e., inverse kinematics.
Revised: This motivates attempting to predict the current action from the current context and the currently observed feedback, i.e., inverse kinematics.

Revised: Inverse Kinematics Traditionally in robotics and computer animation, inverse kinematics is the mathematical process used to recover the movement (action) of a robot/object in the world from some other data such as the position and orientation of a robot manipulator/video of a moving object (in our case, the other data is context and feedback).

Original: Inverse Kinematics We motivate our inverse kinematics strategy using exact expectations.
Revised: We motivate our inverse kinematics strategy using exact expectations.

Original: When acting according to any policy P ( a | x ) , we can imagine trying to predict the action taken given the context and feedback; the posterior distribution is
Revised: When acting according to any policy P ( a | x ) , we can imagine trying to predict the action taken given the context and feedback; the posterior distribution is

Original: [Equation]
Revised: [Equation]

Original: We arrive at the inner product between a reward decoder term ( P ( r | y, x )) and a reward predictor term ( P ( r | a, x ) ).
Revised: We arrive at the inner product between a reward decoder term ( P ( r | y, x )) and a reward predictor term ( P ( r | a, x ) ).

Original: Extreme Event Detection Direct extraction of a reward predictor using maximum likelihood on the action prediction problem with Eq.
Revised: Extreme Event Detection Direct extraction of a reward predictor using maximum likelihood on the action prediction problem with Eq.

Original: (2) is frustrated by two identifiability issues: first, this expression is invariant to a permutation of the rewards on a context dependent basis; and second, the relative scale of two terms being multiplied is not uniquely determined by their product.
Revised: (2) is frustrated by two identifiability issues: first, this expression is invariant to a permutation of the rewards on a context dependent basis; and second, the relative scale of two terms being multiplied is not uniquely determined by their product.

Original: To mitigate the first issue, we assume (cid:80) a P ( r = 0 | a, x ) P ( a | x ) > 12 , i.e., nonzero rewards are rare under
Revised: To mitigate the first issue, we assume (cid:80) a P ( r = 0 | a, x ) P ( a | x ) > 12 , i.e., nonzero rewards are rare under

Original: P ( a | x ) ; and to mitigate the second issue, we assume the feedback can be perfectly decoded, i.e.,
Revised: P ( a | x ) ; and to mitigate the second issue, we assume the feedback can be perfectly decoded, i.e.,

Original: P ( r | y, x ) ∈ { 0 , 1 } .
Revised: P ( r | y, x ) ∈ { 0 , 1 } .

Original: Under these assumptions we have
Revised: Under these assumptions, we have

Original: [Equation]
Revised: [Equation]

Original: Eq.
Revised: Eq.

Original: (3) forms the basis for our extreme event detector: anytime the posterior probability of an action is predicted to be more than twice the prior probability, we deduce r ̸ = 0 .
Revised: (3) forms the basis for our extreme event detector: anytime the posterior probability of an action is predicted to be more than twice the prior probability, we deduce r ̸ = 0 .

Original: Note a feedback merely being apriori rare or frequent (i.e., the magnitude of P ( y | x ) under the policy
Revised: Note a feedback merely being apriori rare or frequent (i.e., the magnitude of P ( y | x ) under the policy

Original: P ( a | x ) ) does not imply that observing such feedback will induce an extreme event detection; rather the feedback must have a probability that strongly depends upon which action is taken.
Revised: P ( a | x ) ) does not imply that observing such feedback will induce an extreme event detection; rather the feedback must have a probability that strongly depends upon which action is taken.

Original: Because feedback is assumed conditionally independent of action given the reward, the only way for feedback to help predict which action is played is via the (action dependence of the) latent reward.
Revised: Because feedback is assumed conditionally independent of action given the reward, the only way for feedback to help predict which action is played is via the (action dependence of the) latent reward.

Original: Extreme Event Disambiguation With 2 latent states, r ̸
Revised: Extreme Event Disambiguation With 2 latent states, r ̸

Original: = 0 =
Revised: = 0 =

Original: ⇒ r
Revised: ⇒ r

Original: = 1 , and we can reduce to a standard contextual bandit with inferred rewards 1 ( P ( a | y, x ) > 2 P ( a | x )) .
Revised: = 1 , and we can reduce to a standard contextual bandit with inferred rewards 1 ( P ( a | y, x ) > 2 P ( a | x )) .

Original: With 3 latent states, r ̸ = 0 = ⇒ r = ± 1 , and additional information is necessary to disambiguate the extreme events.
Revised: With 3 latent states, r ̸ = 0 = ⇒ r = ± 1 , and additional information is necessary to disambiguate the extreme events.

Original: We assume partial reward information is available via a “definitely negative” function 1 DN :
Revised: We assume partial reward information is available via a “definitely negative” function 1 DN :

Original: X ×
Revised: X ×

Original: Y → {− 1 , 0 } where P ( DN ( x, y ) = 

Revised: Y → {− 1 , 0 } where P ( DN ( x, y ) =

Original: | r = 1) =
Revised: | r = 1) =

Original: 1 and P ( DN ( x, y )
Revised: 1 and P ( DN ( x, y )

Original: = − 1 | r = − 1) > 0 .
Revised: = − 1 | r = − 1) > 0 .

Original: This reduces extreme event disambiguation to one-sided learning (Bekker & Davis, 2020) applied only to extreme events, where we try to predict the underlying latent state given ( x, a ) .
Revised: This reduces extreme event disambiguation to one-sided learning (Bekker & Davis, 2020) applied only to extreme events, where we try to predict the underlying latent state given ( x, a ) .

Original: We assume partial labelling is selected completely at random Elkan & Noto (2008) and treat the (constant) negative labelling propensity α as a hyperparameter.
Revised: We assume partial labelling is selected completely at random Elkan & Noto (2008) and treat the (constant) negative labelling propensity α as a hyperparameter.

Original: We arrive at our 3-state reward extractor [Equation]
Revised: We arrive at our 3-state reward extractor ρ ( x, a, y ) =

Original: otherwise
Revised:  

Original: [Equation]
Revised: P ( a | y, x ) ≤ 2 P ( a | x ) − α − P ( a | y, x ) > 2 P ( a | x ) and DN ( x, y ) =

Original:  equivalent to Bekker & Davis (2020, Equation 11).
Revised: − otherwise , (4) equivalent to Bekker & Davis (2020, Equation 11).

Original: Setting α = 1 embeds 2-state IGL.
Revised: Setting α = 1 embeds 2-state IGL.

Original: 1: π ←
Revised: On or Off Policy.

Original: 3: for t do 4:
Revised: # DN ( . . . )

Original: Observe context x t action set A t{+= 
+}
Revised: {+= 
+}

Original: A t
Revised: for

Original:  = K . 5: if Policy then 6:
Revised: # α = 1 for 2 state IGL

Original: P( ·| x t ) ← π. predict ( x t , A t
Revised: P

Original: Play a t ∼ P (·| x t) and observe feedbacky t .
Revised: Play a t ∼ P ( x t observe feedback

Original: 8: else 9:
Revised: 8: else 9:

Original: Observe ( x t, a t , y t , P ( ·| x t )) .
Revised: Observe ( x t a t , y t , P ( ·| x t )

Original: 10: w t ← 1 / ( KP ( a t | x t )) .
Revised: 10: w t ← 1 / ( KP ( a t | x t )) .

Original: # Synthetic uniform distribution 11:ˆ P( a t | y t , x t ) ← IK .
Revised: # Synthetic uniform distribution 11: P

Original: predict(( x t ,y t) , At , at ) .
Revised: predict t , t , A , a

Original: # Predict action probability 12:if K ˆ P( a t| y t ,x t ) ≤ 2 then # ˆ r t =13:π. learn ( x t , a t , A t , r t = 0) 14: else # ˆ r t ̸ =15: if DN ( . . . )
Revised: # Predict action probability 12: P a t y t , # ˆ r t =13: 14: else # ˆ r t ̸ =15: if DN ( . . . )

Original: = 

Revised: =

Original: then 16: π. learn ( x t , a t , A t , r t = 1 , P ( ·| x t )) 17: else # Definitely negative 18: π.
Revised: then 16: π. learn ( x t , a t , A t , r t = 1 , P ( ·| x t )) 17: else # Definitely negative 18: π.

Original: learn  = −  , P 
Revised: learn ( x t , a t , A t , r t = − α − 1 , P ( ·| x t )) 19:

Original: IK .
Revised: IK .

Original: learn  x t  y t )  A t  a t  t 
Revised: learn (( x t , y t ) , A t , a t , w t ) .

Original: Implementation Notes In practice, P ( a | x ) is known but the other probabilities are estimated.
Revised: Implementation Notes In practice, P ( a | x ) is known but the other probabilities are estimated.

Original: ˆ P ( a | y, x ) is estimated online using maximum likelihood on the problem predicting a from ( x, y ) , i.e., on a data stream of tuples (( x, y ) , a ) .
Revised: ˆ P ( a | y, x ) is estimated online using maximum likelihood on the problem predicting a from ( x, y ) , i.e., on a data stream of tuples (( x, y ) , a ) .

Original: The current estimates induce ˆ ρ ( x, a, y ) based upon the plug-in version of Eq.
Revised: The current estimates induce ˆ ρ ( x, a, y ) based upon the plug-in version of Eq.

Original: In this manner, the original data stream of ( x, a, y ) tuples is transformed into stream of ( x, a, ˆ r =
Revised: In this manner, the original data stream of ( x, a, y ) tuples is transformed into stream of ( x, a, ˆ r =

Original: ˆ ρ ( x, a, y )) tuples and reduced to a standard online contextual bandit problem.
Revised: ˆ ρ ( x, a, y )) tuples and reduced to a standard online contextual bandit problem.

Original: As an additional complication, although P ( a | x ) is known, it is typically a good policy under which rewards are not rare (e.g., offline learning with a good historical policy; or acting online according to the policy being learned by the IGL procedure).
Revised: As an additional complication, although P ( a | x ) is known, it is typically a good policy under which rewards are not rare (e.g., offline learning with a good historical policy; or acting online according to the policy being learned by the IGL procedure).

Original: Therefore we use importance weighting to synthesize a uniform action distribution P ( a | x ) from the true action distribution.
Revised: Therefore, we use importance weighting to synthesize a uniform action distribution P ( a | x ) from the true action distribution.

Original: Ultimately we arrive at the procedure of Algorithm 1.
Revised: Ultimately we arrive at the procedure of Algorithm 1.

Original:  E MPIRICAL E VALUATIONS
Revised:  E MPIRICAL E VALUATIONS

Original: Evaluation Settings: Evaluation settings include simulation using a supervised classification dataset, online news recommendation on Facebook, and a production image recommendation scenario.
Revised: Evaluation Settings: Evaluation settings include simulation using a supervised classification dataset, online news recommendation on Facebook, and a production image recommendation scenario.

Original: Abbreviations: Algorithms are denoted by the following abbreviations: Personalized IGL forlatent states ( IGL-P(2) ); Personalized IGL for 3 latent states ( IGL-P(3) ); Contextual Bandits for the Facebook news setting that maximizes for emoji, non-like click-based reactions {+
 When the+} ( CB-emoji ); Contextual Banditsfor the Facebook news setting that maximizes for comment interactions ( CB-comment ).
Revised: Abbreviations: Algorithms are denoted by the following abbreviations: Personalized IGL forlatent states ( IGL-P(2) ); Personalized IGL for 3 latent states ( IGL-P(3) ); Contextual Bandits for the Facebook news setting that maximizes for emoji, non-like click-based reactions {+
 When the+} ( CB-emoji ) ; Contextual Bandits

Original: General Evaluation Setup:
Revised: for the Facebook news setting that maximizes for comment in- teractions ( CB-comment ); Contextual Bandits

Revised: for the

Revised: production

Revised: recommendation

Revised: scenario

Revised: that maximizes for CTR CB-click

Revised: General Evaluation Setup:

Original: At each time step t , the context x t is provided from either the simulator (Section 4.1, Section 4.2) or the logged production data (Section 4.3).
Revised: At each time step t , the context x t is provided from either the simulator (Section 4.1, Section 4.2) or the logged production data (Section 4.3).

Original: The learner then selects an action a t and receives feedback y t .
Revised: The learner then selects an action a t and receives feedback y t .

Original: In these evaluations, each user provides feedback in exactly one interaction and different user feedback signals are mutually exclusive, so that y t is a one-hot 
Revised: In these evaluations, each user provides feedback in exactly one interaction and different user feedback signals are mutually exclusive, so that y t is a one-hot vector.

Original: User Embeddings normal bizarro 2.

Original: Word Embeddings good bad vector.

Original: In simulated environments, the ground truth reward is sometimes used for evaluation but  revealed 
Revised: In simulated environments, the ground truth reward is sometimes used for evaluation but never revealed to the algorithm.

Original: Code: Our code will be made available at { url redacted } for all publicly replicable experiments (i.e., except for the production data).
Revised: Code: Our code will be made available at { url redacted } for all publicly replicable experiments (i.e., except for the production data).

Original: 4. C OVERTYPE IGL S IMULATION
Revised: 4. C OVERTYPE IGL S IMULATION

Original: To highlight that personalized IGL can distinguish between different user communication styles, we create a simulated 2-state IGL scenario from a supervised classification dataset.
Revised: To highlight that personalized IGL can distinguish between different user communication styles, we create a simulated 2-state IGL scenario from a supervised classification dataset.

Original: First, we apply a supervised-to-bandit transform to convert the dataset into a contextual bandit simulation (Bietti et al., 2021), i.e., the algorithm is presented the example features as context, chooses one of the classes as an action, and experiences a binary reward which indicates whether or not it matches the example label.
Revised: First, we apply a supervised-to-bandit transform to convert the dataset into a contextual bandit simulation (Bietti et al., 2021), i.e., the algorithm is presented the example features as context, chooses one of the classes as an action, and experiences a binary reward which indicates whether or not it matches the example label.

Original: In the IGL simulation this reward is experienced but not revealed to the algorithm.
Revised: In the IGL simulation, this reward is experienced but not revealed to the algorithm.

Original: Instead, the latent reward is converted into a feedback signal as follows: each example is assigned one of N different user ids and the user id is revealed to the algorithm as part of the example features.
Revised: Instead, the latent reward is converted into a feedback signal as follows: each example is assigned one of N different user ids and the user id is revealed to the algorithm as part of the example features.

Original: The simulated user will generate feedback in the form of one of M different word ids.
Revised: The simulated user will generate feedback in the form of one of M different word ids.

Original: Unknown to the algorithm, the words are divided equally into “good” and “bad” words, and the users are divided equally into “normal” and “bizarro” users.
Revised: Unknown to the algorithm, the words are divided equally into “good” and “bad” words, and the users are divided equally into “normal” and “bizarro” users.

Original: “Normal” users indicate positive and zero reward via “good” and “bad” words respectively, while “bizarro” users employ the exact opposite communication convention.
Revised: “Normal” users indicate positive and zero reward via “good” and “bad” words respectively, while “bizarro” users employ the exact opposite communication convention.

Original:  We simulated using the Covertype (Blackard & Dean, 1999) dataset with M = N = 100 , and an (inverse kinematics) model class which embedded both user and word ids into a 2 dimensional space.
Revised: [Figure] We simulated using the Covertype (Blackard & Dean, 1999) dataset with M = N = 100 , and an (inverse kinematics) model class which embedded both user and word ids into a 2 dimensional space.

Original: Fig.
Revised: Fig.

Original: 2 demonstrates  both the user population and the words are cleanly separated into two latent groups.
Revised: 2 demonstrates that both the user population and the words are cleanly separated into two latent groups.

Original: Additional results showcasing the learning curves for inverse kinematics, reward and policy learning are shown in ?? .
Revised: Additional results showcasing the learning curves for inverse kinematics, reward and policy learning are shown in Appendix A.1 along with more detailed description of the model parameters.

Original: 4. F AIRNESS IN F ACEBOOK N EWS R ECOMMENDATION
Revised: 4. F AIRNESS IN F ACEBOOK N EWS R ECOMMENDATION

Original: Personalized reward learning is the key to more fair recommender systems.
Revised: Personalized reward learning is the key to more fair recommender systems.

Original: Previous work (Neophytou et al., 2022) suggests that inconsistent performance in recommender systems across user subgroups arises due to differences in user communication modalities.
Revised: Previous work (Neophytou et al., 2022) suggests that inconsistent performance in recommender systems across user subgroups arises due to differences in user communication modalities.

Original: We now test this hypothesis in the setting of Facebook news recommendation.
Revised: We now test this hypothesis in the setting of Facebook news recommendation.

Original: Our simulations are built on a dataset (Martinchek, 2016) of all posts by the official Facebook pages of 3 popular news outlets (Fox News,
Revised: Our simulations are built on a dataset (Martinchek, 2016) of all posts by the official Facebook pages of 3 popular news outlets (Fox News,

Original: The
Revised: The

Original: Huffington Post and TIME Magazine) that span the political spectrum.
Revised: Huffington Post and TIME Magazine) that span the political spectrum.

Original: Posts range from May to
Revised: Posts range from May to

Original: November 2016 and contain text content information, as well as logged interaction counts, which include comments and shares, as well as diverse click-based reactions (see Fig. 3).
Revised: November 2016 and contain text content information, as well as logged interaction counts, which include comments and shares, as well as diverse click-based reactions (see Fig. 3).

Original: Constructing a hand-engineered reward signal using these feedbacks is difficult, and Facebook itself came under fire for utilizing reward weights that disproportionately promote toxic, low quality news.
Revised: Constructing a hand-engineered reward signal using these feedbacks is difficult, and Facebook itself came under fire for utilizing reward weights that disproportionately promote toxic, low quality news.

Original: One highly criticized iteration of the reward ranking algorithm treated emoji reactions as five times more valuable than likes (Merrill & Oremus, 2021).
Revised: One highly criticized iteration of the reward ranking algorithm treated emoji reactions as five times more valuable than likes (Merrill & Oremus, 2021).

Original: Future iterations of theranking algorithm promoted comments, in an attempt to bolster “meaningful social interactions” (Hagey & Jeff Horwitz, 2021).
Revised: Future iterations of the algorithm promoted comments, in an attempt to bolster “meaningful social interactions” (Hagey & Jeff Horwitz, 2021).

Original: Our experiments evaluate the performance of CB algorithms using these two reward functions, referring to them as CB-emoji and CB-comment .
Revised: Our experiments evaluate the performance of CB algorithms using these two reward functions, referring to them as CB-emoji and CB-comment .

Original: We model the news recommendation problem as a 3 latent  state problem, with readers of different news outlets as different contexts.
Revised: We model the news recommendation problem as a 3 latent reward state problem, with readers of different news outlets as different contexts.

Original: Given a curated selection of posts, the goal of the learner is to select the best article to show to the reader.
Revised: Given a curated selection of posts, the goal of the learner is to select the best article to show to the reader.

Original: The learner can leverage action features including the post type (link, video, photo, status or event) as well as embeddings of the text content that were generated using pre-trained transformers (Reimers & Gurevych, 2019).
Revised: The learner can leverage action features including the post type (link, video, photo, status or event) as well as embeddings of the text content that were generated using pre-trained transformers (Reimers & Gurevych, 2019).

Original: User feedback is drawn from a fixed probability distribution (unknown to the algorithms) that depends on the user type and latent reward of the chosen action.
Revised: User feedback is drawn from a fixed probability distribution (unknown to the algorithms) that depends on the user type and latent reward of the chosen action.

Original: As an approximation of the true latent reward signal, we use low dimensional embeddings of the different news outlets combined with aggregate statistics from the post feedback to categorize whether the users had a positive ( r = 1 ), neutral ( r = 0 ) or negative experience ( r = − 1 ) with the post.
Revised: As an approximation of the true latent reward signal, we use low dimensional embeddings of the different news outlets combined with aggregate statistics from the post feedback to categorize whether the users had a positive ( r = 1 ), neutral ( r = 0 ) or negative experience ( r = − 1 ) with the post.

Original: This categorization is not available to the evaluated algorithms.
Revised: This categorization is not available to the evaluated algorithms.

Original: Finally, we implement IGL-P(3) with the angry reaction as a negative oracle to disambiguate the positive and negative reward states.
Revised: Finally, we implement IGL-P(3) with the angry reaction as a negative oracle to disambiguate the positive and negative reward states.

Original: [Figure]
Revised: [Figure]

Original: Fig.
Revised: Fig.

Original: 4 shows the results of our online news recommendation experiments.
Revised: 4 shows the results of our online news recommendation experiments.

Original: While the performance of both CB-emoji and CB-comment varies significantly across the different reader groups,
Revised: While the performance of both CB-emoji and CB-comment varies significantly across the different reader groups,

Original: IGL-P(3) maintains relatively stable performance for both positive and negative rewards.
Revised: IGL-P(3) maintains relatively stable performance for both positive and negative rewards.

Original: The
Revised: The

Original: CB algorithm that maximizes emoji reactions achieves the best performance for TIME readers at the cost of very bad performance for the Fox News and Huffington Post Readers.
Revised: CB algorithm that maximizes emoji reactions achieves the best performance for TIME readers at the cost of very bad performance for the Fox News and Huffington Post Readers.

Original: On the other hand, the CB algorithm that maximizes comment engagement achieves best performance with Fox
Revised: On the other hand, the CB algorithm that maximizes comment engagement achieves best performance with Fox

Original: News readers, however it still performs worse than IGL-P(3) .
Revised: News readers, however it still performs worse than IGL-P(3) .

Original: Finally, our simulations show that the CB-comment objective that was introduced to decrease low quality news actually significantly increased it for the Fox News reader population.
Revised: Finally, our simulations show that the CB-comment objective that was introduced to decrease low quality news actually significantly increased it for the Fox News reader population.

Revised: Additional details of this experiment, including model parameters and learning curves are available in Appendix A.

Original: 4. P RODUCTION R ESULTS
Revised: 4. P RODUCTION R ESULTS

Original:  Our production setting is a real world image recommendation system that serves hundreds of millions of users.
Revised: [Table] [Table] Our production setting is a real world image recommendation system that serves hundreds of millions of users.

Original: In our recommendation system interface, users provide feedback in the form of clicks,[Table] likes, dislikes or no feedback.
Revised: In our recommendation system interface, users provide feedback in the form of clicks, likes, dislikes or no feedback.

Original: All four signals are mutually exclusive and the user only provides one feedback after each interaction.
Revised: All four signals are mutually exclusive and the user only provides one feedback after each interaction.

Original: For these experiments, we use data that spans millions of interactions.
Revised: For these experiments, we use data that spans millions of interactions.

Original: The production baseline is a contextual bandit algorithm with a hand-engineered multi-task reward function which dominates approaches that only use click feedback 3 .
Revised: The production baseline is a contextual bandit algorithm with a hand-engineered multi-task reward function which dominates approaches that only use click feedback 4 .

Original: Consequently, any improvements over the production policy imply improvement over any bandit algorithm optimizing for click feedback.
Revised: Consequently, any improvements over the production policy imply improvement over any bandit algorithm optimizing for click feedback.

Original: We implement IGL-P(2) and IGL-P(3) and report the performance as relative lift metrics over the production baseline.
Revised: We implement IGL-P(2) and IGL-P(3) and report the performance as relative lift metrics over the production baseline.

Original: Unlike the simulation setting, we no longer have access to the user’s latent reward after each interaction.
Revised: Unlike the simulation setting, we no longer have access to the user’s latent reward after each interaction.

Original: As a result, we evaluate IGL by comparing all feedback signals.
Revised: As a result, we evaluate IGL by comparing all feedback signals.

Original: An increase in both clicks and likes, and a decrease in dislikes, are considered desirable outcomes.
Revised: An increase in both clicks and likes, and a decrease in dislikes, are considered desirable outcomes.

Original: Table 1 shows the results of our empirical study.
Revised: Table 1 shows the results of our empirical study.

Original: IGL-P(2) exhibits an inability to avoid extreme negative events.
Revised: IGL-P(2) exhibits an inability to avoid extreme negative events.

Original: Although the true latent state is unknown, IGL-P(2) is Pareto-dominated due to an increase in dislikes.
Revised: Although the true latent state is unknown, IGL-P(2) is Pareto-dominated due to an increase in dislikes.

Original: IGL-P(3) does not exhibit this pathology.
Revised: IGL-P(3) does not exhibit this pathology.

Original: These results indicate the utility of a 3 latent state model in real world recommendation systems.
Revised: These results indicate the utility of a 3 latent state model in real world recommendation systems.

Original:   R ELATED W ORK 
Revised: To clearly illustrate the improvement of IGL-P(3) and provide an additional benchmark, we also report that IGL-P(3) significantly outperforms a contextual bandit policy CB-Click using the industry standard CTR reward in Table 2.  R ELATED W ORK 5. E LIMINATING REWARD ENGINEERING Online learning algorithms in supervised learning, contextual bandits, reinforcement learning (Bishop & Nasrabadi, 2006; Kober et al., 2013; Zhou, 2015; Bzdok et al., 2018; Hoi et al., 2021; Brunton & Kutz, 2022; Alsubari et al., 2022) all optimize over hand-crafted scalar rewards.

Original: Recommender systems are a well-studied field due to their direct link to product revenues (Naumov et al., 2019; Steck et al., 2021).
Revised: Partial label learning (PLL) approaches (Feng & An, 2019; Gong et al., 2022; Wang et al., 2022) in supervised learning learn strategies when the underlying ground-truth label are noisy. Branches of reinforcement learning like inverse reinforcement learning (Fu et al., 2017; Arora & Doshi, 2021), imitation learning (Hussein et al., 2017; Oh et al., 2018), robust reinforcement learning (Zhou et al., 2021; Panaganti et al., 2022) exists that learn from historical data without access to the underlying true reward models.

Original: The rapid growth of online content has generated interest in
Revised: However these methods require expert demonstration data, are incapable of generalizing Chen et al. (2022) to different feedback signals, and generally need expensive compute and enormous sampling capabilities for learning. In this context, IGL (Xie et al., 2021; 2022) is a novel paradigm, where a learner’s goal is to optimally interact with the environment with no explicit reward or demonstrations to ground its policies. Our proposed approach to IGL, IGL-P(3) further enables learning of personalized reward functions for different feedback signals.

Revised: 5. R ECOMMENDER S YSTEMS

Revised: Recommender systems are a well-studied field due to their direct link to product revenues (Naumov et al., 2019; Steck et al., 2021).

Original: ML-based solutions (Li et al., 2011) that are able to offer more diverse personalized recommendations tothe internet users.
Revised: The rapid growth of online content has generated interest in MLbased solutions (Li et al., 2011) that are able to offer more diverse personalized recommendations to internet users.

Original: Traditional vanilla recommendation approaches can be divided into three types.
Revised: Traditional vanilla recommendation approaches can be divided into three types.

Original: Content-based approaches (Balabanovi ´ c & Shoham, 1997; IJntema et al., 2010; Kompan & Bielikov ´ a, 2010; Lops et al., 2019; Argyriou et al., 2020; Javed et al., 2021) maintain representations for users based on their content and recommend new content with good similarity metrics for particular users.
Revised: Content-based approaches maintain representations for users based on their content and recommend new content with good similarity metrics for particular users (Balabanovi´c & Shoham, 1997; IJntema et al., 2010; Kompan & Bielikov´a, 2010; Lops et al., 2019; Argyriou et al., 2020; Javed et al., 2021).

Original: In contrast, collaborative filtering approaches (Balabanovi ´ c & Shoham, 1997; Schafer et al., 2007; Hu et al., 2008; Argyriou et al., 2020; Steck & Liang, 2021) employ user rating predictions based on historical consumed content and underlying user similarities.
Revised: In contrast, collaborative filtering approaches employ user rating predictions based on historical consumed content and underlying user similarities (Balabanovi´c & Shoham, 1997; Schafer et al., 2007; Hu et al., 2008; Argyriou et al., 2020; Steck & Liang, 2021).

Original: Finally, there are hybrid approaches (Balabanovi ´ c & Shoham, 1997; Funakoshi & Ohguro, 2000; Burke, 2007; Argyriou et al., 2020; Javed et al., 2021) that combine the previous two contrasting approaches to better represent user profiles for improved recommendations.
Revised: Finally, there are hybrid approaches that combine the previous two contrasting approaches to better represent user profiles for improved recommendations (Balabanovi´c & Shoham, 1997; Funakoshi & Ohguro, 2000; Burke, 2007; Argyriou et al., 2020; Javed et al., 2021).

Original: Our work is a significant departure from these approaches, in that we learn representations for users via their content interaction history for improved diverse personalized recommendations.
Revised: Our work is a significant departure from these approaches, in that we learn representations of user’s communication style via their content interaction history for improved diverse personalized recommendations.

Original: Recommendation as a contextual bandits problem has a rich history (Li et al., 2010; 2011; Bouneffouf et al., 2020).

Original: Typically, implicit signals such as the CTR metric are meticulously incorporated into manually-engineered reward functions (Li et al., 2010; 2011; Bouneffouf et al., 2020).

Original: Variants include using ensembles of contextual bandits Tang et al.

Original: (2014), applying collaborative filtering approaches Gentile et al.

Original: ; Wu et al.

Original: (2016), and using content-based hybrid methods (Li et al., 2010; Ding et al., 2021).

Original: Other works formulate recommendation systems as a reinforcement learning problem (Zou et al., 2019; Lin et al., 2021; Afsar et al., 2021).

Original: All of these crucially depend on the implicit signals for driving the recommendation systems, but suffer from the disconnect that these implicit signals have with respect to the true user satisfaction (Section 1), which hinders the practicality of these approaches.

Original: Alternatively, inverse reinforcement learning based recommendation systems (Chen et al., 2021b; Hu et al., 2022) learn the complex reward structure through demonstrations of expert recommendations.

Original: However inverse reinforcement learning is incapable of generalizing Chen et al.

Original: (2022) to different feedback signals, often posed as an optimization problem to imitate the expert demonstrations that may itself be bad recommendation agents, and generally need expensive compute and enormous sampling capabilities for learning.

Original: Our work is significantly different from these in the following way: (i) weformulate and propose a new recommendation system based on the IGL paradigm, (ii) we leverage the capability of IGL learning rewards based off of implicit and explicit feedback signals and avoid the costly, inefficient, status quo process of reward engineering, and (iii) we  propose a novel personalized IGL algorithm based on the inverse kinematics strategy as described in Section 3.
Revised: Our work is significantly different from these in the following ways: (i) we propose a novel personalized IGL algorithm based on the inverse kinematics strategy as described in Section 3, (ii) we leverage the capability of IGL learning rewards based off of implicit and explicit feedback signals and avoid the costly, inefficient, status quo process of reward engineering, and (iii) we formulate and propose a new recommendation system based on the IGL paradigm.

Original: The works that are closest to ours are Xie et al.
Revised: The works that are closest to ours are Xie et al.

Original: (2021; 2022) which introduce and solve for the IGL paradigm under different assumptions.
Revised: (2021; 2022) which introduce and solve for the IGL paradigm under different assumptions.

Original: However, we propose a personalized IGL algorithm for recommendation systems with improved reward predictor models, more practical assumptions on feedback signals, and more intricacies described in Section 2.2.1.
Revised: However, we propose a personalized IGL algorithm for recommendation systems with improved reward predictor models, more practical assumptions on feedback signals, and more intricacies described in Section 2.2.1.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We evaluated the proposed personalized IGL approach ( IGL-P ) in three different settings: (1) A simulation using a supervised classification dataset shows that IGL-P can learn to successfully distinguish between different communication modalities; (2) A simulation for online news recommendation based on real data from Facebook users shows that IGL-P leverages insights about different communication modalities to learn better policies and achieve fairness with consistent performance among diverse user groups; (3) A real-world experiment deployed in an image recommendation product showcases that the proposed method outperforms the hand-engineered reward baseline, and succeeds in a practical application.
Revised: We evaluated the proposed personalized IGL approach ( IGL-P ) in three different settings: (1) A simulation using a supervised classification dataset shows that IGL-P can learn to successfully distinguish between different communication modalities; (2) A simulation for online news recommendation based on real data from Facebook users shows that IGL-P leverages insights about different communication modalities to learn better policies and achieve fairness with consistent performance among diverse user groups; (3) A real-world experiment deployed in an image recommendation product showcases that the proposed method outperforms the hand-engineered reward baseline, and succeeds in a practical application.

Original: This work assumes that user may communicate in different ways, but a given user expresses (dis)satisfaction or indifference to all content in the same way.
Revised: This work assumes that users may communicate in different ways, but a given user expresses (dis)satisfaction or indifference to all content in the same way.

Original: This assumption was critical to deriving the inverse kinematics approach, but in practice user feedback can also depend upon content (Freeman et al., 2020).
Revised: This assumption was critical to deriving the inverse kinematics approach, but in practice user feedback can also depend upon content (Freeman et al., 2020).

Original: IGL with arbitrary joint content-action dependence of feedback is intractable, but plausibly there exists a tractable IGL setting with a constrained joint content-action dependence which is a better fit for recommendation scenarios.
Revised: IGL with arbitrary joint content-action dependence of feedback is intractable, but plausibly there exists a tractable IGL setting with a constrained joint content-action dependence which is a better fit for recommendation scenarios.

Original: Furthermore, although we established that utility of a three state model over a two state model in our experiments, perhaps more than three states is necessary for more complex recommendation scenarios.
Revised: Furthermore, although we established the utility of a three state model over a two state model in our experiments, perhaps more than three states is necessary for more complex recommendation scenarios.

Original: We also explore the possibilities of achieving fairness through personalized reward learning. Our findings build off observations in the literature and are empirical results showing the potential for consistent performance using IGL-P . By utilizing personalized rewards, unlike alternative approaches, IGL is not incentivized to sacrifice performance for sub-populations in order to achieve better average performance across all users.
Revised: Although we consider the application of recommender systems, personalized reward learning can benefit any application suffering from a one-size-fits-all approach. One candidate domain is the detection of underlying medical conditions and development of new treatments and interventions.

Original: Theoretical guarantees of IGL-P fairness remain an interesting future direction. Potential obstacles to guaranteed fairness might arise due to insufficient data from underrepresented populations, as well as scarcity of a negative oracle signal in  the 3 latent state setting.
Revised: Systematic misdiagnoses in subgroups with symptoms different than the “norm” are tragically standard in medicine. IGL-P can empower medical practitioners to address systematic inequality by tuning diagnostic tools via personalization and moving past an oblivious diagnostic approach.

Original: Although IGL-P provably learns even with very rare presence of a negative oracle signal, inconsistent performance across user subsets can arise due to faster learning and convergence for populations that utilize the negative oracle signal more frequently.
Revised: Other potential applications are learned brain-computer and human-computer interfaces (Xie et al., 2022). One example is a self-calibrating eye tracker where people’s response to erroneous agent control is idiosyncratic (Gao et al., 2021).

Original: E THICS S TATEMENT
Revised: E THICS S TATEMENT

Original: In our paper, we use two real-world interaction datasets.
Revised: In our paper, we use two real-world interaction datasets.

Original: The first is a publicly available dataset of interactions with public Facebook news pages.
Revised: The first is a publicly available dataset of interactions with public Facebook news pages.

Original: All interactions are anonymous and the identities of users are excluded from the dataset, preserving their privacy.
Revised: All interactions are anonymous and the identities of users are excluded from the dataset, preserving their privacy.

Original: Similarly, our production data does not include user information and was used with the consent and permission of all relevent parties.
Revised: Similarly, our production data does not include user information and was used with the consent and permission of all relevant parties.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: We have taken considerable measures to ensure the results are as reproducible as possible.
Revised: We have taken considerable measures to ensure the results are as reproducible as possible.

Original: We have provided the code to replicate our experiment results (except for the production results in
Revised: We have provided the code to replicate our experiment results (except for the production results in

Original: Section 4.3) as part of the supplementary material.
Revised: Section 4.3) as part of the supplementary material.

Original: The code will be made publicly available at { url redacted } .
Revised: The code will be made publicly available at { url redacted } .

Original: We used publicly available datasets for our simulated experiments in Section 4.
Revised: We used publicly available datasets for our simulated experiments in Section 4.

Original: (Blackard & Dean, 1999) and Section 4.2 Martinchek (2016).
Revised: (Blackard & Dean, 1999) and Section 4.2 Martinchek (2016).

Original: The experiment code for Section 4.1, when executed, will automatically download the dataset.
Revised: The experiment code for Section 4.1, when executed, will automatically download the dataset.

Original: The dataset for Section 4.2 is included as part of our supplementary material.
Revised: The dataset for Section 4.2 is included as part of our supplementary material.

Original: Finally, our supplementary material also includes a conda environment file to help future researchers recreate our development environment on their machines when running the experiments.
Revised: Finally, our supplementary material also includes a conda environment file to help future researchers recreate our development environment on their machines when running the experiments.