Original: H IDDEN I NCENTIVES FOR A UTO - INDUCED D ISTRIBUTIONAL S HIFT
Revised: H IDDEN I NCENTIVES FOR A UTO - INDUCED D ISTRIBUTIONAL S HIFT

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Decisions made by machine learning systems have increasing inﬂuence on the world, yet it is common for machine learning algorithms to assume that no such inﬂuence exists.
Revised: Decisions made by machine learning systems have increasing inﬂuence on the world, yet it is common for machine learning algorithms to assume that no such inﬂuence exists.

Original: An example is the use of the i.i.d.
Revised: An example is the use of the i.i.d.

Original: assumption in content recommendation.
Revised: assumption in content recommendation.

Original: In fact, the (choice of) content displayed can change users’ perceptions and preferences, or even drive them away, causing a shift in the distribution of users.
Revised: In fact, the (choice of) content displayed can change users’ perceptions and preferences, or even drive them away, causing a shift in the distribution of users.

Original: We introduce the term auto-induced distributional shift (ADS) to describe the phenomenon of an algorithm causing a change in the distribution of its own inputs.
Revised: We introduce the term auto-induced distributional shift (ADS) to describe the phenomenon of an algorithm causing a change in the distribution of its own inputs.

Original: Our goal is to ensure that machine learning systems do not leverage ADS to increase performance when doing so could be undesirable.
Revised: Our goal is to ensure that machine learning systems do not leverage ADS to increase performance when doing so could be undesirable.

Original: We demonstrate that changes to the learning algorithm, such as the introduction of meta-learning, can cause hidden incentives for auto-induced distributional shift (HI-ADS) to be revealed.
Revised: We demonstrate that changes to the learning algorithm, such as the introduction of meta-learning, can cause hidden incentives for auto-induced distributional shift (HI-ADS) to be revealed.

Original: To address this issue, we introduce ‘unit tests’ and a mitigation strategy for HI-ADS, as well as a toy environment for modelling real-world issues with HIADS in content recommendation, where we demonstrate that strong meta-learners achieve gains in performance via ADS.
Revised: To address this issue, we introduce ‘unit tests’ and a mitigation strategy for HI-ADS, as well as a toy environment for modelling real-world issues with HIADS in content recommendation, where we demonstrate that strong meta-learners achieve gains in performance via ADS.

Original: We show meta-learning and Q-learning both sometimes fail unit tests, but pass when using our mitigation strategy.
Revised: We show meta-learning and Q-learning both sometimes fail unit tests, but pass when using our mitigation strategy.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Consider a content recommendation system whose performance is measured by accuracy of predicting what users will click.
Revised: Consider a content recommendation system whose performance is measured by accuracy of predicting what users will click.

Original: This system can achieve better performance by either 1) making better predictions, or 2) changing the distribution of users such that predictions are easier to make.
Revised: This system can achieve better performance by either 1) making better predictions, or 2) changing the distribution of users such that predictions are easier to make.

Original: We propose the term auto-induced distributional shift (ADS) to describe this latter kind of distributional shift, caused by the algorithm’s own predictions or behaviour (Figure 1).
Revised: We propose the term auto-induced distributional shift (ADS) to describe this latter kind of distributional shift, caused by the algorithm’s own predictions or behaviour (Figure 1).

Original: ADS are not inherently bad, and are sometimes even desirable.
Revised: ADS are not inherently bad, and are sometimes even desirable.

Original: But they can cause problems if they occur unexpectedly.
Revised: But they can cause problems if they occur unexpectedly.

Original: It is typical in machine learning (ML) to assume (e.g. via the i.i.d. assumption) that (2) will not happen.
Revised: It is typical in machine learning (ML) to assume (e.g. via the i.i.d. assumption) that (2) will not happen.

Original: However, given the increasing real-world use of ML algorithms, we believe it is important to model and experimentally observe what happens when assumptions like this are violated.
Revised: However, given the increasing real-world use of ML algorithms, we believe it is important to model and experimentally observe what happens when assumptions like this are violated.

Original: This is the motivation of our work.
Revised: This is the motivation of our work.

Original: In many cases, including news recommendation, we would consider (2) a form of cheating —the algorithm changed the task rather than solving it as intended.
Revised: In many cases, including news recommendation, we would consider (2) a form of cheating —the algorithm changed the task rather than solving it as intended.

Original: We care which means the algorithm used to solve the problem (e.g. (1) and/or (2)), but we only told it about the ends , so it didn’t know not to ’cheat’.
Revised: We care which means the algorithm used to solve the problem (e.g. (1) and/or (2)), but we only told it about the ends , so it didn’t know not to ’cheat’.

Original: This is an example of a speciﬁcation problem (Leike et al., 2017; Ortega et al., 2018): a problem which arises from a discrepancy between the performance metric (maximize accuracy) and “what we really meant”: in this case, to maximize accuracy via (1).
Revised: This is an example of a speciﬁcation problem (Leike et al., 2017; Ortega et al., 2018): a problem which arises from a discrepancy between the performance metric (maximize accuracy) and “what we really meant”: in this case, to maximize accuracy via (1).

Original: Ideally, we’d like to quantify the desirability of all possible means, e.g. assign appropriate rewards to all potential strategies and “side-effects”, but this is intractable for real-world settings.
Revised: Ideally, we’d like to quantify the desirability of all possible means, e.g. assign appropriate rewards to all potential strategies and “side-effects”, but this is intractable for real-world settings.

Original: Using human feedback to learn reward functions which account for such impacts is a promising approach to specifying desired behavior (Leike et al., 2018; Christiano et al., 2017).
Revised: Using human feedback to learn reward functions which account for such impacts is a promising approach to specifying desired behavior (Leike et al., 2018; Christiano et al., 2017).

Original: But the same issue can arise whenever human feedback is used in training: one means of improving performance could be to alter human preferences, making them easier to satisfy.
Revised: But the same issue can arise whenever human feedback is used in training: one means of improving performance could be to alter human preferences, making them easier to satisfy.

Original: Thus in this work, we pursue a complementary approach: managing learners’ incentives .
Revised: Thus in this work, we pursue a complementary approach: managing learners’ incentives .

Original: A learner has an incentive to behave in a certain way when doing so can increase performance (e.g. accuracy or reward).
Revised: A learner has an incentive to behave in a certain way when doing so can increase performance (e.g. accuracy or reward).

Original: Informally, we say an incentive is hidden when the learner behaves as if it were not present.
Revised: Informally, we say an incentive is hidden when the learner behaves as if it were not present.

Original: But we note that changes to the learning algorithm or training regime could cause previously hidden incentives to be revealed, resulting in unexpected and potentially undesirable behaviour.
Revised: But we note that changes to the learning algorithm or training regime could cause previously hidden incentives to be revealed, resulting in unexpected and potentially undesirable behaviour.

Original: Managing incentives (e.g. controlling which incentives are hidden/ revealed) can allow algorithm designers to disincentivize broad classes of strategies (such as any that rely on manipulating human preferences) without knowing their exact instantiation.
Revised: Managing incentives (e.g. controlling which incentives are hidden/ revealed) can allow algorithm designers to disincentivize broad classes of strategies (such as any that rely on manipulating human preferences) without knowing their exact instantiation.

Original: The goal of our work is to provide insights and practical tools for understanding and managing incentives, speciﬁcally hidden incentives for auto-induced distributional shift : HI-ADS .
Revised: The goal of our work is to provide insights and practical tools for understanding and managing incentives, speciﬁcally hidden incentives for auto-induced distributional shift : HI-ADS .

Original: To study which conditions cause HI-ADS to be revealed, we present unit tests for detecting HI-ADS in supervised learning (SL) and reinforcement learning (RL).
Revised: To study which conditions cause HI-ADS to be revealed, we present unit tests for detecting HI-ADS in supervised learning (SL) and reinforcement learning (RL).

Original: We also create an environment that models
Revised: We also create an environment that models

Original: ADS in news recommendation, illustrating possible effects of revealing HI-ADS in this setting.
Revised: ADS in news recommendation, illustrating possible effects of revealing HI-ADS in this setting.

Original: The unit tests both have two means by which the learner can improve performance: one which creates
Revised: The unit tests both have two means by which the learner can improve performance: one which creates

Original: ADS and one which does not.
Revised: ADS and one which does not.

Original: The intended method of improving performance is one that does not induce ADS; the other is ’hidden’ and we want it to remain hidden.
Revised: The intended method of improving performance is one that does not induce ADS; the other is ’hidden’ and we want it to remain hidden.

Original: A learner "fails" the unit test if it nonetheless pursues the incentive to increase performance via ADS.
Revised: A learner "fails" the unit test if it nonetheless pursues the incentive to increase performance via ADS.

Original: In both the RL and SL unit tests, we ﬁnd that introducing an outer-loop of meta-learning (e.g. Population-Based Training (PBT)
Revised: In both the RL and SL unit tests, we ﬁnd that introducing an outer-loop of meta-learning (e.g. Population-Based Training (PBT)

Original: Jaderberg et al.
Revised: Jaderberg et al.

Original: (2017)) can lead to high levels of failure.
Revised: (2017)) can lead to high levels of failure.

Original: Similarly, recommender systems trained with PBT induce larger drifts in user base and user interests.
Revised: Similarly, recommender systems trained with PBT induce larger drifts in user base and user interests.

Original: These results suggest that failure of our unit tests indicates that an algorithm is prone to revealing HI-ADS in other settings.
Revised: These results suggest that failure of our unit tests indicates that an algorithm is prone to revealing HI-ADS in other settings.

Original: Finally, we propose and test a mitigation strategy we call context swapping .
Revised: Finally, we propose and test a mitigation strategy we call context swapping .

Original: The strategy consists of rotating learners through different environments throughout learning, so that they can’t see the results or correlations of their actions in one environment over longer time horizons.
Revised: The strategy consists of rotating learners through different environments throughout learning, so that they can’t see the results or correlations of their actions in one environment over longer time horizons.

Original: This effectively mitigates
Revised: This effectively mitigates

Original: HI-ADS in our unit test environments, but did not work well in content recommendation experiments.
Revised: HI-ADS in our unit test environments, but did not work well in content recommendation experiments.

Original:  B ACKGROUND 2. M ETA - LEARNING AND POPULATION BASED TRAINING
Revised:  B ACKGROUND 2. M ETA - LEARNING AND POPULATION BASED TRAINING

Original: Meta-learning is the use of machine learning techniques to learn machine learning algorithms.
Revised: Meta-learning is the use of machine learning techniques to learn machine learning algorithms.

Original: This involves instantiating multiple learning scenarios which run in an inner loop (IL) , while an outer loop (OL) uses the outcomes of the inner loop(s) as data-points from which to learn which learning algorithms are most effective (Metz et al., 2019).
Revised: This involves instantiating multiple learning scenarios which run in an inner loop (IL) , while an outer loop (OL) uses the outcomes of the inner loop(s) as data-points from which to learn which learning algorithms are most effective (Metz et al., 2019).

Original: The number of IL steps per OL step is called the interval .
Revised: The number of IL steps per OL step is called the interval .

Original: Many recent works focus on multi-task meta-learning, where the OL seeks to ﬁnd learning rules that generalize to unseen tasks by training the IL on a distribution of tasks (Finn et al., 2017).
Revised: Many recent works focus on multi-task meta-learning, where the OL seeks to ﬁnd learning rules that generalize to unseen tasks by training the IL on a distribution of tasks (Finn et al., 2017).

Original: Single-task meta-learning includes learning an optimizer for a single task
Revised: Single-task meta-learning includes learning an optimizer for a single task

Original: (Gong et al., 2018), and adaptive methods for selecting models (Kalousis, 2000) or setting hyperparameters (Snoek et al., 2012).
Revised: (Gong et al., 2018), and adaptive methods for selecting models (Kalousis, 2000) or setting hyperparameters (Snoek et al., 2012).

Original: For simplicity in this initial study we focus on single-task meta-learning.
Revised: For simplicity in this initial study we focus on single-task meta-learning.

Original: Population-based training (PBT; Jaderberg et al., 2017) is a meta-learning algorithm that trains multiple learners L 1 , ..., L n in parallel, after each interval ( T steps of IL) applying an evolutionary
Revised: Population-based training (PBT; Jaderberg et al., 2017) is a meta-learning algorithm that trains multiple learners L 1 , ..., L n in parallel, after each interval ( T steps of IL) applying an evolutionary

Original: OL step which consists of: (1) Evaluate the performance of each learner, (2) Replace both parameters and hyperparameters of 20% lowest-performing learners with copies of those from the 20%
Revised: OL step which consists of: (1) Evaluate the performance of each learner, (2) Replace both parameters and hyperparameters of 20% lowest-performing learners with copies of those from the 20%

Original: high-
Revised: high-

Original: performing learners (EXPLOIT).
Revised: performing learners (EXPLOIT).

Original: (3) Randomly perturb the hyperparameters (but not the parameters) of all
Revised: (3) Randomly perturb the hyperparameters (but not the parameters) of all

Original: learners (
Revised: learners (

Original: EXPLORE
Revised: EXPLORE

Original: 2. D ISTRIBUTIONAL SHIFT AND CONTENT RECOMMENDATION
Revised: 2. D ISTRIBUTIONAL SHIFT AND CONTENT RECOMMENDATION

Original: In general, distributional shift refers to change of the data distribution over time.
Revised: In general, distributional shift refers to change of the data distribution over time.

Original: In supervised learning with data x and labels y , this can be more speciﬁcally described as dataset shift: change in the joint distribution of P ( x , y ) between the training and test sets (Moreno-Torres et al., 2012;
Revised: In supervised learning with data x and labels y , this can be more speciﬁcally described as dataset shift: change in the joint distribution of P ( x , y ) between the training and test sets (Moreno-Torres et al., 2012;

Original: Quionero-Candela et al., 2009).
Revised: Quionero-Candela et al., 2009).

Original: As identiﬁed by Moreno-Torres et al.
Revised: As identiﬁed by Moreno-Torres et al.

Original: (2012), two common kinds of shift are: (1) Covariate shift : changing P ( x ) .
Revised: (2012), two common kinds of shift are: (1) Covariate shift : changing P ( x ) .

Original: In the example of content recommendation, this corresponds to changing the user base of the recommendation system.
Revised: In the example of content recommendation, this corresponds to changing the user base of the recommendation system.

Original: For instance, a media outlet which publishes inﬂammatory content may appeal to users with extreme views while alienating more moderate users.
Revised: For instance, a media outlet which publishes inﬂammatory content may appeal to users with extreme views while alienating more moderate users.

Original: This self-selection effect (Kayhan, 2015) may appear to a recommendation system as an increase in performance, leading to a feedback effect, as previously noted by Shah et al.
Revised: This self-selection effect (Kayhan, 2015) may appear to a recommendation system as an increase in performance, leading to a feedback effect, as previously noted by Shah et al.

Original: This type of feedback effect has been identiﬁed as contributing to ﬁlter bubbles and radicalization (Pariser, 2011; Kayhan, 2015).
Revised: This type of feedback effect has been identiﬁed as contributing to ﬁlter bubbles and radicalization (Pariser, 2011; Kayhan, 2015).

Original: (2) Concept shift : changing P ( y | x ) .
Revised: (2) Concept shift : changing P ( y | x ) .

Original: In the example of content recommendation, this corresponds to changing a given user’s interest in different kinds of content.
Revised: In the example of content recommendation, this corresponds to changing a given user’s interest in different kinds of content.

Original: For example, exposure to a fake news story has been shown to increase the perceived accuracy of (and thus presumably future interest in) the content, an example of the illusory truth effect (Pennycook et al., 2019).
Revised: For example, exposure to a fake news story has been shown to increase the perceived accuracy of (and thus presumably future interest in) the content, an example of the illusory truth effect (Pennycook et al., 2019).

Original: For further details on such effects in content recommendation, see Appendix 1.
Revised: For further details on such effects in content recommendation, see Appendix 8.

Original:  A UTO - INDUCED D ISTRIBUTION S HIFT (ADS)
Revised:  A UTO - INDUCED D ISTRIBUTION S HIFT (ADS)

Original: Auto-induced distribution shift (ADS) is distributional shift caused by an algorithm’s behaviour.
Revised: Auto-induced distribution shift (ADS) is distributional shift caused by an algorithm’s behaviour.

Original: This is in contrast to distributional shift which would happen even if the learner were not present - e.g. for a crash prediction algorithm trained on data from the summer, encountering snowy roads is an example of distributional shift, but not auto-induced distributional shift (ADS).
Revised: This is in contrast to distributional shift which would happen even if the learner were not present - e.g. for a crash prediction algorithm trained on data from the summer, encountering snowy roads is an example of distributional shift, but not auto-induced distributional shift (ADS).

Original: [Figure]
Revised: [Figure]

Original: We emphasize that ADS are not inherently bad or good; often ADS can even be desirable: consider an algorithm meant to alert drivers of imminent collisions.
Revised: We emphasize that ADS are not inherently bad or good; often ADS can even be desirable: consider an algorithm meant to alert drivers of imminent collisions.

Original: If it works well, such a system will help drivers avoid crashing, thus making self-refuting predictions which result in ADS.
Revised: If it works well, such a system will help drivers avoid crashing, thus making self-refuting predictions which result in ADS.

Original: What separates desirable and undesirable ADS?
Revised: What separates desirable and undesirable ADS?

Original: The collision-alert system alters its data distribution in a way that is aligned with the goal of fewer collisions, whereas the news manipulation results in changes that are misaligned with the goal of better predicting existing users’ interests (Leike et al., 2018).
Revised: The collision-alert system alters its data distribution in a way that is aligned with the goal of fewer collisions, whereas the news manipulation results in changes that are misaligned with the goal of better predicting existing users’ interests (Leike et al., 2018).

Original: In reinforcement learning (RL), ADS are typically encouraged as a means to increase performance.
Revised: In reinforcement learning (RL), ADS are typically encouraged as a means to increase performance.

Original: On the other hand, in supervised learning (SL), the i.i.d.
Revised: On the other hand, in supervised learning (SL), the i.i.d.

Original: assumption precludes ADS in theory.
Revised: assumption precludes ADS in theory.

Original: In practice, however, the possibility of using ADS to increase performance (and thus an incentive to do so) often remains.
Revised: In practice, however, the possibility of using ADS to increase performance (and thus an incentive to do so) often remains.

Original: For instance, this occurs in online learning.
Revised: For instance, this occurs in online learning.

Original: In our experiments, we explicitly model such situations where i.i.d.
Revised: In our experiments, we explicitly model such situations where i.i.d.

Original: assumptions are violated: We study the behavior of SL and myopic
Revised: assumptions are violated: We study the behavior of SL and myopic

Original: RL algorithms, in environments designed to include incentives for ADS, in order to understand when incentives are effectively hidden.
Revised: RL algorithms, in environments designed to include incentives for ADS, in order to understand when incentives are effectively hidden.

Original: Fig.
Revised: Fig.

Original: 2 contrasts these settings with typical RL and SL.
Revised: 2 contrasts these settings with typical RL and SL.

Original:  I NCENTIVES
Revised:  I NCENTIVES

Original: For our study of incentives, we use the following terminology: an incentive for a behavior (e.g. an action, a classiﬁcation, etc.) is present (not absent ) to the extent that the behaviour will increase performance (e.g. reward, accuracy, etc.)
Revised: For our study of incentives, we use the following terminology: an incentive for a behavior (e.g. an action, a classiﬁcation, etc.) is present (not absent ) to the extent that the behaviour will increase performance (e.g. reward, accuracy, etc.)

Original: (Everitt & Hutter, 2019).
Revised: (Everitt & Hutter, 2019).

Original: This incentive is revealed to (not hidden from) a learner if it would, at higher than chance levels, learn to perform the behavior given sufﬁcient capacity and training experience.
Revised: This incentive is revealed to (not hidden from) a learner if it would, at higher than chance levels, learn to perform the behavior given sufﬁcient capacity and training experience.

Original: The incentive is pursued (not eschewed ) by a learner if it actually performs the incentivized behaviour.
Revised: The incentive is pursued (not eschewed ) by a learner if it actually performs the incentivized behaviour.

Original: Note even when an incentive is revealed, it may not be pursued, e.g. due to limited capacity and/or data, or simply chance.
Revised: Note even when an incentive is revealed, it may not be pursued, e.g. due to limited capacity and/or data, or simply chance.

Original: See Fig 3.
Revised: See Fig 3.

Original: For example, in content recommendation, the incentive to drive users away is present if some user types are easier to predict than others.
Revised: For example, in content recommendation, the incentive to drive users away is present if some user types are easier to predict than others.

Original: But this incentive may be hidden from the learner by using a myopic algorithm, e.g. one that does not see the effects of its actions on the distribution of users.
Revised: But this incentive may be hidden from the learner by using a myopic algorithm, e.g. one that does not see the effects of its actions on the distribution of users.

Original: The incentive might instead be revealed to the outer loop of a meta-learning algorithm like PBT, which does see the effects of learner’s actions.
Revised: The incentive might instead be revealed to the outer loop of a meta-learning algorithm like PBT, which does see the effects of learner’s actions.

Original: Even when this incentive is revealed, however, it might not end up being pursued .
Revised: Even when this incentive is revealed, however, it might not end up being pursued .

Original: For example, this could happen if predicting which recommendations will drive away users is too difﬁcult a learning problem, or if the incentive to do so is dominated by other incentives (e.g. change individual users’ interests, or improve accuracy of predictions).
Revised: For example, this could happen if predicting which recommendations will drive away users is too difﬁcult a learning problem, or if the incentive to do so is dominated by other incentives (e.g. change individual users’ interests, or improve accuracy of predictions).

Original: In general, it may be difﬁcult to determine empirically which incentives are revealed, because failure to pursue an incentive can be due to limited capacity, insufﬁcient training, and/or random chance.
Revised: In general, it may be difﬁcult to determine empirically which incentives are revealed, because failure to pursue an incentive can be due to limited capacity, insufﬁcient training, and/or random chance.

Original: To address this challenge, we devise extremely simple environments (“unit tests”), where we can be conﬁdent that revealed incentives will be pursued.
Revised: To address this challenge, we devise extremely simple environments (“unit tests”), where we can be conﬁdent that revealed incentives will be pursued.

Original: 4. H IDDEN I NCENTIVES FOR A UTO - INDUCED D ISTRIBUTIONAL SHIFT (HI
Revised: 4. H IDDEN I NCENTIVES FOR A UTO - INDUCED D ISTRIBUTIONAL SHIFT (HI

Original: -ADS)
Revised: -ADS)

Original: Following from the deﬁnitions in Sections 3 and 4, HI-ADS are incentives for behaviors that cause
Revised: Following from the deﬁnitions in Sections 3 and 4, HI-ADS are incentives for behaviors that cause

Original: Auto-induced Distributional Shift that are hidden from the learner, i.e. the learner would not learn to perform the incentivized behaviors at higher than chance levels, even given inﬁnite capacity and training experience.
Revised: Auto-induced Distributional Shift that are hidden from the learner, i.e. the learner would not learn to perform the incentivized behaviors at higher than chance levels, even given inﬁnite capacity and training experience.

Original: Like ADS, HI-ADS are not necessarily problematic.
Revised: Like ADS, HI-ADS are not necessarily problematic.

Original: Indeed, hiding incentives can be an effective method of inﬂuencing learner behavior.
Revised: Indeed, hiding incentives can be an effective method of inﬂuencing learner behavior.

Original: For example, hiding the incentive to manipulate users from a content recommendation algorithm could prevent it from inﬂuencing users in a way they would not endorse.
Revised: For example, hiding the incentive to manipulate users from a content recommendation algorithm could prevent it from inﬂuencing users in a way they would not endorse.

Original: However, if machine learning practitioners are not aware that incentives are present, or that properties of the learning algorithm are hiding them, then seemingly innocuous changes to the learning algorithm may reveal HI-ADS, and lead to signiﬁcant unexpected changes in behavior.
Revised: However, if machine learning practitioners are not aware that incentives are present, or that properties of the learning algorithm are hiding them, then seemingly innocuous changes to the learning algorithm may reveal HI-ADS, and lead to signiﬁcant unexpected changes in behavior.

Original: Hiding incentives for ADS may seem counter-intuitive and counter-productive in the context of reinforcement learning (RL), where moving towards high-reward states is typically desirable.
Revised: Hiding incentives for ADS may seem counter-intuitive and counter-productive in the context of reinforcement learning (RL), where moving towards high-reward states is typically desirable.

Original: However, for real-world applications of RL, the ultimate goal is not a system that achieves high reward, but rather one that behaves according to the designer’s intentions.
Revised: However, for real-world applications of RL, the ultimate goal is not a system that achieves high reward, but rather one that behaves according to the designer’s intentions.

Original: And as we discussed in the introduction, it can be intractable to design reward functions that perfectly specify intended behavior.
Revised: And as we discussed in the introduction, it can be intractable to design reward functions that perfectly specify intended behavior.

Original: Thus managing (e.g. hiding)
Revised: Thus managing (e.g. hiding)

Original: some incentives can provide a useful tool for speciﬁcation, even in RL.
Revised: some incentives can provide a useful tool for speciﬁcation, even in RL.

Original: We have several reasons for focusing on HI-ADS: (1) The issue of HI-ADS has not yet been identiﬁed, and thus is liable to be neglected in practice.
Revised: We have several reasons for focusing on HI-ADS: (1) The issue of HI-ADS has not yet been identiﬁed, and thus is liable to be neglected in practice.

Original: Indeed, our “unit tests” are the ﬁrst published empirical methodology for assessing whether incentives are hidden or revealed by different learning algorithms.
Revised: Indeed, our “unit tests” are the ﬁrst published empirical methodology for assessing whether incentives are hidden or revealed by different learning algorithms.

Original: (2) Machine learning algorithms are commonly deployed in settings where ADS are present, violating assumptions used to analyze their properties theoretically.
Revised: (2) Machine learning algorithms are commonly deployed in settings where ADS are present, violating assumptions used to analyze their properties theoretically.

Original: This means learners could exploit ADS in unexpected and undesirable ways if incentives for ADS are revealed.
Revised: This means learners could exploit ADS in unexpected and undesirable ways if incentives for ADS are revealed.

Original: Hiding these incentives heuristically (e.g. via off-line training) is a common approach, but potentially brittle (if practitioners don’t understand how HI-ADS could become revealed).
Revised: Hiding these incentives heuristically (e.g. via off-line training) is a common approach, but potentially brittle (if practitioners don’t understand how HI-ADS could become revealed).

Original: In particular, meta-learning can reveal
Revised: In particular, meta-learning can reveal

Original: HI-ADS in online learning settings.
Revised: HI-ADS in online learning settings.

Original: (3) Substantial real-world issues could result from improper management of learner’s incentives.
Revised: (3) Substantial real-world issues could result from improper management of learner’s incentives.

Original: Examples include tampering with human-generated reward signals (Everitt & Hutter, 2018)
Revised: Examples include tampering with human-generated reward signals (Everitt & Hutter, 2018)

Original: (e.g. selecting news articles which manipulate user interests), and creating “self-fulﬁlling prophecies” (e.g. driving up the value of an asset by publicly predicting its value will increase (Armstrong & O’Rorke, 2017)).
Revised: (e.g. selecting news articles which manipulate user interests), and creating “self-fulﬁlling prophecies” (e.g. driving up the value of an asset by publicly predicting its value will increase (Armstrong & O’Rorke, 2017)).

Original: 4. R EMOVING HI-ADS VIA C ONTEXT S WAPPING
Revised: 4. R EMOVING HI-ADS VIA C ONTEXT S WAPPING

Original: We propose a technique called context swapping for removing incentives for ADS.
Revised: We propose a technique called context swapping for removing incentives for ADS.

Original: The technique trains N learners in parallel, and shufﬂes the learners through N different copies of the same (or similar) environments.
Revised: The technique trains N learners in parallel, and shufﬂes the learners through N different copies of the same (or similar) environments.

Original: We use a deterministic permutation of learners in environment copies, so that the i -th learner inhabits the j -th environment on time-steps t where j = ( i + t ) mod N , makes an observation, takes an action, and receives a reward before moving to the next environment.
Revised: We use a deterministic permutation of learners in environment copies, so that the i -th learner inhabits the j -th environment on time-steps t where j = ( i + t ) mod N , makes an observation, takes an action, and receives a reward before moving to the next environment.

Original: When N is larger than the interval of the OL optimizer, each learner inhabits each copy for at most a single time-step before an OL step is applied.
Revised: When N is larger than the interval of the OL optimizer, each learner inhabits each copy for at most a single time-step before an OL step is applied.

Original: Under the assumption that different copies of the environment do not inﬂuence each other, this technique can address HI-ADS in practice, as we show in Sec. 5.1.1.
Revised: Under the assumption that different copies of the environment do not inﬂuence each other, this technique can address HI-ADS in practice, as we show in Sec. 5.1.1.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In Section 5.1, we introduce ‘unit tests’ for HI-ADS.
Revised: In Section 5.1, we introduce ‘unit tests’ for HI-ADS.

Original: Our primary goal with these experiments is to convey a crisp understanding of potential issues caused by revealing HI-ADS.
Revised: Our primary goal with these experiments is to convey a crisp understanding of potential issues caused by revealing HI-ADS.

Original: Put simply, our experiments show that you can have a learner which behaves as intended, and just by using metalearning (e.g. PBT), without changing the performance metric (e.g. loss or rewards), the learner’s behavior can change completely.
Revised: Put simply, our experiments show that you can have a learner which behaves as intended, and just by using metalearning (e.g. PBT), without changing the performance metric (e.g. loss or rewards), the learner’s behavior can change completely.

Original: We also show that context swapping is an effective mitigation technique in these environments.
Revised: We also show that context swapping is an effective mitigation technique in these environments.

Original: On the practical side, the unit tests can be used to compare learning algorithms and diagnose their propensity to reveal incentives.
Revised: On the practical side, the unit tests can be used to compare learning algorithms and diagnose their propensity to reveal incentives.

Original: In Section 5.2, we model a content recommendation system.
Revised: In Section 5.2, we model a content recommendation system.

Original: The goal of these experiments is to demonstrate how HI-ADS could create issues for real-world content recommendation systems such as news feeds, search results, or automated suggestions.
Revised: The goal of these experiments is to demonstrate how HI-ADS could create issues for real-world content recommendation systems such as news feeds, search results, or automated suggestions.

Original: They also validate the usefulness of the unit tests: algorithms that failed the unit tests also reveal HI-ADS in this setting.
Revised: They also validate the usefulness of the unit tests: algorithms that failed the unit tests also reveal HI-ADS in this setting.

Original: We emphasize that ADS takes place in this environment by construction .
Revised: We emphasize that ADS takes place in this environment by construction .

Original: The point of our experiments is that meta-learning can increase the rate and/or extent of ADS, by revealing this incentive.
Revised: The point of our experiments is that meta-learning can increase the rate and/or extent of ADS, by revealing this incentive.

Original: We ﬁnd that context swapping is not effective in this environment, highlighting the need for alternative mitigation strategies.
Revised: We ﬁnd that context swapping is not effective in this environment, highlighting the need for alternative mitigation strategies.

Original: 5. HI-ADS U NIT T ESTS
Revised: 5. HI-ADS U NIT T ESTS

Original: Unit test 1: Supervised Learning.
Revised: Unit test 1: Supervised Learning.

Original: This unit test consists of a simple prediction problem.
Revised: This unit test consists of a simple prediction problem.

Original: There are no inputs, only an underlying state s ∈ { 0 , 1 } , and targets y ∈ R 2 with y 1 , y 2 ∼ N (0 , s ∗ σ 2 ) , N (0 , 1) , with corresponding predictions ˆ y 1 , ˆ y 2 .
Revised: There are no inputs, only an underlying state s ∈ { 0 , 1 } , and targets y ∈ R 2 with y 1 , y 2 ∼ N (0 , s ∗ σ 2 ) , N (0 , 1) , with corresponding predictions ˆ y 1 , ˆ y 2 .

Original: Additionally, s t +1 = 

Revised: Additionally, s t +1 =

Original: iff ˆ y 2 > .
Revised: iff ˆ y 2 > .

Original: We use Mean Squared
Revised: We use Mean Squared

Original: Error as the loss function, so the optimal predictor is ˆ y 1 , ˆ y 2 =
Revised: Error as the loss function, so the optimal predictor is ˆ y 1 , ˆ y 2 =

Original: However, predicting ˆ y 2 > .reduces the variance of ˆ y 1 , i.e. reduces future loss.
Revised: However, predicting ˆ y 2 > .reduces the variance of ˆ y 1 , i.e. reduces future loss.

Original: The baseline/IL predictor learns ˆ y 1 , ˆ y 2 as parameters using SGD with a learning rate of 0.001.
Revised: The baseline/IL predictor learns ˆ y 1 , ˆ y 2 as parameters using SGD with a learning rate of 0.001.

Original: For experiments with meta-learning, PBT is the
Revised: For experiments with meta-learning, PBT is the

Original: OL (with default settings, see Section 2.2), used to tune the learning rate, with negative loss on the ﬁnal time-step of the interval as the performance measure for PBT.
Revised: OL (with default settings, see Section 2.2), used to tune the learning rate, with negative loss on the ﬁnal time-step of the interval as the performance measure for PBT.

Original: Unit test 2: Myopic RL.
Revised: Unit test 2: Myopic RL.

Original: This unit test is based on a modiﬁcation of the prisoner’s dilemma (Prisner, 2014) where an agent plays each round against its past self.
Revised: This unit test is based on a modiﬁcation of the prisoner’s dilemma (Prisner, 2014) where an agent plays each round against its past self.

Original: The reward function is presented in Table 1.
Revised: The reward function is presented in Table 1.

Original: An agent in this environment has a long-term, non-myopic , incentive for cooperation (with its future self), but a current-time-step, myopic , incentive for defection (from its future self).
Revised: An agent in this environment has a long-term, non-myopic , incentive for cooperation (with its future self), but a current-time-step, myopic , incentive for defection (from its future self).

Original: reward a t =
Revised: reward a t =

Original: D a t =
Revised: D a t =

Original: C s t = a t − 1 =
Revised: C s t = a t − 1 =

Original: D − 1 /−s t = a t − 1 =
Revised: D − 1 /−s t = a t − 1 =

Original: C 1 /
Revised: C 1 /

Original: The unit test evaluates whether a agent reveals the non-myopic incentive even when the agent is meant to optimize for the present reward only (i.e. uses discount rate γ = 0 ).
Revised: The unit test evaluates whether a agent reveals the non-myopic incentive even when the agent is meant to optimize for the present reward only (i.e. uses discount rate γ = 0 ).

Original: Naively, we’d expect the nonmyopic incentive to be hidden from the agent in this case, and for the agent to consistently defect ; learning algorithms that do so pass the test.
Revised: Naively, we’d expect the nonmyopic incentive to be hidden from the agent in this case, and for the agent to consistently defect ; learning algorithms that do so pass the test.

Original: But some learning algorithms also fail the unit test, revealing the incentive for the agent to cooperate with its future self.
Revised: But some learning algorithms also fail the unit test, revealing the incentive for the agent to cooperate with its future self.

Original: While aiming for myopic behavior may seem odd, myopic learners have no incentives to cause distributional shift, since it can only improve future performance.
Revised: While aiming for myopic behavior may seem odd, myopic learners have no incentives to cause distributional shift, since it can only improve future performance.

Original: And while making learners myopic may seem like a ’brute-force’ guaranteed way to manage HI-ADS, we show it is in fact non-trivial to implement.
Revised: And while making learners myopic may seem like a ’brute-force’ guaranteed way to manage HI-ADS, we show it is in fact non-trivial to implement.

Original: See Appendix 3. for details and experiments varying the reward structure.
Revised: See Appendix 9. for details and experiments varying the reward structure.

Original: 5.1. HI-ADS U NIT T ESTS EXPERIMENTAL RESULTS AND DISCUSSION
Revised: 5.1. HI-ADS U NIT T ESTS EXPERIMENTAL RESULTS AND DISCUSSION

Original: We ﬁrst show that agents trained with PBT fail the unit tests more often than “vanilla” algorithms which do not use meta-learning.
Revised: We ﬁrst show that agents trained with PBT fail the unit tests more often than “vanilla” algorithms which do not use meta-learning.

Original: We initialize the learning rate log-uniformly between 0 . 01 and 1 . 0 for all experiments (whether using PBT or not).
Revised: We initialize the learning rate log-uniformly between 0 . 01 and 1 . 0 for all experiments (whether using PBT or not).

Original: We expect and conﬁrm that the following two factors lead to higher rates of unit test failure: (1) Shorter intervals: These give the OL more opportunities to inﬂuence the population.
Revised: We expect and conﬁrm that the following two factors lead to higher rates of unit test failure: (1) Shorter intervals: These give the OL more opportunities to inﬂuence the population.

Original: (2) Larger populations: These make outliers with exceptional non-myopic performance more likely, and OL makes them likely to survive and propagate.
Revised: (2) Larger populations: These make outliers with exceptional non-myopic performance more likely, and OL makes them likely to survive and propagate.

Original: The baseline (no meta-learning) algorithms all pass the unit tests: hidden incentives are almost never revealed - see blue curves in Fig.
Revised: The baseline (no meta-learning) algorithms all pass the unit tests: hidden incentives are almost never revealed - see blue curves in Fig.

Original: However, agents trained with meta-learning and large populations often fail the unit tests: see orange curves in top rows of Fig.
Revised: However, agents trained with meta-learning and large populations often fail the unit tests: see orange curves in top rows of Fig.

Original: [Figure]
Revised: [Figure]

Original: cooperate behavior observed in the RL unit test for
Revised: cooperate behavior observed in the RL unit test for

Original: HI-ADS, with two meta-learning algorithms (B1) PBT and (B2) REINFORCE.
Revised: HI-ADS, with two meta-learning algorithms (B1) PBT and (B2) REINFORCE.

Original: Lower is better, since the goal is for non-myopic incentives to remain hidden.
Revised: Lower is better, since the goal is for non-myopic incentives to remain hidden.

Original: Despite the inner loop being fully myopic (simple MLP in the
Revised: Despite the inner loop being fully myopic (simple MLP in the

Original: SL test, γ = 0 in RL test), in all cases outer-loop (OL) optimizers reveal HI-ADS ( top rows ).
Revised: SL test, γ = 0 in RL test), in all cases outer-loop (OL) optimizers reveal HI-ADS ( top rows ).

Original: Context swapping signiﬁcantly mitigates HI-ADS ( bottom rows ).
Revised: Context swapping signiﬁcantly mitigates HI-ADS ( bottom rows ).

Original: Furthermore, we verify that context swapping signiﬁcantly mitigates the effect of
Revised: Furthermore, we verify that context swapping signiﬁcantly mitigates the effect of

Original: HI-ADS in both unit tests, decreasing undesirable behaviour to near-baseline levels - see bottom rows of Fig.
Revised: HI-ADS in both unit tests, decreasing undesirable behaviour to near-baseline levels - see bottom rows of Fig.

Original: This effect can be explained as follows: Because context swapping transfers the beneﬁts of one learner’s action to the next learner to inhabit that environment, it increases the second learner’s ﬁtness, and thereby reduces the relative ﬁtness (as evaluated by
Revised: This effect can be explained as follows: Because context swapping transfers the beneﬁts of one learner’s action to the next learner to inhabit that environment, it increases the second learner’s ﬁtness, and thereby reduces the relative ﬁtness (as evaluated by

Original: PBT’s EXPLOIT step) of the non-myopic cooperate behaviour.
Revised: PBT’s EXPLOIT step) of the non-myopic cooperate behaviour.

Original: We observe some interesting exceptions with the combination of small populations and short PBT intervals: Although context swapping still signiﬁcantly decreases the effect of HIADS, non-myopic cooperate behaviour is observed as much as 20% of the time (for #learners=10, T = 1 ; see bottom-left plot).
Revised: We observe some interesting exceptions with the combination of small populations and short PBT intervals: Although context swapping still signiﬁcantly decreases the effect of HIADS, non-myopic cooperate behaviour is observed as much as 20% of the time (for #learners=10, T = 1 ; see bottom-left plot).

Original: We also observe that PBT reveals HI-ADS even when T = 1 , where the explanation that PBT operates on a longer time horizon than the inner loop does not apply.
Revised: We also observe that PBT reveals HI-ADS even when T = 1 , where the explanation that PBT operates on a longer time horizon than the inner loop does not apply.

Original: We provide a detailed explanation for how this might happen in Appendix 3, but in summary, we hypothesize that there are at least 2 mechanisms by which PBT is revealing HI-ADS: (1) optimizing over a longer timescale, and (2) picking up on the correlation between an agent’s current policy and the underlying state.
Revised: We provide a detailed explanation for how this might happen in Appendix 9.1.2, but in summary, we hypothesize that there are at least 2 mechanisms by which PBT is revealing HI-ADS: (1) optimizing over a longer time-scale, and (2) picking up on the correlation between an agent’s current policy and the underlying state.

Original: Mechanism (2) can be explained informally as reasoning as: “If I’m cooperating, then I was probably cooperating on the last time-step as well, so my reward should be higher”.
Revised: Mechanism (2) can be explained informally as reasoning as: “If I’m cooperating, then I was probably cooperating on the last time-step as well, so my reward should be higher”.

Original: As support for these hypotheses, we run control experiments identifying two algorithms (each sharing only one of these properties) that can fail the unit test.
Revised: As support for these hypotheses, we run control experiments identifying two algorithms (each sharing only one of these properties) that can fail the unit test.

Original: Context swapping remains effective.
Revised: Context swapping remains effective.

Original: (1) Optimizing over a longer time-scale: replacing PBT with REINFORCE as an outer-loop optimizer.
Revised: (1) Optimizing over a longer time-scale: replacing PBT with REINFORCE as an outer-loop optimizer.

Original: The outer-loop optimizes the parameters to maximize the summed reward of the last T time-steps.
Revised: The outer-loop optimizes the parameters to maximize the summed reward of the last T time-steps.

Original: As with PBT, we observe non-myopic behavior, but now only when T > 1 .
Revised: As with PBT, we observe non-myopic behavior, but now only when T > 1 .

Original: This supports our hypothesis that exploitation of HI-ADS is due not to PBT in particular, but just to the introduction of sufﬁciently powerful meta-learning.
Revised: This supports our hypothesis that exploitation of HI-ADS is due not to PBT in particular, but just to the introduction of sufﬁciently powerful meta-learning.

Original: See Fig.
Revised: See Fig.

Original: 5 B2.
Revised: 5 B2.

Original: (2) Exploiting correlation: Q-learning with γ = 0 an (cid:15) = 
Revised: (2) Exploiting correlation: Q-learning with γ = 0 an (cid:15) = 0 .

Original: 1 -greedy behavior policy and no meta-learning .
Revised: 1 -greedy behavior policy and no meta-learning .

Original: If either state was equally likely, the Q-values would be the average of the values in each column in Table 1, so the estimated
Revised: If either state was equally likely, the Q-values would be the average of the values in each column in Table 1, so the estimated

Original: Q (defect) would be larger.
Revised: Q (defect) would be larger.

Original: But the (cid:15) -greedy policy correlates the previous action (i.e. the current state) and current action (so long as the policy did not just change), so the top-left and bottomright entries carry more weight in the estimates, sometimes causing Q (defect) ≈ Q (cooperate) and persistent nonmyopic behavior.
Revised: But the (cid:15) -greedy policy correlates the previous action (i.e. the current state) and current action (so long as the policy did not just change), so the top-left and bottomright entries carry more weight in the estimates, sometimes causing Q (defect) ≈ Q (cooperate) and persistent nonmyopic behavior.

Original: See Fig.for results, Appendix 4.2 for more results, and
Revised: See Fig.for results, Appendix 9.1.4 for more results, and

Original: Appendix 4.1 for experimental details.
Revised: Appendix 9.1.3 for experimental details.

Original: 5. HI-ADS IN CONTENT RECOMMENDATION
Revised: 5. C ONTENT RECOMMENDATION

Original: We now present a toy environment for modeling content recommendation of news articles, which includes the potential for ADS by incorporating the mechanisms mentioned in Sec. 2.2, discussed as contributing factors to the problems of fake news and ﬁlter bubbles.
Revised: We now present a toy environment for modeling content recommendation of news articles, which includes the potential for ADS by incorporating the mechanisms mentioned in Sec. 2.2, discussed as contributing factors to the problems of fake news and ﬁlter bubbles.

Original: Speciﬁcally, the environment assumes that presenting an article to a user can inﬂuence (1) their interest in similar articles, and (2) their propensity to use the recommendation service.
Revised: Speciﬁcally, the environment assumes that presenting an article to a user can inﬂuence (1) their interest in similar articles, and (2) their propensity to use the recommendation service.

Original: These correspond to modeling auto-induced concept shift of users, and auto-induced covariate shift of the user base, respectively (see Sec. 2.2).
Revised: These correspond to modeling auto-induced concept shift of users, and auto-induced covariate shift of the user base, respectively (see Sec. 2.2).

Original: This environment includes the following components, which change over (discrete) time: User type : x t , Article type : y t , User interests : W t (propensity for users of each type to click on articles of each type), and User loyalty : g t (propensity for users of each type to use the platform).
Revised: This environment includes the following components, which change over (discrete) time: User type : x t , Article type : y t , User interests : W t (propensity for users of each type to click on articles of each type), and User loyalty : g t (propensity for users of each type to use the platform).

Original: At each time step t , a user x t is sampled from a categorical distribution, based on the loyalty of the different user types.
Revised: At each time step t , a user x t is sampled from a categorical distribution, based on the loyalty of the different user types.

Original: The recommendation system (a classiﬁer) selects which type of article to present in the top position, and ﬁnally the user ‘clicks’ an article y t , according to their interests.
Revised: The recommendation system (a classiﬁer) selects which type of article to present in the top position, and ﬁnally the user ‘clicks’ an article y t , according to their interests.

Original: User loyalty for user type x t undergoes covariate shift: in accordance with the self-selection effect, g t increases or decreases proportionally to that user type’s interest in the top article.
Revised: User loyalty for user type x t undergoes covariate shift: in accordance with the self-selection effect, g t increases or decreases proportionally to that user type’s interest in the top article.

Original: The interests of user type x t (represented by a column of W t ) undergoing concept shift; in accordance with the illusory truth effect, interest in the topic of the top article chosen by the recommender system always increases.
Revised: The interests of user type x t (represented by a column of W t ) undergoing concept shift; in accordance with the illusory truth effect, interest in the topic of the top article chosen by the recommender system always increases.

Original: 5.2. C ONTENT RECOMMENDATION EXPERIMENTAL RESULTS AND DISCUSSION
Revised: 5.2. C ONTENT RECOMMENDATION EXPERIMENTAL RESULTS AND DISCUSSION

Original: We run 20 trials using an MLP  for the recommender system; see Appendix 3 for experimental details.
Revised: We run 20 trials using an MLP trained with SGD for the recommender system.

Original: We ﬁnd that PBT yields signiﬁcant improvements in training time and accuracy, but also greater distributional shift (Fig. 7).
Revised: We ﬁnd that PBT yields signiﬁcant improvements in training time and accuracy, but also greater distributional shift (Fig. 7).

Original: User base and user interests both change faster with PBT, and user interests change more overall.
Revised: User base and user interests both change faster with PBT, and user interests change more overall.

Original: We  observe that the distributions over user types typically saturate (to a single user type) after a few hundred time-steps (Fig 1 and Fig.
Revised: We measure concept/covariate shift using the cosine distance and KL-divergence, respectively. We observe that the distributions over user types typically saturate (to a single user type) after a few hundred time-steps (Fig 1 and Fig.

Original: 7, Right).
Revised: 7, Right).

Original: We run long enough to reach such states, to demonstrate that the increase in ADS from PBT is not transitory.
Revised: We run long enough to reach such states, to demonstrate that the increase in ADS from PBT is not transitory.

Original: The environment has a number of free parameters, and our results are qualitatively consistent so long as the covariate shift rate ( α 1 ) is faster than the concept shift rate ( α 2 ).
Revised: The environment has a number of free parameters, and our results are qualitatively consistent so long as the covariate shift rate ( α 1 ) is faster than the concept shift rate ( α 2 ).

Original: See Appendix 2 for details.
Revised: See Appendix 9.2.1 for details.

Revised: increases accuracy of predictions faster

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: ADS in practice: We introduce the term ADS, but we are far from the ﬁrst to study it.
Revised: ADS in practice: We introduce the term ADS, but we are far from the ﬁrst to study it.

Original: Caruana et al.
Revised: Caruana et al.

Original: (2015) provide an example of asthmatic patients having lower predicted risk of pneumonia.
Revised: (2015) provide an example of asthmatic patients having lower predicted risk of pneumonia.

Original: Treating asthmatics with pneumonia less aggressively on this basis would be an example of harmful
Revised: Treating asthmatics with pneumonia less aggressively on this basis would be an example of harmful

Original: ADS; the reason they had lower pneumonia risk was because they had received more aggressive care already.
Revised: ADS; the reason they had lower pneumonia risk was because they had received more aggressive care already.

Original: Schulam & Saria (2017) note that such predictive models are commonly used to inform decision-making, and propose modeling counterfactuals (e.g. “how would this patient fare with less aggressive treatment”) to avoid making such self-refuting predictions.
Revised: Schulam & Saria (2017) note that such predictive models are commonly used to inform decision-making, and propose modeling counterfactuals (e.g. “how would this patient fare with less aggressive treatment”) to avoid making such self-refuting predictions.

Original: While their goal is to make accurate predictions in the presence of ADS, our goal is to identify and manage incentives for ADS.
Revised: While their goal is to make accurate predictions in the presence of ADS, our goal is to identify and manage incentives for ADS.

Original: Non-i.i.d bandits:
Revised: Non-i.i.d bandits:

Original: Contextual bandits (Wang et al., 2005) are a common approach to content recommendation (Li et al., 2010).
Revised: Contextual bandits (Wang et al., 2005) are a common approach to content recommendation (Li et al., 2010).

Original: While bandit algorithms typically make the i.i.d.
Revised: While bandit algorithms typically make the i.i.d.

Original: assumption, counter-examples exist (Gheshlaghi Azar et al., 2014; Auer et al., 1995).
Revised: assumption, counter-examples exist (Gheshlaghi Azar et al., 2014; Auer et al., 1995).

Original: Closest to our work is Shah et al.
Revised: Closest to our work is Shah et al.

Original: (2018), who consider covariate shift caused by recommender systems’ recommendations.
Revised: (2018), who consider covariate shift caused by recommender systems’ recommendations.

Original: But while they seek to exploit ADS, our aim is to avoid hidden incentives for exploiting ADS.
Revised: But while they seek to exploit ADS, our aim is to avoid hidden incentives for exploiting ADS.

Original: Safety and incentives: Emergent incentives to inﬂuence the world (such as HI-ADS) are at the heart of many concerns about the safety of advanced AI systems (Omohundro, 2008; Bostrom, 2014).
Revised: Safety and incentives: Emergent incentives to inﬂuence the world (such as HI-ADS) are at the heart of many concerns about the safety of advanced AI systems (Omohundro, 2008; Bostrom, 2014).

Original: Understanding and managing the incentives of learners is also a focus of Armstrong & O’Rourke (2017);
Revised: Understanding and managing the incentives of learners is also a focus of Armstrong & O’Rourke (2017);

Original: Everitt (2018); Everitt et al.
Revised: Everitt (2018); Everitt et al.

Original: ; Cohen et al.
Revised: ; Cohen et al.

Original: While Everitt et al.
Revised: While Everitt et al.

Original: (2019) focus on identifying which incentives are present, we note that incentives may be present and yet not be revealed or pursued - for example, in supervised learning, there is an incentive to make predictions that are over-ﬁt to the test set, but we typically hide the test set from the learner, which effectively hides this incentive.
Revised: (2019) focus on identifying which incentives are present, we note that incentives may be present and yet not be revealed or pursued - for example, in supervised learning, there is an incentive to make predictions that are over-ﬁt to the test set, but we typically hide the test set from the learner, which effectively hides this incentive.

Original: While Carey et al.
Revised: While Carey et al.

Original: Everitt et al.
Revised: Everitt et al.

Original: (2019); Armstrong & O’Rourke (2017) discuss methods of removing problematic incentives, we note in practice incentives are often hidden rather than removed.
Revised: (2019); Armstrong & O’Rourke (2017) discuss methods of removing problematic incentives, we note in practice incentives are often hidden rather than removed.

Original: Our work addresses the efﬁcacy of this approach and ways in which it can fail.
Revised: Our work addresses the efﬁcacy of this approach and ways in which it can fail.

Original: HI-ADS and meta-learning:
Revised: HI-ADS and meta-learning:

Original: We believe our work is the ﬁrst to consider HI-ADS and their relation to meta-learning.
Revised: We believe our work is the ﬁrst to consider HI-ADS and their relation to meta-learning.

Original: A few previous works have some relevance.
Revised: A few previous works have some relevance.

Original: Rabinowitz (2019) documents qualitative differences in learning behavior when meta-learning is applied.
Revised: Rabinowitz (2019) documents qualitative differences in learning behavior when meta-learning is applied.

Original: MacKay et al.
Revised: MacKay et al.

Original: and
Revised: and

Original: Lorraine & Duvenaud (2018) view meta-learning as a bilevel optimization problem, with the inner loop playing a best-response to the outer loop.
Revised: Lorraine & Duvenaud (2018) view meta-learning as a bilevel optimization problem, with the inner loop playing a best-response to the outer loop.

Original: In our work, the outer loop has a greater inﬂuence, and the inner loop often fails to play best-response.
Revised: In our work, the outer loop has a greater inﬂuence, and the inner loop often fails to play best-response.

Original: Sutton et al.
Revised: Sutton et al.

Original: (2007) noted that meta-learning can improve performance by preventing convergence of the inner loop to best response.
Revised: (2007) noted that meta-learning can improve performance by preventing convergence of the inner loop to best response.

Original:  D ISCUSSION AND C ONCLUSION
Revised:  D ISCUSSION AND C ONCLUSION

Original: We identify the phenomenon of auto-induced distributional shift (ADS) and problems that can arise when there are hidden incentives for learners to induce distributional shift (HI-ADS).
Revised: We identify the phenomenon of auto-induced distributional shift (ADS) and problems that can arise when there are hidden incentives for learners to induce distributional shift (HI-ADS).

Original: We show that meta-learning can reveal HI-ADS and lead learners to use ADS as a means of increasing performance.
Revised: We show that meta-learning can reveal HI-ADS and lead learners to use ADS as a means of increasing performance.

Original: Our work highlights the interdisciplinary nature of issues with real-world deployment of ML systems - we show how HI-ADS could play a role in important technosocial issues like ﬁlter bubbles and the propagation of fake news.
Revised: Our work highlights the interdisciplinary nature of issues with real-world deployment of ML systems - we show how HI-ADS could play a role in important technosocial issues like ﬁlter bubbles and the propagation of fake news.

Original: There are a number of potential implications for our work: (1) When
Revised: There are a number of potential implications for our work: (1) When

Original: HI-ADS are a concern, our methodology and environments can be used to help diagnose whether and to what extent the ﬁnal performance/behavior of a learner is due to ADS and/or incentives for
Revised: HI-ADS are a concern, our methodology and environments can be used to help diagnose whether and to what extent the ﬁnal performance/behavior of a learner is due to ADS and/or incentives for

Original: ADS, i.e. to quantify their inﬂuence on that learner.
Revised: ADS, i.e. to quantify their inﬂuence on that learner.

Original: (2) Comparing this quantitative analysis for different algorithms could help us understand which features of algorithms affect their propensity to reveal HI-ADS, and aid in the development of safer and more robust algorithms.
Revised: (2) Comparing this quantitative analysis for different algorithms could help us understand which features of algorithms affect their propensity to reveal HI-ADS, and aid in the development of safer and more robust algorithms.

Original: (3) Characterizing and identifying HI-ADS in these tests is a ﬁrst step to analyzing and mitigating other (problematic) hidden incentives, as well as to developing theoretical understanding of hidden incentives.
Revised: (3) Characterizing and identifying HI-ADS in these tests is a ﬁrst step to analyzing and mitigating other (problematic) hidden incentives, as well as to developing theoretical understanding of hidden incentives.

Original: Broadly speaking, our work emphasizes that the choice of machine learning algorithm plays an important role in speciﬁcation, independently of the choice of performance metric.
Revised: Broadly speaking, our work emphasizes that the choice of machine learning algorithm plays an important role in speciﬁcation, independently of the choice of performance metric.

Original: A learner can use ADS to increase performance according to the intended performance metric , and yet still behave in an undesirable way, if we did not intend the learner to improve performance by that method .
Revised: A learner can use ADS to increase performance according to the intended performance metric , and yet still behave in an undesirable way, if we did not intend the learner to improve performance by that method .

Original: In other words, performance metrics are typically incomplete speciﬁcations: they only specify our goals or ends , while our choice of learning algorithm plays a role in specifying the means by which we intend an learner to achieve those ends.
Revised: In other words, performance metrics are typically incomplete speciﬁcations: they only specify our goals or ends , while our choice of learning algorithm plays a role in specifying the means by which we intend an learner to achieve those ends.

Original: With increasing deployment of ML algorithms in daily life, we believe that (1) understanding incentives and (2) specifying desired/allowed means of improving performance are important avenues of future work to ensure fair, robust, and safe outcomes. 
Revised: With increasing deployment of ML algorithms in daily life, we believe that (1) understanding incentives and (2) specifying desired/allowed means of improving performance are important avenues of future work to ensure fair, robust, and safe outcomes. A PPENDICES C ONTENT RECOMMENDATION IN THE WILD

Revised: Filter bubbles, the spread of fake news, and other techno-social issues are widely reported to be responsible for the rise of populism (Groshek & Koc-Michalska, 2017), increase in racism and prejudice against immigrants and refugees (Noble, 2018), increase in social isolation and suicide (Luxton et al., 2012), and, particularly with reference to the 2016 US elections, are decried as threatening the foundations of democracy (El-Bermawy, 2016).

Revised: Even in 2013, well before the

Revised: American elections, a World Economic Forum report identiﬁed these problems as a global crisis (Lee Howell, 2013).

Revised: We focus on two related issues in which content recommendation algorithms play a role: fake news and ﬁlter bubbles.

Revised: 8. F AKE NEWS

Revised: Fake news (also called false news or junk news) is an extreme version of yellow journalism, propaganda, or clickbait, in which media that is ostensibly providing information focuses on being eye-catching or appealing, at the expense of the quality of research and exposition of factual information.

Revised: Fake news is distinguished by being speciﬁcally and deliberately created to spread falsehoods or misinformation (Merriam-Webster, 2017; Mihailidis & Viotty, 2017).

Revised: Why does fake news spread?

Revised: It may at ﬁrst seem the solution is simply to educate people about the truth, but research tells us the problem is more multifaceted and insidious, due to a combination of related biases and cognitive effects including conﬁrmation bias (people are more likely to believe things that ﬁt with their existing beliefs), priming (exposure to information unconsciously inﬂuences the processing of subsequent information, i.e. seeing something in a credible context makes things seem more credible) and the illusory truth effect (i.e. people are more likely to believe something simply if they are told it is true).

Revised: Allcott & Gentzkow (2017) track about 150 fake news stories during the 2016 US election, and ﬁnd the average American adult saw 1-2 fake news stories, just over half believed the story was true, and likelihood of believing fake news increased with ideological segregation (polarization) of their social media.

Revised: Shao et al.

Revised: (2018) examine the role of social bots in spreading fake news by analyzing 14 million Twitter messages.

Revised: They ﬁnd that bots are far more likely than humans to spread misinformation, and that success of a fake news story (in terms of human retweets) was heavily dependent on whether bots had shared the story.

Revised: Pennycook et al.

Revised: (2019) examine the role of the illusory truth effect in fake news.

Revised: They ﬁnd that even a single exposure to a news story makes people more likely to believe that it is true, and repeat viewings increase this likelihood.

Revised: They ﬁnd that this is not true for extremely implausible statements (e.g. “the world is a perfect cube”), but that “only a small degree of potential plausibility is sufﬁcient for repetition to increase perceived accuracy” of the story.

Revised: The situation is further complicated by peoples’ inability to distinguish promoted content from real news - Amazeen & Wojdynski (2018)

Revised: ﬁnd that fewer than 1/10 people were able to tell when content was an advertisement, even when it was explicitly labelled as such.

Revised: Similarly, Fazio et al.

Revised: ﬁnd that repeated exposure to incorrect trivia make people more likely to believe it, even when they are later able to identify the trivia as incorrect.

Revised: 8. F ILTER BUBBLES

Revised: Filter bubbles, a term coined and popularized by Pariser (2011) are created by positive or negative feedback loops which encourage users or groups of users towards increasing within-group similarity, while driving up between-group dissimilarity.

Revised: The curation of this echo chamber is called selfselection (people are more likely to look for or select things that ﬁt their existing preferences), and favours what Techopedia (2018) calls intellectual isolation.

Revised: In the context of social and political opinions, this is often called the polarization effect (Wikipedia contributors, 2018).

Revised: Filter bubbles can be encouraged by algorithms in two main ways.

Revised: The ﬁrst is the most commonly described: simply by showing content that is similar to what a user has already searched for, search or recommender systems create a positive feedback loop of increasingly-similar content (Pariser, 2011; Kayhan, 2015).

Revised: The second way is similar but opposite - if the predictions of an algorithm are good for a certain group of people, but bad for others, the algorithm can do better on its metrics by driving hard-to-predict users away.

Revised: Then new users to the site will either be turned off entirely, or see an artiﬁcially homogenous community of like-minded peers, a phenomena Shah et al.

Revised: (2018) call positive externalities .

Revised: In a study of 50,000 US-based internet users, Flaxman & Goel (2015) ﬁnd that two things increase with social media and search engine use: (1) exposure of an individual to opposing or different viewpoints, and (2) mean ideological distance between users.

Revised: Many studies cite the ﬁrst result as evidence of the beneﬁts of internet and social media (Robson, 2018; Bakshy et al., 2015), but the correlation of exposure with ideological distances demonstrates that exposure is not enough, and might even be counterproductive.

Revised: Facebook’s own study on ﬁlter bubbles results show that the impact of the news feed algorithm on ﬁlter bubble “size” (a measure of homogeneity of posts relative to a baseline) is almost as large as the impact of friend group composition (Bakshy et al., 2015).

Revised: Kayhan (2015) speciﬁcally study the role of search engines in conﬁrmation bias, and ﬁnd that search context and the similarity of results in search engine results both reinforce existing biases and increase the likelihood of future biased searches.

Revised: Nguyen et al.

Revised: (2014) similarly study the effect of recommender systems on individual users’ content diversity, and ﬁnd that the set of options recommended narrows over time.

Revised: Filter bubbles create an ideal environment for the spread of fake news: they increase the likelihood of repeat viewings of similar content, and because of the illusory truth effect, that content is more likely to be believed and shared (Pennycook et al.

Revised: We are not claiming that HI-ADS are entirely or even mostly responsible for these problems, but we do note that they can play a role that is worth addressing.

Revised: [Figure]

Revised:  E XTRA EXPERIMENTS AND REPRODUCIBILITY DETAILS 9. HI-ADS UNIT TEST

Revised: We used REINFORCE (Williams, 1992) with discount factor γ = 0 as the baseline/IL optimizer.

Revised: PBT (with default settings, see Section 2.2) is used to tune the learning rate, with reward on the ﬁnal time-step of the interval as the performance measure for PBT.

Revised: Formally, the environment is not a 2x2 game (as the original prisoner’s dilemma); it’s a partially observable Markov Decision Process (Åström, 1965; Kaelbling et al., 1998):

Revised: [Equation]

Revised: where I is an indicator function, and β = − 1 / 2 is a parameter controlling the alignment of incentives.

Revised: The initial state is sampled as s 0 ∼ U ( defect , cooperate ) .

Revised: Policies are represented by a single real-valued parameter θ (initialized as θ ∼ N (0 , 1) ) passed through a sigmoid whose output represents P ( a t = defect) .

Revised: 9.1. A LIGNMENT OF INCENTIVES EXPLORATION

Revised: This section presents an exploration of the parameter β , which controls the alignment of incentives in the HI-ADS unit tests (see Table 2).

Revised: To clarify the interpretation of experiments, we distinguish between environments in which myopic (defect) vs. nonmyopic (cooperate) incentives are opposed , orthogonal , or compatible .

Revised: Note that in this unit test myopic behaviour (defection) is what we want to see.

Revised: Incentive-opposed : Optimal myopic behavior is incompatible with optimal nonmyopic behavior (classic prisoner’s dilemma; these experiments are in the main paper).

Revised: Incentive-orthogonal : Optimal myopic behavior may or may not be optimal nonmyopic behavior.

Revised: Incentive-compatible : Optimal myopic behavior is necessarily also optimal nonmyopic behavior.

Revised: We focused on incentive-opposed environment ( β = − 1 / 2 ) in the main paper in order to demonstrate that HI-ADS can be powerful enough to change the behavior of the system in an undesirable way.

Revised: Here we also explore incentive-compatible and incentive-orthogonal environments because they provide useful baselines, helping us distinguish a systematic bias towards nonmyopic behavior from other reasons (such as randomness or optimization issues) for behavior that does not follow a myopically optimal policy.

Revised: 9.1. W ORKING THROUGH A DETAILED EXAMPLE FOR PBT WITH T =

Revised: To help provide intuition on how (mechanistically) PBT could lead to persistent levels of cooperation, we walk through a simple example (with no inner loop).

Revised: Consider PBT with T = 1 and a population of 5 deterministic agents

Revised: A 1 , ..., A 5 playing cooperate and receiving reward of r ( A i )

Revised: = 0 .

Revised: Now suppose A 1 suddenly switches to play defect .

Revised: Then r ( A 1 ) = 1 / 2 on the next time-step (while the other agents’ reward is still 0 ), and so PBT’s EXPLOIT step will copy A 1 (without loss of generality to A 2 ).

Revised: On the following time-step, r ( A 2 ) = 1 / 2 , and r ( A 1 ) = − 1 / 2 , so PBT will clone A 2 to A 1 , and the cycle repeats.

Revised: Similar reasoning applies for larger populations, and T > 1 .

Revised: 9.1. Q- LEARNING EXPERIMENT DETAILS

Revised: We show that, under certain conditions, Q-learning can learn to (primarily) cooperate, and thus fails the HI-ADS unit test.

Revised: We estimate Q-values using the sample-average method, which is guaranteed to

Revised: [Table]

Revised: converge in the fully observed, tabular case (Sutton & Barto, 1998).

Revised: The agent follows the (cid:15) -greedy policy with (cid:15)

Revised: = 0 .

Revised: In order to achieve this result, we additionally start the agent off with one synthetic memory where both state and action are defect and therefor R (defect) = − . 5 , and we hard-code the starting state to be cooperate (which normally only happens 50% of the time).

Revised: Without this kind of an initialization, the agent always learns to defect.

Revised: However, under these conditions, we ﬁnd that 10/30 agents learned to play cooperate most of the time, with Q (cooperate) and Q (defect) both hovering around − 0 .

Revised: 07 , while others learn to always defect, with Q (cooperate)

Revised: ≈ − 0 .

Revised: 92 and

Revised: Q (defect) ≈ − 0 .

Revised: context swapping, however, prevents majority-cooperate behavior from ever emerging, see Figure 11.

Revised: 9.1. Q- LEARNING : FURTHER RESULTS

Revised: To give a more representative picture of how often Q-learning fails the unit test, we run a larger set of experiments with Q-learning, results are in Figure 10.

Revised: It’s possible that the failure of Q-learning is not persistent, since we have not proved otherwise, but we did run much longer experiments and still observe persistent failure, see Figure 9.

Revised: 9. C ONTENT RECOMMENDATION 9.2. E NVIRONMENT DETAILS

Revised: The evironment has the following components:

Revised: User type , x t : categorical variable representing different types of users.

Revised: The content recommender conditions its predictions on the type of the current user.

Revised: User loyalty , g t : the propensity for users of each type to use the platform.

Revised: User x t is sampled from a categorical distribution with parameters given by softmax( g t ) .

Revised: Article type , y t : a categorical variable (one-hot encoding) representing the type of article selected by the user.

Revised: User interests , W t : a matrix whose entries W tx,y represent the average interest user of type x have in articles of type y .

Revised: At each time step t , a user x t is sampled from a categorical distribution (based on the loyalty of the different user types), then the recommendation system selects which type of article to present in the top position, and ﬁnally, the user selects an article.

Revised: The goal of the recommendation system is to predict the likelihood that the user would click on each of the available articles, in order to select the one which is most interesting to the user.

Revised: User loyalty for x t then changes in accordance with the self-selection effect, increasing or decreasing proportionally to their interest in the top article.

Revised: The interests of user type x t (represented by a column of W t ) also change; in accordance with the illusory truth effect, their interest in the topic of the top article (as chosen by the recommender system) always increases.

Revised: Overall, this environment is an extremely crude representation of reality, but it allows us to incorporate both the effects of self-selection (via covariate shift), and the illusory truth effect (via concept shift).

Revised: Formally, this environment is similar to a POMDP \ R, i.e. a POMDP with no reward function, also known as a world model (Armstrong & O’Rourke, 2017; Hadﬁeld-Menell et al., 2017); the difference is that the learner observes the input before acting and only observes the target after acting.

Revised: The states, observations, and actions given below.

Revised: [Equation]

Revised: The state transition function is deﬁned by:

Revised: [Equation]

Revised: Where ˆ y t is the top article as chosen by the recommender, and α 1 , α 2 represent the rate of covariate and concept shift (respectively).

Revised: The update for W t +1 merely increases the interest of user type x t in article type ˆ y t , then normalizes the interests for that user type.

Revised: 9.2. R EPRODUCIBILITY DETAILS

Revised: For these experiments, the recommendation system is a ReLU-MLP with 1 hidden layer of 100 units, trained via supervised learning with SGD (learning rate = 0 . 01 ) to predict which article a user will select.

Revised: Actions are sampled from the MLP’s predictive distribution.

Revised: We apply PBT without any hyperparameter selection (this amounts to just doing the EXPLOIT step), and an interval of 10, selecting on accuracy.

Revised: We use a population of 20 learners (whether applying PBT or not), and match random seeds for the trials with and without PBT.

Revised: We initialize g 1 and W 1 to be the same across the 20 copies of the environment (i.e. the learners start with the same user population), but these values diverge throughout learning.

Revised: For the environment, we set the number of user and article types both to 10.

Revised: Initial user loyalties are randomly sampled from N (0 , 0 . 03) , α 1 = 0 .

Revised: 03 , and α 2 = 0 .

Revised: 9.2. D ETAILS OF EVALUATION

Revised: We measure concept shift (change in P ( y | x ) ) as the cosine distance between each user types’ initial and current interest vectors.

Revised: And we measure covariate shift (change in P ( x ) ) as the KL-divergence between the current and initial user distributions, parametrized by g 1 and g t , respectively.

Revised: Results are presented in 7 (main text).

Revised: In Figure 12, we additionally plot concept shift and covariate shift as a function of accuracy.

Revised: We observe that for both types of ADS, at low levels of accuracy PBT actually causes less shift than occur in baseline agents; HI-ADS are only observed for accuracies above 60%.

Revised: This suggests that only relatively strong performers are able to pick up on the HI-ADS revealed by

Revised: PBT (Fig. 12).

Revised: 9.2. CONTEXT SWAPPING IN CONTENT RECOMMENDATION

Revised: We believe context swapping is not appropriate for the content recommendation environment, since when the environments diverge, optimal behavior may differ across environments.

Revised: Nevertheless, we ran experiments with it for completeness.

Revised: The main effect appears to be to hamper learning when

Revised: PBT is not used, see Figure 13.

Revised: Notably, it does not appear to signiﬁcantly inﬂuence the rate or extent of ADS when combined with PBT.

Revised: 9.2. E XPLORATION OF ENVIRONMENT PARAMETERS

Revised: In Figure 14, we examine the effect of the rate-of-change parameters ( α 1 , α 2 ) of the content recommendation environment on the results provided in the paper.

Revised: As noted there, our results are qualitatively consistent so long as (1) the initial user distribution is approximately uniform, and (2) the covariate shift rate ( α 1 ) is faster than the concept shift rate ( α 2 ).

Revised: These distributions are updated by different mechanisms, and are not directly comparable.

Revised: Concept shift changes the task more radically, requiring a learner to change its predictions, rather than just become accurate on a wider range of inputs.

Revised: We conjecture that changes in P ( y | x ) must therefore be kept smooth enough for the outer loop to have pressure to capitalize on HI-ADS.

Revised: α

Revised: [Figure]

Revised: = 0 .

Revised: 01 , α

Revised: [Figure]

Revised: =

Revised: [Figure]

Revised: .α

Revised: [Figure]

Revised: = 0 .

Revised: 1 , α

Revised: [Figure]

Revised: = 0 .α

Revised: [Figure]

Revised: = 0 .

Revised: 01 , α

Revised: [Figure]

Revised: = 0 .α

Revised: [Figure]

Revised: = 0 .

Revised: 1 , α

Revised: [Figure]

Revised: = 0 .

Revised: [Figure]

Revised: [Figure]

Revised: .α

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]