Original: P URCHASE AS R EWARD : S ESSION - BASED R ECOMMENDATION BY I MAGINATION R ECONSTRUCTION
Revised: P URCHASE AS R EWARD : S ESSION - BASED R ECOMMENDATION BY I MAGINATION R ECONSTRUCTION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: One of the key challenges of session-based recommender systems is to enhance users’ purchase intentions.
Revised: One of the key challenges of session-based recommender systems is to enhance users’ purchase intentions.

Original: In this paper, we formulate the sequential interactions between user sessions and a recommender agent as a Markov Decision Process (MDP).
Revised: In this paper, we formulate the sequential interactions between user sessions and a recommender agent as a Markov Decision Process (MDP).

Original: In practice, the purchase reward is delayed and sparse, and may be buried by clicks, making it an impoverished signal for policy learning.
Revised: In practice, the purchase reward is delayed and sparse, and may be buried by clicks, making it an impoverished signal for policy learning.

Original: Inspired by the prediction error minimization (PEM) and embodied cognition, we propose a simple architecture to augment reward, namely Imagination Reconstruction Network (IRN).
Revised: Inspired by the prediction error minimization (PEM) and embodied cognition, we propose a simple architecture to augment reward, namely Imagination Reconstruction Network (IRN).

Original: Speciﬁcally, IRN enables the agent to explore its environment and learn predictive representations via three key components.
Revised: Speciﬁcally, IRN enables the agent to explore its environment and learn predictive representations via three key components.

Original: The imagination core generates predicted trajectories, i.e., imagined items that users may purchase.
Revised: The imagination core generates predicted trajectories, i.e., imagined items that users may purchase.

Original: The trajectory manager controls the granularity of imagined trajectories using the planning strategies, which balances the long-term rewards and short-term rewards.
Revised: The trajectory manager controls the granularity of imagined trajectories using the planning strategies, which balances the long-term rewards and short-term rewards.

Original: To optimize the action policy, the imagination-augmented executor minimizes the intrinsic imagination error of simulated trajectories by self-supervised reconstruction, while maximizing the extrinsic reward using model-free algorithms.
Revised: To optimize the action policy, the imagination-augmented executor minimizes the intrinsic imagination error of simulated trajectories by self-supervised reconstruction, while maximizing the extrinsic reward using model-free algorithms.

Original: Empirically, IRN promotes quicker adaptation to user interest, and shows improved robustness to the cold-start scenario and ultimately higher purchase performance compared to several baselines.
Revised: Empirically, IRN promotes quicker adaptation to user interest, and shows improved robustness to the cold-start scenario and ultimately higher purchase performance compared to several baselines.

Original: Somewhat surprisingly, IRN using only the purchase reward achieves excellent next-click prediction performance, demonstrating that the agent can "guess what you like" via internal planning.
Revised: Somewhat surprisingly, IRN using only the purchase reward achieves excellent next-click prediction performance, demonstrating that the agent can "guess what you like" via internal planning.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: A good recommender system can enhance both satisfaction for users and proﬁt for content providers (Gomez-Uribe & Hunt, 2016).
Revised: A good recommender system can enhance both satisfaction for users and proﬁt for content providers (Gomez-Uribe & Hunt, 2016).

Original: In many real-world scenarios, the recommender systems make recommendations based only on the current browsing session, given the absence of user proﬁles (because the user is new or not tracked or not logged in, till the ﬁnal purchase step).
Revised: In many real-world scenarios, the recommender systems make recommendations based only on the current browsing session, given the absence of user proﬁles (because the user is new or not tracked or not logged in, till the ﬁnal purchase step).

Original: A session is a group of sequential interactions between a user and the system within a short period of time.
Revised: A session is a group of sequential interactions between a user and the system within a short period of time.

Original: To model this phenomenon, Recurrent Neural Networks (RNNs) were recently employed as session-based recommenders (Hidasi et al., 2016; Jannach & Ludewig, 2017).
Revised: To model this phenomenon, Recurrent Neural Networks (RNNs) were recently employed as session-based recommenders (Hidasi et al., 2016; Jannach & Ludewig, 2017).

Original: For instance, GRU4Rec (Hidasi et al., 2016) utilizes the session-parallel mini-batch training to handle the variable lengths of sessions, and predicts the next action given the sequence of items in the current session.
Revised: For instance, GRU4Rec (Hidasi et al., 2016) utilizes the session-parallel mini-batch training to handle the variable lengths of sessions, and predicts the next action given the sequence of items in the current session.

Original: However, these approaches primarily focus on next-click prediction and model the session data via sequential classiﬁcation, and thus cannot distinguish the different effects of user clicks and purchases.
Revised: However, these approaches primarily focus on next-click prediction and model the session data via sequential classiﬁcation, and thus cannot distinguish the different effects of user clicks and purchases.

Original: In this paper, we consider the session-based recommendation as a Markov Decision Process (MDP), which can take into account both the click reward and the purchase reward (see Figure 1), and leverage
Revised: In this paper, we consider the session-based recommendation as a Markov Decision Process (MDP), which can take into account both the click reward and the purchase reward (see Figure 1), and leverage

Original: Reinforcement Learning (RL) to learn the recommendation strategy.
Revised: Reinforcement Learning (RL) to learn the recommendation strategy.

Original: In practice, several challenges need to be addressed.
Revised: In practice, several challenges need to be addressed.

Original: First, the recommender systems involve large numbers of discrete actions (i.e., items), making current RL algorithms difﬁcult to apply (Dulac-Arnold et al., 2015; Sunehag et al., 2015).
Revised: First, the recommender systems involve large numbers of discrete actions (i.e., items), making current RL algorithms difﬁcult to apply (Dulac-Arnold et al., 2015; Sunehag et al., 2015).

Original: This requires the agent to explore its environment for action feature learning and develop an ability to generalize over unseen actions.
Revised: This requires the agent to explore its environment for action feature learning and develop an ability to generalize over unseen actions.

Original: Second, we found it difﬁcult to specify the click reward and the purchase reward; the policy may be biased by long sessions that contain many user clicks, as RL algorithms maximize the accumulated reward.
Revised: Second, we found it difﬁcult to specify the click reward and the purchase reward; the policy may be biased by long sessions that contain many user clicks, as RL algorithms maximize the accumulated reward.

Original: Besides, real-world recommender systems require quick adaptation to user interest and robustness to the cold-start scenario (i.e., enhancing the purchase performance of short sessions).
Revised: Besides, real-world recommender systems require quick adaptation to user interest and robustness to the cold-start scenario (i.e., enhancing the purchase performance of short sessions).

Original: Therefore, we will be particularly interested in a case where only the purchase is used as reward (click sequences are used as inputs of the imagination core for exploration).
Revised: Therefore, we will be particularly interested in a case where only the purchase is used as reward (click sequences are used as inputs of the imagination core for exploration).

Original: However, the purchase reward is delayed and sparse (one session may contain only one purchase), making it a difﬁcult signal for policy learning.
Revised: However, the purchase reward is delayed and sparse (one session may contain only one purchase), making it a difﬁcult signal for policy learning.

Original: To augment reward and encourage exploration, we present the Imagination Reconstruction Network (IRN), which is inspired by the prediction error minimization (PEM)
Revised: To augment reward and encourage exploration, we present the Imagination Reconstruction Network (IRN), which is inspired by the prediction error minimization (PEM)

Original: (Hohwy, 2016; Friston, 2010;
Revised: (Hohwy, 2016; Friston, 2010;

Original: Lotter et al., 2017) and embodied cognition (Clark, 2013; Burr & Jones, 2016; Seth, 2014; de Bruin & Michael, 2017) from the neuroscience literature.
Revised: Lotter et al., 2017) and embodied cognition (Clark, 2013; Burr & Jones, 2016; Seth, 2014; de Bruin & Michael, 2017) from the neuroscience literature.

Original: The PEM is an increasingly inﬂuential theory that stresses the importance of brain-body-world interactions in cognitive processes, involving perception, action and learning.
Revised: The PEM is an increasingly inﬂuential theory that stresses the importance of brain-body-world interactions in cognitive processes, involving perception, action and learning.

Original: In particular, IRN can be regarded as a proof-of-concept for the PEM from the recommendation perspective, following the ideas in Burr & Jones (2016) and Seth (2014) — the brain utilizes active sensorimotor predictions (or counterfactual predictions) to represent states of affairs in the world in an action-oriented manner.
Revised: In particular, IRN can be regarded as a proof-of-concept for the PEM from the recommendation perspective, following the ideas in Burr & Jones (2016) and Seth (2014) — the brain utilizes active sensorimotor predictions (or counterfactual predictions) to represent states of affairs in the world in an action-oriented manner.

Original: Speciﬁcally, the imagination core of IRN that predicts the future trajectories (i.e., a set of imagined items that user may purchase) conditioned on actions sampled from the imagination policy, can be considered as the generative model of the brain that simulates sensorimotor predictions.
Revised: Speciﬁcally, the imagination core of IRN that predicts the future trajectories (i.e., a set of imagined items that user may purchase) conditioned on actions sampled from the imagination policy, can be considered as the generative model of the brain that simulates sensorimotor predictions.

Original: To update the action policy, the imagination-augmented executor minimizes the intrinsic imagination error of predicted trajectories by self-supervised reconstruction, while maximizing the extrinsic reward using RL, with shared input state or output action representations for predictive learning.
Revised: To update the action policy, the imagination-augmented executor minimizes the intrinsic imagination error of predicted trajectories by self-supervised reconstruction, while maximizing the extrinsic reward using RL, with shared input state or output action representations for predictive learning.

Original: This simulates the active perception (a key aspect of embodied cognition) of the body under the PEM framework, which adapts the agent to possible changes that arise from the ongoing exploratory action.
Revised: This simulates the active perception (a key aspect of embodied cognition) of the body under the PEM framework, which adapts the agent to possible changes that arise from the ongoing exploratory action.

Original: Note that the imagination policy imitates the action policy through distillation or a delayed target network, and thus IRN constructs a loop between brain and body, encouraging the agent to perform actions that can reduce the error in the agent’s ability to predict the future events (Pathak et al., 2017).
Revised: Note that the imagination policy imitates the action policy through distillation or a delayed target network, and thus IRN constructs a loop between brain and body, encouraging the agent to perform actions that can reduce the error in the agent’s ability to predict the future events (Pathak et al., 2017).

Original: IRN equips the agent with a planning module, trajectory manager , that controls the granularity of imagined trajectories using the planning strategies (e.g., breadth- n and depth- m ).
Revised: IRN equips the agent with a planning module, trajectory manager , that controls the granularity of imagined trajectories using the planning strategies (e.g., breadth- n and depth- m ).

Original: Besides, IRN is a combination of model-based planning and self-supervised RL, as the imagined trajectories provide dense training signals for auxiliary task learning (see section 2).
Revised: Besides, IRN is a combination of model-based planning and self-supervised RL, as the imagined trajectories provide dense training signals for auxiliary task learning (see section 2).

Original: The key contributions of this paper are summarized as follows:
Revised: The key contributions of this paper are summarized as follows:

Original: • We formulate the session-based recommendation as a MDP, and leverage deep RL to learn the optimal recommendation policy, and also discuss several challenges when RL is applied.
Revised: • We formulate the session-based recommendation as a MDP, and leverage deep RL to learn the optimal recommendation policy, and also discuss several challenges when RL is applied.

Original: • We consider a special case where only the purchase is used as reward, and then propose the IRN architecture to optimize the sparser but more business-critical purchase signals, which draws inspiration from the theories of cognition science.
Revised: • We consider a special case where only the purchase is used as reward, and then propose the IRN architecture to optimize the sparser but more business-critical purchase signals, which draws inspiration from the theories of cognition science.

Original: • We present a self-supervised reconstruction method for predictive learning, which minimizes the imagination error of simulated trajectories over time.
Revised: • We present a self-supervised reconstruction method for predictive learning, which minimizes the imagination error of simulated trajectories over time.

Original: IRN achieves excellent click and purchase performance even without any external reward (predictive perception (Seth, 2014)).
Revised: IRN achieves excellent click and purchase performance even without any external reward (predictive perception (Seth, 2014)).

Original: • We conduct a comprehensive set of experiments to demonstrate the effectiveness of IRN.
Revised: • We conduct a comprehensive set of experiments to demonstrate the effectiveness of IRN.

Original: Compared to several baselines, IRN improves data efﬁciency, promotes quicker adaptation to user interest, and shows improved robustness to the cold-start scenario and ultimately higher purchase performance.
Revised: Compared to several baselines, IRN improves data efﬁciency, promotes quicker adaptation to user interest, and shows improved robustness to the cold-start scenario and ultimately higher purchase performance.

Original: These are highly valuable properties in an industrial context.
Revised: These are highly valuable properties in an industrial context.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Session-based Recommenders Classical latent factor models (e.g., matrix factorization) break down in the session-based setting, given the absence of user proﬁles.
Revised: Session-based Recommenders Classical latent factor models (e.g., matrix factorization) break down in the session-based setting, given the absence of user proﬁles.

Original: A natural solution is the neighborhood approach like item-to-item recommendation (Mirowski et al., 2016).
Revised: A natural solution is the neighborhood approach like item-to-item recommendation (Mirowski et al., 2016).

Original: In this setting, an item similarity matrix can be precomputed based on co-occurrences of clicked items in sessions.
Revised: In this setting, an item similarity matrix can be precomputed based on co-occurrences of clicked items in sessions.

Original: However, this method only considers the last clicked item of the browsing session for recommendations, ignoring the sequential information of the previous events.
Revised: However, this method only considers the last clicked item of the browsing session for recommendations, ignoring the sequential information of the previous events.

Original: Previous works also attempt to apply MDPs in the recommendation systems (Shani et al., 2002; Tavakol & Brefeld, 2014).
Revised: Previous works also attempt to apply MDPs in the recommendation systems (Shani et al., 2002; Tavakol & Brefeld, 2014).

Original: The main issue is that the state space quickly becomes unmanageable due to the large number of items (IRN employs deep learning to overcome this problem and thus generalizes well to unseen states).
Revised: The main issue is that the state space quickly becomes unmanageable due to the large number of items (IRN employs deep learning to overcome this problem and thus generalizes well to unseen states).

Original: Recently, RNNs have been used with success in this area (Hidasi et al., 2016; Hidasi & Karatzoglou, 2017; Jannach & Ludewig, 2017).
Revised: Recently, RNNs have been used with success in this area (Hidasi et al., 2016; Hidasi & Karatzoglou, 2017; Jannach & Ludewig, 2017).

Original: GRU4Rec (Hidasi et al., 2016) is the ﬁrst application of RNNs to model the session data, which can provide recommendations after each click for new sessions.
Revised: GRU4Rec (Hidasi et al., 2016) is the ﬁrst application of RNNs to model the session data, which can provide recommendations after each click for new sessions.

Original: Under review as a conference paper at ICLR 2019 1 - Input 1 - Input 2 - Densify 3 - Denoising : : :? ? 4   4 5 (cid:18)(cid:83) 3/p
Revised: Under review as a conference paper at ICLR 2019 1 - Input 1 - Input 2 - Densify 3 - Denoising : : :? ? 4   4 5 (cid:18)(cid:83) 3/p

Original: Shallow Representation Deep Representation
Revised: Shallow Representation Deep Representation

Original: Target =
Revised: Target =

Original: SR    e rr o r (Densify)    4 (Densify) e rr o r supervised unsupervised supervised e rr o r
Revised: SR    e rr o r (Densify)    4 (Densify) e rr o r supervised unsupervised supervised e rr o r

Original: B ac k w a r d 4 5 4 4 3 5 3 2 1 5 3 ? 4 ? ? ? ? ? ? ?
Revised: B ac k w a r d 4 5 4 4 3 5 3 2 1 5 3 ? 4 ? ? ? ? ? ? ?

Original: u 1 u 2 u 3 u 4 v 1 v 2 v 3 v 4 v 5 1 1 1 1 ? 1 11 ? 1 1 ?? ?? ? ? ?
Revised: u 1 u 2 u 3 u 4 v 1 v 2 v 3 v 4 v 5 1 1 1 1 ? 1 11 ? 1 1 ?? ?? ? ? ?

Original: u 1 u 2 uuv 1 v 2 v 3 v 4 vitems u s e r s 0.17 0.25 0.33 0.17 0.08 1 

Revised: u 1 u 2 uuv 1 v 2 v 3 v 4 vitems u s e r s 0.17 0.25 0.33 0.17 0.08 1

Original: Stage
Revised: Stage

Original: Stage 3 Stage
Revised: Stage 3 Stage

Original: RA
Revised: RA

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: RLAgent vo t+ o t+ o t+k
Revised: RLAgent vo t+ o t+ o t+k

Original: S t k - s t e p n-rollout
Revised: S t k - s t e p n-rollout

Original: Imag .
Revised: Imag .

Original: Core
Revised: Core

Original: Imag.
Revised: Imag.

Original: Core
Revised: Core

Original: Imagine future
Revised: Imagine future

Original: Policy Network rollout-out reconstruction … softmax
Revised: Policy Network rollout-out reconstruction … softmax

Original: Imag.
Revised: Imag.

Original: Net Env.
Revised: Net Env.

Original: Model
Revised: Model

Original: S t+i =
Revised: S t+i =

Original: {s , o } t+1 t error …
Revised: {s , o } t+1 t error …

Original: R o ll o u t …
Revised: R o ll o u t …

Original: S e ss i o n l og purchase
Revised: S e ss i o n l og purchase

Original: A3Cupdate
Revised: A3Cupdate

Original: RL update
Revised: RL update

Original: Self- supervision a) Imagination Core b) Trajectory Manager c) Imagination-augmented Executor
Revised: Self- supervision a) Imagination Core b) Trajectory Manager c) Imagination-augmented Executor

Original: R o ll o u t
Revised: R o ll o u t

Original: Figure2:IRNarchitecture.sshowninFigure2,whereICisimaginationcoretogeneratetrajectories,andN-stepimaginationrollout,usingthesetrajectoriestoperformencoder-decoder.
Revised: Figure2:IRNarchitecture.sshowninFigure2,whereICisimaginationcoretogeneratetrajectories,andN-stepimaginationrollout,usingthesetrajectoriestoperformencoder-decoder.

Original: 4 IRNArchitecture 116 Inthissectionweincorporatetheimaginationreconstructionmoduleintothemodel-freeagents 117 (e.g.,A3C)inordertoenhancedataefﬁciency,promotemorerobustlearningandultimatelyhigher 118 performanceunderthesparseextrinsicrewards.
Revised: 4 IRNArchitecture 116 Inthissectionweincorporatetheimaginationreconstructionmoduleintothemodel-freeagents 117 (e.g.,A3C)inordertoenhancedataefﬁciency,promotemorerobustlearningandultimatelyhigher 118 performanceunderthesparseextrinsicrewards.

Original: OurIRNimplementsanimagination-augmentedpolicy via three key components (Fig 2).
Revised: OurIRNimplementsanimagination-augmentedpolicy via three key components (Fig 2).

Original: The imagination core (IC) predicts next time steps 120 conditionedonactionssampledfromtheimaginationpolicy ˆ ⇡ .
Revised: The imagination core (IC) predicts next time steps 120 conditionedonactionssampledfromtheimaginationpolicy ˆ ⇡ .

Original: Atatimestep t ,the trajectory 121 manager(TM) determineshowtorolloutthe IC undertheplanningstrategy,andthenproducesimaginedtrajectories ˆ T 1 ,..., ˆ T n ofanobservableworldstate s t .
Revised: Atatimestep t ,the trajectory 121 manager(TM) determineshowtorolloutthe IC undertheplanningstrategy,andthenproducesimaginedtrajectories ˆ T 1 ,..., ˆ T n ofanobservableworldstate s t .

Original: Eachtrajectory ˆ T isasequence 123 ofitems { ˆ i t , ˆ i t +1 ,... } ,thattheagentmaypurchase(orclick)fromthecurrenttime t .
Revised: Eachtrajectory ˆ T isasequence 123 ofitems { ˆ i t , ˆ i t +1 ,... } ,thattheagentmaypurchase(orclick)fromthecurrenttime t .

Original: Finally, 124 the Imagination-augmentedExecutor(I2E) aggregatestheinternaldataresultingfromimagination 125 andexternalrewardingdatatoupdateitsactionpolicy ⇡ .
Revised: Finally, 124 the Imagination-augmentedExecutor(I2E) aggregatestheinternaldataresultingfromimagination 125 andexternalrewardingdatatoupdateitsactionpolicy ⇡ .

Original: Speciﬁcally,I2Eoptimizesthepolicy⇡ bymaximizingtheextrinsicrewardswhileminimizingtheintrinsicimaginationerrors, L = 127 L A 3 C + L IRN .Inprinciple,IRNencouragesexploration(ontheoutputlayer,i.e.,itemrepresentation) 128 andimplicitlylearnstoplan("guesswhatyoulike")viaimaginationrollouts,andthereforepromotes 129 quickadaptationtouserinterest.
Revised: Speciﬁcally,I2Eoptimizesthepolicy⇡ bymaximizingtheextrinsicrewardswhileminimizingtheintrinsicimaginationerrors, L = 127 L A 3 C + L IRN .Inprinciple,IRNencouragesexploration(ontheoutputlayer,i.e.,itemrepresentation) 128 andimplicitlylearnstoplan("guesswhatyoulike")viaimaginationrollouts,andthereforepromotes 129 quickadaptationtouserinterest.

Original: ImaginationCore.
Revised: ImaginationCore.

Original: Inordertosimulateimaginedtrajectories,werelyonenvironmentmodelsthat, 131 giventhepresentstateandacandidateaction,makepredictionsaboutthefuturestates.
Revised: Inordertosimulateimaginedtrajectories,werelyonenvironmentmodelsthat, 131 giventhepresentstateandacandidateaction,makepredictionsaboutthefuturestates.

Original: 2 Ingeneral, 132 wecanemployanenvironmentmodelthatbuildonrecentpopularaction-conditionalnext-step 133 predictors[],andtrainitinanunsupervisedfashionfromagentexperiences.
Revised: 2 Ingeneral, 132 wecanemployanenvironmentmodelthatbuildonrecentpopularaction-conditionalnext-step 133 predictors[],andtrainitinanunsupervisedfashionfromagentexperiences.

Original: However,thepredictorsusuallysufferfrommodelerrors,resultinginpooragentperformance,andneedextracomputational 135 cost(e.g.
Revised: However,thepredictorsusuallysufferfrommodelerrors,resultinginpooragentperformance,andneedextracomputational 135 cost(e.g.

Original: ,pre-training)[error][cost].Besides,thepredictorsmaylearnatrivialidenticalfunction, 136 sincethestatetransitioninagenttrajectoriesisdeterministic,i.e., s t +

Revised: ,pre-training)[error][cost].Besides,thepredictorsmaylearnatrivialidenticalfunction, 136 sincethestatetransitioninagenttrajectoriesisdeterministic,i.e., s t +

Original: =
Revised: =

Original: s t [{ i t } and a t =
Revised: s t [{ i t } and a t =

Original: i t .In 137 thiswork,wederiveastaticenvironmentmodelfromthestatetransition:
Revised: i t .In 137 thiswork,wederiveastaticenvironmentmodelfromthestatetransition:

Original: ˆ s t + ⌧
Revised: ˆ s t + ⌧

Original: +

Revised: +

Original: =
Revised: =

Original: ˆ s t + ⌧ [{ ˆ i t + ⌧ } , 138 ˆ i t + ⌧
Revised: ˆ s t + ⌧ [{ ˆ i t + ⌧ } , 138 ˆ i t + ⌧

Original: =
Revised: =

Original: ˆ a t + ⌧ and ˆ s t =
Revised: ˆ a t + ⌧ and ˆ s t =

Original: s t ,where ⌧ isthelengthoftheimaginedrollout, ˆ a t + ⌧ theoutputactionofthe 139 imaginationpolicy ˆ ⇡ .Duringtraining,thegenerateditem ˆ i t + ⌧ maynotbethetruepurchase,butwe 140 stilluseitforself-supervisedreconstructionof I2E .Thismakestheactionpolicy
Revised: s t ,where ⌧ isthelengthoftheimaginedrollout, ˆ a t + ⌧ theoutputactionofthe 139 imaginationpolicy ˆ ⇡ .Duringtraining,thegenerateditem ˆ i t + ⌧ maynotbethetruepurchase,butwe 140 stilluseitforself-supervisedreconstructionof I2E .Thismakestheactionpolicy

Original: ⇡ morerobusttointrinsicerrorsandforcestheimaginationpolicy ˆ ⇡ togeneratemoreaccurateactions.
Revised: ⇡ morerobusttointrinsicerrorsandforcestheimaginationpolicy ˆ ⇡ togeneratemoreaccurateactions.

Original: 142 Inpractice,theimaginationpolicy ˆ ⇡ canbeobtainedfrompolicydistillation[]orﬁxedtargetnetwork 143 likeDQN[].Theformerdistillstheactionpolicy ⇡ ( s t ; ✓ ) intoasmallerrolloutnetwork ˆ ⇡ ( s t ;ˆ ✓ ) , 144 usingacross-entropyloss, l ⇡, ˆ ⇡ ( s t )=
Revised: 142 Inpractice,theimaginationpolicy ˆ ⇡ canbeobtainedfrompolicydistillation[]orﬁxedtargetnetwork 143 likeDQN[].Theformerdistillstheactionpolicy ⇡ ( s t ; ✓ ) intoasmallerrolloutnetwork ˆ ⇡ ( s t ;ˆ ✓ ) , 144 usingacross-entropyloss, l ⇡, ˆ ⇡ ( s t )=

Original: P a ⇡ ( a | s t ) log ˆ ⇡ ( a | s t ;ˆ ✓ ) .Thelatterusesasharedbutslowly
Revised: P a ⇡ ( a | s t ) log ˆ ⇡ ( a | s t ;ˆ ✓ ) .Thelatterusesasharedbutslowly

Original: 145 changingtargetnetwork ˆ ⇡ ( s t
Revised: 145 changingtargetnetwork ˆ ⇡ ( s t

Original: [Table]
Revised: [Table]

Original: ✓  ) ,where ✓  arepreviousparametersin ⇡
Revised: ✓  ) ,where ✓  arepreviousparametersin ⇡

Original: [Table]
Revised: [Table]

Original: s t
Revised: s t

Original: [Table]
Revised: [Table]

Original: ✓
Revised: ✓

Original: [Table]
Revised: [Table]

Original: 146 actionpolicy ⇡ ,
Revised: 146 actionpolicy ⇡ ,

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Figure2: IRNarchitecture.
Revised: Figure2: IRNarchitecture.

Original: sshowninFigure2,whereICisimaginationcoretogeneratetrajectories, andN-stepimaginationrollout,usingthesetrajectoriestoperformencoder-decoder.
Revised: sshowninFigure2,whereICisimaginationcoretogeneratetrajectories, andN-stepimaginationrollout,usingthesetrajectoriestoperformencoder-decoder.

Original: 4 IRN Architecture
Revised: 4 IRN Architecture

Original: In this section we incorporate the imagination reconstruction module into the model-free agents 117 (e.g.,A3C)inordertoenhancedataefﬁciency,promotemorerobustlearningandultimatelyhigher 118 performanceunderthesparseextrinsicrewards.
Revised: In this section we incorporate the imagination reconstruction module into the model-free agents 117 (e.g.,A3C)inordertoenhancedataefﬁciency,promotemorerobustlearningandultimatelyhigher 118 performanceunderthesparseextrinsicrewards.

Original: OurIRNimplementsanimagination-augmentedpolicy via three key components (Fig 2).
Revised: OurIRNimplementsanimagination-augmentedpolicy via three key components (Fig 2).

Original: The imagination core (IC) predicts next time steps 120 conditioned on actions sampled from the imagination policy ˆ ⇡ .
Revised: The imagination core (IC) predicts next time steps 120 conditioned on actions sampled from the imagination policy ˆ ⇡ .

Original: At a time step t , the trajectory 121 manager (TM) determines how to roll out the IC under the planning strategy, and then producesimaginedtrajectories ˆ T 1 ,..., ˆ T n
Revised: At a time step t , the trajectory 121 manager (TM) determines how to roll out the IC under the planning strategy, and then producesimaginedtrajectories ˆ T 1 ,..., ˆ T n

Original: [Table]
Revised: [Table]

Original: s t .
Revised: s t .

Original: Eachtrajectory ˆ T isasequence 123 of items { ˆ i t , ˆ i t +1 ,... } , that the agent may purchase (or click) from the current time t .
Revised: Eachtrajectory ˆ T isasequence 123 of items { ˆ i t , ˆ i t +1 ,... } , that the agent may purchase (or click) from the current time t .

Original: Finally, 124 the Imagination-augmentedExecutor(I2E) aggregatestheinternaldataresultingfromimaginationand external rewarding data to update its action policy ⇡ .
Revised: Finally, 124 the Imagination-augmentedExecutor(I2E) aggregatestheinternaldataresultingfromimaginationand external rewarding data to update its action policy ⇡ .

Original: Speciﬁcally, I2E optimizes the policy 126 ⇡ by maximizing the
Revised: Speciﬁcally, I2E optimizes the policy 126 ⇡ by maximizing the

Original: [Table]
Revised: [Table]

Original: rewards
Revised: rewards

Original: [Table]
Revised: [Table]

Original: minimizing the intrinsic imagination errors, L
Revised: minimizing the intrinsic imagination errors, L

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 3 C + L IRN .
Revised: 3 C + L IRN .

Original: Inprinciple
Revised: Inprinciple

Original: [Table]
Revised: [Table]

Original: IRNencouragesexploration
Revised: IRNencouragesexploration

Original: [Table]
Revised: [Table]

Original: ontheoutputlayer
Revised: ontheoutputlayer

Original: [Table]
Revised: [Table]

Original: i.e.
Revised: i.e.

Original: [Table]
Revised: [Table]

Original: itemrepresentation)
Revised: itemrepresentation)

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: employ
Revised: employ

Original: [Table]
Revised: [Table]

Original: popular
Revised: popular

Original: [Table]
Revised: [Table]

Original: However
Revised: However

Original: [Table]
Revised: [Table]

Original: cost
Revised: cost

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: Inpractice,theimaginationpolicy ˆ ⇡ canbeobtainedfrompolicydistillation[]orﬁxedtargetnetwork 143 likeDQN[].
Revised: Inpractice,theimaginationpolicy ˆ ⇡ canbeobtainedfrompolicydistillation[]orﬁxedtargetnetwork 143 likeDQN[].

Original: Theformerdistillstheactionpolicy ⇡ ( s t ; ✓ ) intoasmallerrolloutnetwork ˆ ⇡ ( s t ;ˆ ✓ ) , 144 usingacross-entropyloss, l ⇡, ˆ ⇡ ( s t )=
Revised: Theformerdistillstheactionpolicy ⇡ ( s t ; ✓ ) intoasmallerrolloutnetwork ˆ ⇡ ( s t ;ˆ ✓ ) , 144 usingacross-entropyloss, l ⇡, ˆ ⇡ ( s t )=

Original: P a ⇡ ( a | s t ) log ˆ ⇡ ( a | s t ;ˆ ✓ ) .
Revised: P a ⇡ ( a | s t ) log ˆ ⇡ ( a | s t ;ˆ ✓ ) .

Original: Thelatterusesasharedbutslowly 145 changingtargetnetwork ˆ ⇡ ( s t ; ✓  ) ,where ✓  arepreviousparametersin ⇡ ( s t ; ✓ ) .
Revised: Thelatterusesasharedbutslowly 145 changingtargetnetwork ˆ ⇡ ( s t ; ✓  ) ,where ✓  arepreviousparametersin ⇡ ( s t ; ✓ ) .

Original: Byimitatingthe 146 actionpolicy ⇡ ,theimaginedtrajectorieswillbesimilartoagentexperiencesintherealenvironment;thisalsohelps I2E learnapredictiverepresentationofrewardingstates,andinturnshouldallowthe 148 easylearningoftheactionpolicyunderthesparserewardsignals.2 IRNdonotpredicttherewards,sinceitisnotusefulasreportedin[].andinRS,rewardsofdifferentactions arehardtospecify
Revised: Byimitatingthe 146 actionpolicy ⇡ ,theimaginedtrajectorieswillbesimilartoagentexperiencesintherealenvironment;thisalsohelps I2E learnapredictiverepresentationofrewardingstates,andinturnshouldallowthe 148 easylearningoftheactionpolicyunderthesparserewardsignals.2 IRNdonotpredicttherewards,sinceitisnotusefulasreportedin[].andinRS,rewardsofdifferentactions arehardtospecify

Original: IRNarchitecture.
Revised: IRNarchitecture.

Original: sshowninFigure2,whereICisimaginationcoretogeneratetrajectories,
Revised: sshowninFigure2,whereICisimaginationcoretogeneratetrajectories,

Original: ,usingthesetrajectoriestoperformencoder
Revised: ,usingthesetrajectoriestoperformencoder

Original: 4 IRN Architecture
Revised: 4 IRN Architecture

Original: In this section we incorporate the imagination reconstruction module into the model-free agents 117 (e.g.,A3C)inordertoenhancedataefﬁciency,promotemorerobustlearningandultimatelyhigher 118 performanceunderthesparseextrinsicrewards.
Revised: In this section we incorporate the imagination reconstruction module into the model-free agents 117 (e.g.,A3C)inordertoenhancedataefﬁciency,promotemorerobustlearningandultimatelyhigher 118 performanceunderthesparseextrinsicrewards.

Original: OurIRNimplementsanimagination-augmentedpolicy via three key components (Fig 2).
Revised: OurIRNimplementsanimagination-augmentedpolicy via three key components (Fig 2).

Original: The imagination core (IC) predicts next time steps 120 conditioned on actions sampled from the imagination policy ˆ ⇡ .
Revised: The imagination core (IC) predicts next time steps 120 conditioned on actions sampled from the imagination policy ˆ ⇡ .

Original: At a time step t , the trajectory 121 manager (TM) determines how to roll out the IC under the planning strategy, and then producesimaginedtrajectories ˆ T 1 ,..., ˆ T n ofanobservableworldstate s t .
Revised: At a time step t , the trajectory 121 manager (TM) determines how to roll out the IC under the planning strategy, and then producesimaginedtrajectories ˆ T 1 ,..., ˆ T n ofanobservableworldstate s t .

Original: Eachtrajectory ˆ T isasequence 123 of items { ˆ i t , ˆ i t +1 ,... } , that the agent may purchase (or click) from the current time t .
Revised: Eachtrajectory ˆ T isasequence 123 of items { ˆ i t , ˆ i t +1 ,... } , that the agent may purchase (or click) from the current time t .

Original: Finally, 124 the Imagination-augmentedExecutor(I2E) aggregatestheinternaldataresultingfromimaginationand external rewarding data to update its action policy ⇡ .
Revised: Finally, 124 the Imagination-augmentedExecutor(I2E) aggregatestheinternaldataresultingfromimaginationand external rewarding data to update its action policy ⇡ .

Original: Speciﬁcally, I2E optimizes the policy 126 ⇡ by maximizing the extrinsic rewards while minimizing the intrinsic imagination errors, L =
Revised: Speciﬁcally, I2E optimizes the policy 126 ⇡ by maximizing the extrinsic rewards while minimizing the intrinsic imagination errors, L =

Original: L A 3 C + L IRN .
Revised: L A 3 C + L IRN .

Original: Inprinciple,IRNencouragesexploration(ontheoutputlayer,i.e.,itemrepresentation) 128 andimplicitlylearnstoplan("guesswhatyoulike")viaimaginationrollouts,andthereforepromotes 129 quickadaptationtouserinterest.
Revised: Inprinciple,IRNencouragesexploration(ontheoutputlayer,i.e.,itemrepresentation) 128 andimplicitlylearnstoplan("guesswhatyoulike")viaimaginationrollouts,andthereforepromotes 129 quickadaptationtouserinterest.

Original: ImaginationCore.
Revised: ImaginationCore.

Original: Inordertosimulateimaginedtrajectories,werelyonenvironmentmodelsthat, 131 giventhepresentstateandacandidateaction,makepredictionsaboutthefuturestates.
Revised: Inordertosimulateimaginedtrajectories,werelyonenvironmentmodelsthat, 131 giventhepresentstateandacandidateaction,makepredictionsaboutthefuturestates.

Original: 2 Ingeneral, 132 we can employ an environment model that build on recent popular action-conditional next-step 133 predictors[],andtrainitinanunsupervisedfashionfromagentexperiences.
Revised: 2 Ingeneral, 132 we can employ an environment model that build on recent popular action-conditional next-step 133 predictors[],andtrainitinanunsupervisedfashionfromagentexperiences.

Original: However,thepredictorsusuallysufferfrommodelerrors,resultinginpooragentperformance,andneedextracomputational 135 cost(e.g.,pre-training)[error][cost].
Revised: However,thepredictorsusuallysufferfrommodelerrors,resultinginpooragentperformance,andneedextracomputational 135 cost(e.g.,pre-training)[error][cost].

Original: Besides,thepredictorsmaylearnatrivialidenticalfunction, 136 sincethestatetransitioninagenttrajectoriesisdeterministic,i.e., s t +

Revised: Besides,thepredictorsmaylearnatrivialidenticalfunction, 136 sincethestatetransitioninagenttrajectoriesisdeterministic,i.e., s t +

Original: =
Revised: =

Original: s t [{ i t } and a t = i t .
Revised: s t [{ i t } and a t = i t .

Original: In 137 thiswork,wederiveastaticenvironmentmodelfromthestatetransition: ˆ s t + ⌧
Revised: In 137 thiswork,wederiveastaticenvironmentmodelfromthestatetransition: ˆ s t + ⌧

Original: +

Revised: +

Original: =
Revised: =

Original: ˆ s t + ⌧ [{ ˆ i t + ⌧ } ,ˆ i t + ⌧ =ˆ a t + ⌧ and ˆ s t =
Revised: ˆ s t + ⌧ [{ ˆ i t + ⌧ } ,ˆ i t + ⌧ =ˆ a t + ⌧ and ˆ s t =

Original: s t ,where ⌧ isthelengthoftheimaginedrollout, ˆ a t + ⌧ theoutputactionofthe 139 imaginationpolicy ˆ ⇡ .
Revised: s t ,where ⌧ isthelengthoftheimaginedrollout, ˆ a t + ⌧ theoutputactionofthe 139 imaginationpolicy ˆ ⇡ .

Original: Duringtraining,thegenerateditem ˆ i t + ⌧ maynotbethetruepurchase,butwe 140 stilluseitforself-supervisedreconstructionof I2E .Thismakestheactionpolicy
Revised: Duringtraining,thegenerateditem ˆ i t + ⌧ maynotbethetruepurchase,butwe 140 stilluseitforself-supervisedreconstructionof I2E .Thismakestheactionpolicy

Original: ⇡ morerobusttointrinsicerrorsandforcestheimaginationpolicy ˆ ⇡ togeneratemoreaccurateactions.
Revised: ⇡ morerobusttointrinsicerrorsandforcestheimaginationpolicy ˆ ⇡ togeneratemoreaccurateactions.

Original: 142 Inpractice,theimaginationpolicy ˆ ⇡ canbeobtainedfrompolicydistillation[]orﬁxedtargetnetwork 143 likeDQN[].
Revised: 142 Inpractice,theimaginationpolicy ˆ ⇡ canbeobtainedfrompolicydistillation[]orﬁxedtargetnetwork 143 likeDQN[].

Original: Theformerdistillstheactionpolicy ⇡ ( s t ; ✓ ) intoasmallerrolloutnetwork ˆ ⇡ ( s t ;ˆ ✓ ) , 144 usingacross-entropyloss, l ⇡, ˆ ⇡ ( s t )=
Revised: Theformerdistillstheactionpolicy ⇡ ( s t ; ✓ ) intoasmallerrolloutnetwork ˆ ⇡ ( s t ;ˆ ✓ ) , 144 usingacross-entropyloss, l ⇡, ˆ ⇡ ( s t )=

Original: P a ⇡ ( a | s t ) log ˆ ⇡ ( a | s t ;ˆ ✓ ) .
Revised: P a ⇡ ( a | s t ) log ˆ ⇡ ( a | s t ;ˆ ✓ ) .

Original: Thelatterusesasharedbutslowly 145 changingtargetnetwork ˆ ⇡ ( s t ; ✓  ) ,where ✓  arepreviousparametersin ⇡ ( s t ; ✓ ) .
Revised: Thelatterusesasharedbutslowly 145 changingtargetnetwork ˆ ⇡ ( s t ; ✓  ) ,where ✓  arepreviousparametersin ⇡ ( s t ; ✓ ) .

Original: Byimitatingthe 146 actionpolicy ⇡ ,theimaginedtrajectorieswillbesimilartoagentexperiencesintherealenvironment;thisalsohelps I2E learnapredictiverepresentationofrewardingstates,andinturnshouldallowthe 148 easylearningoftheactionpolicyunderthesparserewardsignals.2 IRNdonotpredicttherewards,sinceitisnotusefulasreportedin[].andinRS,rewardsofdifferentactions arehardtospecify
Revised: Byimitatingthe 146 actionpolicy ⇡ ,theimaginedtrajectorieswillbesimilartoagentexperiencesintherealenvironment;thisalsohelps I2E learnapredictiverepresentationofrewardingstates,andinturnshouldallowthe 148 easylearningoftheactionpolicyunderthesparserewardsignals.2 IRNdonotpredicttherewards,sinceitisnotusefulasreportedin[].andinRS,rewardsofdifferentactions arehardtospecify

Original: 4 IRN Architecture
Revised: 4 IRN Architecture

Original: In
Revised: In

Original: ) in order to enhance data efﬁciency, promote more robust learning and ultimately higher 118 performance under the sparse extrinsic rewards.
Revised: ) in order to enhance data efﬁciency, promote more robust learning and ultimately higher 118 performance under the sparse extrinsic rewards.

Original: Our IRN implements an imagination-augmentedpolicy via three key components (Fig 2).
Revised: Our IRN implements an imagination-augmentedpolicy via three key components (Fig 2).

Original: The imagination core (IC) predicts next time steps 120 conditioned on actions sampled from the imagination policy ˆ ⇡ .
Revised: The imagination core (IC) predicts next time steps 120 conditioned on actions sampled from the imagination policy ˆ ⇡ .

Original: At a time step t , the trajectorymanager (TM) determines how to roll out the IC under the planning strategy, and then produces 122 imagined trajectories ˆ
Revised: At a time step t , the trajectorymanager (TM) determines how to roll out the IC under the planning strategy, and then produces 122 imagined trajectories ˆ

Original: T 1 ,..., ˆ T n of an observable world state s t .
Revised: T 1 ,..., ˆ T n of an observable world state s t .

Original: Each trajectory ˆ T is a sequence 123 of items { ˆ i t , ˆ i t +1 ,... } , that the agent may purchase (or click) from the current time t .
Revised: Each trajectory ˆ T is a sequence 123 of items { ˆ i t , ˆ i t +1 ,... } , that the agent may purchase (or click) from the current time t .

Original: Finally, 124 the Imagination-augmented Executor (I2E) aggregates the internal data resulting from imaginationand external rewarding data to update its action policy ⇡ .
Revised: Finally, 124 the Imagination-augmented Executor (I2E) aggregates the internal data resulting from imaginationand external rewarding data to update its action policy ⇡ .

Original: Speciﬁcally, I2E optimizes the policy 126 ⇡ by maximizing the extrinsic rewards while minimizing the intrinsic imagination errors, L =
Revised: Speciﬁcally, I2E optimizes the policy 126 ⇡ by maximizing the extrinsic rewards while minimizing the intrinsic imagination errors, L =

Original: L A 3 C + L IRN .
Revised: L A 3 C + L IRN .

Original: Inprinciple,IRNencouragesexploration(ontheoutputlayer,i.e.,itemrepresentation) 128 and implicitly learns to plan ("guess what you like") via imagination rollouts, and therefore promotes 129 quick adaptation to user interest.
Revised: Inprinciple,IRNencouragesexploration(ontheoutputlayer,i.e.,itemrepresentation) 128 and implicitly learns to plan ("guess what you like") via imagination rollouts, and therefore promotes 129 quick adaptation to user interest.

Original: Imagination Core.
Revised: Imagination Core.

Original: In order to simulate imagined trajectories, we rely on environment models that, 131 given the present state and a candidate action, make predictions about the future states.
Revised: In order to simulate imagined trajectories, we rely on environment models that, 131 given the present state and a candidate action, make predictions about the future states.

Original: In general, 132 we can employ an environment model that build on recent popular action-conditional next-steppredictors [], and train it in an unsupervised fashion from agent experiences.
Revised: In general, 132 we can employ an environment model that build on recent popular action-conditional next-steppredictors [], and train it in an unsupervised fashion from agent experiences.

Original: However, the predictors 134 usually suffer from model errors, resulting in poor agent performance, and need extra computationalcost (e.g., pre-training)
Revised: However, the predictors 134 usually suffer from model errors, resulting in poor agent performance, and need extra computationalcost (e.g., pre-training)

Original: [error][cost].
Revised: [error][cost].

Original: Besides, the predictors may learn a trivial identical function, 136 since the state transition in agent trajectories is deterministic, i.e., s t +

Revised: Besides, the predictors may learn a trivial identical function, 136 since the state transition in agent trajectories is deterministic, i.e., s t +

Original: =
Revised: =

Original: s t [{ i t } and a t = i t .
Revised: s t [{ i t } and a t = i t .

Original: Inthis work, we derive a static environment model from the state transition: ˆ s t + ⌧
Revised: Inthis work, we derive a static environment model from the state transition: ˆ s t + ⌧

Original: +

Revised: +

Original: =
Revised: =

Original: ˆ s t + ⌧ [{ ˆ i t + ⌧ } , 138 ˆ i t + ⌧
Revised: ˆ s t + ⌧ [{ ˆ i t + ⌧ } , 138 ˆ i t + ⌧

Original: =
Revised: =

Original: ˆ a t + ⌧ and ˆ s t =
Revised: ˆ a t + ⌧ and ˆ s t =

Original: s t , where ⌧ is the length of the imagined rollout, ˆ a t + ⌧ the output action of theimagination policy ˆ ⇡ .
Revised: s t , where ⌧ is the length of the imagined rollout, ˆ a t + ⌧ the output action of theimagination policy ˆ ⇡ .

Original: During training, the generated item ˆ i t + ⌧ may not be the true purchase, but we 140 still use it for self-supervised reconstruction of I2E .
Revised: During training, the generated item ˆ i t + ⌧ may not be the true purchase, but we 140 still use it for self-supervised reconstruction of I2E .

Original: This makes the action policy ⇡ more robust tointrinsic errors and forces the imagination policy ˆ ⇡ to generate more accurate actions.
Revised: This makes the action policy ⇡ more robust tointrinsic errors and forces the imagination policy ˆ ⇡ to generate more accurate actions.

Original: Inpractice,theimaginationpolicy ˆ ⇡ canbeobtainedfrompolicydistillation[]orﬁxedtargetnetwork 143 like DQN [].
Revised: Inpractice,theimaginationpolicy ˆ ⇡ canbeobtainedfrompolicydistillation[]orﬁxedtargetnetwork 143 like DQN [].

Original: The former distills the action policy ⇡ ( s t ; ✓ ) into a smaller rollout network ˆ ⇡ ( s t ; ˆ ✓ ) , 144 using a cross-entropy loss, l ⇡, ˆ ⇡ ( s t ) =
Revised: The former distills the action policy ⇡ ( s t ; ✓ ) into a smaller rollout network ˆ ⇡ ( s t ; ˆ ✓ ) , 144 using a cross-entropy loss, l ⇡, ˆ ⇡ ( s t ) =

Original: P a ⇡ ( a | s t ) log ˆ ⇡ ( a | s t ; ˆ ✓ ) .
Revised: P a ⇡ ( a | s t ) log ˆ ⇡ ( a | s t ; ˆ ✓ ) .

Original: The latter uses a shared but slowly 145 changing target network ˆ ⇡ ( s t ; ✓  ) , where ✓  are previous parameters in ⇡ ( s t ; ✓ ) .
Revised: The latter uses a shared but slowly 145 changing target network ˆ ⇡ ( s t ; ✓  ) , where ✓  are previous parameters in ⇡ ( s t ; ✓ ) .

Original: By imitating the 146 actionpolicy ⇡ , theimaginedtrajectories willbe similarto agentexperiencesin thereal environment;this also helps I2E learn a predictive representation of rewarding states, and in turn should allow the 148 easy learning of the action policy under the sparse reward signals.2 IRNdonotpredicttherewards,sinceitisnotusefulasreportedin[].
Revised: By imitating the 146 actionpolicy ⇡ , theimaginedtrajectories willbe similarto agentexperiencesin thereal environment;this also helps I2E learn a predictive representation of rewarding states, and in turn should allow the 148 easy learning of the action policy under the sparse reward signals.2 IRNdonotpredicttherewards,sinceitisnotusefulasreportedin[].

Original: andinRS,rewardsofdifferentactions arehardtospecify
Revised: andinRS,rewardsofdifferentactions arehardtospecify

Original: Forsequentialrecommendations itisimportanttoenhancethepurchasewillingof
Revised: Forsequentialrecommendations itisimportanttoenhancethepurchasewillingof

Original: Recentpopularsession-basedmodelsuseclassiﬁcationmethodstomodel purchasereward wherethepurchaseisusedastherewardsignalforrecommenderagentandusersessionsaretreatedasexternalenvironment.
Revised: Recentpopularsession-basedmodelsuseclassiﬁcationmethodstomodel purchasereward wherethepurchaseisusedastherewardsignalforrecommenderagentandusersessionsaretreatedasexternalenvironment.

Original: Tohandlethesparsepurchasereward,we propose a simple architecture, namely Imagination Reconstruction Network(IRN), which is inspired from the neu-science Prediction Error Minimization.
Revised: Tohandlethesparsepurchasereward,we propose a simple architecture, namely Imagination Reconstruction Network(IRN), which is inspired from the neu-science Prediction Error Minimization.

Original: IRN distill a roll-out policy from action policy and combine it with a staticenvironment model to predict the future observations.
Revised: IRN distill a roll-out policy from action policy and combine it with a staticenvironment model to predict the future observations.

Original: Imaginations are thenused as self-supervised reconstruction signal for action policy when there is nopurchasereward.
Revised: Imaginations are thenused as self-supervised reconstruction signal for action policy when there is nopurchasereward.

Original: Byminimizingtheintrinsicimaginationerrorandmaximizingtheextrinsicpurchaseprobability,IRNcanautomaticallybalanceEEandlearnabetterpredictiverepresentations,resultinginamorerobustactionpolicy.
Revised: Byminimizingtheintrinsicimaginationerrorandmaximizingtheextrinsicpurchaseprobability,IRNcanautomaticallybalanceEEandlearnabetterpredictiverepresentations,resultinginamorerobustactionpolicy.

Original: ExperimentsshowthatIRNcansigniﬁcantlyenhancethepurchase,andachieveasatisfactoryclickperformanceevenwhenwedonottreatclickasrewards.robusttothecold-startscenario(shortsessions).improvethedataefﬁciency.doexplorationwhenitemspaceishuge.1 Introductions T  

Revised: ExperimentsshowthatIRNcansigniﬁcantlyenhancethepurchase,andachieveasatisfactoryclickperformanceevenwhenwedonottreatclickasrewards.robusttothecold-startscenario(shortsessions).improvethedataefﬁciency.doexplorationwhenitemspaceishuge.1 Introductions T

Original: 21 ˆ i tˆ i t +1 23 a T  1 24 ⇡,
Revised: 21 ˆ i tˆ i t +1 23 a T  1 24 ⇡,

Original: Vsession-based recommendation (end with buy)
Revised: Vsession-based recommendation (end with buy)

Original: GRU4REC classiﬁcation problem do not 26 differentiatetheclickandpurchase,
Revised: GRU4REC classiﬁcation problem do not 26 differentiatetheclickandpurchase,

Original: ,wemaypaymoreattentiontotheimprovementsofpurchase.
Revised: ,wemaypaymoreattentiontotheimprovementsofpurchase.

Original: -of-
Revised: -of-

Original: -artmethodsthatconsiderusers’sequentialbehaviorforrecommendation, 29 e sequentialrecommenderswithrecurrentneuralnetworks(
Revised: -artmethodsthatconsiderusers’sequentialbehaviorforrecommendation, 29 e sequentialrecommenderswithrecurrentneuralnetworks(

Original: )orMarkovchains,
Revised: )orMarkovchains,

Original: 30 achievessigniﬁcantlyandconsistentlybetterperformanceonfourreal-worlddatasets.
Revised: 30 achievessigniﬁcantlyandconsistentlybetterperformanceonfourreal-worlddatasets.

Original: Inmanyreal-worldapplications,users’currentinterestsareinﬂuencedbytheirhistoricalbehaviors
Revised: Inmanyreal-worldapplications,users’currentinterestsareinﬂuencedbytheirhistoricalbehaviors

Original: 32 To model this phenomenon, previous sequential recommenders with user historical records.
Revised: 32 To model this phenomenon, previous sequential recommenders with user historical records.

Original: For
Revised: For

Original: Submittedto32ndConferenceonNeuralInformationProcessingSystems
Revised: Submittedto32ndConferenceonNeuralInformationProcessingSystems

Original: (NIPS2018).
Revised: (NIPS2018).

Original: Donotdistribute
Revised: Donotdistribute

Original: AnonymousAuthor(s)
Revised: AnonymousAuthor(s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Forsequentialrecommendations,itisimportanttoenhancethepurchasewillingof
Revised: Forsequentialrecommendations,itisimportanttoenhancethepurchasewillingof

Original: users
Revised: users

Original: .Recentpopularsession
Revised: .Recentpopularsession

Original: basedmodelsuseclassiﬁcationmethodstomodel thesequentialactionsclassiﬁcation,whichdonotexplicitdifferentiatetheclickand purchasereward
Revised: basedmodelsuseclassiﬁcationmethodstomodel thesequentialactionsclassiﬁcation,whichdonotexplicitdifferentiatetheclickand purchasereward

Original: Inthispaper,weformulatesequentialrecommendationasMDP wherethepurchaseisusedastherewardsignalforrecommenderagentanduser
Revised: Inthispaper,weformulatesequentialrecommendationasMDP wherethepurchaseisusedastherewardsignalforrecommenderagentanduser

Original: sessionsaretreatedasexternalenvironment.
Revised: sessionsaretreatedasexternalenvironment.

Original: Tohandlethesparsepurchasereward weproposeasimplearchitecture,namelyImaginationReconstructionNetwork
Revised: Tohandlethesparsepurchasereward weproposeasimplearchitecture,namelyImaginationReconstructionNetwork

Original: (IRN), which is inspired from the neu-science Prediction Error Minimization.
Revised: (IRN), which is inspired from the neu-science Prediction Error Minimization.

Original: IRN distill a roll-out policy from action policy and combine it with a static environment model to predict the future observations.
Revised: IRN distill a roll-out policy from action policy and combine it with a static environment model to predict the future observations.

Original: Imaginations are then usedasself-supervisedreconstructionsignalforactionpolicywhenthereisno purchasereward
Revised: Imaginations are then usedasself-supervisedreconstructionsignalforactionpolicywhenthereisno purchasereward

Original: .Byminimizingtheintrinsicimaginationerrorandmaximizingthe extrinsicpurchaseprobability,IRNcanautomaticallybalanceEEandlearnabetter predictiverepresentations,resultinginamorerobustactionpolicy
Revised: .Byminimizingtheintrinsicimaginationerrorandmaximizingthe extrinsicpurchaseprobability,IRNcanautomaticallybalanceEEandlearnabetter predictiverepresentations,resultinginamorerobustactionpolicy

Original: .Experiments showthatIRNcansigniﬁcantlyenhancethepurchase,andachieveasatisfactory clickperformanceevenwhenwedonottreatclickasrewards
Revised: .Experiments showthatIRNcansigniﬁcantlyenhancethepurchase,andachieveasatisfactory clickperformanceevenwhenwedonottreatclickasrewards

Original: .robusttothecold
Revised: .robusttothecold

Original: -startscenario(shortsessions
Revised: -startscenario(shortsessions

Original: ).improvethedataefﬁciency.doexplorationwhenitemspaceishuge.
Revised: ).improvethedataefﬁciency.doexplorationwhenitemspaceishuge.

Original: 1 Introduction
Revised: 1 Introduction

Original: [Figure]
Revised: [Figure]

Original: T
Revised: T

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: i
Revised: i

Original: AnonymousAuthor(s)
Revised: AnonymousAuthor(s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: AfﬁliationAddress
Revised: AfﬁliationAddress

Original: Abstract
Revised: Abstract

Original: Forsequentialrecommendations,itisimportanttoenhancethepurchasewillingof 1 users.
Revised: Forsequentialrecommendations,itisimportanttoenhancethepurchasewillingof 1 users.

Original: Recentpopularsession-basedmodelsuseclassiﬁcationmethodstomodel 2 thesequentialactionsclassiﬁcation,whichdonotexplicitdifferentiatetheclickandpurchasereward.
Revised: Recentpopularsession-basedmodelsuseclassiﬁcationmethodstomodel 2 thesequentialactionsclassiﬁcation,whichdonotexplicitdifferentiatetheclickandpurchasereward.

Original: Inthispaper,weformulatesequentialrecommendationasMDP, 4 wherethepurchaseisusedastherewardsignalforrecommenderagentanduser 5 sessionsaretreatedasexternalenvironment.
Revised: Inthispaper,weformulatesequentialrecommendationasMDP, 4 wherethepurchaseisusedastherewardsignalforrecommenderagentanduser 5 sessionsaretreatedasexternalenvironment.

Original: Tohandlethesparsepurchasereward,weproposeasimplearchitecture,namelyImaginationReconstructionNetwork 7 (IRN), which is inspired from the neu-science Prediction Error Minimization.
Revised: Tohandlethesparsepurchasereward,weproposeasimplearchitecture,namelyImaginationReconstructionNetwork 7 (IRN), which is inspired from the neu-science Prediction Error Minimization.

Original: 8 IRN distill a roll-out policy from action policy and combine it with a static 9 environment model to predict the future observations.
Revised: 8 IRN distill a roll-out policy from action policy and combine it with a static 9 environment model to predict the future observations.

Original: Imaginations are thenusedasself-supervisedreconstructionsignalforactionpolicywhenthereisno 11 purchasereward.
Revised: Imaginations are thenusedasself-supervisedreconstructionsignalforactionpolicywhenthereisno 11 purchasereward.

Original: Byminimizingtheintrinsicimaginationerrorandmaximizingthe 12 extrinsicpurchaseprobability,IRNcanautomaticallybalanceEEandlearnabetterpredictiverepresentations,resultinginamorerobustactionpolicy.
Revised: Byminimizingtheintrinsicimaginationerrorandmaximizingthe 12 extrinsicpurchaseprobability,IRNcanautomaticallybalanceEEandlearnabetterpredictiverepresentations,resultinginamorerobustactionpolicy.

Original: Experiments 14 showthatIRNcansigniﬁcantlyenhancethepurchase,andachieveasatisfactory 15 clickperformanceevenwhenwedonottreatclickasrewards.robusttothecold-startscenario(shortsessions).
Revised: Experiments 14 showthatIRNcansigniﬁcantlyenhancethepurchase,andachieveasatisfactory 15 clickperformanceevenwhenwedonottreatclickasrewards.robusttothecold-startscenario(shortsessions).

Original: 17 improvethedataefﬁciency.doexplorationwhenitemspaceishuge.1 Introductions T  

Revised: 17 improvethedataefﬁciency.doexplorationwhenitemspaceishuge.1 Introductions T

Original: 21 ˆ i tˆ i t +1 23 a T  1⇡,V
Revised: 21 ˆ i tˆ i t +1 23 a T  1⇡,V

Original: 25 session-based recommendation
Revised: 25 session-based recommendation

Original: (end with buy).
Revised: (end with buy).

Original: using GRU4REC classiﬁcation problem do not 26 differentiatetheclickandpurchase,inpractice,wemaypaymoreattentiontotheimprovementsofpurchase. 

Revised: using GRU4REC classiﬁcation problem do not 26 differentiatetheclickandpurchase,inpractice,wemaypaymoreattentiontotheimprovementsofpurchase.

Original: Comparedwithstate-of-the-artmethodsthatconsiderusers’sequentialbehaviorforrecommendation
Revised: Comparedwithstate-of-the-artmethodsthatconsiderusers’sequentialbehaviorforrecommendation

Original: Submittedto32ndConferenceonNeuralInformationProcessingSystems
Revised: Submittedto32ndConferenceonNeuralInformationProcessingSystems

Original: Purchase as Reward: Session-based Recommendation with Imagination Reconstruction Network
Revised: Purchase as Reward: Session-based Recommendation with Imagination Reconstruction Network

Original: Abstract
Revised: Abstract

Original: ,weproposeasimplearchitecture,
Revised: ,weproposeasimplearchitecture,

Original: 7 (IRN which is inspired from the neu science Prediction Error
Revised: 7 (IRN which is inspired from the neu science Prediction Error

Original: 8 IRN distill a roll out policy from action policy and combine it with a static 9 environment model to predict the future observations.
Revised: 8 IRN distill a roll out policy from action policy and combine it with a static 9 environment model to predict the future observations.

Original: Imaginations are then
Revised: Imaginations are then

Original: Purchase as Reward: Session-based Recommendation with Imagination Reconstruction Network
Revised: Purchase as Reward: Session-based Recommendation with Imagination Reconstruction Network

Original: AnonymousAuthor(s)
Revised: AnonymousAuthor(s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: [Table]
Revised: [Table]

Original: Purchase
Revised: Purchase

Original: Reward: Session based Recommendation
Revised: Reward: Session based Recommendation

Original: (s)
Revised: (s)

Original: AfﬁliationAddress
Revised: AfﬁliationAddress

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Figure2: IRNarchitecture.
Revised: Figure2: IRNarchitecture.

Original: sshowninFigure2,whereICisimaginationcoretogeneratetrajectories, and N-step imagination rollout, using these trajectories to perform encoder-decoder.
Revised: sshowninFigure2,whereICisimaginationcoretogeneratetrajectories, and N-step imagination rollout, using these trajectories to perform encoder-decoder.

Original: 4 IRN Architecture
Revised: 4 IRN Architecture

Original: In this section we incorporate the imagination reconstruction module into
Revised: In this section we incorporate the imagination reconstruction module into

Original: [Table]
Revised: [Table]

Original: agents
Revised: agents

Original: [Table]
Revised: [Table]

Original: enhance data
Revised: enhance data

Original: [Table]
Revised: [Table]

Original: promote more robust learning and ultimately higher 118 performance under the sparse extrinsic rewards
Revised: promote more robust learning and ultimately higher 118 performance under the sparse extrinsic rewards

Original: [Table]
Revised: [Table]

Original: Our IRN implements an imagination-augmentedpolicy via three key components
Revised: Our IRN implements an imagination-augmentedpolicy via three key components

Original: [Table]
Revised: [Table]

Original: Fig
Revised: Fig

Original: [Table]
Revised: [Table]

Original: core
Revised: core

Original: [Table]
Revised: [Table]

Original: IC
Revised: IC

Original: [Table]
Revised: [Table]

Original: next time steps 120 conditioned on actions sampled from the imagination policy ˆ ⇡
Revised: next time steps 120 conditioned on actions sampled from the imagination policy ˆ ⇡

Original: [Table]
Revised: [Table]

Original: time step t , the trajectorymanager (TM) determines how to roll out the IC under the planning strategy, and then produces 122 imagined trajectories ˆ
Revised: time step t , the trajectorymanager (TM) determines how to roll out the IC under the planning strategy, and then produces 122 imagined trajectories ˆ

Original: T 1 ,..., ˆ T n of an observable world state s t .
Revised: T 1 ,..., ˆ T n of an observable world state s t .

Original: Each trajectory ˆ T is a sequence 123 of items { ˆ i t , ˆ i t +1 ,... } , that the agent may purchase (or click) from the current time t .
Revised: Each trajectory ˆ T is a sequence 123 of items { ˆ i t , ˆ i t +1 ,... } , that the agent may purchase (or click) from the current time t .

Original: Finally, 124 the Imagination-augmented Executor (I2E) aggregates the internal data resulting from imaginationand external rewarding data to update its action policy ⇡ .
Revised: Finally, 124 the Imagination-augmented Executor (I2E) aggregates the internal data resulting from imaginationand external rewarding data to update its action policy ⇡ .

Original: Speciﬁcally, I2E optimizes the policy 126 ⇡ by maximizing the extrinsic rewards while minimizing the intrinsic imagination errors, L =
Revised: Speciﬁcally, I2E optimizes the policy 126 ⇡ by maximizing the extrinsic rewards while minimizing the intrinsic imagination errors, L =

Original: L A 3 C + L IRN .
Revised: L A 3 C + L IRN .

Original: Inprinciple,IRNencouragesexploration(ontheoutputlayer,i.e.,itemrepresentation) 128 and implicitly learns to plan ("guess what you like") via imagination rollouts, and therefore promotes 129 quick adaptation to user interest.
Revised: Inprinciple,IRNencouragesexploration(ontheoutputlayer,i.e.,itemrepresentation) 128 and implicitly learns to plan ("guess what you like") via imagination rollouts, and therefore promotes 129 quick adaptation to user interest.

Original: Imagination Core.
Revised: Imagination Core.

Original: In order to simulate imagined trajectories, we rely on environment models that, 131 given the present state and a candidate action, make predictions about the future states.
Revised: In order to simulate imagined trajectories, we rely on environment models that, 131 given the present state and a candidate action, make predictions about the future states.

Original: In general, 132 we can employ an environment model that build on recent popular action-conditional next-steppredictors [], and train it in an unsupervised fashion from agent experiences.
Revised: In general, 132 we can employ an environment model that build on recent popular action-conditional next-steppredictors [], and train it in an unsupervised fashion from agent experiences.

Original: However, the predictors 134 usually suffer from model errors, resulting in poor agent performance, and need extra computationalcost (e.g., pre-training)
Revised: However, the predictors 134 usually suffer from model errors, resulting in poor agent performance, and need extra computationalcost (e.g., pre-training)

Original: [error][cost].
Revised: [error][cost].

Original: Besides, the predictors may learn a trivial identical function, 136 since the state transition in agent trajectories is deterministic, i.e., s t +

Revised: Besides, the predictors may learn a trivial identical function, 136 since the state transition in agent trajectories is deterministic, i.e., s t +

Original: =
Revised: =

Original: s t [{ i t } and a t = i t .
Revised: s t [{ i t } and a t = i t .

Original: Inthis work, we derive a static environment model from the state transition: ˆ s t + ⌧
Revised: Inthis work, we derive a static environment model from the state transition: ˆ s t + ⌧

Original: +

Revised: +

Original: =
Revised: =

Original: ˆ s t + ⌧ [{ ˆ i t + ⌧ } , 138 ˆ i t + ⌧
Revised: ˆ s t + ⌧ [{ ˆ i t + ⌧ } , 138 ˆ i t + ⌧

Original: =
Revised: =

Original: ˆ a t + ⌧ and ˆ s t =
Revised: ˆ a t + ⌧ and ˆ s t =

Original: s t , where ⌧ is the length of the imagined rollout, ˆ a t + ⌧ the output action of theimagination policy ˆ ⇡ .
Revised: s t , where ⌧ is the length of the imagined rollout, ˆ a t + ⌧ the output action of theimagination policy ˆ ⇡ .

Original: During training, the generated item ˆ i t + ⌧ may not be the true purchase, but we 140 still use it for self-supervised reconstruction of I2E .
Revised: During training, the generated item ˆ i t + ⌧ may not be the true purchase, but we 140 still use it for self-supervised reconstruction of I2E .

Original: This makes the action policy ⇡ more robust tointrinsic errors and forces the imagination policy ˆ ⇡ to generate more accurate actions.
Revised: This makes the action policy ⇡ more robust tointrinsic errors and forces the imagination policy ˆ ⇡ to generate more accurate actions.

Original: Inpractice,theimaginationpolicy ˆ ⇡ canbeobtainedfrompolicydistillation[]orﬁxedtargetnetwork 143 like DQN [].
Revised: Inpractice,theimaginationpolicy ˆ ⇡ canbeobtainedfrompolicydistillation[]orﬁxedtargetnetwork 143 like DQN [].

Original: The former distills the action policy ⇡ ( s t ; ✓ ) into a smaller rollout network ˆ ⇡ ( s t ; ˆ ✓ ) , 144 using a cross-entropy loss, l ⇡, ˆ ⇡ ( s t ) =
Revised: The former distills the action policy ⇡ ( s t ; ✓ ) into a smaller rollout network ˆ ⇡ ( s t ; ˆ ✓ ) , 144 using a cross-entropy loss, l ⇡, ˆ ⇡ ( s t ) =

Original: P a ⇡ ( a | s t ) log ˆ ⇡ ( a | s t ; ˆ ✓ ) .
Revised: P a ⇡ ( a | s t ) log ˆ ⇡ ( a | s t ; ˆ ✓ ) .

Original: The latter uses a shared but slowly 145 changing target network ˆ ⇡ ( s t ; ✓  ) , where ✓  are previous parameters in ⇡ ( s t ; ✓ ) .
Revised: The latter uses a shared but slowly 145 changing target network ˆ ⇡ ( s t ; ✓  ) , where ✓  are previous parameters in ⇡ ( s t ; ✓ ) .

Original: By imitating the 146 actionpolicy ⇡ , theimaginedtrajectories willbe similarto agentexperiencesin thereal environment;this also helps I2E learn a predictive representation of rewarding states, and in turn should allow the 148 easy learning of the action policy under the sparse reward signals.2 IRNdonotpredicttherewards,sinceitisnotusefulasreportedin[].
Revised: By imitating the 146 actionpolicy ⇡ , theimaginedtrajectories willbe similarto agentexperiencesin thereal environment;this also helps I2E learn a predictive representation of rewarding states, and in turn should allow the 148 easy learning of the action policy under the sparse reward signals.2 IRNdonotpredicttherewards,sinceitisnotusefulasreportedin[].

Original: andinRS,rewardsofdifferentactions arehardtospecify
Revised: andinRS,rewardsofdifferentactions arehardtospecify

Original: Figure2: IRNarchitecture.
Revised: Figure2: IRNarchitecture.

Original: sshowninFigure2,whereICisimaginationcoretogeneratetrajectories, and N-step imagination rollout, using these trajectories to perform encoder-decoder.
Revised: sshowninFigure2,whereICisimaginationcoretogeneratetrajectories, and N-step imagination rollout, using these trajectories to perform encoder-decoder.

Original: 4 IRN Architecture
Revised: 4 IRN Architecture

Original: In this section we incorporate the imagination reconstruction module into the model-free agents 117 (e.g., A3C) in order to enhance data efﬁciency, promote more robust learning and ultimately higher 118 performance under the sparse extrinsic rewards.
Revised: In this section we incorporate the imagination reconstruction module into the model-free agents 117 (e.g., A3C) in order to enhance data efﬁciency, promote more robust learning and ultimately higher 118 performance under the sparse extrinsic rewards.

Original: Our IRN implements an imagination-augmentedpolicy via three key components (Fig 2).
Revised: Our IRN implements an imagination-augmentedpolicy via three key components (Fig 2).

Original: The imagination core (IC) predicts next time steps 120 conditioned on actions sampled from the imagination policy ˆ ⇡ .
Revised: The imagination core (IC) predicts next time steps 120 conditioned on actions sampled from the imagination policy ˆ ⇡ .

Original: At a time step t , the trajectorymanager (TM) determines how to roll out the IC under the planning strategy, and then produces 122 imagined trajectories ˆ
Revised: At a time step t , the trajectorymanager (TM) determines how to roll out the IC under the planning strategy, and then produces 122 imagined trajectories ˆ

Original: T 1 ,..., ˆ T n of an observable world state s t .
Revised: T 1 ,..., ˆ T n of an observable world state s t .

Original: Each trajectory ˆ T is a sequence 123 of items { ˆ i t , ˆ i t +1 ,... } , that the agent may purchase (or click) from the current time t .
Revised: Each trajectory ˆ T is a sequence 123 of items { ˆ i t , ˆ i t +1 ,... } , that the agent may purchase (or click) from the current time t .

Original: Finally, 124 the Imagination-augmented Executor (I2E) aggregates the internal data resulting from imaginationand external rewarding data to update its action policy ⇡ .
Revised: Finally, 124 the Imagination-augmented Executor (I2E) aggregates the internal data resulting from imaginationand external rewarding data to update its action policy ⇡ .

Original: Speciﬁcally, I2E optimizes the policy 126 ⇡ by maximizing the extrinsic rewards while minimizing the intrinsic imagination errors, L =
Revised: Speciﬁcally, I2E optimizes the policy 126 ⇡ by maximizing the extrinsic rewards while minimizing the intrinsic imagination errors, L =

Original: L A 3 C + L IRN .
Revised: L A 3 C + L IRN .

Original: Inprinciple,IRNencouragesexploration(ontheoutputlayer,i.e.,itemrepresentation) 128 and implicitly learns to plan ("guess what you like") via imagination rollouts, and therefore promotes 129 quick adaptation to user interest.
Revised: Inprinciple,IRNencouragesexploration(ontheoutputlayer,i.e.,itemrepresentation) 128 and implicitly learns to plan ("guess what you like") via imagination rollouts, and therefore promotes 129 quick adaptation to user interest.

Original: Imagination Core.
Revised: Imagination Core.

Original: In order to simulate imagined trajectories, we rely on environment models that, 131 given the present state and a candidate action, make predictions about the future states.
Revised: In order to simulate imagined trajectories, we rely on environment models that, 131 given the present state and a candidate action, make predictions about the future states.

Original: In general, 132 we can employ an environment model that build on recent popular action-conditional next-steppredictors [], and train it in an unsupervised fashion from agent experiences.
Revised: In general, 132 we can employ an environment model that build on recent popular action-conditional next-steppredictors [], and train it in an unsupervised fashion from agent experiences.

Original: However, the predictors 134 usually suffer from model errors, resulting in poor agent performance, and need extra computationalcost (e.g., pre-training)
Revised: However, the predictors 134 usually suffer from model errors, resulting in poor agent performance, and need extra computationalcost (e.g., pre-training)

Original: [error][cost].
Revised: [error][cost].

Original: Besides, the predictors may learn a trivial identical function, 136 since the state transition in agent trajectories is deterministic, i.e., s t +

Revised: Besides, the predictors may learn a trivial identical function, 136 since the state transition in agent trajectories is deterministic, i.e., s t +

Original: =
Revised: =

Original: s t [{ i t } and a t = i t .
Revised: s t [{ i t } and a t = i t .

Original: Inthis work, we derive a static environment model from the state transition: ˆ s t + ⌧
Revised: Inthis work, we derive a static environment model from the state transition: ˆ s t + ⌧

Original: +

Revised: +

Original: =
Revised: =

Original: ˆ s t + ⌧ [{ ˆ i t + ⌧ } , 138 ˆ i t + ⌧
Revised: ˆ s t + ⌧ [{ ˆ i t + ⌧ } , 138 ˆ i t + ⌧

Original: =
Revised: =

Original: ˆ a t + ⌧ and ˆ s t =
Revised: ˆ a t + ⌧ and ˆ s t =

Original: s t , where ⌧ is the length of the imagined rollout, ˆ a t + ⌧ the output action of theimagination policy ˆ ⇡ .
Revised: s t , where ⌧ is the length of the imagined rollout, ˆ a t + ⌧ the output action of theimagination policy ˆ ⇡ .

Original: During training, the generated item ˆ i t + ⌧ may not be the true purchase, but we 140 still use it for self-supervised reconstruction of I2E .
Revised: During training, the generated item ˆ i t + ⌧ may not be the true purchase, but we 140 still use it for self-supervised reconstruction of I2E .

Original: This makes the action policy ⇡ more robust tointrinsic errors and forces the imagination policy ˆ ⇡ to generate more accurate actions.
Revised: This makes the action policy ⇡ more robust tointrinsic errors and forces the imagination policy ˆ ⇡ to generate more accurate actions.

Original: Inpractice,theimaginationpolicy ˆ ⇡ canbeobtainedfrompolicydistillation[]orﬁxedtargetnetwork 143 like DQN [].
Revised: Inpractice,theimaginationpolicy ˆ ⇡ canbeobtainedfrompolicydistillation[]orﬁxedtargetnetwork 143 like DQN [].

Original: The former distills the action policy ⇡ ( s t ; ✓ ) into a smaller rollout network ˆ ⇡ ( s t ; ˆ ✓ ) , 144 using a cross-entropy loss, l ⇡, ˆ ⇡ ( s t ) =
Revised: The former distills the action policy ⇡ ( s t ; ✓ ) into a smaller rollout network ˆ ⇡ ( s t ; ˆ ✓ ) , 144 using a cross-entropy loss, l ⇡, ˆ ⇡ ( s t ) =

Original: P a ⇡ ( a | s t ) log ˆ ⇡ ( a | s t ; ˆ ✓ ) .
Revised: P a ⇡ ( a | s t ) log ˆ ⇡ ( a | s t ; ˆ ✓ ) .

Original: The latter uses a shared but slowly 145 changing target network ˆ ⇡ ( s t ; ✓  ) , where ✓  are previous parameters in ⇡ ( s t ; ✓ ) .
Revised: The latter uses a shared but slowly 145 changing target network ˆ ⇡ ( s t ; ✓  ) , where ✓  are previous parameters in ⇡ ( s t ; ✓ ) .

Original: By imitating the 146 actionpolicy ⇡ , theimaginedtrajectories willbe similarto agentexperiencesin thereal environment;this also helps I2E learn a predictive representation of rewarding states, and in turn should allow the 148 easy learning of the action policy under the sparse reward signals.2 IRNdonotpredicttherewards,sinceitisnotusefulasreportedin[].
Revised: By imitating the 146 actionpolicy ⇡ , theimaginedtrajectories willbe similarto agentexperiencesin thereal environment;this also helps I2E learn a predictive representation of rewarding states, and in turn should allow the 148 easy learning of the action policy under the sparse reward signals.2 IRNdonotpredicttherewards,sinceitisnotusefulasreportedin[].

Original: andinRS,rewardsofdifferentactions arehardtospecify
Revised: andinRS,rewardsofdifferentactions arehardtospecify

Original: RA { item 8, item 5, item 1, item 2 }
Revised: RA { item 8, item 5, item 1, item 2 }

Original: S
Revised: S

Original: [Figure]
Revised: [Figure]

Original: click purchase
Revised: click purchase

Original: [Figure]
Revised: [Figure]

Original: v 1 v
Revised: v 1 v

Original: [Figure]
Revised: [Figure]

Original: v n
Revised: v n

Original: [Figure]
Revised: [Figure]

Original: However, GRU4Rec utilizes the session-parallel mini-batch training to handle the variable lengths of sessions; this trick cannot effectively capture sequentiality of sessions, since the network is trained using the BP algorithm (not BPTT for RNNs).
Revised: However, GRU4Rec utilizes the session-parallel mini-batch training to handle the variable lengths of sessions; this trick cannot effectively capture sequentiality of sessions, since the network is trained using the BP algorithm (not BPTT for RNNs).

Original: These models primarily focus on next-click prediction and model the click-streams via sequential classiﬁcation, while here we aim at modeling the purchase behavior and enhancing users’ purchase intentions.
Revised: These models primarily focus on next-click prediction and model the click-streams via sequential classiﬁcation, while here we aim at modeling the purchase behavior and enhancing users’ purchase intentions.

Original: Besides, IRN is built on RL, which encodes sequentiality of states into the value function.
Revised: Besides, IRN is built on RL, which encodes sequentiality of states into the value function.

Original: Imagination-augmented Agents All approaches incorporating off-policy experience (e.g., imagined trajectories) generated by a learned model can be categorized into model-based reinforcement learning (Racanière et al., 2017; Pascanu et al., 2017; Silver et al., 2017; Sutton, 1991).
Revised: Imagination-augmented Agents All approaches incorporating off-policy experience (e.g., imagined trajectories) generated by a learned model can be categorized into model-based reinforcement learning (Racanière et al., 2017; Pascanu et al., 2017; Silver et al., 2017; Sutton, 1991).

Original: By using an internal model of the world, the agent can generalize to unseen states, remain valid in the real environment, and exploit additional training signals to improve data efﬁciency.
Revised: By using an internal model of the world, the agent can generalize to unseen states, remain valid in the real environment, and exploit additional training signals to improve data efﬁciency.

Original: However, the performance of model-based agents usually suffers from model errors resulting from function approximation.
Revised: However, the performance of model-based agents usually suffers from model errors resulting from function approximation.

Original: I2As (Racanière et al., 2017) were proposed to address this issue.
Revised: I2As (Racanière et al., 2017) were proposed to address this issue.

Original: I2As augment model-free agents with imagination and use an interpretation module to handle imperfect predictions.
Revised: I2As augment model-free agents with imagination and use an interpretation module to handle imperfect predictions.

Original: The imagined trajectories of I2As are provided as additional context (i.e., input features) to a policy network, while the proposed IRN uses the trajectories as additional training signals for self-supervised reconstruction.
Revised: The imagined trajectories of I2As are provided as additional context (i.e., input features) to a policy network, while the proposed IRN uses the trajectories as additional training signals for self-supervised reconstruction.

Original: Self-supervised Reinforcement Learning
Revised: Self-supervised Reinforcement Learning

Original: In many real-world scenarios, reward is extremely sparse and delayed, and the agent updates its policy only if it reaches a pre-deﬁned goal state.
Revised: In many real-world scenarios, reward is extremely sparse and delayed, and the agent updates its policy only if it reaches a pre-deﬁned goal state.

Original: To model this phenomenon, self-supervised reinforcement learning have often been used, which accelerates the acquisition of a useful representation with auxiliary task learning (Jaderberg et al., 2016;
Revised: To model this phenomenon, self-supervised reinforcement learning have often been used, which accelerates the acquisition of a useful representation with auxiliary task learning (Jaderberg et al., 2016;

Original: Pathak et al., 2017; Mirowski et al., 2016; Shelhamer et al., 2016).
Revised: Pathak et al., 2017; Mirowski et al., 2016; Shelhamer et al., 2016).

Original: Speciﬁcally, auxiliary tasks provide additional losses for feature learning, and can be trained instantaneously using the self-supervision from the environment.
Revised: Speciﬁcally, auxiliary tasks provide additional losses for feature learning, and can be trained instantaneously using the self-supervision from the environment.

Original: For instance, UNREAL (Jaderberg et al., 2016) maximizes many other pseudo-reward functions simultaneously, e.g., pixel change control, with a common representation shared by all tasks.
Revised: For instance, UNREAL (Jaderberg et al., 2016) maximizes many other pseudo-reward functions simultaneously, e.g., pixel change control, with a common representation shared by all tasks.

Original: In contrast, the proposed IRN do not require the external supervision from the environment, i.e., self-supervised reconstruction is performed on internal imagined trajectories.
Revised: In contrast, the proposed IRN do not require the external supervision from the environment, i.e., self-supervised reconstruction is performed on internal imagined trajectories.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: We interpret the sequential recommendation task based on the standard reinforcement learning setting:
Revised: We interpret the sequential recommendation task based on the standard reinforcement learning setting:

Original: An recommender agent (RA) interacts with an environment E (or user sessions) by sequentially choosing a list of recommendation items over a number of discrete time steps, so as to maximize its cumulative reward.
Revised: An recommender agent (RA) interacts with an environment E (or user sessions) by sequentially choosing a list of recommendation items over a number of discrete time steps, so as to maximize its cumulative reward.

Original: As shown in Figure 1, we model this problem as a Markov Decision Process (MDP), which consists of a tuple of ﬁve elements ( S, A, P, R, γ ):
Revised: As shown in Figure 1, we model this problem as a Markov Decision Process (MDP), which consists of a tuple of ﬁve elements ( S, A, P, R, γ ):

Original: State space S :
Revised: State space S :

Original: A state s t ∈ S is deﬁned as the previous items that a user clicked/purchased in one session.
Revised: A state s t ∈ S is deﬁned as the previous items that a user clicked/purchased in one session.

Original: Speciﬁcally, the initial state s 1 contains the ﬁrst item i 0 of one session.
Revised: Speciﬁcally, the initial state s 1 contains the ﬁrst item i 0 of one session.

Original: The items in s t = { i 0 , i 1 , ..., i t − 1 } are sorted in chronological order.
Revised: The items in s t = { i 0 , i 1 , ..., i t − 1 } are sorted in chronological order.

Original: Action space A : An action a t ∈ A is to recommend items to a user at time t according to its policy π , where π is a mapping from s t to a t .
Revised: Action space A : An action a t ∈ A is to recommend items to a user at time t according to its policy π , where π is a mapping from s t to a t .

Original: We assume that the RA only recommends one item to the user each time, since we use the observed click/purchase sequences for off-policy training.
Revised: We assume that the RA only recommends one item to the user each time, since we use the observed click/purchase sequences for off-policy training.

Original: During off-policy evaluation, we can recommend a list of K candidates to the user.
Revised: During off-policy evaluation, we can recommend a list of K candidates to the user.

Original: Reward R : After the RA takes an action a t at the state s t , i.e., recommending an item to a user, the user browses this item and provides her feedback (click or purchase).
Revised: Reward R : After the RA takes an action a t at the state s t , i.e., recommending an item to a user, the user browses this item and provides her feedback (click or purchase).

Original: The agent receives a scalar reward r ( s t , a t ) according to the user’s feedback.
Revised: The agent receives a scalar reward r ( s t , a t ) according to the user’s feedback.

Original: We also deﬁne the k -step return starting from state s t as G t,t + k
Revised: We also deﬁne the k -step return starting from state s t as G t,t + k

Original: ( s t ) =
Revised: ( s t ) =

Original: (cid:80) t
Revised: (cid:80) t

Original: +
Revised: +

Original: k j = t γ j − t r ( s j , a j ) , where γ ∈ [0 , 1] is a discounting factor.
Revised: k j = t γ j − t r ( s j , a j ) , where γ ∈ [0 , 1] is a discounting factor.

Original: Transition probability P : Transition probability p ( s t +1 | s t , a t ) deﬁnes the probability of state transition from s t to s t +1 when the RA takes action a t .
Revised: Transition probability P : Transition probability p ( s t +1 | s t , a t ) deﬁnes the probability of state transition from s t to s t +1 when the RA takes action a t .

Original: In this case, the state transition is deterministic after taking the ground-true action a t = i t , i.e., p ( s t +1 | s t , i t )
Revised: In this case, the state transition is deterministic after taking the ground-true action a t = i t , i.e., p ( s t +1 | s t , i t )

Original: = 1 and s t +1 =
Revised: = 1 and s t +1 =

Original: s t ∪ { i t } .
Revised: s t ∪ { i t } .

Original: The goal of the RA is to ﬁnd an optimal policy π ∗ , such that V π ∗ ( s 1 ) ≥ V π ( s 1 ) for all policies π and start state s 1 , where V π ( s t ) is the expected return for a state s t when following a policy π , i.e.,
Revised: The goal of the RA is to ﬁnd an optimal policy π ∗ , such that V π ∗ ( s 1 ) ≥ V π ( s 1 ) for all policies π and start state s 1 , where V π ( s t ) is the expected return for a state s t when following a policy π , i.e.,

Original: V π ( s t ) =
Revised: V π ( s t ) =

Original: E s j>t ∼ S,a j>t ∼ π [ G t, ∞ ( s t )] .
Revised: E s j>t ∼ S,a j>t ∼ π [ G t, ∞ ( s t )] (or E s ∼ π [ G t, ∞ ( s t )] for simplicity).

Original: Asynchronous Advantage Actor-Critic.
Revised: Asynchronous Advantage Actor-Critic.

Original: This paper builds upon the A3C algorithm, an actor-critic approach that constructs a policy network π ( a | s ; θ ) and a value function network V ( s ; θ v ) , with all non-output layers shared (Mnih et al., 2016).
Revised: This paper builds upon the A3C algorithm, an actorcritic approach that constructs a policy network π ( a | s ; θ ) and a value function network V ( s ; θ v ) , with all non-output layers shared (Mnih et al., 2016).

Original: The policy and the value function are adjusted towards the bootstrapped k -step return G t,t + k
Revised: The policy and the value function are adjusted towards the bootstrapped k -step return G t,t + k

Original: ( s t )
Revised: ( s t )

Original: + γ k +1 V ( s t
Revised: + γ k +1 V ( s t

Revised: + k +1 ; θ v ) , L A 3 C

Revised: =

Original:  + k +1 ; θ v ) , using an entropy regularization penalty to encourage exploration, H (  π  ( s t , · ; θ ) .
Revised: L π + L V R , where L π = E s ∼ π [ G 1 , ∞ ( s 1 )] and L V R = E s ∼ π,a ∼ π [ A ( s t , a t )] .

Original: To increase the probability of rewarding actions, A3C applies an update g ( θ ) to the parameters θ using an unbiased estimation (Sutton et al., 2000):
Revised: The advantage function A ( s t , a t ) (Baird III, 1993) is computed as the difference of the bootstrapped k -step return and the current state value estimate:

Original: [Equation]
Revised: [Equation]

Original: where  the advantage function A ( s t , a t ) (Baird III, 1993) is computed as the difference of the bootstrapped k -step return and the current state value estimate:
Revised: where θ − v are the parameters of the previous target network. To increase the probability of rewarding actions, A3C applies an update g ( θ ) to the parameters θ using an unbiased estimation (Sutton et al., 2000):

Original: [Equation]
Revised: [Equation]

Original: The value function V ( s ; θ v ) is updated following the recursive deﬁnition of the Bellman Equation,
Revised: The value function V ( s ; θ v ) is updated following the recursive deﬁnition of the Bellman Equation,

Original: V ( s t ; θv ) = E s
Revised: V ( s t ; θ

Original: [Equation]
Revised: [Equation]

Original: >t [Equation] S (cid:2) G t,t + k
Revised: ) =

Original: [Equation]
Revised: [Equation]

Original: st ) + γ k +1 V ( s t + k +1 ; θ
Revised: s

Original: [Equation]
Revised: [Equation]

Original: ) (cid:3) .

Original: Then g ( θ v ) is obtained by minimizing a squared error between the target return and the current value estimate:
Revised: Then g ( θ v ) is obtained by minimizing a squared error between the target return and the current value estimate:

Original: [Equation]
Revised: [Equation]

Original: In A3C multiple agents interact in parallel, with multiple instances of the environment.
Revised: In A3C multiple agents interact in parallel, with multiple instances of the environment.

Original: The asynchronous execution accelerates and stabilizes learning.
Revised: The asynchronous execution accelerates and stabilizes learning.

Original: In practice, we combine A3C with the session-parallel mini-batches proposed in (Hidasi et al., 2016).
Revised: In practice, we combine A3C with the session-parallel mini-batches proposed in (Hidasi et al., 2016).

Original: Each instance of the agent interacts with multiple sessions simultaneously, gathering M samples from different sessions at a time step.
Revised: Each instance of the agent interacts with multiple sessions simultaneously, gathering M samples from different sessions at a time step.

Original: After k steps, the agent updates its policy and value network according to Eq.
Revised: After k steps, the agent updates its policy and value network according to Eq.

Original: (1)(3), using k ∗ M samples.
Revised: (2)(3), using k ∗ M samples.

Original: This decorrelates updates between samples of one session in the instance level.
Revised: This decorrelates updates between samples of one session in the instance level.

Original: Besides, to build the A3C agent, we employ an LSTM that jointly approximates both policy π and value function
Revised: Besides, to build the A3C agent, we employ an LSTM that jointly approximates both policy π and value function

Original: V , given the one-hot vectors of previous items clicked/purchased as inputs.
Revised: V , given the one-hot vectors of previous items clicked/purchased as inputs.

Original:  IRN A RCHITECTURE
Revised:  IRN A RCHITECTURE

Original: In this section we incorporate the imagination reconstruction module into the model-free agents (e.g., A3C) in order to enhance data efﬁciency, promote more robust learning and ultimately higher performance under the sparse extrinsic reward.
Revised: In this section we incorporate the imagination reconstruction module into the model-free agents (e.g., A3C) in order to enhance data efﬁciency, promote more robust learning and ultimately higher performance under the sparse extrinsic reward.

Original: Our IRN implements an imagination-augmented policy via three key components (Figure 2).
Revised: Our IRN implements an imagination-augmented policy via three key components (Figure 2).

Original: The imagination core (IC) predicts the next time
Revised: The imagination core (IC) predicts the next time

Original: steps conditioned on actions sampled from the imagination policy ˆ π .
Revised: steps conditioned on actions sampled from the imagination policy ˆ π .

Original: At a time step t , the trajectory manager (TM) determines how to roll out the IC under the planning strategy, and then produces imagined trajectories ˆ T 1 , . .
Revised: At a time step t , the trajectory manager (TM) determines how to roll out the IC under the planning strategy, and then produces imagined trajectories ˆ T 1 , . .

Original: , ˆ T n of an observable world state s t .
Revised: , ˆ T n of an observable world state s t .

Original: Each trajectory ˆ T j is a sequence of items { ˆ i j,t , ˆ i j,t +1 , . . .
Revised: Each trajectory ˆ T j is a sequence of items { ˆ i j,t , ˆ i j,t +1 , . . .

Original: } , that users may purchase (or click) from the current time t .
Revised: } , that users may purchase (or click) from the current time t .

Original: The Imaginationaugmented Executor (IAE) aggregates the internal data resulting from imagination and external rewarding data to update its action policy π .
Revised: The Imaginationaugmented Executor (IAE) aggregates the internal data resulting from imagination and external rewarding data to update its action policy π .

Original: Speciﬁcally, the IAE optimizes the policy π by maximizing the extrinsic reward while minimizing the intrinsic imagination error.
Revised: Speciﬁcally, the IAE optimizes the policy π by maximizing the extrinsic reward while minimizing the intrinsic imagination error.

Original: In principle, IRN encourages exploration and learns predictive representations via imagination rollouts, which promotes quick adaptation to user interest and robustness to the cold-start scenario.
Revised: In principle, IRN encourages exploration and learns predictive representations via imagination rollouts, which promotes quick adaptation to user interest and robustness to the cold-start scenario.

Original: 1 - Input 1 - Input 2 - Densify 3 -
Revised: 1 - Input 1 - Input 2 - Densify 3 -

Original: Denoising : : : ? ?
Revised: Denoising : : : ? ?

Original: 4   4 5 (cid:18)(cid:83) 3/p 

Revised: 4   4 5 (cid:18)(cid:83) 3/p

Original: Shallow Representation Deep Representation
Revised: Shallow Representation Deep Representation

Original: Target =
Revised: Target =

Original: SR 5 0 3 0 

Revised: SR 5 0 3 0

Original: e rr o r (Densify) 5 0 3 0 4 (Densify) e rr o r supervised unsupervised supervised e rr o r B
Revised: e rr o r (Densify) 5 0 3 0 4 (Densify) e rr o r supervised unsupervised supervised e rr o r B

Original: ac
Revised: ac

Original: k
Revised: k

Original: w a r d 4 5 4 45 31 5? 4 ? ? ? ? ? ? ?
Revised: w a r d 4 5 4 45 31 5? 4 ? ? ? ? ? ? ?

Original: u 1 u 2 u 3 u 4 v 1 v 2 vvv  1? ? ?
Revised: u 1 u 2 u 3 u 4 v 1 v 2 vvv  1? ? ?

Original: u 1 u 2 u 3 uvvvvv 5 items u s e r s 0.17 0.25 0.33 0.17 0.08 

Revised: u 1 u 2 u 3 uvvvvv 5 items u s e r s 0.17 0.25 0.33 0.17 0.08

Original: Stage 2 Stage 3 Stage 1 RLAgent RLAgent < item 0 > itemvv… v n < item 0, item 2 > < item 0, item 2, item 1 > S 0 S 

Revised: Stage 2 Stage 3 Stage 1 RLAgent RLAgent < item 0 > itemvv… v n < item 0, item 2 > < item 0, item 2, item 1 > S 0 S

Original: Fo r w a r d
Revised: Fo r w a r d

Original: Sitem 1 v… vv n
Revised: Sitem 1 v… vv n

Original: RecommenderAgent
Revised: RecommenderAgent

Original: User Session a t Action s r
Revised: User Session a t Action s r

Original: Reward r t+ s t+
Revised: Reward r t+ s t+

Original: RLAgent vo t+ o t+ o t+k
Revised: RLAgent vo t+ o t+ o t+k

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: n-rollout
Revised: n-rollout

Original: [Figure]
Revised: [Figure]

Original: Policy
Revised: Policy

Original: [Figure]
Revised: [Figure]

Original: rollout-out reconstruction
Revised: rollout-out reconstruction

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: sshowninFigure2,whereICisimaginationcoretogeneratetrajectories, and N-step imagination rollout, using these trajectories to perform encoder-decoder.
Revised: sshowninFigure2,whereICisimaginationcoretogeneratetrajectories, and N-step imagination rollout, using these trajectories to perform encoder-decoder.

Original: In this section we incorporate the imagination reconstruction module into the model-free agents 117 (e.g., A3C) in order to enhance data efﬁciency, promote more robust learning and ultimately higherperformance under the sparse extrinsic rewards.
Revised: In this section we incorporate the imagination reconstruction module into the model-free agents 117 (e.g., A3C) in order to enhance data efﬁciency, promote more robust learning and ultimately higherperformance under the sparse extrinsic rewards.

Original: Our IRN implements an imagination-augmented
Revised: Our IRN implements an imagination-augmented

Original: [Table]
Revised: [Table]

Original: policy via three key components (Fig 2).
Revised: policy via three key components (Fig 2).

Original: The imagination core
Revised: The imagination core

Original: [Table]
Revised: [Table]

Original: IC)
Revised: IC)

Original: [Table]
Revised: [Table]

Original: conditioned on actions sampled from the
Revised: conditioned on actions sampled from the

Original: [Table] [Table]
Revised: [Table] [Table]

Original: manager (TM) determines how to roll out the IC under the planning strategy, and then produces
Revised: manager (TM) determines how to roll out the IC under the planning strategy, and then produces

Original: [Table]
Revised: [Table]

Original: of an observable world state s t .
Revised: of an observable world state s t .

Original: Each trajectory ˆ T
Revised: Each trajectory ˆ T

Original: [Table]
Revised: [Table]

Original: a sequence
Revised: a sequence

Original: [Table]
Revised: [Table]

Original: of
Revised: of

Original: [Table] [Table]
Revised: [Table] [Table]

Original: implicitly
Revised: implicitly

Original: [Table] [Table]
Revised: [Table] [Table]

Original: imagination
Revised: imagination

Original: [Table]
Revised: [Table]

Original: , and therefore promotes
Revised: , and therefore promotes

Original: [Table]
Revised: [Table]

Original: to
Revised: to

Original: [Table]
Revised: [Table]

Original: , we rely
Revised: , we rely

Original: [Table] [Table]
Revised: [Table] [Table]

Original: we
Revised: we

Original: [Table]
Revised: [Table]

Original: ], and
Revised: ], and

Original: [Table]
Revised: [Table]

Original: experiences.
Revised: experiences.

Original: However, the
Revised: However, the

Original: [Table]
Revised: [Table]

Original: usually suffer from
Revised: usually suffer from

Original: [Table]
Revised: [Table]

Original: errors, resulting in poor agent performance, and need extra
Revised: errors, resulting in poor agent performance, and need extra

Original: [Table]
Revised: [Table]

Original: Figure 2: IRN architecture.
Revised: Figure 2: IRN architecture.

Original: s shown in Figure 2, where IC is imagination core to generate trajectories, and N-step imagination rollout, using these trajectories to perform encoder-decoder.
Revised: s shown in Figure 2, where IC is imagination core to generate trajectories, and N-step imagination rollout, using these trajectories to perform encoder-decoder.

Original: 4 IRN Architecture 

Revised: 4 IRN Architecture

Original: In this section we incorporate the imagination reconstruction module into the model-free agents 117 (e.g., A3C) in order to enhance data efﬁciency, promote more robust learning and ultimately higher 118 performance under the sparse extrinsic rewards.
Revised: In this section we incorporate the imagination reconstruction module into the model-free agents 117 (e.g., A3C) in order to enhance data efﬁciency, promote more robust learning and ultimately higher 118 performance under the sparse extrinsic rewards.

Original: Our IRN implements an imagination-augmented 119 policy via three key components (Fig 2).
Revised: Our IRN implements an imagination-augmented 119 policy via three key components (Fig 2).

Original: The imagination core (IC) predicts next time steps 120 conditioned on actions sampled from the imagination policy ˆ ⇡ .
Revised: The imagination core (IC) predicts next time steps 120 conditioned on actions sampled from the imagination policy ˆ ⇡ .

Original: At a time step t , the trajectory 121 manager (TM) determines how to roll out the IC under the planning strategy, and then produces 122 imagined trajectories ˆ T 1 ,..., ˆ T n of an observable world state s t .
Revised: At a time step t , the trajectory 121 manager (TM) determines how to roll out the IC under the planning strategy, and then produces 122 imagined trajectories ˆ T 1 ,..., ˆ T n of an observable world state s t .

Original: Each trajectory ˆ T is a sequence 123 of items { ˆ i t , ˆ i t +1 ,... } , that the agent may purchase (or click) from the current time t .
Revised: Each trajectory ˆ T is a sequence 123 of items { ˆ i t , ˆ i t +1 ,... } , that the agent may purchase (or click) from the current time t .

Original: Finally, 124 the Imagination-augmented Executor (I2E) aggregates the internal data resulting from imagination 125 and external rewarding data to update its action policy ⇡ .
Revised: Finally, 124 the Imagination-augmented Executor (I2E) aggregates the internal data resulting from imagination 125 and external rewarding data to update its action policy ⇡ .

Original: Speciﬁcally, I2E optimizes the policy 126 ⇡ by maximizing the extrinsic rewards while minimizing the intrinsic imagination errors, L = 127 L A 3 C + L IRN .
Revised: Speciﬁcally, I2E optimizes the policy 126 ⇡ by maximizing the extrinsic rewards while minimizing the intrinsic imagination errors, L = 127 L A 3 C + L IRN .

Original: In principle, IRN encourages exploration (on the output layer, i.e., item representation) 128 and implicitly learns to plan ("guess what you like") via imagination rollouts, and therefore promotes 129 quick adaptation to user interest.
Revised: In principle, IRN encourages exploration (on the output layer, i.e., item representation) 128 and implicitly learns to plan ("guess what you like") via imagination rollouts, and therefore promotes 129 quick adaptation to user interest.

Original: Imagination Core.
Revised: Imagination Core.

Original: In order to simulate imagined trajectories, we rely on environment models that, 131 given the present state and a candidate action, make predictions about the future states.
Revised: In order to simulate imagined trajectories, we rely on environment models that, 131 given the present state and a candidate action, make predictions about the future states.

Original: In general,we can employ an environment model that build on recent popular action-conditional next-step 133 predictors [], and train it in an unsupervised fashion from agent experiences.
Revised: In general,we can employ an environment model that build on recent popular action-conditional next-step 133 predictors [], and train it in an unsupervised fashion from agent experiences.

Original: However, the predictorsusually suffer from model errors, resulting in poor agent performance, and need extra computational 135 cost (e.g., pre-training)
Revised: However, the predictorsusually suffer from model errors, resulting in poor agent performance, and need extra computational 135 cost (e.g., pre-training)

Original: [error][cost].
Revised: [error][cost].

Original: Besides, the predictors may learn a trivial identical function,since the state transition in agent trajectories is deterministic, i.e., s t +1 =
Revised: Besides, the predictors may learn a trivial identical function,since the state transition in agent trajectories is deterministic, i.e., s t +1 =

Original: 137 this work,
Revised: 137 this work,

Original: we
Revised: we

Original: 138 ˆ
Revised: 138 ˆ

Original: i
Revised: i

Original: + ⌧ =
Revised: + ⌧ =

Original: may
Revised: may

Original: 140 still use errors
Revised: 140 still use errors

Original: In practice, or
Revised: In practice, or

Original: 143 like DQN former
Revised: 143 like DQN former

Original: using
Revised: using

Original: P
Revised: P

Original: 147 this also helps allow
Revised: 147 this also helps allow

Original: easy
Revised: easy

Original: 2 IRN are specify
Revised: 2 IRN are specify

Original: Figure 2: IRN architecture.
Revised: Figure 2: IRN architecture.

Original: s shown in Figure 2, where IC is imagination core to generate trajectories, and N-step imagination rollout, using these trajectories to perform encoder-decoder.
Revised: s shown in Figure 2, where IC is imagination core to generate trajectories, and N-step imagination rollout, using these trajectories to perform encoder-decoder.

Original: 4 IRN Architecture 

Revised: 4 IRN Architecture

Original: In this section we incorporate the imagination reconstruction module into the model-free agents 117 (e.g., A3C) in order to enhance data efﬁciency, promote more robust learning and ultimately higher 118 performance under the sparse extrinsic rewards.
Revised: In this section we incorporate the imagination reconstruction module into the model-free agents 117 (e.g., A3C) in order to enhance data efﬁciency, promote more robust learning and ultimately higher 118 performance under the sparse extrinsic rewards.

Original: Our IRN implements an imagination-augmented 119 policy via three key components (Fig 2).
Revised: Our IRN implements an imagination-augmented 119 policy via three key components (Fig 2).

Original: The imagination core (IC) predicts next time steps 120 conditioned on actions sampled from the imagination policy ˆ ⇡ .
Revised: The imagination core (IC) predicts next time steps 120 conditioned on actions sampled from the imagination policy ˆ ⇡ .

Original: At a time step t , the trajectory 121 manager (TM) determines how to roll out the IC under the planning strategy, and then produces 122 imagined trajectories ˆ T 1 ,..., ˆ T n of an observable world state s t .
Revised: At a time step t , the trajectory 121 manager (TM) determines how to roll out the IC under the planning strategy, and then produces 122 imagined trajectories ˆ T 1 ,..., ˆ T n of an observable world state s t .

Original: Each trajectory ˆ T is a sequence 123 of items { ˆ i t , ˆ i t +1 ,... } , that the agent may purchase (or click) from the current time t .
Revised: Each trajectory ˆ T is a sequence 123 of items { ˆ i t , ˆ i t +1 ,... } , that the agent may purchase (or click) from the current time t .

Original: Finally, 124 the Imagination-augmented Executor (I2E) aggregates the internal data resulting from imagination 125 and external rewarding data to update its action policy ⇡ .
Revised: Finally, 124 the Imagination-augmented Executor (I2E) aggregates the internal data resulting from imagination 125 and external rewarding data to update its action policy ⇡ .

Original: Speciﬁcally, I2E optimizes the policy 126 ⇡ by maximizing the extrinsic rewards while minimizing the intrinsic imagination errors,
Revised: Speciﬁcally, I2E optimizes the policy 126 ⇡ by maximizing the extrinsic rewards while minimizing the intrinsic imagination errors,

Original: L =
Revised: L =

Original: L A 3 C + L IRN .
Revised: L A 3 C + L IRN .

Original: In principle, encourages exploration (on the output layer, i.e., item representation) 128 and implicitly learns to plan ("guess what you like") via imagination rollouts, and therefore promotes to user interest.
Revised: In principle, encourages exploration (on the output layer, i.e., item representation) 128 and implicitly learns to plan ("guess what you like") via imagination rollouts, and therefore promotes to user interest.

Original: Imagination Core.
Revised: Imagination Core.

Original: In order to simulate imagined trajectories, we rely on environment models that, 131 given the present state and a candidate action, make predictions about the future states.
Revised: In order to simulate imagined trajectories, we rely on environment models that, 131 given the present state and a candidate action, make predictions about the future states.

Original: In general,we can employ an environment model that build on recent popular action-conditional next-step 133 predictors [], and train it in an unsupervised fashion from agent experiences.
Revised: In general,we can employ an environment model that build on recent popular action-conditional next-step 133 predictors [], and train it in an unsupervised fashion from agent experiences.

Original: However, the predictorsusually suffer from model errors, resulting in poor agent performance, and need extra computational 135 cost (e.g., pre-training)
Revised: However, the predictorsusually suffer from model errors, resulting in poor agent performance, and need extra computational 135 cost (e.g., pre-training)

Original: [error][cost].
Revised: [error][cost].

Original: Besides, the predictors may learn a trivial identical function,since the state transition in agent trajectories is deterministic, i.e., s t +

Revised: Besides, the predictors may learn a trivial identical function,since the state transition in agent trajectories is deterministic, i.e., s t +

Original: =
Revised: =

Original: s t [ { i t } and a t = i t .
Revised: s t [ { i t } and a t = i t .

Original: In 137 this work, we derive a static environment model from the state transition: ˆ s t + ⌧
Revised: In 137 this work, we derive a static environment model from the state transition: ˆ s t + ⌧

Original: +

Revised: +

Original: =
Revised: =

Original: ˆ s t + ⌧ [ { ˆ i t + ⌧ } ,ˆ i t + ⌧ = ˆ a t + ⌧ and ˆ s t =
Revised: ˆ s t + ⌧ [ { ˆ i t + ⌧ } ,ˆ i t + ⌧ = ˆ a t + ⌧ and ˆ s t =

Original: s t , where ⌧ is the length of the imagined rollout, ˆ a t + ⌧ the output action of the 139 imagination policy ˆ ⇡ .
Revised: s t , where ⌧ is the length of the imagined rollout, ˆ a t + ⌧ the output action of the 139 imagination policy ˆ ⇡ .

Original: During training, the generated item ˆ i t + ⌧ may not be the true purchase, but westill use it for self-supervised reconstruction of I2E .
Revised: During training, the generated item ˆ i t + ⌧ may not be the true purchase, but westill use it for self-supervised reconstruction of I2E .

Original: This makes the action policy ⇡ more robust to 141 intrinsic errors and forces the imagination policy ˆ ⇡ to generate more accurate actions.
Revised: This makes the action policy ⇡ more robust to 141 intrinsic errors and forces the imagination policy ˆ ⇡ to generate more accurate actions.

Original: In practice, the imagination policy ˆ ⇡ can be obtained from policy distillation [] or ﬁxed target network 143 like DQN [].
Revised: In practice, the imagination policy ˆ ⇡ can be obtained from policy distillation [] or ﬁxed target network 143 like DQN [].

Original: The former distills the action policy ⇡ ( s t ; ✓ ) into a smaller rollout network ˆ ⇡ ( s t ; ˆ ✓ ) ,using a cross-entropy loss, l ⇡, ˆ ⇡ ( s t ) =
Revised: The former distills the action policy ⇡ ( s t ; ✓ ) into a smaller rollout network ˆ ⇡ ( s t ; ˆ ✓ ) ,using a cross-entropy loss, l ⇡, ˆ ⇡ ( s t ) =

Original: P a ⇡ ( a | s t ) log ˆ ⇡ ( a | s t ; ˆ ✓ ) .
Revised: P a ⇡ ( a | s t ) log ˆ ⇡ ( a | s t ; ˆ ✓ ) .

Original: The latter uses a shared but slowly 145 changing target network ˆ ⇡ ( s t ; ✓  ) , where ✓  are previous parameters in ⇡ ( s t ; ✓ ) .
Revised: The latter uses a shared but slowly 145 changing target network ˆ ⇡ ( s t ; ✓  ) , where ✓  are previous parameters in ⇡ ( s t ; ✓ ) .

Original: By imitating theaction policy ⇡ , the imagined trajectories will be similar to agent experiences in the real environment; 147 this also helps I2E learn a predictive representation of rewarding states, and in turn should allow theeasy learning of the action policy under the sparse reward signals.2 IRN do not predict the rewards, since it is not useful as reported in [].
Revised: By imitating theaction policy ⇡ , the imagined trajectories will be similar to agent experiences in the real environment; 147 this also helps I2E learn a predictive representation of rewarding states, and in turn should allow theeasy learning of the action policy under the sparse reward signals.2 IRN do not predict the rewards, since it is not useful as reported in [].

Original: and in RS, rewards of different actions are hard to specify
Revised: and in RS, rewards of different actions are hard to specify

Original: ˆ ⇡ a step t the
Revised: ˆ ⇡ a step t the

Original: , and then producesimagined trajectories ˆ T 1 , . .
Revised: , and then producesimagined trajectories ˆ T 1 , . .

Original: , ˆ T n of an observable world state s t .
Revised: , ˆ T n of an observable world state s t .

Original: Each trajectory ˆ T is a sequenceof items { ˆ i t , ˆ i t +1 , . . .
Revised: Each trajectory ˆ T is a sequenceof items { ˆ i t , ˆ i t +1 , . . .

Original: } , that the agent may purchase (or click) from the current time t .
Revised: } , that the agent may purchase (or click) from the current time t .

Original: Finally,the Imagination-augmented Executor (I2E) aggregates the internal data resulting from imaginationand external rewarding data to update its action policy ⇡ .
Revised: Finally,the Imagination-augmented Executor (I2E) aggregates the internal data resulting from imaginationand external rewarding data to update its action policy ⇡ .

Original: Speciﬁcally, I2E optimizes the policy⇡ by maximizing the extrinsic rewards while minimizing the intrinsic imagination errors, L =
Revised: Speciﬁcally, I2E optimizes the policy⇡ by maximizing the extrinsic rewards while minimizing the intrinsic imagination errors, L =

Original: L A 3 C + L IRN .
Revised: L A 3 C + L IRN .

Original: In principle, IRN encourages exploration (on the output layer, i.e., item representation)and implicitly learns to plan ("guess what you like") via imagination rollouts, and therefore promotesquick adaptation to user interest.
Revised: In principle, IRN encourages exploration (on the output layer, i.e., item representation)and implicitly learns to plan ("guess what you like") via imagination rollouts, and therefore promotesquick adaptation to user interest.

Original: Imagination Core.
Revised: Imagination Core.

Original: In order to simulate imagined trajectories, we rely on environment models that,given the present state and a candidate action, make predictions about the future states.
Revised: In order to simulate imagined trajectories, we rely on environment models that,given the present state and a candidate action, make predictions about the future states.

Original: In general,we can employ an environment model that build on recent popular action-conditional next-steppredictors [], and train it in an unsupervised fashion from agent experiences.
Revised: In general,we can employ an environment model that build on recent popular action-conditional next-steppredictors [], and train it in an unsupervised fashion from agent experiences.

Original: However, the predictorsusually suffer from model errors, resulting in poor agent performance, and need extra computationalcost (e.g., pre-training)
Revised: However, the predictorsusually suffer from model errors, resulting in poor agent performance, and need extra computationalcost (e.g., pre-training)

Original: [error][cost].
Revised: [error][cost].

Original: Besides, the predictors may learn a trivial identical function,since the state transition in agent trajectories is deterministic, i.e., s t +

Revised: Besides, the predictors may learn a trivial identical function,since the state transition in agent trajectories is deterministic, i.e., s t +

Original: =
Revised: =

Original: s t [ { i t } and a t = i t .
Revised: s t [ { i t } and a t = i t .

Original: Inthis work, we derive a static environment model from the state transition: ˆ s t + ⌧
Revised: Inthis work, we derive a static environment model from the state transition: ˆ s t + ⌧

Original: +

Revised: +

Original: =
Revised: =

Original: ˆ s t + ⌧ [ { ˆ i t + ⌧ } , 138 ˆ i t + ⌧
Revised: ˆ s t + ⌧ [ { ˆ i t + ⌧ } , 138 ˆ i t + ⌧

Original: =
Revised: =

Original: ˆ a t + ⌧ and ˆ s t =
Revised: ˆ a t + ⌧ and ˆ s t =

Original: s t , where ⌧ is the length of the imagined rollout, ˆ a t + ⌧ the output action of theimagination policy ˆ ⇡ .
Revised: s t , where ⌧ is the length of the imagined rollout, ˆ a t + ⌧ the output action of theimagination policy ˆ ⇡ .

Original: During training, the generated item ˆ i t + ⌧ may not be the true purchase, but westill use it for self-supervised reconstruction of I2E .
Revised: During training, the generated item ˆ i t + ⌧ may not be the true purchase, but westill use it for self-supervised reconstruction of I2E .

Original: This makes the action policy ⇡ more robust tointrinsic errors and forces the imagination policy ˆ ⇡ to generate more accurate actions.
Revised: This makes the action policy ⇡ more robust tointrinsic errors and forces the imagination policy ˆ ⇡ to generate more accurate actions.

Original: In practice, the imagination policy ˆ ⇡ can be obtained from policy distillation [] or ﬁxed target networklike DQN [].
Revised: In practice, the imagination policy ˆ ⇡ can be obtained from policy distillation [] or ﬁxed target networklike DQN [].

Original: The former distills the action policy ⇡ ( s t ; ✓ ) into a smaller rollout network ˆ ⇡ ( s t ; ˆ ✓ ) ,using a cross-entropy loss, l ⇡, ˆ ⇡ ( s t ) =
Revised: The former distills the action policy ⇡ ( s t ; ✓ ) into a smaller rollout network ˆ ⇡ ( s t ; ˆ ✓ ) ,using a cross-entropy loss, l ⇡, ˆ ⇡ ( s t ) =

Original: P a ⇡ ( a | s t ) log ˆ ⇡ ( a | s t ; ˆ ✓ ) .
Revised: P a ⇡ ( a | s t ) log ˆ ⇡ ( a | s t ; ˆ ✓ ) .

Original: The latter uses a shared but slowlychanging target network ˆ ⇡ ( s t ; ✓  ) , where ✓  are previous parameters in ⇡ ( s t ; ✓ ) .
Revised: The latter uses a shared but slowlychanging target network ˆ ⇡ ( s t ; ✓  ) , where ✓  are previous parameters in ⇡ ( s t ; ✓ ) .

Original: By imitating theaction policy ⇡ , the imagined trajectories will be similar to agent experiences in the real environment;this also helps I2E learn a predictive representation of rewarding states, and in turn should allow theeasy learning of the action policy under the sparse reward signals.
Revised: By imitating theaction policy ⇡ , the imagined trajectories will be similar to agent experiences in the real environment;this also helps I2E learn a predictive representation of rewarding states, and in turn should allow theeasy learning of the action policy under the sparse reward signals.

Original: Purchase as Reward: Session-based Recommendation with Imagination Reconstruction Network Anonymous Author(s) AfﬁliationAddress email Abstract For sequential recommendations, it is important to enhance the purchase willing ofusers.
Revised: Purchase as Reward: Session-based Recommendation with Imagination Reconstruction Network Anonymous Author(s) AfﬁliationAddress email Abstract For sequential recommendations, it is important to enhance the purchase willing ofusers.

Original: Recent popular session-based models use classiﬁcation methods to modelthe sequential actions classiﬁcation, which do not explicit differentiate the click andpurchase reward.
Revised: Recent popular session-based models use classiﬁcation methods to modelthe sequential actions classiﬁcation, which do not explicit differentiate the click andpurchase reward.

Original: In this paper, we formulate sequential recommendation as MDP,where the purchase is used as the reward signal for recommender agent and usersessions are treated as external environment.
Revised: In this paper, we formulate sequential recommendation as MDP,where the purchase is used as the reward signal for recommender agent and usersessions are treated as external environment.

Original: To handle the sparse purchase reward,we propose a simple architecture, namely Imagination Reconstruction Network(IRN), which is inspired from the neu-science Prediction Error Minimization.
Revised: To handle the sparse purchase reward,we propose a simple architecture, namely Imagination Reconstruction Network(IRN), which is inspired from the neu-science Prediction Error Minimization.

Original: IRN distill a roll-out policy from action policy and combine it with a staticenvironment model to predict the future observations.
Revised: IRN distill a roll-out policy from action policy and combine it with a staticenvironment model to predict the future observations.

Original: Imaginations are thenused as self-supervised reconstruction signal for action policy when there is nopurchase reward.
Revised: Imaginations are thenused as self-supervised reconstruction signal for action policy when there is nopurchase reward.

Original: By minimizing the intrinsic imagination error and maximizing theextrinsic purchase probability, IRN can automatically balance EE and learn a betterpredictive representations, resulting in a more robust action policy.
Revised: By minimizing the intrinsic imagination error and maximizing theextrinsic purchase probability, IRN can automatically balance EE and learn a betterpredictive representations, resulting in a more robust action policy.

Original: Experimentsshow that IRN can signiﬁcantly enhance the purchase, and achieve a satisfactoryclick performance even when we do not treat click as rewards.robust to the cold-start scenario (short sessions).improve the data efﬁciency.do exploration when item space is huge.
Revised: Experimentsshow that IRN can signiﬁcantly enhance the purchase, and achieve a satisfactoryclick performance even when we do not treat click as rewards.robust to the cold-start scenario (short sessions).improve the data efﬁciency.do exploration when item space is huge.

Original: Introductions T  

Revised: Introductions T

Original: i t i t +
Revised: i t i t +

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: signiﬁcantly and consistently better performance on
Revised: signiﬁcantly and consistently better performance on

Original: [Table]
Revised: [Table]

Original: many real world applications, users’ current interests are
Revised: many real world applications, users’ current interests are

Original: [Table]
Revised: [Table]

Original: model this phenomenon, previous sequential
Revised: model this phenomenon, previous sequential

Original: [Table]
Revised: [Table]

Original: to 32nd
Revised: to 32nd

Original: Purchase as Reward: Session-based Recommendation with Imagination Reconstruction Network Anonymous Author(s) AfﬁliationAddress email
Revised: Purchase as Reward: Session-based Recommendation with Imagination Reconstruction Network Anonymous Author(s) AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: For sequential recommendations, it is important to enhance the purchase willing ofusers
Revised: For sequential recommendations, it is important to enhance the purchase willing ofusers

Original: Recent popular session-based models use classiﬁcation methods model the sequential actions classiﬁcation, which do not explicit differentiate the click and purchase reward.
Revised: Recent popular session-based models use classiﬁcation methods model the sequential actions classiﬁcation, which do not explicit differentiate the click and purchase reward.

Original: In this paper, we formulate sequential recommendation as MDP the purchase is used as the reward signal for recommender agent and sessions are treated as environment.
Revised: In this paper, we formulate sequential recommendation as MDP the purchase is used as the reward signal for recommender agent and sessions are treated as environment.

Original: To handle reward propose is inspired from environment model to predict future
Revised: To handle reward propose is inspired from environment model to predict future

Original: [Table]
Revised: [Table]

Original: as self-supervised reconstruction signal for action policy when there is no
Revised: as self-supervised reconstruction signal for action policy when there is no

Original: [Table]
Revised: [Table]

Original: purchasereward.
Revised: purchasereward.

Original: Byminimizingtheintrinsicimaginationerrorandmaximizingthe
Revised: Byminimizingtheintrinsicimaginationerrorandmaximizingthe

Original: [Table]
Revised: [Table]

Original: extrinsic purchase probability, IRN can automatically balance EE and learn a better
Revised: extrinsic purchase probability, IRN can automatically balance EE and learn a better

Original: [Table]
Revised: [Table]

Original: predictive representations, resulting in a more robust action policy.
Revised: predictive representations, resulting in a more robust action policy.

Original: Experiments
Revised: Experiments

Original: [Table]
Revised: [Table]

Original: show that IRN can signiﬁcantly enhance the purchase, and achieve a satisfactory
Revised: show that IRN can signiﬁcantly enhance the purchase, and achieve a satisfactory

Original: [Table]
Revised: [Table]

Original: click performance even when we do not treat click as rewards.
Revised: click performance even when we do not treat click as rewards.

Original: [Table]
Revised: [Table]

Original: robust to the cold-start scenario (short sessions).
Revised: robust to the cold-start scenario (short sessions).

Original: [Table]
Revised: [Table]

Original: improve the data efﬁciency.
Revised: improve the data efﬁciency.

Original: [Table]
Revised: [Table]

Original: do exploration when item space is huge.
Revised: do exploration when item space is huge.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: session-
Revised: session-

Original: [Table] [Table]
Revised: [Table] [Table]

Original: with buy).
Revised: with buy).

Original: using GRU4REC classiﬁcation
Revised: using GRU4REC classiﬁcation

Original: [Table]
Revised: [Table]

Original: the click and
Revised: the click and

Original: [Table]
Revised: [Table]

Original: , in practice, we may pay more attention to the improvements
Revised: , in practice, we may pay more attention to the improvements

Original: [Table]
Revised: [Table]

Original: Compared with state-of-the-art methods that consider users’ sequential behavior for recommendation,
Revised: Compared with state-of-the-art methods that consider users’ sequential behavior for recommendation,

Original: [Table]
Revised: [Table]

Original: e
Revised: e

Original: [Table]
Revised: [Table]

Original: g., sequential recommenders with recurrent neural networks (RNN) or Markov chains,
Revised: g., sequential recommenders with recurrent neural networks (RNN) or Markov chains,

Original: [Table]
Revised: [Table]

Original: achieves signiﬁcantly and consistently better performance on four real-world datasets.
Revised: achieves signiﬁcantly and consistently better performance on four real-world datasets.

Original: [Table]
Revised: [Table]

Original: In many real-world applications, users’ current interests are inﬂuenced by their historical behaviors.
Revised: In many real-world applications, users’ current interests are inﬂuenced by their historical behaviors.

Original: [Table]
Revised: [Table]

Original: To model this phenomenon, previous
Revised: To model this phenomenon, previous

Original: [Table]
Revised: [Table]

Original: user historical records.
Revised: user historical records.

Original: For
Revised: For

Original: [Table]
Revised: [Table]

Original: Submitted to 32nd Conference on Neural Information Processing Systems (NIPS 2018).
Revised: Submitted to 32nd Conference on Neural Information Processing Systems (NIPS 2018).

Original: Do not distribute.
Revised: Do not distribute.

Original: Purchase as Reward: Session-based Recommendation with Imagination Reconstruction Network AnonymousAuthor(s)
Revised: Purchase as Reward: Session-based Recommendation with Imagination Reconstruction Network AnonymousAuthor(s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Forsequentialrecommendations,itisimportanttoenhancethepurchasewillingofusers.
Revised: Forsequentialrecommendations,itisimportanttoenhancethepurchasewillingofusers.

Original: Recentpopularsession-basedmodelsuseclassiﬁcationmethodstomodelthesequentialactionsclassiﬁcation,whichdonotexplicitdifferentiatetheclickandpurchasereward.
Revised: Recentpopularsession-basedmodelsuseclassiﬁcationmethodstomodelthesequentialactionsclassiﬁcation,whichdonotexplicitdifferentiatetheclickandpurchasereward.

Original: Inthispaper,weformulatesequentialrecommendationasMDP,wherethepurchaseisusedastherewardsignalforrecommenderagentandusersessionsaretreatedasexternalenvironment.
Revised: Inthispaper,weformulatesequentialrecommendationasMDP,wherethepurchaseisusedastherewardsignalforrecommenderagentandusersessionsaretreatedasexternalenvironment.

Original: Tohandlethesparsepurchasereward,we propose a simple architecture, namely Imagination Reconstruction Network(IRN), which is inspired from the neu-science Prediction Error Minimization.
Revised: Tohandlethesparsepurchasereward,we propose a simple architecture, namely Imagination Reconstruction Network(IRN), which is inspired from the neu-science Prediction Error Minimization.

Original: IRN distill a roll-out policy from action policy and combine it with a staticenvironment model to predict the future observations.
Revised: IRN distill a roll-out policy from action policy and combine it with a staticenvironment model to predict the future observations.

Original: Imaginations are thenused as self-supervised reconstruction signal for action policy when there is nopurchasereward.
Revised: Imaginations are thenused as self-supervised reconstruction signal for action policy when there is nopurchasereward.

Original: Byminimizingtheintrinsicimaginationerrorandmaximizingthe
Revised: Byminimizingtheintrinsicimaginationerrorandmaximizingthe

Original: [Figure]
Revised: [Figure]

Original: extrinsicpurchaseprobability,IRNcanautomaticallybalanceEEandlearnabetter
Revised: extrinsicpurchaseprobability,IRNcanautomaticallybalanceEEandlearnabetter

Original: [Table]
Revised: [Table]

Original: predictiverepresentations,resultinginamorerobustactionpolicy.
Revised: predictiverepresentations,resultinginamorerobustactionpolicy.

Original: Experiments
Revised: Experiments

Original: [Figure]
Revised: [Figure]

Original: showthatIRNcansigniﬁcantlyenhancethepurchase,andachieveasatisfactory
Revised: showthatIRNcansigniﬁcantlyenhancethepurchase,andachieveasatisfactory

Original: [Figure]
Revised: [Figure]

Original: clickperformanceevenwhenwedonottreatclickasrewards.
Revised: clickperformanceevenwhenwedonottreatclickasrewards.

Original: [Figure]
Revised: [Figure]

Original: robusttothecold-startscenario(shortsessions).
Revised: robusttothecold-startscenario(shortsessions).

Original: [Figure]
Revised: [Figure]

Original: improvethedataefﬁciency
Revised: improvethedataefﬁciency

Original: [Figure]
Revised: [Figure]

Original: doexplorationwhenitemspaceishuge.
Revised: doexplorationwhenitemspaceishuge.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: achievessigniﬁcantlyandconsistentlybetterperformanceonfourreal
Revised: achievessigniﬁcantlyandconsistentlybetterperformanceonfourreal

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: To model
Revised: To model

Original: [Table]
Revised: [Table]

Original: phenomenon, previous
Revised: phenomenon, previous

Original: [Table]
Revised: [Table]

Original: recommenders with
Revised: recommenders with

Original: [Table]
Revised: [Table]

Original: historical records
Revised: historical records

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: Submittedto32ndConferenceonNeuralInformationProcessingSystems(NIPS2018).
Revised: Submittedto32ndConferenceonNeuralInformationProcessingSystems(NIPS2018).

Original: Donotdistribute.
Revised: Donotdistribute.

Original: Purchase as Reward: Session-based Recommendation with Imagination Reconstruction Network Anonymous Author(s) AfﬁliationAddress email Abstract For sequential recommendations, it is important to enhance the purchase willing ofusers.
Revised: Purchase as Reward: Session-based Recommendation with Imagination Reconstruction Network Anonymous Author(s) AfﬁliationAddress email Abstract For sequential recommendations, it is important to enhance the purchase willing ofusers.

Original: Recent popular session-based models use classiﬁcation methods to modelthe sequential actions classiﬁcation, which do not explicit differentiate the click andpurchase reward.
Revised: Recent popular session-based models use classiﬁcation methods to modelthe sequential actions classiﬁcation, which do not explicit differentiate the click andpurchase reward.

Original: In this paper, we formulate sequential recommendation as MDP,where the purchase is used as the reward signal for recommender agent and usersessions are treated as external environment.
Revised: In this paper, we formulate sequential recommendation as MDP,where the purchase is used as the reward signal for recommender agent and usersessions are treated as external environment.

Original: To handle the sparse purchase reward,we propose a simple architecture, namely Imagination Reconstruction Network(IRN), which is inspired from the neu-science Prediction Error Minimization.
Revised: To handle the sparse purchase reward,we propose a simple architecture, namely Imagination Reconstruction Network(IRN), which is inspired from the neu-science Prediction Error Minimization.

Original: IRN distill a roll-out policy from action policy and combine it with a staticenvironment model to predict the future observations.
Revised: IRN distill a roll-out policy from action policy and combine it with a staticenvironment model to predict the future observations.

Original: Imaginations are thenused as self-supervised reconstruction signal for action policy when there is nopurchase reward.
Revised: Imaginations are thenused as self-supervised reconstruction signal for action policy when there is nopurchase reward.

Original: By minimizing the intrinsic imagination error and maximizing theextrinsic purchase probability, IRN can automatically balance EE and learn a betterpredictive representations, resulting in a more robust action policy.
Revised: By minimizing the intrinsic imagination error and maximizing theextrinsic purchase probability, IRN can automatically balance EE and learn a betterpredictive representations, resulting in a more robust action policy.

Original: Experiments
Revised: Experiments

Original: [Figure]
Revised: [Figure]

Original: show that IRN can signiﬁcantly enhance the purchase, and achieve a satisfactoryclick performance even when we do not treat click as rewards.
Revised: show that IRN can signiﬁcantly enhance the purchase, and achieve a satisfactoryclick performance even when we do not treat click as rewards.

Original: [Figure]
Revised: [Figure]

Original: robust to the cold-start scenario (short sessions).
Revised: robust to the cold-start scenario (short sessions).

Original: [Figure]
Revised: [Figure]

Original: improve the data efﬁciency.
Revised: improve the data efﬁciency.

Original: [Figure]
Revised: [Figure]

Original: do exploration when item space is huge.
Revised: do exploration when item space is huge.

Original: [Figure]
Revised: [Figure]

Original: 1 Introduction
Revised: 1 Introduction

Original: [Figure]
Revised: [Figure]

Original: session-based recommendation (end with buy).
Revised: session-based recommendation (end with buy).

Original: using GRU4REC classiﬁcation problem do not 26 differentiate the click and purchase, in practice, we may pay more attention to the improvements of
Revised: using GRU4REC classiﬁcation problem do not 26 differentiate the click and purchase, in practice, we may pay more attention to the improvements of

Original: [Figure]
Revised: [Figure]

Original: purchase.
Revised: purchase.

Original: Compared with state-of-the-art methods that consider users’ sequential behavior for recommendation, 29 e.g., sequential recommenders with recurrent neural networks (RNN) or Markov chains, our method
Revised: Compared with state-of-the-art methods that consider users’ sequential behavior for recommendation, 29 e.g., sequential recommenders with recurrent neural networks (RNN) or Markov chains, our method

Original: [Figure]
Revised: [Figure]

Original: achieves signiﬁcantly and consistently better performance on four real-world datasets.
Revised: achieves signiﬁcantly and consistently better performance on four real-world datasets.

Original: [Figure]
Revised: [Figure]

Original: In many real-world applications, users’ current interests are inﬂuenced by their historical behaviors.
Revised: In many real-world applications, users’ current interests are inﬂuenced by their historical behaviors.

Original: 32 To model this phenomenon, previous sequential recommenders with user historical records.
Revised: 32 To model this phenomenon, previous sequential recommenders with user historical records.

Original: For
Revised: For

Original: Submitted
Revised: Submitted

Original: 32nd Conference on Neural Information Processing Systems (NIPS 2018).
Revised: 32nd Conference on Neural Information Processing Systems (NIPS 2018).

Original: Do not distribute.
Revised: Do not distribute.

Original: Purchase as Reward: Session-based Recommendation with Imagination Reconstruction Network Anonymous Author(s) AfﬁliationAddress email Abstract For sequential recommendations, it is important to enhance the purchase willing ofusers.
Revised: Purchase as Reward: Session-based Recommendation with Imagination Reconstruction Network Anonymous Author(s) AfﬁliationAddress email Abstract For sequential recommendations, it is important to enhance the purchase willing ofusers.

Original: Recent popular session-based models use classiﬁcation methods to modelthe sequential actions classiﬁcation, which do not explicit differentiate the click andpurchase reward.
Revised: Recent popular session-based models use classiﬁcation methods to modelthe sequential actions classiﬁcation, which do not explicit differentiate the click andpurchase reward.

Original: In this paper, we formulate sequential recommendation as MDP,where the purchase is used as the reward signal for recommender agent and usersessions are treated as external environment.
Revised: In this paper, we formulate sequential recommendation as MDP,where the purchase is used as the reward signal for recommender agent and usersessions are treated as external environment.

Original: To handle the sparse purchase reward,we propose a simple architecture, namely Imagination Reconstruction Network(IRN), which is inspired from the neu-science Prediction Error Minimization.
Revised: To handle the sparse purchase reward,we propose a simple architecture, namely Imagination Reconstruction Network(IRN), which is inspired from the neu-science Prediction Error Minimization.

Original: IRN distill a roll-out policy from action policy and combine it with a staticenvironment model to predict the future observations.
Revised: IRN distill a roll-out policy from action policy and combine it with a staticenvironment model to predict the future observations.

Original: Imaginations are thenused as self-supervised reconstruction signal for action policy when there is nopurchase reward.
Revised: Imaginations are thenused as self-supervised reconstruction signal for action policy when there is nopurchase reward.

Original: By minimizing the intrinsic imagination error and maximizing theextrinsic purchase probability, IRN can automatically balance EE and learn a betterpredictive representations, resulting in a more robust action policy.
Revised: By minimizing the intrinsic imagination error and maximizing theextrinsic purchase probability, IRN can automatically balance EE and learn a betterpredictive representations, resulting in a more robust action policy.

Original: Experimentsshow that IRN can signiﬁcantly enhance the purchase, and achieve a satisfactoryclick performance even when we do not treat click as rewards.
Revised: Experimentsshow that IRN can signiﬁcantly enhance the purchase, and achieve a satisfactoryclick performance even when we do not treat click as rewards.

Original: [Figure]
Revised: [Figure]

Original: robust to the cold-start scenario (short sessions).
Revised: robust to the cold-start scenario (short sessions).

Original: [Figure]
Revised: [Figure]

Original: improve the data efﬁciency.
Revised: improve the data efﬁciency.

Original: [Figure]
Revised: [Figure]

Original: do exploration when item space is huge.
Revised: do exploration when item space is huge.

Original: [Figure]
Revised: [Figure]

Original: 1 Introduction
Revised: 1 Introduction

Original: [Figure]
Revised: [Figure]

Original: GRU4REC classiﬁcation problem do not
Revised: GRU4REC classiﬁcation problem do not

Original: [Figure]
Revised: [Figure]

Original: the click and
Revised: the click and

Original: , in practice, we may pay more attention to the improvements of
Revised: , in practice, we may pay more attention to the improvements of

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Compared with state-of-the-art methods that consider users’ sequential behavior for recommendation
Revised: Compared with state-of-the-art methods that consider users’ sequential behavior for recommendation

Original: [Figure]
Revised: [Figure]

Original: g sequential recommenders with recurrent neural networks (RNN) or Markov chains, our method
Revised: g sequential recommenders with recurrent neural networks (RNN) or Markov chains, our method

Original: [Figure]
Revised: [Figure]

Original: achieves signiﬁcantly and consistently better performance on
Revised: achieves signiﬁcantly and consistently better performance on

Original: [Table]
Revised: [Table]

Original: Purchase
Revised: Purchase

Original: as Reward: Session-based Recommendation with Imagination Reconstruction Network
Revised: as Reward: Session-based Recommendation with Imagination Reconstruction Network

Original: Anonymous Author(s) AfﬁliationAddress email Abstract For sequential recommendations, it is important to enhance the purchase willing ofusers.
Revised: Anonymous Author(s) AfﬁliationAddress email Abstract For sequential recommendations, it is important to enhance the purchase willing ofusers.

Original: Recent popular session-based models use classiﬁcation methods to modelthe sequential actions classiﬁcation, which do not explicit differentiate the click andpurchase reward.
Revised: Recent popular session-based models use classiﬁcation methods to modelthe sequential actions classiﬁcation, which do not explicit differentiate the click andpurchase reward.

Original: In this paper, we formulate sequential recommendation as MDP,where the purchase is used as the reward signal for recommender agent and usersessions are treated as external environment.
Revised: In this paper, we formulate sequential recommendation as MDP,where the purchase is used as the reward signal for recommender agent and usersessions are treated as external environment.

Original: To handle the sparse purchase reward,we propose a simple architecture, namely Imagination Reconstruction Network
Revised: To handle the sparse purchase reward,we propose a simple architecture, namely Imagination Reconstruction Network

Original: [Table]
Revised: [Table]

Original: (IRN), which is inspired from the neu-science Prediction Error Minimization.
Revised: (IRN), which is inspired from the neu-science Prediction Error Minimization.

Original: IRN distill a roll-out policy from action policy and combine it with a staticenvironment model to predict the future observations.
Revised: IRN distill a roll-out policy from action policy and combine it with a staticenvironment model to predict the future observations.

Original: Imaginations are thenused as self-supervised reconstruction signal for action policy when there is nopurchase reward.
Revised: Imaginations are thenused as self-supervised reconstruction signal for action policy when there is nopurchase reward.

Original: By minimizing the intrinsic imagination error and maximizing theextrinsic purchase probability, IRN can automatically balance EE and learn a betterpredictive representations, resulting in a more robust action policy.
Revised: By minimizing the intrinsic imagination error and maximizing theextrinsic purchase probability, IRN can automatically balance EE and learn a betterpredictive representations, resulting in a more robust action policy.

Original: Experimentsshow that IRN can signiﬁcantly enhance the purchase, and achieve a satisfactoryclick performance even when we do not treat click as rewards.
Revised: Experimentsshow that IRN can signiﬁcantly enhance the purchase, and achieve a satisfactoryclick performance even when we do not treat click as rewards.

Original: [Table]
Revised: [Table]

Original: robust to the cold-
Revised: robust to the cold-

Original: [Table]
Revised: [Table]

Original: scenario (
Revised: scenario (

Original: [Table]
Revised: [Table]

Original: ).improve the data efﬁciency.
Revised: ).improve the data efﬁciency.

Original: [Table]
Revised: [Table]

Original: do exploration when item space is huge.
Revised: do exploration when item space is huge.

Original: [Table]
Revised: [Table]

Original: Introduction
Revised: Introduction

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: Purchase as Reward: Session-based Recommendation with Imagination Reconstruction Network Anonymous Author(s) AfﬁliationAddress email Abstract For sequential recommendations, it is important to enhance the purchase willing ofusers.
Revised: Purchase as Reward: Session-based Recommendation with Imagination Reconstruction Network Anonymous Author(s) AfﬁliationAddress email Abstract For sequential recommendations, it is important to enhance the purchase willing ofusers.

Original: Recent popular session-based models use classiﬁcation methods to modelthe sequential actions classiﬁcation, which do not explicit differentiate the click andpurchase reward.
Revised: Recent popular session-based models use classiﬁcation methods to modelthe sequential actions classiﬁcation, which do not explicit differentiate the click andpurchase reward.

Original: In this paper, we formulate sequential recommendation as MDP,where the purchase is used as the reward signal for recommender agent and usersessions are treated as external environment.
Revised: In this paper, we formulate sequential recommendation as MDP,where the purchase is used as the reward signal for recommender agent and usersessions are treated as external environment.

Original: To handle the sparse purchase reward,we propose a simple architecture, namely Imagination Reconstruction Network(IRN), which is inspired from the neu-science Prediction Error Minimization.
Revised: To handle the sparse purchase reward,we propose a simple architecture, namely Imagination Reconstruction Network(IRN), which is inspired from the neu-science Prediction Error Minimization.

Original: [Table]
Revised: [Table]

Original: policy and combine it with a staticenvironment model to predict the future observations.
Revised: policy and combine it with a staticenvironment model to predict the future observations.

Original: Imaginations are thenused as self-supervised reconstruction signal for action policy when there is nopurchase reward.
Revised: Imaginations are thenused as self-supervised reconstruction signal for action policy when there is nopurchase reward.

Original: By minimizing the intrinsic imagination error and maximizing extrinsic purchase probability, IRN can automatically balance EE and learn a betterpredictive representations resulting in a more robust action policy
Revised: By minimizing the intrinsic imagination error and maximizing extrinsic purchase probability, IRN can automatically balance EE and learn a betterpredictive representations resulting in a more robust action policy

Original: Experimentsshow that IRN can signiﬁcantly enhance the purchase, and achieve a satisfactoryclick performance even when we do not treat click as rewards.robust to the cold-start scenario (short sessions).improve the data efﬁciency.do exploration is huge.
Revised: Experimentsshow that IRN can signiﬁcantly enhance the purchase, and achieve a satisfactoryclick performance even when we do not treat click as rewards.robust to the cold-start scenario (short sessions).improve the data efﬁciency.do exploration is huge.

Original: 1 Introduction
Revised: 1 Introduction

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: Figure 2: IRN architecture.
Revised: Figure 2: IRN architecture.

Original: s shown in Figure 2, where IC is imagination core to generate trajectories, and N-step imagination rollout, using these trajectories to perform encoder-decoder.
Revised: s shown in Figure 2, where IC is imagination core to generate trajectories, and N-step imagination rollout, using these trajectories to perform encoder-decoder.

Original: 4 IRN Architecture 

Revised: 4 IRN Architecture

Original: In this section we incorporate the imagination reconstruction module into the model-free agents 117 (e.g., A3C) in order to enhance data efﬁciency, promote more robust learning and ultimately higher 118 performance under the sparse extrinsic rewards.
Revised: In this section we incorporate the imagination reconstruction module into the model-free agents 117 (e.g., A3C) in order to enhance data efﬁciency, promote more robust learning and ultimately higher 118 performance under the sparse extrinsic rewards.

Original: Our IRN implements an imagination-augmented 119 policy via
Revised: Our IRN implements an imagination-augmented 119 policy via

Original: [Table]
Revised: [Table]

Original: 120 conditioned on actions sampled from the imagination policy ˆ ⇡ .
Revised: 120 conditioned on actions sampled from the imagination policy ˆ ⇡ .

Original: At a time step t , the trajectorymanager (TM) determines how to roll out the IC under the planning strategy, and then producesimagined trajectories ˆ T 1 , . .
Revised: At a time step t , the trajectorymanager (TM) determines how to roll out the IC under the planning strategy, and then producesimagined trajectories ˆ T 1 , . .

Original: , ˆ T n of an observable world state s t .
Revised: , ˆ T n of an observable world state s t .

Original: Each trajectory ˆ T is a sequenceof items { ˆ i t , ˆ i t +1 , . . .
Revised: Each trajectory ˆ T is a sequenceof items { ˆ i t , ˆ i t +1 , . . .

Original: } , that the agent may purchase (or click) from the current time t .
Revised: } , that the agent may purchase (or click) from the current time t .

Original: Finally,the Imagination-augmented Executor (I2E) aggregates the internal data resulting from imaginationand external rewarding data to update its action policy ⇡ .
Revised: Finally,the Imagination-augmented Executor (I2E) aggregates the internal data resulting from imaginationand external rewarding data to update its action policy ⇡ .

Original: Speciﬁcally, I2E optimizes the policy⇡ by maximizing the extrinsic rewards while minimizing the intrinsic imagination errors, L =
Revised: Speciﬁcally, I2E optimizes the policy⇡ by maximizing the extrinsic rewards while minimizing the intrinsic imagination errors, L =

Original: L A 3 C + L IRN .
Revised: L A 3 C + L IRN .

Original: In principle, IRN encourages exploration (on the output layer, i.e., item representation)and implicitly learns to plan ("guess what you like") via imagination rollouts, and therefore promotesquick adaptation to user interest.
Revised: In principle, IRN encourages exploration (on the output layer, i.e., item representation)and implicitly learns to plan ("guess what you like") via imagination rollouts, and therefore promotesquick adaptation to user interest.

Original: Imagination Core.
Revised: Imagination Core.

Original: In order to simulate imagined trajectories, we rely on environment models that,given the present state and a candidate action, make predictions about the future states.
Revised: In order to simulate imagined trajectories, we rely on environment models that,given the present state and a candidate action, make predictions about the future states.

Original: In general,we can employ an environment model that build on recent popular action-conditional next-steppredictors [], and train it in an unsupervised fashion from agent experiences.
Revised: In general,we can employ an environment model that build on recent popular action-conditional next-steppredictors [], and train it in an unsupervised fashion from agent experiences.

Original: However, the predictorsusually suffer from model errors, resulting in poor agent performance, and need extra computationalcost (e.g., pre-training)
Revised: However, the predictorsusually suffer from model errors, resulting in poor agent performance, and need extra computationalcost (e.g., pre-training)

Original: [error][cost].
Revised: [error][cost].

Original: Besides, the predictors may learn a trivial identical function,since the state transition in agent trajectories is deterministic, i.e., s t +

Revised: Besides, the predictors may learn a trivial identical function,since the state transition in agent trajectories is deterministic, i.e., s t +

Original: =
Revised: =

Original: s t [ { i t } and a t = i t .
Revised: s t [ { i t } and a t = i t .

Original: Inthis work, we derive a static environment model from the state transition: ˆ s t + ⌧
Revised: Inthis work, we derive a static environment model from the state transition: ˆ s t + ⌧

Original: +

Revised: +

Original: =
Revised: =

Original: ˆ s t + ⌧ [ { ˆ i t + ⌧ } , 138 ˆ i t + ⌧
Revised: ˆ s t + ⌧ [ { ˆ i t + ⌧ } , 138 ˆ i t + ⌧

Original: =
Revised: =

Original: ˆ a t + ⌧ and ˆ s t =
Revised: ˆ a t + ⌧ and ˆ s t =

Original: s t , where ⌧ is the length of the imagined rollout, ˆ a t + ⌧ the output action of theimagination policy ˆ ⇡ .
Revised: s t , where ⌧ is the length of the imagined rollout, ˆ a t + ⌧ the output action of theimagination policy ˆ ⇡ .

Original: During training, the generated item ˆ i t + ⌧ may not be the true purchase, but westill use it for self-supervised reconstruction of I2E .
Revised: During training, the generated item ˆ i t + ⌧ may not be the true purchase, but westill use it for self-supervised reconstruction of I2E .

Original: This makes the action policy ⇡ more robust tointrinsic errors and forces the imagination policy ˆ ⇡ to generate more accurate actions.
Revised: This makes the action policy ⇡ more robust tointrinsic errors and forces the imagination policy ˆ ⇡ to generate more accurate actions.

Original: In practice, the imagination policy ˆ ⇡ can be obtained from policy distillation [] or ﬁxed target networklike DQN [].
Revised: In practice, the imagination policy ˆ ⇡ can be obtained from policy distillation [] or ﬁxed target networklike DQN [].

Original: The former distills the action policy ⇡ ( s t ; ✓ ) into a smaller rollout network ˆ ⇡ ( s t ; ˆ ✓ ) ,using a cross-entropy loss, l ⇡, ˆ ⇡ ( s t ) =
Revised: The former distills the action policy ⇡ ( s t ; ✓ ) into a smaller rollout network ˆ ⇡ ( s t ; ˆ ✓ ) ,using a cross-entropy loss, l ⇡, ˆ ⇡ ( s t ) =

Original: P a ⇡ ( a | s t ) log ˆ ⇡ ( a | s t ; ˆ ✓ ) .
Revised: P a ⇡ ( a | s t ) log ˆ ⇡ ( a | s t ; ˆ ✓ ) .

Original: The latter uses a shared but slowlychanging target network ˆ ⇡ ( s t ; ✓  ) , where ✓  are previous parameters in ⇡ ( s t ; ✓ ) .
Revised: The latter uses a shared but slowlychanging target network ˆ ⇡ ( s t ; ✓  ) , where ✓  are previous parameters in ⇡ ( s t ; ✓ ) .

Original: By imitating theaction policy ⇡ , the imagined trajectories will be similar to agent experiences in the real environment;this also helps I2E learn a predictive representation of rewarding states, and in turn should allow theeasy learning of the action policy under the sparse reward signals.2 IRN are hard to specify
Revised: By imitating theaction policy ⇡ , the imagined trajectories will be similar to agent experiences in the real environment;this also helps I2E learn a predictive representation of rewarding states, and in turn should allow theeasy learning of the action policy under the sparse reward signals.2 IRN are hard to specify

Original: Figure 2: IRN architecture.
Revised: Figure 2: IRN architecture.

Original: s shown in Figure 2, where IC is imagination core to generate trajectories, and N-step imagination rollout, using these trajectories to perform encoder-decoder.
Revised: s shown in Figure 2, where IC is imagination core to generate trajectories, and N-step imagination rollout, using these trajectories to perform encoder-decoder.

Original: 4 IRN Architecture 

Revised: 4 IRN Architecture

Original: In this section we incorporate the imagination reconstruction module into the model-free agents 117 (e.g., A3C) in order to enhance data efﬁciency, promote more robust learning and ultimately higher 118 performance under the sparse extrinsic rewards.
Revised: In this section we incorporate the imagination reconstruction module into the model-free agents 117 (e.g., A3C) in order to enhance data efﬁciency, promote more robust learning and ultimately higher 118 performance under the sparse extrinsic rewards.

Original: Our IRN implements an imagination-augmented 119 policy via three key components (Fig 2).
Revised: Our IRN implements an imagination-augmented 119 policy via three key components (Fig 2).

Original: The imagination core (IC) predicts next time steps 120 conditioned on actions sampled from the imagination policy ˆ ⇡ .
Revised: The imagination core (IC) predicts next time steps 120 conditioned on actions sampled from the imagination policy ˆ ⇡ .

Original: At a time step t , the trajectory 121 manager (TM) determines how to roll out the IC under the planning strategy, and then produces 122 imagined trajectories ˆ T 1 , . .
Revised: At a time step t , the trajectory 121 manager (TM) determines how to roll out the IC under the planning strategy, and then produces 122 imagined trajectories ˆ T 1 , . .

Original: , ˆ T n of an observable world state s t .
Revised: , ˆ T n of an observable world state s t .

Original: Each trajectory ˆ T is a sequenceof items { ˆ i t , ˆ i t +1 , . . .
Revised: Each trajectory ˆ T is a sequenceof items { ˆ i t , ˆ i t +1 , . . .

Original: } , that the agent may purchase (or click) from the current time t .
Revised: } , that the agent may purchase (or click) from the current time t .

Original: Finally,the Imagination-augmented Executor (I2E) aggregates the internal data resulting from imaginationand external rewarding data to update its action policy ⇡ .
Revised: Finally,the Imagination-augmented Executor (I2E) aggregates the internal data resulting from imaginationand external rewarding data to update its action policy ⇡ .

Original: Speciﬁcally, I2E optimizes the policy⇡ by maximizing the extrinsic rewards while minimizing the intrinsic imagination errors, L =
Revised: Speciﬁcally, I2E optimizes the policy⇡ by maximizing the extrinsic rewards while minimizing the intrinsic imagination errors, L =

Original: L A 3 C + L IRN .
Revised: L A 3 C + L IRN .

Original: In principle, IRN encourages exploration (on the output layer, i.e., item representation)and implicitly learns to plan ("guess what you like") via imagination rollouts, and therefore promotesquick adaptation to user interest.
Revised: In principle, IRN encourages exploration (on the output layer, i.e., item representation)and implicitly learns to plan ("guess what you like") via imagination rollouts, and therefore promotesquick adaptation to user interest.

Original: Imagination Core.
Revised: Imagination Core.

Original: In order to simulate imagined trajectories, we rely on environment models that,given the present state and a candidate action, make predictions about the future states.
Revised: In order to simulate imagined trajectories, we rely on environment models that,given the present state and a candidate action, make predictions about the future states.

Original: In general,we can employ an environment model that build on recent popular action-conditional next-steppredictors [], and train it in an unsupervised fashion from agent experiences.
Revised: In general,we can employ an environment model that build on recent popular action-conditional next-steppredictors [], and train it in an unsupervised fashion from agent experiences.

Original: However, the predictorsusually suffer from model errors, resulting in poor agent performance, and need extra computationalcost (e.g., pre-training)
Revised: However, the predictorsusually suffer from model errors, resulting in poor agent performance, and need extra computationalcost (e.g., pre-training)

Original: [error][cost].
Revised: [error][cost].

Original: Besides, the predictors may learn a trivial identical function,since the state transition in agent trajectories is deterministic, i.e., s t +

Revised: Besides, the predictors may learn a trivial identical function,since the state transition in agent trajectories is deterministic, i.e., s t +

Original: =
Revised: =

Original: s t [ { i t } and a t = i t .
Revised: s t [ { i t } and a t = i t .

Original: Inthis work, we derive a static environment model from the state transition: ˆ s t + ⌧
Revised: Inthis work, we derive a static environment model from the state transition: ˆ s t + ⌧

Original: +

Revised: +

Original: =
Revised: =

Original: ˆ s t + ⌧ [ { ˆ i t + ⌧ } , 138 ˆ i t + ⌧
Revised: ˆ s t + ⌧ [ { ˆ i t + ⌧ } , 138 ˆ i t + ⌧

Original: =
Revised: =

Original: ˆ a t + ⌧ and ˆ s t =
Revised: ˆ a t + ⌧ and ˆ s t =

Original: s t , where ⌧ is the length of the imagined rollout, ˆ a t + ⌧ the output action of theimagination policy ˆ ⇡ .
Revised: s t , where ⌧ is the length of the imagined rollout, ˆ a t + ⌧ the output action of theimagination policy ˆ ⇡ .

Original: During training, the generated item ˆ i t + ⌧ may not be the true purchase, but westill use it for self-supervised reconstruction of I2E .
Revised: During training, the generated item ˆ i t + ⌧ may not be the true purchase, but westill use it for self-supervised reconstruction of I2E .

Original: This makes the action policy ⇡ more robust tointrinsic errors and forces the imagination policy ˆ ⇡ to generate more accurate actions.
Revised: This makes the action policy ⇡ more robust tointrinsic errors and forces the imagination policy ˆ ⇡ to generate more accurate actions.

Original: In practice, the imagination policy ˆ ⇡ can be obtained from policy distillation [] or ﬁxed target networklike DQN [].
Revised: In practice, the imagination policy ˆ ⇡ can be obtained from policy distillation [] or ﬁxed target networklike DQN [].

Original: The former distills the action policy ⇡ ( s t ; ✓ ) into a smaller rollout network ˆ ⇡ ( s t ; ˆ ✓ ) ,using a cross-entropy loss, l ⇡, ˆ ⇡ ( s t ) =
Revised: The former distills the action policy ⇡ ( s t ; ✓ ) into a smaller rollout network ˆ ⇡ ( s t ; ˆ ✓ ) ,using a cross-entropy loss, l ⇡, ˆ ⇡ ( s t ) =

Original: P a ⇡ ( a | s t ) log ˆ ⇡ ( a | s t ; ˆ ✓ ) .
Revised: P a ⇡ ( a | s t ) log ˆ ⇡ ( a | s t ; ˆ ✓ ) .

Original: The latter uses a shared but slowlychanging target network ˆ ⇡ ( s t ; ✓  ) , where ✓  are previous parameters in ⇡ ( s t ; ✓ ) .
Revised: The latter uses a shared but slowlychanging target network ˆ ⇡ ( s t ; ✓  ) , where ✓  are previous parameters in ⇡ ( s t ; ✓ ) .

Original: By imitating theaction policy ⇡ , the imagined trajectories will be similar to agent experiences in the real environment;this also helps I2E learn a predictive representation of rewarding states, and in turn should allow theeasy learning of the action policy under the sparse reward signals.2 IRN do not predict the rewards, since it is not useful as reported in [].
Revised: By imitating theaction policy ⇡ , the imagined trajectories will be similar to agent experiences in the real environment;this also helps I2E learn a predictive representation of rewarding states, and in turn should allow theeasy learning of the action policy under the sparse reward signals.2 IRN do not predict the rewards, since it is not useful as reported in [].

Original: and in RS, rewards of different actions are hard to specify
Revised: and in RS, rewards of different actions are hard to specify

Original: Imagination Core
Revised: Imagination Core

Original: In order to simulate imagined trajectories, we rely on environment models that, given the present state and a candidate action, make predictions about the future states.
Revised: In order to simulate imagined trajectories, we rely on environment models that, given the present state and a candidate action, make predictions about the future states.

Original: In general, we can employ an environment model that build on action-conditional next-step predictors (Oh et al., 2015), and train it in an unsupervised fashion from agent experiences.
Revised: In general, we can employ an environment model that build on action-conditional next-step predictors (Oh et al., 2015), and train it in an unsupervised fashion from agent experiences.

Original: However, the predictors usually suffer from model errors, resulting in poor agent performance, and require extra computational cost (e.g., pre-training).
Revised: However, the predictors usually suffer from model errors, resulting in poor agent performance, and require extra computational cost (e.g., pre-training).

Original: Besides, the predictors may learn a trivial identical function, since the state transition in agent trajectories (or session data) is deterministic, i.e., s t +1 =
Revised: Besides, the predictors may learn a trivial identical function, since the state transition in agent trajectories (or session data) is deterministic, i.e., s t +1 =

Original: s t ∪ { i t } and i t = a t .
Revised: s t ∪ { i t } and i t = a t .

Original: In this work, we derive a static environment model from the state transition: ˆ s t
Revised: In this work, we derive a static environment model from the state transition: ˆ s t

Original: + τ +

Revised: + τ +

Original: =
Revised: =

Original: ˆ s t
Revised: ˆ s t

Original: + τ ∪ { ˆ i t + τ } , ˆ i t
Revised: + τ ∪ { ˆ i t + τ } , ˆ i t

Original: + τ = ˆ a t
Revised: + τ = ˆ a t

Original: + τ and ˆ s t =
Revised: + τ and ˆ s t =

Original: s t , where τ is the length of the imagined rollout, ˆ a t + τ the output action of the imagination policy ˆ π .
Revised: s t , where τ is the length of the imagined rollout, ˆ a t + τ the output action of the imagination policy ˆ π .

Original: During training, the generated item ˆ i t + τ may not be the true purchase/click, but we still use it for self-supervised reconstruction.
Revised: During training, the generated item ˆ i t + τ may not be the true purchase/click, but we still use it for self-supervised reconstruction.

Original: This makes the action policy π more robust to intrinsic errors and forces the imagination policy ˆ π to generate more accurate actions.
Revised: This makes the action policy π more robust to intrinsic errors and forces the imagination policy ˆ π to generate more accurate actions.

Original: In practice, the imagination policy ˆ π can be obtained from policy distillation (Racanière et al., 2017) or a ﬁxed target network like DQN (Mnih et al., 2015).
Revised: In practice, the imagination policy ˆ π can be obtained from policy distillation (Racanière et al., 2017) or a ﬁxed target network like DQN (Mnih et al., 2015).

Original: The former distills the action policy π ( s t ; θ ) into a smaller rollout network ˆ π ( s t ; ˆ θ ) , using a cross-entropy loss, l π, ˆ π ( s t ) =
Revised: The former distills the action policy π ( s t ; θ ) into a smaller rollout network ˆ π ( s t ; ˆ θ ) , using a cross-entropy loss, l π, ˆ π ( s t ) =

Original: (cid:80)
Revised: (cid:80)

Original: [Equation]
Revised: [Equation]

Original: The latter uses a shared but slowly changing network ˆ π ( s t ; θ − ) , where θ − are previous parameters in π ( s t ; θ ) .
Revised: The latter uses a shared but slowly changing network ˆ π ( s t ; θ − ) , where θ − are previous parameters in π ( s t ; θ ) .

Original: By imitating the action policy π , the imagined trajectories will be similar to agent experiences in the real environment; this also helps IAE learn predictive representations of rewarding states, and in turn should allow the easy learning of the action policy under the sparse reward signals.
Revised: By imitating the action policy π , the imagined trajectories will be similar to agent experiences in the real environment; this also helps IAE learn predictive representations of rewarding states, and in turn should allow the easy learning of the action policy under the sparse reward signals.

Original: Trajectory Manager The TM rolls out the IC over multiple time steps into the future, generating multiple imagined trajectories with the present information.
Revised: Trajectory Manager The TM rolls out the IC over multiple time steps into the future, generating multiple imagined trajectories with the present information.

Original: Additionally, various planning strategies are supported for trajectory simulation: breadth- n , depth- m and their combination.
Revised: Additionally, various planning strategies are supported for trajectory simulation: breadth- n , depth- m and their combination.

Original: For breadth- n imagination, the TM generates n trajectories, ˆ T 1 , . . .
Revised: For breadth- n imagination, the TM generates n trajectories, ˆ T 1 , . . .

Original: , ˆ T n , over one time step from the current state s t , i.e., ˆ T j
Revised: , ˆ T n , over one time step from the current state s t , i.e., ˆ T j

Original: =
Revised: =

Original: ˆ i j,t } .
Revised: ˆ i j,t } .

Original: Empirically, the IAE using breadth- n imagination will motivate the agent to focus on short-term events and predict the next step more accurately (e.g., enhancing the next-click prediction performance even when we do not formalize the click event as reward).
Revised: Empirically, the IAE using breadth- n imagination will motivate the agent to focus on short-term events and predict the next step more accurately (e.g., enhancing the next-click prediction performance even when we do not formalize the click event as reward).

Original: For depth- m imagination, the TM generates only one trajectory ˆ T 1 through m time steps, i.e., ˆ T 1 = {
Revised: For depth- m imagination, the TM generates only one trajectory ˆ T 1 through m time steps, i.e., ˆ T 1 = {

Original: ˆ i 1 ,t , . . .
Revised: ˆ i 1 ,t , . . .

Original: , ˆ i 1 ,t + m − 1 } .
Revised: , ˆ i 1 ,t + m − 1 } .

Original: This enables the agent to learn to plan the long-term future, and thus recommend items that yield high rewards (purchases).
Revised: This enables the agent to learn to plan the long-term future, and thus recommend items that yield high rewards (purchases).

Original: Finally, we can also achieve the trade-off between breadth- n and depth- m to balance the long-term rewards and short-term rewards.
Revised: Finally, we can also achieve the trade-off between breadth- n and depth- m to balance the long-term rewards and short-term rewards.

Original: Speciﬁcally, we generate n trajectories, and each has a depth m , i.e., { ˆ T }
Revised: Speciﬁcally, we generate n trajectories, and each has a depth m , i.e., { ˆ T }

Original: = {{ ˆ i 1 ,t , . .
Revised: = {{ ˆ i 1 ,t , . .

Original: , ˆ i 1 ,t + m − 1 } , . .
Revised: , ˆ i 1 ,t + m − 1 } , . .

Original: , { ˆ i n,t , . . .
Revised: , { ˆ i n,t , . . .

Original: , ˆ i n,t + m − 1 }} .
Revised: , ˆ i n,t + m − 1 }} .

Original: Imagination-augmented Executor As mentioned before, the IAE uses external rewarding data and internal imagined trajectories to update its action policy π .
Revised: Imagination-augmented Executor As mentioned before, the IAE uses external rewarding data and internal imagined trajectories to update its action policy π .

Original: For the j -th trajectory, ˆ T j
Revised: For the j -th trajectory, ˆ T j

Original: =
Revised: =

Original: ˆ i j,t , . . .
Revised: ˆ i j,t , . . .

Original: , ˆ i j,t + m − 1 } , we deﬁne a multi-step reconstruction objective using the mean squared error:
Revised: , ˆ i j,t + m − 1 } , we deﬁne a multi-step reconstruction objective using the mean squared error:

Original: [Equation]
Revised: [Equation]

Original: where ˆ T j,τ is the τ -th imagined item, φ ( · ) is the input encoder shared by π (for joint feature learning),
Revised: where ˆ T j,τ is the τ -th imagined item, φ ( · ) is the input encoder shared by π (for joint feature learning),

Original: AE is the autoencoder that reconstructs the input feature, and the discounting factor γ is used to mimic Bellman type operations.
Revised: AE is the autoencoder that reconstructs the input feature, and the discounting factor γ is used to mimic Bellman type operations.

Original: In practice, we found that action representation learning (i.e., the output weights of π ) is crucial to the ﬁnal performance due to the large size of candidate items.
Revised: In practice, we found that action representation learning (i.e., the output weights of π ) is crucial to the ﬁnal performance due to the large size of candidate items.

Original: Therefore, we use the one-hot transformation as φ ( · ) and replace AE with the policy π (excluding the ﬁnal softmax function), and only back-propagate errors in the positions of imagined items.
Revised: Therefore, we use the one-hot transformation as φ ( · ) and replace AE with the policy π (excluding the ﬁnal softmax function), and only back-propagate errors in the positions of imagined items.

Original: Speciﬁcally, for an imagined item, the mean squared error is computed between one and its activation value through π ; errors for other items are turned to zero.
Revised: Speciﬁcally, for an imagined item, the mean squared error is computed between one and its activation value through π ; errors for other items are turned to zero.

Original: In this case, the policy π is optimized not only to predict purchases accurately but also to minimize the reconstruction error of imagined items over time.
Revised: In this case, the policy π is optimized not only to predict purchases accurately but also to minimize the reconstruction error of imagined items over time.

Original: Take a session for example, { i 0 , i 1 , ..., i q − 1 , i q } ( i q is the ﬁnal purchased item), π is trained t + 1 times using imagination reconstruction and once using A3C updating (for the purchase event); the overall reconstruction loss for this session is deﬁned as L IRN =
Revised: Take a session for example, { i 0 , i 1 , ..., i q − 1 , i q } ( i q is the ﬁnal purchased item), π is trained t + 1 times using imagination reconstruction and once using A3C updating (for the purchase event); the overall reconstruction loss for this session is deﬁned as L IRN =

Original: [Equation]
Revised: [Equation]

Original: There are several advantages associated with the imagination reconstruction.
Revised: There are several advantages associated with the imagination reconstruction.

Original: First, imagined trajectories provide auxiliary signals for reward augmentation.
Revised: First, imagined trajectories provide auxiliary signals for reward augmentation.

Original: This speeds up policy learning when extrinsic reward is delayed and sparse.
Revised: This speeds up policy learning when extrinsic reward is delayed and sparse.

Original: Second, by using a shared policy network, IAE enables exploration and exploitation, and thus improves feature learning when the number of actions is large.
Revised: Second, by using a shared policy network, IAE enables exploration and exploitation, and thus improves feature learning when the number of actions is large.

Original: Third, compared with agents that predict the next observations for robust learning (Mirowski et al., 2016), our IAE reconstructs the imagined trajectories generated by the TM over time for predictive learning.
Revised: Third, compared with agents that predict the next observations for robust learning (Mirowski et al., 2016), our IAE reconstructs the imagined trajectories generated by the TM over time for predictive learning.

Original: When external reward is provided, IAE can be considered as a process of goal-oriented learning or semi-supervised learning.
Revised: When external reward is provided, IAE can be considered as a process of goal-oriented learning or semi-supervised learning.

Original: This self-supervised reconstruction approach also achieves excellent click and purchase prediction performance even without any external reward (unsupervised learning in this case, where inputs and output targets used for training π are all counterfactual predictions, and the input states are transformed through actions in order to match predictions, i.e., predictive perception in Seth (2014)).
Revised: This self-supervised reconstruction approach also achieves excellent click and purchase prediction performance even without any external reward (unsupervised learning in this case, where inputs and output targets used for training π are all counterfactual predictions, and the input states are transformed through actions in order to match predictions, i.e., predictive perception in Seth (2014)).

Original:  E XPERIMENTS 5. E XPERIMENTAL S ETUP
Revised:  E XPERIMENTS 5. E XPERIMENTAL S ETUP

Original: We evaluate the proposed model on the dataset of ACM RecSys 2015 Challenge 2 , which contains click-streams that sometimes end with purchase events.
Revised: We evaluate the proposed model on the dataset of ACM RecSys 2015 Challenge 2 , which contains click-streams that sometimes end with purchase events.

Original: The purchase reward and the click reward (if used) are empirically set as 5 and 1, respectively.
Revised: The purchase reward and the click reward (if used) are empirically set as 5 and 1, respectively.

Original: Focusing on the most recent events has shown to be effective (Jannach & Ludewig, 2017); therefore we collect the latest one month of data and keep sessions that contain purchases.
Revised: Focusing on the most recent events has shown to be effective (Jannach & Ludewig, 2017); therefore we collect the latest one month of data and keep sessions that contain purchases.

Original: We follow the preprocessing steps in Hidasi et al.
Revised: We follow the preprocessing steps in Hidasi et al.

Original: and use the sessions of the last three day for testing (we also trained IRN and baselines on the full six month training set, with slightly poorer results; the relative improvements remained similar).
Revised: and use the sessions of the last three day for testing (we also trained IRN and baselines on the full six month training set, with slightly poorer results; the relative improvements remained similar).

Original: The training set contains 72274 sessions of 683530 events, and the test set contains 7223 sessions of 63100 events, and the number of items is 9167.
Revised: The training set contains 72274 sessions of 683530 events, and the test set contains 7223 sessions of 63100 events, and the number of items is 9167.

Original: We also derive a separate validation set from the training set, with sessions of the last day in the training set.
Revised: We also derive a separate validation set from the training set, with sessions of the last day in the training set.

Original: The evaluation is done by incrementally adding the previous observed event to the session and checking the rank of the next event.
Revised: The evaluation is done by incrementally adding the previous observed event to the session and checking the rank of the next event.

Original: We adopt Recall and
Revised: We adopt Recall and

Original: Mean Reciprocal Rank (MRR) for top-
Revised: Mean Reciprocal Rank (MRR) for top-

Original: K evaluations, and take the averaged scores over all events in the test set.
Revised: K evaluations, and take the averaged scores over all events in the test set.

Original: We repeat this procedure 5 times and report the average performance.
Revised: We repeat this procedure 5 times and report the average performance.

Original: Without special mention, we set K to 5 for both metrics.
Revised: Without special mention, we set K to 5 for both metrics.

Original: Besides, we build an environment using session-parallel mini-batches, where the agent interacts with multiple sessions simultaneously (see section 3).
Revised: Besides, we build an environment using session-parallel mini-batches, where the agent interacts with multiple sessions simultaneously (see section 3).

Original: Baselines We choose various baseline agents for comparison, including: (1) BPR (Rendle et al., 2009), a pairwise ranking approach, widely applied as a benchmark; (2) GRU4Rec (Hidasi et al., 2016), a RNN-based approach for session-based recommendations with a BPR-max loss function (note that original GRU4Rec gives much lower purchase performance, thus we only use the clicked items from the same mini-batch as negative examples); (3) CKNN (Jannach & Ludewig, 2017), a session-based KNN method, which incorporates heuristics to sample similar past sessions as neighbors; (4) A3C-F and A3C-P, the base agents without imagination, using the click and purchase
Revised: Baselines We choose various baseline agents for comparison, including: (1) BPR (Rendle et al., 2009), a pairwise ranking approach, widely applied as a benchmark; (2) GRU4Rec (Hidasi et al., 2016), a RNN-based approach for session-based recommendations with a BPR-max loss function (note that original GRU4Rec gives much lower purchase performance, thus we only use the clicked items from the same mini-batch as negative examples); (3) CKNN (Jannach & Ludewig, 2017), a session-based KNN method, which incorporates heuristics to sample similar past sessions as neighbors; (4) A3C-F and A3C-P, the base agents without imagination, using the click and purchase

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: IRN-P
Revised: IRN-P

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: reward (-F) or only the purchase reward (-P); (5) IRN-F and IRN-P, the proposed models that augment
Revised: reward (-F) or only the purchase reward (-P); (5) IRN-F and IRN-P, the proposed models that augment

Original: A3C with imagiantion; (6) PRN-P, an A3C agent that reconstructs the previous observed trajectories (i.e., click/purchase sequences), using the purchase reward.
Revised: A3C with imagiantion; (6) PRN-P, an A3C agent that reconstructs the previous observed trajectories (i.e., click/purchase sequences), using the purchase reward.

Original: Architecture We implemented IRN via Tensorﬂow 3 , which will be released publicly upon acceptance.
Revised: Architecture We implemented IRN via Tensorﬂow 3 , which will be released publicly upon acceptance.

Original: We use grid search to tune hyperparameters of IRN and compared baselines on the validation set.
Revised: We use grid search to tune hyperparameters of IRN and compared baselines on the validation set.

Original: Speciﬁcally, the input state s t is passed through a LSTM with 256 units which takes in the one-hot representation of recent clicked/purchased items.
Revised: Speciﬁcally, the input state s t is passed through a LSTM with 256 units which takes in the one-hot representation of recent clicked/purchased items.

Original: The output of the LSTM layer is fed into two separate fully connected layers with linear projections, to predict the value function and the action.
Revised: The output of the LSTM layer is fed into two separate fully connected layers with linear projections, to predict the value function and the action.

Original: A softmax layer is added on top of the action output to generate the probability of 9167 actions.
Revised: A softmax layer is added on top of the action output to generate the probability of 9167 actions.

Original: The discounting value γ is 0.99.
Revised: The discounting value γ is 0.99.

Original: The imagination policy ˆ π is obtained from π using the ﬁxed target network, and the weights of ˆ π are updated after every 500 iterations.
Revised: The imagination policy ˆ π is obtained from π using the ﬁxed target network, and the weights of ˆ π are updated after every 500 iterations.

Original: Without special mentioned, TM employs the combination of breadth-2 and width-2 for internal planning.
Revised: Without special mentioned, TM employs the combination of breadth-2 and width-2 for internal planning.

Original: The imagination reconstruction is performed every one environment step.
Revised: The imagination reconstruction is performed every one environment step.

Original: The A3C updating is performed with immediate purchase reward (when found) or 3-step returns (when click reward is used).
Revised: The A3C updating is performed with immediate purchase reward (when found) or 3-step returns (when click reward is used).

Original: Besides, weights of
Revised: Besides, weights of

Original: IRN are initialized using Xavier-initializer (Glorot & Bengio, 2010) and trained via Adam optimizer (Kingma & Ba, 2014) with the learning rate and the batch size set to 0.001 and 128, respectively.
Revised: IRN are initialized using Xavier-initializer (Glorot & Bengio, 2010) and trained via Adam optimizer (Kingma & Ba, 2014) with the learning rate and the batch size set to 0.001 and 128, respectively.

Original: 5. R ESULTS
Revised: 5. R ESULTS

Original: We ﬁrst evaluate the top- K recommendation performance.
Revised: We ﬁrst evaluate the top- K recommendation performance.

Original: The experimental results are summarized in Table 1.
Revised: The experimental results are summarized in Table 1.

Original: From the purchase performance comparison, we get:
Revised: From the purchase performance comparison, we get:

Original: • A3C-P has already outperformed classical session-based recommenders (BPR, CKNN and GRU4Rec) on Recall metrics and achieved comparable results on MRR metrics.
Revised: • A3C-P has already outperformed classical session-based recommenders (BPR, CKNN and GRU4Rec) on Recall metrics and achieved comparable results on MRR metrics.

Original: GRU4Rec gives poor purchase performance, as it focuses on next-click prediction.
Revised: GRU4Rec gives poor purchase performance, as it focuses on next-click prediction.

Original: • Comparing IRN-P with A3C-P, we can see that the purchase (and click) performance can be signiﬁcantly improved with imagination reconstruction, demonstrating that IRN-P can guess what you like via internal planning and learn predictive representations.
Revised: • Comparing IRN-P with A3C-P, we can see that the purchase (and click) performance can be signiﬁcantly improved with imagination reconstruction, demonstrating that IRN-P can guess what you like via internal planning and learn predictive representations.

Original: • IRN-P consistently outperforms IRN-F, and A3C-P also outperforms A3C-F for purchase prediction.
Revised: • IRN-P consistently outperforms IRN-F, and A3C-P also outperforms A3C-F for purchase prediction.

Original: This demonstrates that purchase events can better characterize user interest, and the agents may be biased if clicks are used as reward.
Revised: This demonstrates that purchase events can better characterize user interest, and the agents may be biased if clicks are used as reward.

Original: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 Number of Iters (x2000) 0.
Revised: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 Number of Iters (x2000) 0.

Original: R e c a ll @validation Recall@
Revised: R e c a ll @validation Recall@

Original: IRN-P-0.0IRN-P-0.05IRN-P-0.1IRN-P-1.
Revised: IRN-P-0.0IRN-P-0.05IRN-P-0.1IRN-P-1.

Original: [Figure]
Revised: [Figure]

Original: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 Number of Iters (x2000) 0.
Revised: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 Number of Iters (x2000) 0.

Original: M RR @validation MRR@
Revised: M RR @validation MRR@

Original: IRN-P-0.0IRN-P-0.05IRN-P-0.1IRN-P-1.
Revised: IRN-P-0.0IRN-P-0.05IRN-P-0.1IRN-P-1.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: • Comparing PRN-P with A3C-P and IRN-P, we found that reconstructing the previous actual trajectories (i.e., click-streams) also improves the purchase performance (compared to A3C
Revised: • Comparing PRN-P with A3C-P and IRN-P, we found that reconstructing the previous actual trajectories (i.e., click-streams) also improves the purchase performance (compared to A3C

Original: - P
Revised: - P

Original: This
Revised: This

Original: is
Revised: is

Original: because that PRN-
Revised: because that PRN-

Original: P
Revised: P

Original: can
Revised: can

Original: learn better representations for
Revised: learn better representations for

Original: clicked items,
Revised: clicked items,

Original: and user
Revised: and user

Original: purchases
Revised: purchases

Original: are
Revised: are

Original: sometimes
Revised: sometimes

Original: contained in the click-
Revised: contained in the click-

Original: streams.
Revised: streams.

Original: Besides, IRN-
Revised: Besides, IRN-

Original: P outperforms
Revised: P outperforms

Original: PRN-
Revised: PRN-

Original: P
Revised: P

Original: , since PRN
Revised: , since PRN

Original: -P
Revised: -P

Original: introduces stronger supervision and may not
Revised: introduces stronger supervision and may not

Original: know
Revised: know

Original: what is the ﬁnal
Revised: what is the ﬁnal

Original: goal
Revised: goal

Original: while the
Revised: while the

Original: imagination
Revised: imagination

Original: reconstruction
Revised: reconstruction

Original: (without any real
Revised: (without any real

Original: trajectories) performs semi
Revised: trajectories) performs semi

Original: -supervised learning, which promotes more robust policy learning
Revised: -supervised learning, which promotes more robust policy learning

Original: From the click performance comparison, we get:
Revised: From the click performance comparison, we get:

Original: • GRU4Rec achieves excellent next-click performance (e.g., top-5 and top-10) compared to BPR and CKNN, as it models the session data via sequential classiﬁcation.
Revised: • GRU4Rec achieves excellent next-click performance (e.g., top-5 and top-10) compared to BPR and CKNN, as it models the session data via sequential classiﬁcation.

Original: • A3C-F performs much better than A3C-P and GRU4Rec.
Revised: • A3C-F performs much better than A3C-P and GRU4Rec.

Original: This indicates that RL-based recommenders trained on clicks can generate actions that better preserve the sequential property, possibly due to the accumulated click reward (of longer sessions).
Revised: This indicates that RL-based recommenders trained on clicks can generate actions that better preserve the sequential property, possibly due to the accumulated click reward (of longer sessions).

Original: • Somewhat interesting, IRN-P signiﬁcantly outperforms A3C-P, and gets comparable results
Revised: • Somewhat interesting, IRN-P signiﬁcantly outperforms A3C-P, and gets comparable results

Original: like IRN-F and A3C-F. This demonstrates that the IRN-P agent may learn to plan and reconstruct the previous clicked trajectories even when only the purchase reward is provided.
Revised: like IRN-F and A3C-F. This demonstrates that the IRN-P agent may learn to plan and reconstruct the previous clicked trajectories even when only the purchase reward is provided.

Original: Varying the degree of purchase reward sparsity We now explore the robustness of four RLbased recommenders to different purchase reward density.
Revised: Varying the degree of purchase reward sparsity We now explore the robustness of four RLbased recommenders to different purchase reward density.

Original: We randomly sample a d proportion of purchase events from the training set.
Revised: We randomly sample a d proportion of purchase events from the training set.

Original: The click events remain unchanged.
Revised: The click events remain unchanged.

Original: As shown in Table 2,
Revised: As shown in Table 2,

Original: A3C-F and IRN-F are robust to different purchase sparsity, since purchases are sometimes contained in the click sequences.
Revised: A3C-F and IRN-F are robust to different purchase sparsity, since purchases are sometimes contained in the click sequences.

Original: IRN using only the click reward for policy learning can also enhance the purchase prediction performance (see d = 0 ).
Revised: IRN using only the click reward for policy learning can also enhance the purchase prediction performance (see d = 0 ).

Original: While the performance of A3C-P degrades with sparser purchase reward, the proposed IRN-P achieves comparable performance; the imagination reconstruction promotes predictive learning of rewarding states.
Revised: While the performance of A3C-P degrades with sparser purchase reward, the proposed IRN-P achieves comparable performance; the imagination reconstruction promotes predictive learning of rewarding states.

Original: To our surprise, we have found that IRN-P performs well even without any external reward from the environment (i.e., predictive perception, see A3C-F and IRN-P with d = 0 ).
Revised: To our surprise, we have found that IRN-P performs well even without any external reward from the environment (i.e., predictive perception, see A3C-F and IRN-P with d = 0 ).

Original: Minimizing the imagination error of predictive trajectories over time enables the agent to learn sequential patterns in an unsupervised fashion.
Revised: Minimizing the imagination error of predictive trajectories over time enables the agent to learn sequential patterns in an unsupervised fashion.

Original: Figure 3 compares the performance of IRN-P on different reward sparsity setting, where one epoch contains nearly 5000 iterations.
Revised: Figure 3 compares the performance of IRN-P on different reward sparsity setting, where one epoch contains nearly 5000 iterations.

Original: We can observe that the performance of all models is gradually improved, and
Revised: We can observe that the performance of all models is gradually improved, and

Original: IRN-P with a larger d learns faster, indicating better exploration and exploitation.
Revised: IRN-P with a larger d learns faster, indicating better exploration and exploitation.

Original: Note that IRN-P with d = 0 will adversely decrease the performance due to the local over-training.
Revised: Note that IRN-P with d = 0 will adversely decrease the performance due to the local over-training.

Original: In extreme cases,
Revised: In extreme cases,

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: First
Revised: First

Original: [Table]
Revised: [Table]

Original: Purchase
Revised: Purchase

Original: [Table]
Revised: [Table]

Original: a ﬁnal purchase decision would be unknown, the imagination reconstruction may be applied without external reward, but we can use the click prediction performance for validation and early stopping.
Revised: a ﬁnal purchase decision would be unknown, the imagination reconstruction may be applied without external reward, but we can use the click prediction performance for validation and early stopping.

Original: Effectiveness of the trajectory manager We then analyze the effectiveness of different planners of the TM .
Revised: Effectiveness of the trajectory manager We then analyze the effectiveness of different planners of the TM .

Original: Table 3 shows the best results obtained with IRN-P when using alternative measurements.
Revised: Table 3 shows the best results obtained with IRN-P when using alternative measurements.

Original: Note that the purchase event in one session is usually the last user interaction, and "First" means that the second event is evaluated separately (the ﬁrst clicked item is used as the initial state).
Revised: Note that the purchase event in one session is usually the last user interaction, and "First" means that the second event is evaluated separately (the ﬁrst clicked item is used as the initial state).

Original: We can observe that, different planners equip the agent with different prediction capacity.
Revised: We can observe that, different planners equip the agent with different prediction capacity.

Original: For instance,
Revised: For instance,

Original: IRN-P with a larger n performs better on First and Click metrics, indicating that the agent with breadth- n planning focuses more on short-term rewards.
Revised: IRN-P with a larger n performs better on First and Click metrics, indicating that the agent with breadth- n planning focuses more on short-term rewards.