From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper15/Authors
title: Clarification 
comment: Thank you for your question. We consider reaching problems in a very general view. Reaching problems are inherently spatiotemporal since we try to reach a given state (space) as fast as possible by making a good decision at every time step  (temporal axis). We achieve this by using a predictive model that given an observation and an action distribution estimates the state change distribution (i.e., the evolution of space over time). Note that our approach to model the full state change distribution through a quantile network can be applied to other spatiotemporal problems were the stochastic aspect of change is important.

From: ~Ransalu_Senanayake1
comment: Interesting paper! Can you briefly clarify why this is a spatiotemporal problem? Does the decision making process considers both space and time? Thanks!
title: Clarification

From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper15/Authors
title: Thank you for the constructive feedback!
comment: Thank you for the constructive feedback. Here a short answer to your concerns:

- The methodology section [...]

"Q" in this paper stands for "Q"uantile function and has no relation to Q(uality)-function in Q-learning. In contrast to related work [4,5] we do not perform Q-learning but rather have an actor-critic approach (since Q-learning is by design only applicable to discrete action spaces). We follow a typical actor critic set-up [14] with a network trained as critic to approximate the value function (trained with L_c) and a network trained to (implicitly) model the action distribution (trained with L_a und L_mon). We see that the naming of the quantile function modeling the action distribution is subobtimal and will change this in the revision. We can also include pseudo code to clarify the implementation. Note that our code is available at https://goo.gl/rMuadg (Anonymous Google Drive for double blind review), in case you are interested in the exact implementation.

 - The goal of [...]

The goal of the agent is to reach the 0 point (origin of the d-dimensional space). We agree that this should be stated and will include it in the revision. 

- Learning state-change distributions [...]

Our agent takes advantage of spatial information (vector instead of scalar reward) to reach a decision (action) at every time step (temporal axis). The paper addresses the spatiotemporal decision making problem in a very general view and proposes a solution to improve sample complexity. Additionally, as you mention, we show how a state change predictor (PCPN in the paper) can be learned using quantile regression. This technique can be applied to any stochastic spatio-temporal process model, where modeling the full next-state distribution is beneficial. We will make this relation clearer in the revision.

- Notation: Why not use "x"  or "s", instead of "p", for position, which means state in this context? Since the paper deals with different types of probability distributions, using "p" for position sometimes is confusing.

Thanks for the feedback. We agree and will change this.

- The meaning of "alignment" is not clear at first. Alginment of vectors can be interpreted in terms of zeroing the angle between them, which only makes sense in an inner product space. However, alignment in this paper means matching dimensionality between two vector spaces. That should be clarified in the introduction.

Thanks for the feedback. We agree that a short definition is needed to avoid confusion. We will include this in the revision.

- Not clear why general metric spaces with dimensionality greater than 3 are mainly mathematical constructs as stated in the conclusion section.

We live in a 3 dimensional world, therefore we can easily imagine scenarios in up to 3 dimensional metric spaces.  The advantage of using vector rewards increases with the dimensionality and at some point training with vector rewards is not only beneficial but essential to learn something. What we wanted to say with the sentence is that even though one might struggle to imagine a high dimensional space, the concept is essential in mathematics, physics and computer science.

From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper15/AnonReviewer2
evaluation: This paper presents a reinforcement learning method based on quantile regression and vector rewards. The method applies neural networks to learn action distributions via quantile regression to guide an agent in reinforcement learning tasks. The idea of combining vector rewards and quantile regression is interesting. However, there are some presentation issues that difficult reading the paper and understanding the value of its contributions at first.

Major issues:
- The methodology section is a bit convoluted. The meaning of "Q_\theta" and its relation to the usual action value function "Q" in standard RL is not clear in the text. The connections between the different loss functions and optimisation goals is also confusing. Replacing parts of the text by an algorithm table and explaining it step by step might help.
 - The goal of the agent in Sec. 4's experiments is not clear. I assume it is simply to reach a given goal location in the hypercube, but that's not clearly specified.
- Learning state-change distributions, as the PCPN does, can be useful when dealing with spatiotemporal processes. However, a discussion on how the contributions of this paper connect to the area of spatiotemporal modelling and decision making is missing.

Minor issues:
- Notation: Why not use "x"  or "s", instead of "p", for position, which means state in this context? Since the paper deals with different types of probability distributions, using "p" for position sometimes is confusing.
- The meaning of "alignment" is not clear at first. Alginment of vectors can be interpreted in terms of zeroing the angle between them, which only makes sense in an inner product space. However, alignment in this paper means matching dimensionality between two vector spaces. That should be clarified in the introduction.
- Not clear why general metric spaces with dimensionality greater than 3 are mainly mathematical constructs as stated in the conclusion section.

From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper15/AnonReviewer2
relevance: 3: borderline
impact: 3: borderline
novelty: 4: high