Paper:
Modeling sequential data has become more and more important in practice. Some applications are autonomous driving, virtual sensors and weather forecasting. To model such systems, so called recurrent models are frequently used. In this paper we introduce several new Deep Recurrent Gaussian Process (DRGP) models based on the Sparse Spectrum Gaussian Process (SSGP) and the improved version, called Variational Sparse Spectrum Gaussian Process (VSSGP). We follow the recurrent structure given by an existing DRGP based on a specific variational sparse Nyström approximation, the recurrent Gaussian Process (RGP). Similar to previous work, we also variationally integrate out the input-space and hence can propagate uncertainty through the Gaussian Process (GP) layers. Our approach can deal with a larger class of covariance functions than the RGP, because its spectral nature allows variational integration in all stationary cases. Furthermore, we combine the (Variational) Sparse Spectrum ((V)SS) approximations with a well known inducing-input regularization framework. For the DRGP extension of these combined approximations and the simple (V)SS approximations an optimal variational distribution exists. We improve over current state of the art methods in prediction accuracy for experimental data-sets used for their evaluation and introduce a new data-set for engine control, named Emission.

Review:
Overall Score: 7/10.
Confidence Score: 7/10.
Detailed Comments: This paper introduces various Deep Recurrent Gaussian Process (DRGP) models based on the Sparse Spectrum Gaussian Process (SSGP) models and the Variational Sparse Spectrum Gaussian Process (VSSGP) models. This is a good paper and proposed models are very sound so I recommend for acceptance although as main weakness I can say that is very technical so it can be difficult to follow. Adding more intuitive ideas, motivation and maybe a figure for each step would be a solution. Apart from that it is a really good paper, congratulations.
Related to: RNN models and Sparse Nystrom approximation.
Strengths: Models are very sound, solutions are solid, the proposed methodology is correct and the empirical results and experiments are valid and properly done.
Weaknesses: It is too difficult to follow and it is written in an extreme technical way. More intuitions and a proper motivation both in the abstract and introduction may be put in order to make the paper easier to read and, hence, more used by researchers and data scientists.
Does this submission add value to the ICLR community? : Yes it does, the experiments show the efficiency of the proposed methods in some scenarios and are valid methodologies.
Quality:
Is this submission technically sound?: Yes it is.
Are claims well supported by theoretical analysis or experimental results?: Experimental results prove empirically the methods and appendixes show the analysis performed in a clear and elegant way.
Is this a complete piece of work or work in progress?: Complete piece of work.
Are the authors careful and honest about evaluating both the strengths and weaknesses of their work?: Yes, and I would enfatize that I have liked that some experiments are won by other methods such as GP-LSTM, they are very honest.
Clarity:
Is the submission clearly written?: Yes, but it is difficult for newcomers due to the reasons that I have stated before.
Is it well organized?: Yes it is.
Does it adequately inform the reader?: Yes it is.
Originality:
Are the tasks or methods new?: Yes, they are sound.
Is the work a novel combination of well-known techniques?: Yes it is.
Is it clear how this work differs from previous contributions?: Yes.
Is related work adequately cited?: Yes, being a strength of the paper.
Significance:
Are the results important?: I would argue that they are and are a clear alternative to consider in order to solve these problems.
Are others likely to use the ideas or build on them?: If the paper is written in a more friendly way, yes.
Does the submission address a difficult task in a better way than previous work?: Yes I think.
Does it advance the state of the art in a demonstrable way?: Yes, empirically.
Arguments for acceptance: Models are very sound, solutions are solid, the proposed methodology is correct and the empirical results and experiments are valid and properly done
Arguments against acceptance: Clarity of the paper.
Minor issues and typos:
-> (V)SS not defined before being used.
-> Abstract should be rewritten adding a motivation and focusing more on the problems being solved and less in the details of the solutions.
-> Recurrent indexes that go backwards (i) of Eq. 1. should be explained why are going backwards before being used like that. Newcomers may be confused.
-> Section 2 writing style lacks a bit of cohesion, relating the paragraphs may be a solution.
-> Q is not defined in section 3.1 paragraph 1.
-> A valid covariance function must produce a PSD matrix, put that in section 3.1. 
-> I do not see how U marginalizes in Eq. 7, kind of confused about that, I think that it should be p(y|X,U).
-> Section 3.4 statistics should be explained.
Reading thread and authors response rebuttal decision:
=================================================
I consider that the authors have perfomed a good rebuttal and reading the other messages and the authors response I also consider that my issue with clarity is solved. Hence, I upgrade my score to 7 and recommend the paper for publication.