Paper:
Orthogonal Random Features
We present an intriguing discovery related to Random Fourier Features: in Gaussian kernel approximation, replacing the random Gaussian matrix by a properly scaled random orthogonal matrix significantly decreases kernel approximation error. We call this technique Orthogonal Random Features (ORF), and provide theoretical and empirical justification for this behavior. Motivated by this discovery, we further propose Structured Orthogonal Random Features (SORF), which uses a class of structured discrete orthogonal matrices to speed up the computation. The method reduces the time cost from
where d is the data dimensionality, with almost no compromise in kernel approximation quality compared to ORF. Experiments on several datasets verify the effectiveness of ORF and SORF over the existing methods. We also provide discussions on using the same type of discrete orthogonal structure for a broader range of applications.

Review:
This paper presents a very interesting finding, that using orthogonalized random projections significantly reduces the variance (error) of Gaussian kernel approximation. The author(s) further propose a structured variant to reduce the computational cost from O(d^2) to O(d log(d)) using the Walsh-Hadamard transformation. Some theoretical insights are presented to justify the procedure proposed. The gain over existing solutions is demonstrated with numerical experiments. 
This is a well-written paper, the motivations and main results are clearly stated in the main text, and technical details are put in the supplements. I do however have a few questions needs to to clarified.   1). Line 98, what does it mean by 'uniformly chosen random orthonormal basis' ? I do not think the set of orthonormal basis is compact, how can there be an uniform measure defined on it?  2). Line 100, again, 'uniformly from N(0,I_d)' does not make sense. And why does w_i follow a Gaussian distribution? It does not seem to be trivial. Please clarify.   3). For very high dimensional data (large d), the random projections will be effectively orthogonal. Does the proposed scheme offer any advantage other than the computational efficiency in such a setting.   4). Line 308 (Supp), the extra term f(z) seems very large, but it is omitted in Eq. (3) (line 108). And in Figure 3, there is this 'theoretical curve' for the variance. I wonder how D is set in this Figure and how is this theoretical curve computed?  And following might be some typos or negligences:   Line 86 : I presume the 'orthogonal' actually means 'orthonormal'?  Line 108 : Ratio of variance, shouldn't the 'K(x,y)' encapsulated by 'Var[]'?  Line 202-206 : I guess poly(x) means polynomial, but I could not find them in the main text or supplements.   ==================================================================   I have read the author(s)' rebuttal which is also well-written, and my major concerns have been clarified. My score remains unchanged and I recommend the area chair to accept this paper.