Paper:
We propose the Fixed Grouping Layer (FGL); a novel feedforward layer designed to incorporate the inductive bias of structured smoothness into a deep learning model. FGL achieves this goal by connecting nodes across layers based on spatial similarity. The use of structured smoothness, as implemented by FGL, is motivated by applications to structured spatial data, which is, in turn, motivated by domain knowledge. The proposed model architecture outperforms conventional neural network architectures across a variety of simulated and real datasets with structured smoothness.

Review:
"Towards a Deep Network Architecture for Structured Smoothness" proposes a new layer, dubbed FGL, specifically focused on structured smoothness.
The paper clearly discusses the proposed layer, introduces the necessary formulation, and demonstrates the effectiveness of this layer in a suite of practical applications in fMRI analysis. The writing is clear, the method is well formulated, and performance compared to relevant and recent benchmarks is strong. Thus most of the questions remaining are about the finer details of the proposed layer and method.
What is the sensitivity of FGL to non-optimal group proposals for A? Given the authors reference Aydore et. al., do randomized (or perhaps meta-optimized) groupings perform poorly? Are there any potential ways to jointly-or-iteratively learn the groupings? Is Ward clustering the primary practical method for getting the groupings?
Can the authors discuss a bit the relationship and potential tradeoffs between "early" segmentation architectures (such as FGL), and those often favored in semantic segmentation ("late" segmentation)? Is there a particular reason why "late" segmentation is particularly a poor choice in brain imaging?
My primary concerns come in relation to other methods for structured smoothness such as CRF/MRF, and the trade-offs and drawbacks of imperfect or downright wrong A, and alternative methods for practically finding good A matrices (potentially with an eye to applications outside fMRI). Addressing some or all of these in a section or two of the text body would potentially raise my score.
Some small errors:
Optimg -> Opting Introduction
Guassian -> Gaussian section 3.2
Potential useful references:
CRF as RNN https://arxiv.org/abs/1502.03240
Efficient Piecewise Training https://www.cv-foundation.org/openaccess/content_cvpr_2016/html/Lin_Efficient_Piecewise_Training_CVPR_2016_paper.html
Deep Learning Markov Random Field for Semantic Segmentation https://arxiv.org/abs/1606.07230
Pixel Adaptive Convolutional Neural Networks http://openaccess.thecvf.com/content_CVPR_2019/html/Su_Pixel-Adaptive_Convolutional_Neural_Networks_CVPR_2019_paper.html