Paper:
Convolutional networks are not aware of an object's geometric variations, which leads to inefficient utilization of model and data capacity. To overcome this issue, recent works on deformation modeling seek to spatially reconfigure the data towards a common arrangement such that semantic recognition suffers less from deformation. This is typically done by augmenting static operators with learned free-form sampling grids in the image space, dynamically tuned to the data and task for adapting the receptive field. Yet adapting the receptive field does not quite reach the actual goal -what really matters to the network is the effective receptive field (ERF), which reflects how much each pixel contributes. It is thus natural to design other approaches to adapt the ERF directly during runtime. In this work, we instantiate one possible solution as Deformable Kernels (DKs), a family of novel and generic convolutional operators for handling object deformations by directly adapting the ERF while leaving the receptive field untouched. At the heart of our method is the ability to resample the original kernel space towards recovering the deformation of objects. This approach is justified with theoretical insights that the ERF is strictly determined by data sampling locations and kernel values. We implement DKs as generic drop-in replacements of rigid kernels and conduct a series of empirical studies whose results conform with our theories. Over several tasks and standard base models, our approach compares favorably against prior works that adapt during runtime. In addition, further experiments suggest a working mechanism orthogonal and complementary to previous works.

Review:
This work presents the idea of deformable kernels (DKs). As opposed to rigid kernels in standard convolutional networks, DKs allow each of their grid locations to be moved around in a larger kernel field. The offset by which a DK grid cell is moved is computed conditioned on the input to the network. To motivate the idea of DKs, the authors give some background on convolution, receptive and effective receptive fields (ERFs). The authors argue that since ERFs are spatially porous and irregularly distributed, one way to model them is to convolve square grids of input with DKs, which are composed of samples drawn from larger kernels. The authors define the concept of global and local DKs. They further contrast DKs with spatial sampling (deformable convolutions) and argue that although conceptually similar, both approaches are complementary to each other and can be used in combination in practice. Numerical experiments show competitive performance of DKs on image classification and object detection tasks. In the end empirical analysis is performed to analyze the characteristics of DKs.
I am unfamiliar with prior work in this direction, but the idea of DKs seems to be conceptually appealing and as the authors point-out, their approach can be seen as an alternative to spatial sampling for modeling deformations. Unfortunately the authors get hand-wavy when in Sec 3.2, they claim that the idea of subsampling kernels "roughly generalizes" to non-linear networks. I don't see how they can generalize what they present beyond piece-wise linear networks. I appreciate the effort to give a background on ERFs and describe (local and global) DKs, but in my opinion, technical sections of the paper partly very obscure. For instance it is not entirely clear how the offset predictors are trained, how exactly the sampling is used, details of architecture etc. 
Empirically the method does not seem to offer a significant performance boost. Also, while the authors sell the idea of subsampling kernels, but the finding that kernel sizes beyond 4x4 don't seem to offer any benefit make the idea practically questionable. 
The idea of DKs seems relevant, but both conceptually and empirically it seems very close to deformable convolutions. The authors need to clearly present their work, including its shortcomings (i.e., generalization or not beyond linear networks).