Paper: Kernelised Normalising Flows (ICLR-2024)

Reviewer comment (Theory):
"This makes kernelised flows especially promising for learning in the low-data regime, as their model complexity naturally scales with dataset size and does not over-parametrise as much as neural networks", however there remains the previously mentioned caveat that the number of layers L is still a hyperparameter and increases model complexity. One can still over-parameterise with increasing L, or am I wrong?

Author response:
**""This makes kernelised flows especially promising for learning in the low-data regime, as their model complexity naturally scales with dataset size and does not over-parametrise as much as neural networks", however there remains the previously mentioned caveat that the number of layers L is still a hyperparameter and increases model complexity. One can still over-parameterise with increasing L, or am I wrong?"**

You are certainly correct in that an excessive number of layers can still lead to overparametrisation in our method. As the context of the sentence you referenced indicates, this only pertains to the "per-layer complexity" of kernelised flows. We added a qualification to the paragraph. Additonally, we would like to point that overparameterisation in the context of neural-net-based coupling layers is primarily attributed to deep neural networks within layers(per-layer complexity,k). In contrast use of shallow networks in coupling layers often lack the expressive power needed for effective scaling and translation parameters.

Reviewer comment (Evaluation):
Methods like FFJORD (\cf Figure 2 in FFJORD) report better results compared to the proposed approach (as shown in Table 1). The performance advantage of FFJORD is even more apparent in case of the challenging discontinuous checkerboard dataset in Figure 3 (supplementary). It is not clear if the proposed model has the modelling capacity to capture complex distributions.

Author response:
In the toy experiments(Figure 1), our focus was on illustrating the effectiveness of kernelisation under the same architecture.  With toy examples in the discontinued densities (figure 3) our objective was to showcase how composite kernels within the same architecture could introduce inductive biases. The performance could be further enhanced by exploring composition of kernels or with more data. Our comparisons in the toy datasets used the same architecture (with and without our kernelisation). 

 We acknowledge that FFJORD in some cases can yield better performance, but it should be noted that FFJORD is a continuous time normalising and the performance gain comes with very high training time as reported in Table 10. Throughout our comparisons(especially for the toy datasets), we primarily contrasted our approach with unkernelised versions of the same architectures, unless non-feasible, as seen in the low data regime experiments where overparameterisation led to overfitting, leading us to use FFJORD(having better performance than Glow or RealNVP as shown in Table 2) for meaningful comparison.

Reviewer comment (Experiments):
The proposed method is outperformed significantly by FFJORD, although FFJORD uses more parameters as reported in Table 4. Can the performance of the proposed method be improved in Table 4 by increasing the number of parameters?

Author response:
We would like to highlight that whilst FFJORD significantly outperforms on large-datasets (except Gas and Miniboone dataset) , it under performs when the number of datapoints is really low as shown in Table 4. Additionally, FFJORD takes significantly longer training periods even for small/toy datasets. This efficiency aspect is a crucial consideration, especially in scenarios where computational resources are limited or training time is a critical factor.
 
 Regarding the question of improving our method's performance by increasing the number of parameters, we posit that the kernelised architectures may derive advantages, especially when the increased parameters result from adding more layers(similar to neural-net counterparts) rather than learning an excess of auxiliary points. However, a more promising direction would be using the composite kernels (or even deep kernel learning). As discussed in the discontinued densities section, we briefly illustrate how simple models, inherently unable to capture complex distributions, can be readily augmented by composite kernels to enhance performance. We believe, this holds substantial promise for further advancements of kernelised architectures.

Reviewer comment (Experiments):
For the experiments on the 2D toy datasets in Table 1, it is not clear which NN based approach is employed. Furthermore, the number of data samples used for training for all models in Table 1 should also be reported.

Author response:
In the experiments on the 2D toy datasets presented in Table 1, we employed Glow-based architectures for both approaches. We trained these datasets for 1000 training steps, with the data synthesised at every training step (as done in existing implementations). We will add these details in an updated version of the manuscript.

Reviewer comment (Experiments):
The method is motivated by alluding to medical settings in Section 1 and 5.4, where data availability is often limited. However, the method is never applied to any medical data.

Author response:
We have now incorporated an evaluation of our method's performance on a medical dataset, demonstrating its practical applicability in a real life application. Please refer to Section D of the appendix for the results..

Reviewer comment (Experiments):
It is not clear if the proposed model can be applied to complex data distributions such as images. The qualitative examples of samples generated by the proposed model trained on the Kuzushiji-MNIST dataset as shown in Figure 4-6 are not promising.

Author response:
The primary objective of our VAE experiments was to demonstrate improvement over the Evidence Lower Bound (ELBO) in the specified settings. It's important to note that the small latent dimension in our experimental setup impacts the visualisation quality and a bigger latent dimension could potentially improve the quality of generated samples.

Reviewer comment (Experiments):
The paper should include a more detailed comparison with FFJORD especially in case of 2D synthetic datasets such as the discontinuous checkerboard dataset.

Author response:
In the toy experiments(Figure 1), our focus was on illustrating the effectiveness of kernelisation under the same architecture.  With toy examples in the discontinued densities (figure 3) our objective was to showcase how composite kernels within the same architecture could introduce inductive biases. The performance could be further enhanced by exploring composition of kernels or with more data. Our comparisons in the toy datasets used the same architecture (with and without our kernelisation). 

 We acknowledge that FFJORD in some cases can yield better performance, but it should be noted that FFJORD is a continuous time normalising and the performance gain comes with very high training time as reported in Table 10. Throughout our comparisons(especially for the toy datasets), we primarily contrasted our approach with unkernelised versions of the same architectures, unless non-feasible, as seen in the low data regime experiments where overparameterisation led to overfitting, leading us to use FFJORD(having better performance than Glow or RealNVP as shown in Table 2) for meaningful comparison.

Reviewer comment (Experiments):
The paper should discuss in more detail if the performance of the proposed approach can be further boosted by scaling the number of parameters.

Author response:
We would like to highlight that whilst FFJORD significantly outperforms on large-datasets (except Gas and Miniboone dataset) , it under performs when the number of datapoints is really low as shown in Table 4. Additionally, FFJORD takes significantly longer training periods even for small/toy datasets. This efficiency aspect is a crucial consideration, especially in scenarios where computational resources are limited or training time is a critical factor.
 
 Regarding the question of improving our method's performance by increasing the number of parameters, we posit that the kernelised architectures may derive advantages, especially when the increased parameters result from adding more layers(similar to neural-net counterparts) rather than learning an excess of auxiliary points. However, a more promising direction would be using the composite kernels (or even deep kernel learning). As discussed in the discontinued densities section, we briefly illustrate how simple models, inherently unable to capture complex distributions, can be readily augmented by composite kernels to enhance performance. We believe, this holds substantial promise for further advancements of kernelised architectures.

Reviewer comment (Experiments):
The paper should discuss in more detail if the proposed approach is applicable to complex datasets such as images.

Author response:
The primary focus in this work was on making normalising flows work well in the low-data regime, where we believe our method has significant potential and practical applications. Notably, our proposed approach serves as a direct replacement for MLP-based coupling layers, traditionally employed for non-image data. In contrast, Glow-based models use to the convolutional neural networks (CNNs) for image generation. Adapting the current method for convolutional neural networks is beyond the scope of current study and reserved for future works. 

 However, we showcase image generation via applicability of our kernelisation to the MixerFlow architecture, a coupling layer architecture for image generation that does not rely on convolutions for weight sharing. Our kernelised method can be directly applied to coupling layer architectures for image generation that do not rely on convolutions for weight sharing. The results of this application are presented in appendix E. It's important to note that for fair comparison, we maintained the same architecture, demonstrating that kernelisation can be successfully applied to image generation tasks with improvements over the neural-net-based architecture. Additionally, we would like to highlight that visually better image generation with flows require larger models and are typically trained on multiple GPUs. Our tabular models are trained on CPUs whereas image models are trained on a single Colab GPUs. Consequently, we have evaluated smaller models for image generation. Training bigger models with fine-grained hyperpaprameter tuning can help generate better visualisation, which is typically the case with flow-based image generation.

Reviewer comment (Experiments):
In Fig 1, authors should add results from more methods, like FFJORD.

Author response:
**"In Fig 1, authors should add results from more methods, like FFJORD"**

In the toy experiments(Figure 1), our focus was on illustrating the effectiveness of kernelisation under the same architecture.  With toy examples in the discontinued densities (figure 3) our objective was to showcase how composite kernels within the same architecture could introduce inductive biases; therefore we did not add any other baselines/FFJORD. Whilst we acknowledge that FFJORD may yield superior performance in certain cases, it is important to note that FFJORD represents a continuous-time normalising flow solved via neural ODEs and takes significantly longer to train as shown in Table 10. Throughout our comparisons, we primarily contrasted our approach with unkernelised versions of the same architectures, unless not feasible, as seen in the low data regime experiments where overparameterisation led to overfitting, leading us to use FFJORD(having better performance than Glow or RealNVP) for meaningful comparison.

Reviewer comment (Experiments):
How does the model work on a spiral 2D dataset?

Author response:
**"How does the model work on a spiral 2D dataset?"**

Just like with other toy datasets, kernelisation contributes to the improved performance of the model on the spiral dataset.
| Method      | Ours            | params |
|-------------|-----------------|--------|
| **Ours**    | **-3.24**      | **1.8K**| 
| NN-based      | -3.29          | 18.4K   | 

 We will include these results in an updated version of the manuscript.

Reviewer comment (Experiments):
In the main paper, we do not have any image datasets. In the appendix, we have Kuzushiji-MNIST dataset. Authors should evaluate the model on MNIST, CIFAR, and CELEBA data especially when we compare methods with Glow.

Author response:
**"In the main paper, we do not have any image datasets. In the appendix, we have Kuzushiji-MNIST dataset. Authors should evaluate the model on MNIST, CIFAR, and CELEBA data special when we compare methods with Glow."**

 The primary focus in this work was on making normalizing flows work well in the low-data regime, where we believe our method has significant potential and practical applications. Notably, our proposed approach serves as a direct replacement for MLP-based coupling layers, traditionally employed for non-image data. In contrast, Glow-based models use to the convolutional neural networks (CNNs) for image generation. Adapting the current method for convolutional neural networks is beyond the scope of current study and reserved for future works. 

 However, we showcase image generation via applicability of our kernelisation to the MixerFlow architecture, a coupling layer architecture for image generation that does not rely on convolutions for weight sharing. Our kernelised method can be directly applied to coupling layer architectures for image generation that do not rely on convolutions for weight sharing. The results of this application are presented in appendix E. It's important to note that for fair comparison, we maintained the same architecture, demonstrating that kernelisation can be successfully applied to image generation tasks with improvements over the neural-net-based architecture. Additonally, we would like to highlight that visually better image generation with flows require larger models and are typically trained on multiple GPUs. Our tabular models are trained on CPUs whereas image models are trained on a single Colab GPUs. Consequently, we have evaluated smaller models for image generation. Training bigger models with fine-grained hyperparameter tuning can help generate better visualisation, which is typically the case with flow-based image generation.

Reviewer comment (Experiments):
How the method works on large image datasets.

Author response:
Thank you for your response. We would like to gently point out that our work focuses on the kernelisation of coupling layer architectures, encompassing not only Glow but also RealNVP, VAE with flow-based prior and posterior, and MixerFlow. 

Additionally, we would like to clarify that we have evaluated our kernelisation on large-scale datasets(e.g. 1.6 million examples in the Power dataset) as well as small datasets (500 examples subsets or 3240 examples on the Biomarker dataset) to show the versatility and efficacy of our method.