pwc-archive/papers-with-abstracts huggingface dataset:
[Re] Spatial-Adaptive Network for Single Image Denoising
Authors: Sami Menteş, Furkan Kınlı, Barış Özcan, Furkan Kıraç
Date: 2021-01-31 00:00:00
Proceeding: RC 2020
Tasks: CPU, Denoising, GPU, Image Denoising, SSIM

Scope of Reproducibility 

The original paper proposes an encoder-decoder structure exploiting a residual spatialadaptive block and a context block to capture multi-scale information for achieving the state-of-the-art on real and synthetic noise removal. 

Methodology 

We have implemented the model, namely SADNet, from scratch in PyTorch as described in the paper, and also adopted the training loop and proposed blocks from the authorʼs code. Since the weight initialization of proposed blocks was not implicitly defined in the paper, we have decided to use the default initialization method for convolutional layers in PyTorch (i.e. Kaiming). Experiments have been completed on a single RTX 2080 Ti in 3 days for each, and it requires ∼3GB GPU memory for training, and ∼8GB CPU memory for loading the data, due to the file structure of datasets. 

Results 

We have achieved to reproduce the results qualitatively and quantitatively on synthetic and noise removal tasks. SADNet has the capacity to learn to remove the synthetic and real noise in images, and it produces visually-plausible outputs even after a few epochs. Moreover, we have employed SSIM and PSNR metrics to measure the quantitative performance for all settings. The quantitative results on both tasks are on-par when compared to the reported results in the paper.

What was easy 

The code was open-source, and implemented in PyTorch, hence adopting the training loop and proposed blocks to our implementation facilitated our reproduction study. The loss function is straightforward and the architecture has a U-Net-like structure, so that we could achieve to implement the architecture in a fair time. 

What was difficult 

Due to the lack of compatibility with the current versions of PyTorch and TorchVision and the dependency on an external CUDA implementation of deformable convolutions, we have encountered several issues during our implementation. Then, we have considered to re-implement residual spatial-adaptive block and context block from scratch for deferring these dependencies, however, we could not achieve it just by referring to the paper in limited time. Therefore, we have decided to directly use the provided blocks as in the authorʼs code. 

Communication with original authors 

We did not make any contact with the authors since we achieved to solve the issues encountered during the implementation of SADNet by examining the authorʼs code.
[Re] Deep Fair Clustering for Visual Learning
Authors: Pauline Baanders, Chris Al Gerges, Nienke Reints, Tobias Teule
Date: 2021-01-31 00:00:00
Proceeding: RC 2020
Tasks: Clustering

Scope of Reproducibility

 Deep Fair Clustering (DFC) aims to provide a clustering algorithm that is fair, clusteringfavourable, and which can be used on high-dimensional and large-scale data. In existing frameworks there is a trade-off between clustering quality and fairness. In this report we aim to reproduce a selection of the results of DFC; using two of four datasets and all four metrics that were used in the original paper, namely accuracy, Normalized Mutual Information (NMI), balance and entropy. We use the authorsʼ implementation and check whether it is consistent with the description in the paper. As extensions to the original paper we look into the effects of 1) using no pretrained cluster centers, 2) using different divergence functions as clustering regularizers and 3) using non-binary/corrupted sensitive attributes. 

Methodology 

The open source code of the authors has been used. The datasets and data-preprocessing has been done with our code, since the authors did not provide the datasets in their code. Also the pretrained Variational Autoencoder (VAE) dataset had to be re-implemented for the Color Reverse MNIST . For the extensions we wrote extra functions. For measuring the influence of discarding the pretrained cluster centers, the code was already provided by the authors. 

Results 

For the MNIST-USPS dataset, we report similar accuracy and NMI values that are within 1.2% and 0.5% of the values reported in the original paper. However, the balance and entropy differed significantly, where our results were within 73.1% and 30.3% of the original values respectively. For the Color Reverse MNIST dataset, we report similar values on accuracy, balance and entropy, which are within 5.3%, 2.6% and 0.2% respectively. Only the value of the NMI differed significantly, name within 12.9% of the original value In general, our results still support the main claim of the original paper, even though on some metrics the results differ significantly.

What was easy 

The open source code of the authors was beneficial; it was well structured and ordered into multiple files. Furthermore, the code to use randomly initialized instead of pretrained cluster centers was already provided. 

What was difficult 

First of all, the main difficulty in reproducing the paper was caused by the coding style; due to the lack of comments it was difficult to get a good understanding of the code. Secondly, we were required to download the data ourselves. However, these filenames and labels did not correspond to the included txt-files by the authors. Therefore, the model did not learn and we regenerated train_mnist.txt and train_usps.txt. Finally, the authors only included pretrained models for the MNIST-USPS dataset. As a consequence, we had to pre-train some parts of the DFC algorithm for the Color Reverse MNIST dataset.
[Re] Parameterized Explainer for Graph Neural Network
Authors: Lars Holdijk, Maarten Boon, Stijn Henckens, Lysander de Jong
Date: 2021-01-31 00:00:00
Proceeding: RC 2020
Tasks: Graph Classification, Graph Neural Network

Scope of Reproducibility 

In this work we perform a replication study of the paper Parameterized Explainer for Graph Neural Network. The replication experiment focuses on three main claims: (1) Is it possible to reimplement the proposed method in a different framework? (2) Do the main claims with respect to the GNNExplainer hold? (3) Is the used evaluation method a valid method for explaining the classification decision by Graph Neural Networks? 

Methodology 

The authorsʼ TensorFlow code was largely used as starting point for our reimplementation in PyTorch. However, large parts of the evaluation setup were missing and differences were found between the listed configurations in the paper and the code. As a result, our codebase contains a large portion of novel code and introduces a different method for tracking experimental configurations. Using the new codebase all experiments are replicated. In addition to this, a short ablation study is performed. 

Results 

Due to numerous inconsistencies between code and paper, it is not possible to replicate the original results using the paper alone. With help of the original codebase, a number of the original results can be retrieved. The main comparison claim of the paper, to improve over the preceding GNNExplainer, does hold. However, after performing the replication experiments, some questions regarding the validity of the used evaluation setup in the original paper remain. 

What was easy 

The method proposed by the authors for explaining the Graph Neural Networks is easy to comprehend and intuitive. Re-implementation of the method is straightforward using a modern deep learning framework. The datasets used for the experimental setup were all provided together with their codebase.

What was difficult 

The main difficulty arose from the difference between the experimental configurations discussed in the paper and implemented in the code. There were a number of small inconsistencies (eg. incorrect hyperparameter settings), but also some major ones (eg. using batch-normalization in training mode during evaluation). This issue was worsened by the fractured reporting of configurations in the code. 

Communication with original authors 

Contact was made with the authors on two occasions. During the first exchange the authors confirmed a number of clarifying questions and confirmed that the configurations as presented in the codebase were to be used instead of those provided in the paper. In the second exchange our reservations concerning the used experimental evaluation were conveyed to the authors. The authors did not share our concerns.
[Re] Reproducing 'Identifying through flows for recovering latent representations'
Authors: Geerten Rijsdijk, Bart de Rooij, Wouter Zwerink, Hidde Lekanne gezegd Deprez
Date: 2021-01-31 00:00:00
Proceeding: RC 2020
Tasks: Representation Learning

Scope of Reproducibility 

The authors claim to introduce a model for recovering the latent representation of observed data, that outperforms the state-of-the-art method for this task; namely the iVAE model. They claim that iFlow outperforms iVAE both in preservation of the original geometry of source manifold and correlation per dimension of the latent space. 

Methodology 

To reproduce the results of the paper, the main experiments are reproduced and the figures are recreated. To do so, we largely worked with code from the repository belonging to the original paper. We added plotting functionality as well as various bug fixes and optimisations. Additionally, attempts were made to improve the iVAE by making it more complex and fixing a mistake in its implementation.We also tried to investigate possible correlation between the structure of the dataset and the performance. All code used is publicly available at https://github.com/HiddeLekanne/Reproducibility-Challenge-iFlow. 

Results 

The obtained mean and standard deviation of the MCC over 100 seeds are within 1 percent of the results reported in the paper. The iFlow model obtained a mean MCC score of 0.718 (0.067). Efforts to improve and correct the baseline increased the mean MCC score from 0.483 (0.059) to 0.556 (0.061). The performance, however, remains worse than the performance of iFlow, further supporting the authorsʼ claim that the iFlow implementation is correct and more effective than iVAE. 

What was easy 

The GitHub repository associated with the paper provided most necessary code and ran with only minor changes. The code included all model implementations and data generation. The script that was used to obtain results was provided, which allowed us to determine which exact hyperparameters were used with experiments on the iFlow models. Overall, the code was well organised and the structure was easy to follow.

What was difficult 

The specific versions of the Python libraries used were unknown, which made it infeasible to achieve the exact results from the paper when running on the same seeds. The code used to create figures 1-3 in the original paper was missing and had to be recreated. Furthermore, the long time the models needed to train made experimentation with e.g., different hyperparameters challenging. Finally, the code was largely undocumented. 

Communication with original authors 

Communication with the authors was attempted but could not be established.
[Re] Warm-Starting Neural Network Training
Authors: Amirkeivan Mohtashami, Ehsan Pajouheshgar, Klim Kireev
Date: 2021-01-31 00:00:00
Proceeding: RC 2020
Tasks: Data Augmentation

Scope of Reproducibility 

We reproduce the results of the paper ”On Warm-Starting Neural Network Training.” In many real-world applications, the training data is not readily available and is accumulated over time. As training models from scratch is a time-consuming task, it is preferred to use warm-starting, i.e., using the already existing models as the starting point to obtain faster convergence. This paper investigates the effect of warm-starting on the final modelʼs performance. It identifies a noticeable gap between warm-started and randomly-initialized models, hereafter referenced as the warm-starting gap. Furthermore, they provide a solution to mitigate this side-effect. In addition to reproducing the original paperʼs results, we propose an alternative solution and assess its effectiveness. 

Methodology 

We reproduced almost every figure and table in the main text and some of those in the appendix. We used our implementation to produce these results. In case of a mismatch of the results, we also investigated the cause and proposed possible explanations. We mainly used GPUs to train our models using infrastructure offered by public clouds and those that were available to us privately. 

Results 

Most of our results closely match the reported results in the original paper. Therefore, we confirm that the warm-starting gap exists in certain settings and that the ShrinkPerturb method successfully reduces or eliminates this gap. However, in some cases, we were not able to completely reproduce their results. By investigating the root of such mismatches, we provide another solution to avoid this gap. In particular, we show that data augmentation also helps to reduce the warm-starting gap. 

What was easy 

The experiments described in the paper were based on regular training of neural networks on a portion of widely-used datasets, possibly from a pre-trained model. Therefore implementing each experiment was relatively easy to do. Furthermore, since many of the parameters were reported in the original paper, we did not need much tuning in most experiments. Finally, it is straightforward to implement and use the proposed solution. 

What was difficult 

Though implementing each experiment is relatively simple, the numerosity of experiments proved to be slightly challenging. In particular, each of the online experiments in the original setting requires training a deep network to convergence more than 30 times. In these cases, we sometimes changed the settings, sacrificing granularity to reduce computation time. However, these changes did not affect the interpretability of the final results. 

Communication with original authors 

We briefly communicated with the authors to clarify the experimentsʼ details, such as the convergence conditions.
[Re] Reproducibility study - Does enforcing diversity in hidden states of LSTM-Attention models improve transparency?
Authors: Frank Verhoef, Pieter Bouwman, Yun Li, Rogier van der Weerd
Date: 2021-01-31 00:00:00
Proceeding: RC 2020
Tasks: Diversity, Navigate

Scope of Reproducibility

For this reproducibility study, we focus on the main claims made

in this paper:
• The attention weights in standard LSTM attention models do not provide faithful and plausible explanations for its predictions. This is potentially because the
conicity of the LSTM hidden vectors is high.
• Two methods can be applied to reduce conicity: Orthogonalization and Diversity Driven Training. When applying these methods, the resulting attention weights offer more faithful and plausible explanations of the modelʼs predictions, without sacrificing model performance.

Methodology

The paper includes a link to a repository with the code used to generate its results. We follow four investigative routes: (i) Replication: we rerun experiments on datasets from the paper in order to replicate the results, and add the results that are missing in the paper; (ii) Code review: we scrutinize the code to validate its correctness; (iii) Evaluation methodology: we extend the set of evaluation metrics used in the paper with the LIME method, in an attempt to resolve inconclusive results; (iv) Generalization to other architectures: we test whether the authorsʼ claims apply to variations of the base model (more complex forms of attention and a BiLSTM encoder).

Results

We confirm that the Orthogonal and Diversity LSTM achieve similar accuracies as the Vanilla LSTM, while lowering conicity. However, we cannot reproduce the results of several of the experiments in the paper that underlie their claim of better transparency. In addition, a close inspection of the code base reveals some potentially problematic inconsistencies. Despite this, under certain conditions, we do confirm that the Orthogonal and Diversity LSTM can be useful methods to increase transparency. How to formulate these conditions more generally remains unclear and deserves further research. The single input sequence tasks appear to benefit most from the methods. For these tasks, the attention mechanism does not play a critical role for achieving performance.

What was easy/difficult

The codebase of the authors is accessible and can be run easily, with good facilities to prepare datasets and define configurations. The Orthogonalization and Diversity Training methods are well explained in the paper and mostly cleanly implemented. The larger datasets (Amazon and CNN) are difficult to run due to memory requirements and compute times. The codebase can be hard to navigate, a consequence of the choice to accommodate a large variation of models and datasets in one framework.

Communication with original authors

We reached out to the authors on a fundamental but unexplained choice in the model architecture but unfortunately did not hear back before the deadline of our assignment.
[Re] Neural Networks Fail to Learn Periodic Functions and How to Fix It
Authors: Mayur Arvind, Mustansir Mama
Date: 2021-01-31 00:00:00
Proceeding: RC 2020
Tasks: Sentiment Analysis

Scope of Reproducibility

Neural Networks Fail to Learn Periodic Functions and How to Fix It [1] demonstrates experimentally that standard activations such as ReLU, tanh, sigmoid and their variants all fail to learn how to extrapolate simple periodic functions. The original paper goes on to propose a new activation, which the authors name the snake function. The central claims of the paper are two-fold: (1) The properties of the activation functions are carried over to the neural networks. A tanh network will be smooth and extrapolates to a constant function, while ReLU extrapolates in a linear way. Standard neural networks with conventional activation functions are insufficient for extrapolating periodic functions. (2) The proposed activation function manages to learn periodic functions while being able to optimize as well as conventional activation functions. While both experimental proof and theoretical justifications are provided for the claims, we shall only be concerned with testing the claims via experimental means.

Methodology

While one of the authors was contacted to clarify certain difficulties, the reproduction of all experiments was completed using only the information provided in the paper. With one exception, the links to all datasets used were also provided in the original paper. This allowed us to implement most experiments from scratch.

Results

We were able to successfully replicate experiments supporting the central claim of the paper, that the proposed snake non-linearity can learn periodic functions. We also analyze the suitability of the snake activation for other tasks like generative modeling and sentiment analysis.

What was easy

Many experiments included descriptions of the neural network architectures and graphs showcasing performance, giving us a clear benchmark to compare our results against.

Links to datasets for all experiments, barring one, were also included in the paper itself.

What was difficult

Data for the human body temperature experiment was not available. Proper implementation details were not given for initializing the weights in neural networks with snake and using snake with RNNs.

Communication with original authors

Liu Ziyin, one of the authors, was contacted to provide the dataset used for the human body temperature experiment, elaborate upon the implementation of variance correction and provide the implementation of RNNs using snake. Liu provided the GitHub link to the authorsʼ original code for the human body temperature, market index, and extrapolation experiments. Liu also provided an explanation on how to implement variance correction. While the code for the RNN implementation using the snake activation was not made public, a screenshot of the same was provided. We thank the authors for their assistance.
[Re] Explaining Groups of Points in Low-Dimensional Representations
Authors: Rajeev Verma, Jim Wagemans, Paras Dahal, Auke Elfrink
Date: 2021-01-31 00:00:00
Proceeding: RC 2020
Tasks: Clustering, Interpretable Machine Learning

Scope of Reproducibility

This report covers our reproduction of the paper ʼExplaining Low dimensional Representationʼ [1] by Plumb et al. In this paper, a method (Transitive Global Translations, TGT) is proposed for explaining different clusters in low dimensional representations of high dimensional data. They show their method outperforms the Difference Between the Means (DBM) method, is consistent in explaining differences with few features and matches real patterns in data. We verify these claims by reproducing their experiments and testing their method on new data. We also investigate the use of more complex transformations to explain differences between clusters.

Methodology

We reproduce the original experiments using their source code. We also replicate their findings by re-implementing the authorsʼ method in PyTorch [2] and evaluating on two of the dataset used in the paper and two new ones. Furthermore, we compare TGT with our own extension of TGT, which uses a larger class of transformations.

Results

We were able to reproduce their results using their code, yielding mostly similar results. TGT generally outperforms DBM, especially when explanations use few features. TGT is consistent in terms of the features to which it attributes cluster differences, across different sparsity levels. TGT matches real patterns in data. When extending the types of functions used for explanations, performance did not improve significantly, suggesting translations make for adequate explanations. However, the scaling extension shows promising performance on the modified synthetic data to recover the original signal.

What was easy

The easiest part was running the existing code with the pre-trained model files. The original authors had set up their code base in an organized manner with clear instructions.

What was difficult

The first difficulty that we encounter was finding the right environment. The source code depends on deprecated functionality. The clustering method they used, had to be re implemented for us to use it in our replication. Another difficulty was the selection of clusters. The authors did not prove a consistent method for selecting clusters in a latent space representation. When retraining the provided models, we get a latent space representation different to the original experiments. The clusters have to be manually selected. The metrics that they used to evaluate their explanations are also depend on the clustering. This means that there is some variability in the exact verification of reproducibility.

Communication with original authors

We asked the original authors for clarification on how to choose the ϵ hyper-parameter. However, it became apparent that we had misread, and the procedure is indeed adequately reported in the paper.