pwc-archive/papers-with-abstracts huggingface dataset:
[Re] Explaining Groups of Points in Low-Dimensional Representations
Authors: Damiaan J W Reijnaers, Daniël B van de Pavert, Giguru Scheuer, Liang Huang
Date: 2021-01-31 00:00:00
Proceeding: RC 2020
Tasks: Dimensionality Reduction, Interpretable Machine Learning

Scope of Reproducibility

In this paper we present an analysis and elaboration of [1], in which an algorithm is posed by Plumb et al. for the purpose of finding human-understandable explanations in terms of given explainable features of input data for differences between groups of points occurring in a lower-dimensional representation of that input data.

Methodology

We have upgraded the original code provided by the authors such that it is compatible with recent versions of popular deep learning frameworks, namely the TensorFlow 2.x- and PyTorch 1.7.x-libraries. Furthermore, we have created our own implementation of the algorithm in which we have incorporated additional experiments in order to evaluate the algorithmʼs relevance in the scope of different dimensionality reduction techniques and differently structured data. We have performed the same experiments as described in the original paper using both the upgraded version of the code and our own implementation taking the authorsʼ code and paper as references.

Results

The results presented in [1] were reproducible, both by using the provided code and our own implementation. Our additional experiments have highlighted several limitations of the explanatory algorithm in question: the algorithm severely relies on the shape and variance of the clusters present in the data (and, if applicable, the method used to label these clusters), and highly non-linear dimensionality reduction algorithms perform worse in terms of explainability.

What was easy

The authors have provided an implementation that cleanly separates different experiments on different datasets and the core functional methodology. Given a working environment, it is easy to reproduce the experiments performed in [1].

What was difficult

Minor difficulties were experienced in setting up the required environment for running the code provided by Plumb et al. locally (i.e. trivial changes in the code such as the usage of absolute paths and obtaining external dependencies). Evidently, it was timeconsuming to rewrite all corresponding code, including the architecture for the variational auto-encoder provided by an external package, scvis 0.1.02.

Communication with original authors

No communication with the original authors was required to reproduce their work.
[Re] Reproducibility report of "Interpretable Complex-Valued Neural Networks for Privacy Protection"
Authors: Arsen Sheverdin, Noud Corten, Alko Knijff, Georg Lange
Date: 2021-01-31 00:00:00
Proceeding: RC 2020
Tasks: CPU, GPU

Scope of Reproducibility

The original work by Xiang et al.1 claimed (1) that complex-valued DNNs effectively increase the difficulty of inferring inputs for the adversary attacks compared to the baseline. In addition, Xiang et al.1 stated that the (2) proposed privacy-protecting complexvalued DNN effectively preserves the accuracy when compared to the baseline.

Methodology

Since the original paperʼs code was not published, all of the codebase was written independently from scratch, based solemnly on how it was described in the paper. We mostly used a Nvidiaʼs RTX 2060 Super as the GPU and a AMD Ryzen 3600x as the CPU. The runtime of each model was highly dependant on the architecture used. The runtimes for each model can be found in Table 2.

Results

In contrast to the first claim, we have discovered that for most of the architectures, reconstruction errors for the attacks are quite low, which means that in our models the first claim is not supported. We also found that for most of the models, the classification error is somewhat higher than those provided in the paper. However, these indeed relate to the original work and partially support the second claim of the authors.

What was easy

Authors of the original paper utilized famous architectures for some of architecturesʼ parts, such as ResNet and LeNet, that were well explained and defined in the literature. In addition, authors, provided formulas on the modified rotation-invariant Complex DNN modules (ReLU, max pooling etc.), implementation of which was relatively straightforward. The paper was based on the openly available datasets.

What was difficult

The paper did not provide any information on the architecture of the critic for the WGAN, along with the architecture of the angle discriminator utilized in inversion attack 1. It also does not provide any information about crucial hyperparameters, such as the k value used for k-anonimity.

Communication with original authors

We did not contact the original authors of the publication.
[Re] Replication Study of 'Generative causal explanations of black-box classifiers'
Authors: Ivo Verhoeven, Xinyi Chen, Qingzhi Hu, Mario Holubar
Date: 2021-01-31 00:00:00
Proceeding: RC 2020
Tasks: Fairness

Scope of Reproducibility

We verify the outcome of the methodology proposed in the article, which attempts to provide post-hoc causal explanations for black-box classifiers through causal reference. This is achieved by replicating the code step by step, according to the descriptions in the paper. All the claims in the paper have been examined, and we provide additional metric to evaluate the portability, expressive power, algorithmic complexity and the data fidelity of their framework. We have further extended their analyses to consider all benchmark datasets used, confirming results.

Methodology

We use the same architecture and (hyper)parameters for replication. However, the code has a different structure and we provide a more efficient implementation for the measure of information flow. In addition, Algorithm 1 in the original paper is not implemented in their repository, so we have also implemented Algorithm 1 ourselves and further extend their framework to another domain (text data), although unsuccessfully. Furthermore, we make a detailed table in the paper to show the time used to produce the results for different experiments reproduced. All models were trained on Nvidia GeForce GTX 1080 GPUs provided by Surfsaraʼs LISA cluster computing service at university of Amsterdam1.

Results

We reproduced the framework in the original paper and verified the main claims made by the authors in the original paper. However, the GCE model in extension study did not manage to separate causal factors and non-causal factors for a text classifier due to the complexity of fine-tuning the model.

What was easy

The original paper comes with extensive appendices, many of which contain crucial details for implementation and understanding of the intended function. The authors provide code for most of the experiments presented in the paper. Although at the beginning their code repository was not functional, we use it as a reference to re-implement our code. The author also updated their code two weeks after we start our own implementation, which made it easy for us to verify the correctness of our re-implementation.

What was difficult

The codebase the authors provided was initially unusable, with missing or renamed imports, hardcoded filepaths and an all-around convoluted structure. Additionally, the description of Algorithm 1 is quite vague and no implementation of it was given. Beyond this, computational expense was a serious issue, given the need for inefficient training steps, and re-iterating training several times for hyperparameter search.

Communication with original authors

This reproducibility study is part of a course on fairness, accountability, confidentiality and transparency in AI. Since it is a course project where we interacted with other group in the forum, and another group also working with this paper has reached out to the authors about problems with the initial repository, we did not find necessary to do it again.
[Re] Training Binary Neural Networks using the Bayesian Learning Rule
Authors: Prateek Garg, Lakshya Singhal, Ashish Sardana
Date: 2021-01-31 00:00:00
Proceeding: RC 2020
Tasks: Continual Learning, Semantic Segmentation

Scope of Reproducibility

We try to verify the performance of our re-implementation of the BayesBiNN optimizer on various classification and regression benchmarks. We also implemented the STE optimizer which was the central baseline model used in the paper. Finally, we tried to evaluate the results of BayesBiNN on the continual learning benchmark to get a better insight.

Methodology

We developed our separate code-base, consisting of an end-to-end trainer with a Keraslike interface, for the reproduction which includes the implementation of the BayesBiNN and STE optimizer. We did refer to the authorʼs code open-sourced on GitHub to get some insights about the hyperparameters and other doubts that emerged during code development.

Results

We reproduced the accuracy of the BayesBiNN optimizer within less than 0.5% of the originally reported value, which upholds the conclusion that it performs nearly as well as its full precision counterpart in classification tasks. When we tried this in a semantic segmentation context, we found that the results were very underwhelming and in contrast with the seemingly good results by the STE optimizer even with much hyperparameter tuning. We can conclude that, like other Bayesian methods, it is difficult to train BayesBiNN on more complex tasks.

What was easy

After we worked out the mathematics behind the BayesBiNN approach, we developed a pseudo-code for the optimization process which along with references from the authorʼs code, helped us a lot in our reproduction study.

What was difficult

Some of the hyperparameters were not mentioned by the authors in their paper so it was difficult to approximate the values of those parameters. The lack of resources was the next big difficulty that we faced.

Communication with original authors

We had a very fruitful conversation with the authors, which helped us in better understanding the BayesBiNN approach and its extension to the segmentation domain. The detailed pointers are given at the end of this report.
[Re] On end-to-end 6{DoF} object pose estimation and robustness to object scale
Authors: Georgios Nikolaos Albanis, Nikolaos Zioulis, Anargyros Chatzitofis, Anastasios Dimou, Dimitrios Zarpalas, Petros Daras
Date: 2021-01-31 00:00:00
Proceeding: RC 2020
Tasks: 6D Pose Estimation, 6D Pose Estimation using RGB, GPU, Keypoint Estimation, Object, Pose Estimation

Scope of Reproducibility

This report contains a set of experiments that seek to reproduce the claims of two recent works related to keypoint estimation, one specific to 6DoF object pose estimation, and the other presenting a generic architectural improvement for keypoint estimation but demonstrated in human pose estimation. More specifically, in the backpropagatable PnP [1], the authors claim that incorporating geometric optimization in a deep-learning pipeline and predicting an objectʼs pose in an end-to-end manner yields improved performance. On the other hand, HigherHRNet [2] introduces a novel heatmap aggregation method that allows for scale-aware pose estimations, offering higher keypoint localization accuracy for small scale objects.

Methodology

We used the publicly provided code where available, adapting it to fit into a model development kit to facilitate our experiments. We used a dataset fit for validating both claims simultaneously, and designed a set of experiments based on the published methodologies, but also went beyond seeking to validate the higher level concepts. Our experiments were conducted on a Nvidia 2080 12 GB GPU with an average training time of 14 hours.

Results

We reproduce the claims of both papers by conducting several experiments in the UAVA dataset [3]. The integration of a differentiable geometric module within an keypointbased object pose estimation model improved its performance in metrics. We additionally verify that this is the case for other differentiable PnP implementations (i.e. EPnP). Further, our results indicate that indeed HigherHRNet improves keypoint localisation performance on small scale objects.

What was easy

Both papers provided publicly available implementations. In addition, many different variations were also found online. Finally, the papers themselves were very clearly written, offering insights on various important details.

What was difficult

The main issue that required more effort was identifying the appropriate weights for BPnP [1] in order to balance the different optimization objectives. As expected, this varies for the context that it is applied (task, dataset) and the values presented in the paper did not work in our case. Sub-optimal selection of weights leads to convergence issues.

Communication with original authors

We communicated with the authors of [1] through GitHub, and we would like to thank them as they provided a fast and detailed response. Furthermore, their responsiveness to past issues had already provided a nice knowledge base regarding reproduction.
[Re] A Reproduction of Ensemble Distribution Distillation
Authors: Toomas Liiv, Einar Lennelöv, Aron Noren
Date: 2021-01-31 00:00:00
Proceeding: RC 2020
Tasks: Image Classification, Object Detection, Out of Distribution (OOD) Detection

Scope of Reproducibility 

The authors claim that their proposed method is able to, given an ensemble of deep neural networks, capture the uncertainty estimation and decomposition capabilities of the ensemble into a single model. The authors also claim that this only results in a small reduction in classification performance compared to the ensemble. We examine these claims by reproducing most of the authorsʼ experiments on the CIFAR-10 dataset. 

Methodology 

The proposed method was re-implemented in tf.keras. The surrounding data pipelines, pre-processing, and experimentation code were also re-implemented. As in the original paper, the models were based on VGG-16 networks trained from scratch with random initialization. Training and evaluation was done on two consumer-grade GPUs, for a total of 273 hours. 

Results 

Our findings support the authorsʼ central claims. In terms of uncertainty estimation our EnD2 achieved (99 ± 1) % of the AUC-ROC of our ensemble on the OOD-detection task. The corresponding value in the original paper was (100±1) %. In terms of classification our EnD2 had (16 ± 1)% higher error than our ensemble. The corresponding values in the original paper was (11 ± 6)%. Other metrics showed similar agreement, but, significantly, in the OOD-detection task our EnD performed at least as well as our EnD2 . This is in stark contrast with the original paper. We also took a novel approach to visualizing the uncertainty decomposition by plotting the resulting distributions on a simplex, offering a visual explanation to some surprising results in the original paper, while mostly supporting the authorsʼ intuitive justifications for the model. 

What was easy 

The original paper features a thorough mathematical formulation of the method, aiding conceptual understanding. The datasets used by the authors are publicly available. The use of the simpler datasets also meant that it was computationally feasible for us to reproduce these results. The base model used is well known with several implementation available, allowing us to focus on the novel aspects of the method. 

What was difficult 

While the theoretical explanations of the method are excellent, we initially found it hard to translate this into an implementation. Our difficulty was likely caused by our inexperience with the subject matter. Nonetheless, a pseudocode, such as the one we have provided, would havee simplified the re-implementation. We were not able to reproduce the results on some of the datasets due to limited computational resources. 

Communication with original authors 

We did not contact the original authors directly, but we did refer to a public GitHub and blog post created by one of the authors. At the same time as submitting this report to the ML Reproducibility Challenge 2020 we also sent a copy to the authors and asked for their feedback.
[Re] Hamiltonian Generative Networks
Authors: Carles Balsells Rodas, Oleguer Canal, Federico Taschin
Date: 2021-01-31 00:00:00
Proceeding: RC 2020
Tasks: GPU, Reinforcement Learning

Scope of Reproducibility

The main objective of the paper is to ”learn the Hamiltonian dynamics of simple physical systems from high-dimensional observations without restrictive domain assumptions”. To do so, the authors train a generative model that reconstructs an inputted sequence of images of the evolution of some physical system. For instance, they learn the dynamics of a pendulum, a body-spring system, and 2,3-bodies. In addition to these environments, we further expand the testing on two new environments and we explore architecture tweaks looking for performance gains.

Methodology

We implement the project with Python using Pytorch [1] as a deep learning library. Previous to ours, there was no public implementation of this work. Thus, we had to write the code of the simulated environments, the deep models, and the training process. The code can be found in this repository: https://github.com/CampusAI/HamiltonianGenerative-Networks A single training takes around 4 hours and 1910MB of GPU memory (NVIDIA GeForce RTX2080Ti).

Results

We found the modelʼs input-output data slightly unclear in the original paper. First, it seems that the model reconstructs the same sequence that has been inputted. Nevertheless, further discussion with the authors seems to indicate that they input the first few frames to the network and reconstructed the rest of the rollout. We test both approaches and analyze the results. We generally obtain comparable results to those of the original authors when just reconstructing the input sequence (30% average absolute relative error w.r.t. to their reported values) and worse results when trying to reconstruct unseen frames (107% error). In this report, we include our intuition on possible reasons that would explain these observations.

What was easy

The architecture of the model and training procedure was easy to understand from the paper. Besides, creating simulation environments similar to those of the original authors was also straightforward.

What was difficult

While the overall model architecture and data generation were easy to understand, we encountered the optimization to be especially tricky to perform. In particular, finding a good balance between the reconstruction loss and KL divergence loss was challenging. We implemented GECO [2] to dynamically adapt the Lagrange multiplier but it proved to be surprisingly brittle to its hyper-parameters, resulting in very unstable behavior. We were unable to identify the cause of the problem and ended up training with simpler techniques such as using a fixed Lagrange multiplier as presented in [3].

Communication with original authors

We exchanged around 6 emails with doubts and answers with the original authors.
[Re] Improving Multi-hop Question Answering over Knowledge Graphs using Knowledge Base Embeddings
Authors: Jishnu Jaykumar P, Ashish Sardana
Date: 2021-01-31 00:00:00
Proceeding: RC 2020
Tasks: Answer Selection, Graph Embedding, Knowledge Graph Embedding, Knowledge Graph Embeddings, Knowledge Graphs, Link Prediction, Multi-hop Question Answering, Navigate, Question Answering

Scope of Reproducibility

Our work consists of four parts:
1. Reproducing the results from [1].
2. Exploring the effect of various knowledge graph embedding models in the Knowledge Graph Embedding module.
3. Exploring the effect of various transformer models in the Question Embedding
module.
4. Verifying the importance of the Relation Matching (RM) module.
Based on the code shared by the authors, we have reproduced the results for EmbedKGQA[1]. We have not performed relation matching deliberately to validate point-4.

Methodology

We have used the code provided by [1] with some customization for reproducibility. In addition to making the codebase more modular and easy to navigate, we have made changes to incorporate different transformers in the question embedding module. QuestionAnswering models were trained from scratch as no pre-trained models were available for our particular dataset. The code for this work is available on GitHub (See page footer for the link).

Results

We were able to reproduce the Hits@1 to be within ±2.4% of the reported value (in most cases). Anomalies were observed in 2 cases.
1. In MetaQA-KG-Full (3-hop) dataset.
2. WebQSP-KG-Full dataset.
From our experiments on the QA model, we have found that a recent transformer architecture, SBERT[2] produced better accuracy than the original paper. Replacing RoBERTa[3] with SBERT[2] increased the absolute accuracy by ≈3.4% and ≈0.6% in the half KG and the full KG case respectively. (KG: Knowledge Graph, ”≈”: Approximately)

What was easy

As the code was open-sourced, we didnʼt have to implement the paper giving us the liberty to customize the codebase to focus on the authorʼs claim validation, perform extended experiments and explore shared as well as new models. In addition to this, pretrained KG embedding models were shared which helped in the reproduction experiment.

What was difficult

The lack of comprehensive documentation along with missing comments defining functions/classes/attributes etc. made it laborious to review the code and modify it. In addition to large training times for question answering models, the knowledge graph embeddings also required a significant amount of computing resources.

Communication with original authors

We had a couple of virtual meetings with Apoorv Saxena, the primary author of EmbedKGQA[1].
iCompass at NLP4IF-2021–Fighting the COVID-19 Infodemic
Authors: Wassim Henia, Oumayma Rjab, Hatem Haddad, Chayma Fourati
Date: 2021-06-01 00:00:00

This paper provides a detailed overview of the system and its outcomes, which were produced as part of the NLP4IF Shared Task on Fighting the COVID-19 Infodemic at NAACL 2021. This task is accomplished using a variety of techniques. We used state-of-the-art contextualized text representation models that were fine-tuned for the downstream task in hand. ARBERT, MARBERT,AraBERT, Arabic ALBERT and BERT-base-arabic were used. According to the results, BERT-base-arabic had the highest 0.784 F1 score on the test set.
Data-driven sparse polynomial chaos expansion for models with dependent inputs
Authors: Zhanlin Liu, Youngjun Choe
Date: 2021-01-20 00:00:00
Tasks: Computational Efficiency

Polynomial chaos expansions (PCEs) have been used in many real-world engineering applications to quantify how the uncertainty of an output is propagated from inputs. PCEs for models with independent inputs have been extensively explored in the literature. Recently, different approaches have been proposed for models with dependent inputs to expand the use of PCEs to more real-world applications. Typical approaches include building PCEs based on the Gram-Schmidt algorithm or transforming the dependent inputs into independent inputs. However, the two approaches have their limitations regarding computational efficiency and additional assumptions about the input distributions, respectively. In this paper, we propose a data-driven approach to build sparse PCEs for models with dependent inputs. The proposed algorithm recursively constructs orthonormal polynomials using a set of monomials based on their correlations with the output. The proposed algorithm on building sparse PCEs not only reduces the number of minimally required observations but also improves the numerical stability and computational efficiency. Four numerical examples are implemented to validate the proposed algorithm.