pwc-archive/papers-with-abstracts huggingface dataset:
Is RLHF More Difficult than Standard RL? A Theoretical Perspective
Date: 2023-09-21 00:00:00
Proceeding: NeurIPS 2023 11

Reinforcement learning from Human Feedback (RLHF) learns from preference signals, while standard Reinforcement Learning (RL) directly learns from reward signals. Preferences arguably contain less information than rewards, which makes preference-based RL seemingly more difficult. This paper theoretically proves that, for a wide range of preference models, we can solve preference-based RL directly using existing algorithms and techniques for reward-based RL, with small or no extra costs. Specifically, (1) for preferences that are drawn from reward-based probabilistic models, we reduce the problem to robust reward-based RL that can tolerate small errors in rewards; (2) for general arbitrary preferences where the objective is to find the von Neumann winner, we reduce the problem to multiagent reward-based RL which finds Nash equilibria for factored Markov games under a restricted set of policies. The latter case can be further reduce to adversarial MDP when preferences only depend on the final state. We instantiate all reward-based RL subroutines by concrete provable algorithms, and apply our theory to a large class of models including tabular MDPs and MDPs with generic function approximation. We further provide guarantees when K-wise comparisons are available.Submission Number: 14534
[Re] Fairness Guarantees under Demographic Shift
Date: 2023-09-21 00:00:00
Proceeding: NeurIPS 2023 11

Scope of Reproducibility: The original authors'  main contribution is the family of Shifty algorithms, which can guarantee that certain fairness constraints will hold with high confidence even after a demographic shift in the deployment population occurs. They claim that Shifty provides these high-confidence fairness guarantees without a loss in model performance, given enough training data.

Methodology: The code provided by the original paper was used, and only some small adjustments needed to be made in order to reproduce the experiments. All model specifications and hyperparameters from the original implementation were used. Extending beyond reproducing the original paper, we investigated the sensibility of Shifty to the size of the bounding intervals limiting the possible demographic shift, and ran shifty with an additional optimization method.

Results: Our results approached the results reported in the original paper. They supported the claim that \textit{Shifty} reliably guarantees fairness under demographic shift, but could not verify that Shifty performs at no loss of accuracy. 

What was easy: The theoretical framework laid out in the original paper was well explained and supported by additional formulas and proofs in the appendix. Further, the authors provided clear instructions on how to run the experiments and provided necessary hyperparameters.

What was difficult: While an open-source implementation of Shifty was provided and was debugged with relatively low time investment, the code did not contain extensive documentation and was complex to understand. It was therefore difficult to verify that each part of the code functions as expected and to expand upon the existing experiments. Further, certain hyperparameter and model specifications deviated between the provided code and the original paper, which made it challenging to know which specifications to apply when reproducing.

Communication with original authors: The first author of the original paper was contacted, but we have yet to receive a reply.Paper Url: https://iclr.cc/virtual/2022/poster/6666Paper Review Url: https://openreview.net/forum?id=wbPObLm6ueAPaper Venue: ICLR 2022Confirmation: The report pdf is generated from the provided camera ready Google Colab script, The report metadata is verified from the camera ready Google Colab script, The report contains correct author information., The report contains link to code and SWH metadata., The report follows the ReScience latex style guides as in the Reproducibility Report Template (https://paperswithcode.com/rc2022/registration)., The report contains the Reproducibility Summary in the first page., The latex .zip file is verified from the camera ready Google Colab scriptLatex:  zipJournal: ReScience Volume 9 Issue 2 Article 13Doi: https://www.doi.org/10.5281/zenodo.8173680Code: https://archive.softwareheritage.org/swh:1:dir:436c48ce9cf36b10ee3cdcd537a06c9df2cd53cc
Inference for Gaussian Processes with Matern Covariogram on Compact Riemannian Manifolds
Date: 2023-09-21 00:00:00
Tasks: Gaussian Processes, PAC learning, regression

In this work, we aim to characterize the statistical complexity of realizable regression both in the PAC learning setting and the online learning setting. Previous work had established the sufficiency of finiteness of the fat shattering dimension for PAC learnability and the necessity of finiteness of the scaled Natarajan dimension, but little progress had been made towards a more complete characterization since  the work of Simon 1997 (SICOMP '97). To this end,  we first introduce a minimax instance optimal learner for realizable regression and propose a novel dimension that both qualitatively and quantitatively characterizes which classes of real-valued predictors are learnable.  We then identify a combinatorial dimension related to the graph dimension that characterizes ERM learnability in the realizable setting. Finally, we establish a necessary condition for learnability based on a combinatorial dimension related to the DS dimension, and conjecture that it may also be sufficient in this context. Additionally, in the context of online learning we provide a dimension that characterizes the minimax instance optimal cumulative loss up to a constant factor and design an optimal online learner for realizable regression, thus resolving an open question raised by Daskalakis and Golowich in STOC '22.
Easy Bayesian Transfer Learning with Informative Priors
Date: 2023-09-21 00:00:00
Proceeding: NeurIPS 2023 11

REPRODUCIBILITY SUMMARY

Scope of Reproducibility
In this work, we study the reproducibility of the paper: Pre-Train Your Loss: Easy Bayesian Transfer Learning with Informative Priors. The paper proposes a three-step pipeline for replacing standard transfer learning with a pre-trained prior. The first step is training a prior, the second is re-scaling of a prior, and the third is inference.
The authors claim that increasing the rank and the scaling factor improves performance on the downstream task. They also argue that using Bayesian learning with informative prior leads to a more data-efficient and improved performance compared to standard SGD transfer learning or using non-informative prior. We reproduce the main claims on one of the four data sets in the paper.

Methodology
We used a combination of the authors' and our code. The authors provided a training pipeline for the user but not the code to fully reproduce the paper. We modified the training pipeline to suit our needs and created a testing pipeline to evaluate the models. We reproduced the results for the Oxford-102-Flowers data set on an Nvidia RTX 3070 GPU using approximately 310 GPU hours for the main results.

Results
Our results confirm most of the claims tested, although we could not achieve the exact same accuracy due to missing hyper-parameters. We reproduced the trend in how scaling the prior impacts the performance and how a learned prior outperforms a non-learned prior.
On contrary, we could not reproduce the effect of rank in low-rank covariance approximation on model performance, as well as the beneficial boost in performance of Bayesian learning compared to the standard SGD.

What was easy
The authors' implementation provides various training and logging parameters. It is also helpful that the authors provided both the learned priors and scripts for the download, split and pre-processing of the data sets used in the study.

What was difficult
Setting the environment for the used packages to work correctly was difficult. Although many parameters are available for running the pipeline, their descriptions are misguiding, therefore a lot of time went into clarifying the parameter function and debugging different settings. The training also took a while, especially when training 5 models per data point. 

Communication with original authors
We contacted the authors via e-mail about their pipeline and their use of hyper-parameters but did not hear back.Paper Url: https://openreview.net/forum?id=ao30zaT3YLPaper Review Url: https://openreview.net/forum?id=ao30zaT3YLPaper Venue: ICML 2022
Reproducibility Study of ”Label-Free Explainability for Unsupervised Models”
Date: 2023-09-21 00:00:00
Proceeding: NeurIPS 2023 11

In this work, we present our reproducibility study of "Label-Free Explainability for Unsupervised Models", a paper that introduces two post‐hoc explanation techniques for neural networks: (1) label‐free feature importance and (2) label‐free example importance. Our study focuses on the reproducibility of the authors’ most important claims: (i) perturbing features with the highest importance scores causes higherlatent shift than perturbing random pixels, (ii) label‐free example importance scores help to identify training examples that are highly related to a given test example, (iii) unsupervised models trained on different tasks show moderate correlation among the highest scored features and (iv) low correlation in example scores measured on a fixed set of data points, and (v) increasing the disentanglement with β in a β‐VAE does not imply that latent units will focus on more different features. We reviewed the authors’ code, checked if the implementation of experiments matched with the paper, and also ran all experiments. The results are shown to be reproducible. Moreover, we extended the codebase in order to run the experiments on more datasets, and to test the claims with other experiments.Paper Url: https://arxiv.org/abs/2203.01928Paper Venue: Other venue (not in list)Venue Name: MLRC 2022Confirmation: The report pdf is generated from the provided camera ready Google Colab script, The report metadata is verified from the camera ready Google Colab script, The report contains correct author information., The report contains link to code and SWH metadata., The report follows the ReScience latex style guides as in the Reproducibility Report Template (https://paperswithcode.com/rc2022/registration)., The report contains the Reproducibility Summary in the first page., The latex .zip file is verified from the camera ready Google Colab scriptLatex:  zipJournal: ReScience Volume 9 Issue 2 Article 24Doi: https://www.doi.org/10.5281/zenodo.8173711Code: https://archive.softwareheritage.org/swh:1:dir:0df843f2fc8c0868968429afb8908db7d3a76a3c
[Re] Variational Neural Cellular Automata
Date: 2023-09-21 00:00:00
Proceeding: NeurIPS 2023 11

The main claim of the paper being reproduced is that the proposed Variational Neural Cellular Automata (VNCA) architecture, composed of a convolutional encoder and a Neural Cellular Automata (NCA)-based decoder, is able to generate high-quality samples.
The paper presents two variants of this VNCA decoder: the doubling VNCA variant that is claimed to have a simple latent space, and the non-doubling VNCA variant that is claimed to be optimized for damage recovery and stability over many steps.

To reproduce the results, we re-implemented all of the VNCA models and a fully-convolutional baseline in JAX, by using the descriptions given in the paper. We then followed the same experimental setup and hyperparameter choices as in the original paper. 

All of the models were trained on a TPU v3-8 provided by Kaggle, with a total budget of around 4 TPU hours, not counting unreported experiments.

All but one of the figures and results from the original study were possible to reproduce. The obtained Evidence Lower Bound (ELBO) of the doubling VNCA was within $0.3\%$ of the stated and for the non-doubling VNCA the ELBO was within $1.8\%$ and the observed damage recovery was similar. We were however not able to reproduce the t-SNE reduction experiment for the baseline and were therefore not able to show the VNCA decoder having a cleaner t-SNE separation than the baseline.

The implementation of the convolutional baseline and most parts of the NCA-based decoder were straightforward to re-implement based on the description provided in the paper.

One of the difficulties faced in the reproduction study was obtaining the labels of the binarized MNIST dataset used in the original paper since it officially is provided without labels. This made it unclear how the original paper got the labels for the latent space visualization. 
Additionally, implementing the pool for the non-doubling version of the VNCA model with a distributed training setup was challenging as it required operations between TPU cores.

No communication was made with the original authors.Paper Url: https://arxiv.org/abs/2201.12360Paper Review Url: https://openreview.net/forum?id=7fFO4cMBx_9Paper Venue: ICLR 2022Confirmation: The report pdf is generated from the provided camera ready Google Colab script, The report metadata is verified from the camera ready Google Colab script, The report contains correct author information., The report contains link to code and SWH metadata., The report follows the ReScience latex style guides as in the Reproducibility Report Template (https://paperswithcode.com/rc2022/registration)., The report contains the Reproducibility Summary in the first page., The latex .zip file is verified from the camera ready Google Colab scriptLatex:  zipJournal: ReScience Volume 9 Issue 2 Article 31Doi: https://www.doi.org/10.5281/zenodo.8173729Code: https://archive.softwareheritage.org/swh:1:dir:3de126d09281c3dab4749dc8ac9330e53a22c558
$\mathbf{\mathbb{E}^{FWI}}$: Multiparameter Benchmark Datasets for Elastic Full Waveform Inversion of Geophysical Properties
Date: 2023-09-26 00:00:00
Proceeding: NeurIPS 2023 11

Elastic geophysical properties (such as P- and S-wave velocities) are of great importance to various subsurface applications like CO$_2$ sequestration and energy exploration (e.g., hydrogen and geothermal). Elastic full waveform inversion (FWI) is widely applied for characterizing reservoir properties. In this paper, we introduce $\mathbf{\mathbb{E}^{FWI}}$, a comprehensive benchmark dataset that is specifically designed for elastic FWI. $\mathbf{\mathbb{E}^{FWI}}$ encompasses 8 distinct datasets that cover diverse subsurface geologic structures (flat, curve, faults, etc). The benchmark results produced by three different deep learning methods are provided. In contrast to our previously presented dataset (pressure recordings) for acoustic FWI (referred to as OpenFWI), the seismic dataset in $\mathbf{\mathbb{E}^{FWI}}$ has both vertical and horizontal components. Moreover, the velocity maps in $\mathbf{\mathbb{E}^{FWI}}$ incorporate both P- and S-wave velocities. While the multicomponent data and the added S-wave velocity make the data more realistic, more challenges are introduced regarding the convergence and computational cost of the inversion. We conduct comprehensive numerical experiments to explore the relationship between P-wave and S-wave velocities in seismic data. The relation between  P- and S-wave velocities provides crucial insights into the subsurface properties such as lithology, porosity, fluid content, etc.  We anticipate that $\mathbf{\mathbb{E}^{FWI}}$ will facilitate future research on multiparameter inversions and stimulate endeavors in several critical research topics of carbon-zero and new energy exploration. All datasets, codes and relevant information can be accessed through our website at https://efwi-lanl.github.io/
$\mathcal{M}^4$: A Unified XAI Benchmark for Faithfulness Evaluation of Feature Attribution Methods across Metrics, Modalities and Models
Date: 2023-09-26 00:00:00
Proceeding: NeurIPS 2023 11

While Explainable Artificial Intelligence (XAI) techniques have been widely studied to explain predictions made by deep neural networks, the way to evaluate the faithfulness of explanation results remains challenging, due to the heterogeneity of explanations for various models and the lack of ground-truth explanations. This paper introduces an XAI benchmark named $\mathcal{M}^4$, which allows evaluating various input feature attribution methods using the same set of faithfulness metrics across multiple data modalities (images and texts) and network structures (ResNets, MobileNets, Transformers). A taxonomy for the metrics has been proposed as well. We first categorize commonly used XAI evaluation metrics into three groups based on the ground truth they require. We then implement classic and state-of-the-art feature attribution methods using InterpretDL and conduct extensive experiments to compare methods and gain insights. Extensive experiments have been conducted to provide holistic evaluations as benchmark baselines. Several interesting observations are noticed for designing attribution algorithms. The implementation of state-of-the-art explanation methods and evaluation metrics of $\mathcal{M}^4$ is publicly available at \url{https://github.com/PaddlePaddle/InterpretDL}.
[Re] CrossWalk: Fairness-enhanced Node Representation Learning
Date: 2023-09-26 00:00:00
Proceeding: NeurIPS 2023 11

Scope of Reproducibility
This work aims to reproduce the findings of the paper “CrossWalk: Fairness-enhanced Node Representation Learning” by investigating the two main claims made by the authors about CrossWalk, which suggest that (i) CrossWalk enhances fairness in three graph algorithms, while only suffering from small decreases in performance, and that (ii) CrossWalk preserves the necessary structural properties of the graph while reducing disparity.

Methodology
The authors made the CrossWalk repository available, which contained most of the datasets used for their experimentation, and the scripts needed to run the experiments. However, the codebase lacked documentation and was missing logic for running all experiments and visualizing the results. We, therefore, re-implement their code from scratch and deploy it as a python package which can be run to obtain all the showcased results. 

Results
Our work suggests that the first claim of the paper, which states that Crosswalk minimizes disparity and thus enhances fairness is partially reproducible, and only for the tasks of Node classification and Influence maximization as the parameters specified in the paper do not always yield similar results. Then, the second claim of the paper which states that Crosswalk attains the necessary structural properties of the graph is fully reproducible through our experiments.

What was easy
The original paper contained the necessary information about hyperparameters, which coupled with the publicly available repository made it straightforward to refactor the code and understand the idea of the proposed method. 

What was difficult
The difficulty stems from the lack of structure and documentation in the provided code which made the original experiments hard to reproduce. Furthermore, there were missing files in the provided datasets. Also, some experiments were not reproducible at all through the provided code. One more important aspect is that the experiments are CPU intensive which made the reproducibility even harder.

Communication with original authors
Albeit rather late, the authors provided meaningful feedback on our questions about implementation details and initial results.Paper Url: https://arxiv.org/abs/2105.02725Paper Venue: AAAI 2022Confirmation: The report pdf is generated from the provided camera ready Google Colab script, The report metadata is verified from the camera ready Google Colab script, The report contains correct author information., The report contains link to code and SWH metadata., The report follows the ReScience latex style guides as in the Reproducibility Report Template (https://paperswithcode.com/rc2022/registration)., The report contains the Reproducibility Summary in the first page., The latex .zip file is verified from the camera ready Google Colab scriptLatex:  zipJournal: ReScience Volume 9 Issue 2 Article 39Doi: https://www.doi.org/10.5281/zenodo.8173749Code: https://archive.softwareheritage.org/swh:1:dir:6367a5e042f3b14cd6d86349199e08597fc0f3f6
[Re] Pure Noise to the Rescue of Insufficient Data
Date: 2023-09-21 00:00:00
Proceeding: NeurIPS 2023 11

Scope of Reproducibility — We examine the main claims of the original paper [1], which
states that in an image classification task with imbalanced training data, (i) using pure
noise to augment minority‐class images encourages generalization by improving minority‐
class accuracy. This method is paired with (ii) a new batch normalization layer that
normalizes noise images using affine parameters learned from natural images, which
improves the model’s performance. Moreover, (iii) this improvement is robust to vary‐
ing levels of data augmentation. Finally, the authors propose that (iv) adding pure noise
images can improve classification even on balanced training data.

Methodology — We implemented the training pipeline from the description of the paper
using PyTorch and integrated authors’ code snippets for sampling pure noise images
and batch normalizing noise and natural images separately. All of our experiments were
run on a machine from a cloud computing service with one NVIDIA RTX A5000 Graphics
Card and had a total computational time of approximately 432 GPU hours.

Results — We reproduced the main claims that (i) oversampling with pure noise improves
generalization by improving the minority‐class accuracy, (ii) the proposed batch nor‐
malization (BN) method outperforms baselines, (iii) and this improvement is robust
across data augmentations. Our results also support that (iv) adding pure noise images
can improve classification on balanced training data. However, additional experiments
suggest that the performance improvement from OPeN may be more orthogonal to the
improvement caused by a bigger network or more complex data augmentation.

What was easy — The code snippet in the original paper was thoroughly documented and
was easy to use. The authors also clearly documented most of the hyperparameters that
were used in the main experiments.

What was difficult — The repo linked in the original paper was not populated yet. As a re‐
sult, we had to retrieve the CIFAR‐10‐LT dataset from previous works [2, 3], re‐implement
WideResNet [4], and the overall training pipeline.

Communication with original authors — We contacted the authors for clarifications on the
implementation details of the algorithm. Prior works had many important implemen‐
tation details such as linear learning rate warmup or deferred oversampling, so we con‐
firmed with the authors on whether these methods were used.Paper Url: https://proceedings.mlr.press/v162/zada22aPaper Venue: ICML 2022