pwc-archive/papers-with-abstracts huggingface dataset:
SoK: How Robust is Image Classification Deep Neural Network Watermarking? (Extended Version)
Authors: Nils Lukas, Edward Jiang, Xinda Li, Florian Kerschbaum
Date: 2021-08-11 00:00:00
Tasks: image-classification, Image Classification

Deep Neural Network (DNN) watermarking is a method for provenance verification of DNN models. Watermarking should be robust against watermark removal attacks that derive a surrogate model that evades provenance verification. Many watermarking schemes that claim robustness have been proposed, but their robustness is only validated in isolation against a relatively small set of attacks. There is no systematic, empirical evaluation of these claims against a common, comprehensive set of removal attacks. This uncertainty about a watermarking scheme's robustness causes difficulty to trust their deployment in practice. In this paper, we evaluate whether recently proposed watermarking schemes that claim robustness are robust against a large set of removal attacks. We survey methods from the literature that (i) are known removal attacks, (ii) derive surrogate models but have not been evaluated as removal attacks, and (iii) novel removal attacks. Weight shifting and smooth retraining are novel removal attacks adapted to the DNN watermarking schemes surveyed in this paper. We propose taxonomies for watermarking schemes and removal attacks. Our empirical evaluation includes an ablation study over sets of parameters for each attack and watermarking scheme on the CIFAR-10 and ImageNet datasets. Surprisingly, none of the surveyed watermarking schemes is robust in practice. We find that schemes fail to withstand adaptive attacks and known methods for deriving surrogate models that have not been evaluated as removal attacks. This points to intrinsic flaws in how robustness is currently evaluated. We show that watermarking schemes need to be evaluated against a more extensive set of removal attacks with a more realistic adversary model. Our source code and a complete dataset of evaluation results are publicly available, which allows to independently verify our conclusions.
A spectral-based analysis of the separation between two-layer neural networks and linear methods
Authors: Lei Wu, Jihao Long
Date: 2021-08-10 00:00:00

We propose a spectral-based approach to analyze how two-layer neural networks separate from linear methods in terms of approximating high-dimensional functions. We show that quantifying this separation can be reduced to estimating the Kolmogorov width of two-layer neural networks, and the latter can be further characterized by using the spectrum of an associated kernel. Different from previous work, our approach allows obtaining upper bounds, lower bounds, and identifying explicit hard functions in a united manner. We provide a systematic study of how the choice of activation functions affects the separation, in particular the dependence on the input dimension. Specifically, for nonsmooth activation functions, we extend known results to more activation functions with sharper bounds. As concrete examples, we prove that any single neuron can instantiate the separation between neural networks and random feature models. For smooth activation functions, one surprising finding is that the separation is negligible unless the norms of inner-layer weights are polynomially large with respect to the input dimension. By contrast, the separation for nonsmooth activation functions is independent of the norms of inner-layer weights.
A Brief Review of Machine Learning Techniques for Protein Phosphorylation Sites Prediction
Authors: Farzaneh Esmaili, Mahdi Pourmirzaei, Shahin Ramazi, Elham Yavari
Date: 2021-08-10 00:00:00
Tasks: BIG-bench Machine Learning, Prediction

Post-translational modifications (PTMs) have vital roles in extending the functional diversity of proteins and as a result, regulating diverse cellular processes in prokaryotic and eukaryotic organisms. Phosphorylation modification is a vital PTM that occurs in most proteins and plays significant roles in many biological processes. Disorders in the phosphorylation process lead to multiple diseases including neurological disorders and cancers. At first, this study comprehensively reviewed all databases related to phosphorylation sites (p-sites). Secondly, we introduced all steps regarding dataset creation, data preprocessing and method evaluation in p-sites prediction. Next, we investigated p-sites prediction methods which fall into two computational and Machine Learning (ML) groups. Additionally, it was shown that there are basically two main approaches for p-sites prediction by ML: conventional and End-to-End learning, which were given an overview for both of them. Moreover, this study introduced the most important feature extraction techniques which have mostly been used in ML approaches. Finally, we created three test sets from new proteins related to the 2022th released version of the dbPTM database based on general and human species. After evaluating available online tools on the test sets, results showed that the performance of online tools for p-sites prediction are quite weak on new reported phospho-proteins.
A Study of Social and Behavioral Determinants of Health in Lung Cancer Patients Using Transformers-based Natural Language Processing Models
Authors: Zehao Yu, Xi Yang, Chong Dang, Songzi Wu, Prakash Adekkanattu, Jyotishman Pathak, Thomas J. George, William R. Hogan, Yi Guo, Jiang Bian, Yonghui Wu
Date: 2021-08-10 00:00:00

Social and behavioral determinants of health (SBDoH) have important roles in shaping people's health. In clinical research studies, especially comparative effectiveness studies, failure to adjust for SBDoH factors will potentially cause confounding issues and misclassification errors in either statistical analyses and machine learning-based models. However, there are limited studies to examine SBDoH factors in clinical outcomes due to the lack of structured SBDoH information in current electronic health record (EHR) systems, while much of the SBDoH information is documented in clinical narratives. Natural language processing (NLP) is thus the key technology to extract such information from unstructured clinical text. However, there is not a mature clinical NLP system focusing on SBDoH. In this study, we examined two state-of-the-art transformer-based NLP models, including BERT and RoBERTa, to extract SBDoH concepts from clinical narratives, applied the best performing model to extract SBDoH concepts on a lung cancer screening patient cohort, and examined the difference of SBDoH information between NLP extracted results and structured EHRs (SBDoH information captured in standard vocabularies such as the International Classification of Diseases codes). The experimental results show that the BERT-based NLP model achieved the best strict/lenient F1-score of 0.8791 and 0.8999, respectively. The comparison between NLP extracted SBDoH information and structured EHRs in the lung cancer patient cohort of 864 patients with 161,933 various types of clinical notes showed that much more detailed information about smoking, education, and employment were only captured in clinical narratives and that it is necessary to use both clinical narratives and structured EHRs to construct a more complete picture of patients' SBDoH factors.
Causal Order Identification to Address Confounding: Binary Variables
Authors: Joe Suzuki, Yusuke Inaoka
Date: 2021-08-10 00:00:00
Tasks: Mutual Information Estimation

This paper considers an extension of the linear non-Gaussian acyclic model (LiNGAM) that determines the causal order among variables from a dataset when the variables are expressed by a set of linear equations, including noise. In particular, we assume that the variables are binary. The existing LiNGAM assumes that no confounding is present, which is restrictive in practice. Based on the concept of independent component analysis (ICA), this paper proposes an extended framework in which the mutual information among the noises is minimized. Another significant contribution is to reduce the realization of the shortest path problem. The distance between each pair of nodes expresses an associated mutual information value, and the path with the minimum sum (KL divergence) is sought. Although $p!$ mutual information values should be compared, this paper dramatically reduces the computation when no confounding is present. The proposed algorithm finds the globally optimal solution, while the existing locally greedily seek the order based on hypothesis testing. We use the best estimator in the sense of Bayes/MDL that correctly detects independence for mutual information estimation. Experiments using artificial and actual data show that the proposed version of LiNGAM achieves significantly better performance, particularly when confounding is present.
Arbitrage-Free Implied Volatility Surface Generation with Variational Autoencoders
Authors: Brian Ning, Sebastian Jaimungal, Xiaorong Zhang, Maxime Bergeron
Date: 2021-08-10 00:00:00

We propose a hybrid method for generating arbitrage-free implied volatility (IV) surfaces consistent with historical data by combining model-free Variational Autoencoders (VAEs) with continuous time stochastic differential equation (SDE) driven models. We focus on two classes of SDE models: regime switching models and L\'evy additive processes. By projecting historical surfaces onto the space of SDE model parameters, we obtain a distribution on the parameter subspace faithful to the data on which we then train a VAE. Arbitrage-free IV surfaces are then generated by sampling from the posterior distribution on the latent space, decoding to obtain SDE model parameters, and finally mapping those parameters to IV surfaces. We further refine the VAE model by including conditional features and demonstrate its superior generative out-of-sample performance.
On the Distinction Between "Conditional Average Treatment Effects" (CATE) and "Individual Treatment Effects" (ITE) Under Ignorability Assumptions
Authors: Brian G. Vegetabile
Date: 2021-08-10 00:00:00

Recent years have seen a swell in methods that focus on estimating "individual treatment effects". These methods are often focused on the estimation of heterogeneous treatment effects under ignorability assumptions. This paper hopes to draw attention to the fact that there is nothing necessarily "individual" about such effects under ignorability assumptions and isolating individual effects may require additional assumptions. Such individual effects, more often than not, are more precisely described as "conditional average treatment effects" and confusion between the two has the potential to hinder advances in personalized and individualized effect estimation.
Flow-based SVDD for anomaly detection
Authors: Marcin Sendera, Marek Śmieja, Łukasz Maziarka, Łukasz Struski, Przemysław Spurek, Jacek Tabor
Date: 2021-08-10 00:00:00
Tasks: Anomaly Detection, One-class classifier

We propose FlowSVDD -- a flow-based one-class classifier for anomaly/outliers detection that realizes a well-known SVDD principle using deep learning tools. Contrary to other approaches to deep SVDD, the proposed model is instantiated using flow-based models, which naturally prevents from collapsing of bounding hypersphere into a single point. Experiments show that FlowSVDD achieves comparable results to the current state-of-the-art methods and significantly outperforms related deep SVDD methods on benchmark datasets.
Analysis of ODE2VAE with Examples
Authors: Batuhan Koyuncu
Date: 2021-08-10 00:00:00
Tasks: Inductive Bias

Deep generative models aim to learn underlying distributions that generate the observed data. Given the fact that the generative distribution may be complex and intractable, deep latent variable models use probabilistic frameworks to learn more expressive joint probability distributions over the data and their low-dimensional hidden variables. Learning complex probability distributions over sequential data without any supervision is a difficult task for deep generative models. Ordinary Differential Equation Variational Auto-Encoder (ODE2VAE) is a deep latent variable model that aims to learn complex distributions over high-dimensional sequential data and their low-dimensional representations. ODE2VAE infers continuous latent dynamics of the high-dimensional input in a low-dimensional hierarchical latent space. The hierarchical organization of the continuous latent space embeds a physics-guided inductive bias in the model. In this paper, we analyze the latent representations inferred by the ODE2VAE model over three different physical motion datasets: bouncing balls, projectile motion, and simple pendulum. Through our experiments, we explore the effects of the physics-guided inductive bias of the ODE2VAE model over the learned dynamical latent representations. We show that the model is able to learn meaningful latent representations to an extent without any supervision.
Retiring Adult: New Datasets for Fair Machine Learning
Authors: Frances Ding, Moritz Hardt, John Miller, Ludwig Schmidt
Date: 2021-08-10 00:00:00
Proceeding: NeurIPS 2021 12
Tasks: BIG-bench Machine Learning, Fairness

Although the fairness community has recognized the importance of data, researchers in the area primarily rely on UCI Adult when it comes to tabular data. Derived from a 1994 US Census survey, this dataset has appeared in hundreds of research papers where it served as the basis for the development and comparison of many algorithmic fairness interventions. We reconstruct a superset of the UCI Adult data from available US Census sources and reveal idiosyncrasies of the UCI Adult dataset that limit its external validity. Our primary contribution is a suite of new datasets derived from US Census surveys that extend the existing data ecosystem for research on fair machine learning. We create prediction tasks relating to income, employment, health, transportation, and housing. The data span multiple years and all states of the United States, allowing researchers to study temporal shift and geographic variation. We highlight a broad initial sweep of new empirical insights relating to trade-offs between fairness criteria, performance of algorithmic interventions, and the role of distribution shift based on our new datasets. Our findings inform ongoing debates, challenge some existing narratives, and point to future research directions. Our datasets are available at https://github.com/zykls/folktables.
Excited state, non-adiabatic dynamics of large photoswitchable molecules using a chemically transferable machine learning potential
Authors: Simon Axelrod, Eugene Shakhnovich, Rafael Gómez-Bombarelli
Date: 2021-08-10 00:00:00

Light-induced chemical processes are ubiquitous in nature and have widespread technological applications. For example, photoisomerization can allow a drug with a photo-switchable scaffold such as azobenzene to be activated with light. In principle, photoswitches with desired photophysical properties like high isomerization quantum yields can be identified through virtual screening with reactive simulations. In practice, these simulations are rarely used for screening, since they require hundreds of trajectories and expensive quantum chemical methods to account for non-adiabatic excited state effects. Here we introduce a diabatic artificial neural network (DANN) based on diabatic states to accelerate such simulations for azobenzene derivatives. The network is six orders of magnitude faster than the quantum chemistry method used for training. DANN is transferable to azobenzene molecules outside the training set, predicting quantum yields for unseen species that are correlated with experiment. We use the model to virtually screen 3,100 hypothetical molecules, and identify novel species with extremely high predicted quantum yields. The model predictions are confirmed using high accuracy non-adiabatic dynamics. Our results pave the way for fast and accurate virtual screening of photoactive compounds.
Optimal learning of quantum Hamiltonians from high-temperature Gibbs states
Authors: Jeongwan Haah, Robin Kothari, Ewin Tang
Date: 2021-08-10 00:00:00
Tasks: Vocal Bursts Intensity Prediction

We study the problem of learning a Hamiltonian $H$ to precision $\varepsilon$, supposing we are given copies of its Gibbs state $\rho=\exp(-\beta H)/\operatorname{Tr}(\exp(-\beta H))$ at a known inverse temperature $\beta$. Anshu, Arunachalam, Kuwahara, and Soleimanifar (Nature Physics, 2021, arXiv:2004.07266) recently studied the sample complexity (number of copies of $\rho$ needed) of this problem for geometrically local $N$-qubit Hamiltonians. In the high-temperature (low $\beta$) regime, their algorithm has sample complexity poly$(N, 1/\beta,1/\varepsilon)$ and can be implemented with polynomial, but suboptimal, time complexity. In this paper, we study the same question for a more general class of Hamiltonians. We show how to learn the coefficients of a Hamiltonian to error $\varepsilon$ with sample complexity $S = O(\log N/(\beta\varepsilon)^{2})$ and time complexity linear in the sample size, $O(S N)$. Furthermore, we prove a matching lower bound showing that our algorithm's sample complexity is optimal, and hence our time complexity is also optimal. In the appendix, we show that virtually the same algorithm can be used to learn $H$ from a real-time evolution unitary $e^{-it H}$ in a small $t$ regime with similar sample and time complexity.
Post-hoc Interpretability for Neural NLP: A Survey
Authors: Andreas Madsen, Siva Reddy, Sarath Chandar
Date: 2021-08-10 00:00:00
Tasks: Survey

Neural networks for NLP are becoming increasingly complex and widespread, and there is a growing concern if these models are responsible to use. Explaining models helps to address the safety and ethical concerns and is essential for accountability. Interpretability serves to provide these explanations in terms that are understandable to humans. Additionally, post-hoc methods provide explanations after a model is learned and are generally model-agnostic. This survey provides a categorization of how recent post-hoc interpretability methods communicate explanations to humans, it discusses each method in-depth, and how they are validated, as the latter is often a common concern.
Icelandic Parallel Abstracts Corpus
Authors: Haukur Barri Símonarson, Vésteinn Snæbjarnarson
Date: 2021-08-11 00:00:00
Tasks: NMT, Sentence, Translation

We present a new Icelandic-English parallel corpus, the Icelandic Parallel Abstracts Corpus (IPAC), composed of abstracts from student theses and dissertations. The texts were collected from the Skemman repository which keeps records of all theses, dissertations and final projects from students at Icelandic universities. The corpus was aligned based on sentence-level BLEU scores, in both translation directions, from NMT models using Bleualign. The result is a corpus of 64k sentence pairs from over 6 thousand parallel abstracts.
Position-based Contributive Embeddings for Aspect-Based Sentiment Analysis
Authors: Zijian Zhang, Chenxin Zhang, Jiangfeng Li, Qinpei Zhao
Date: 2021-08-11 00:00:00
Tasks: Aspect-Based Sentiment Analysis, Aspect-Based Sentiment Analysis (ABSA), Position, Sentence, Sentiment Analysis

Aspect-based sentiment analysis (ABSA), exploring sentiment polarity of aspect-given sentence, is a fine-grained task in the field of nature language processing. Previously researches typically tend to predict polarity based on the meaning of aspect and opinions. However, those approaches mainly focus on considering relations implicitly at the word level, ignore the historical impact of other positional words when the aspect appears in a certain position. Therefore, we propose a Position-based Contributive Embeddings (PosCE) to highlight the historical reference to special position aspect. Contribution of each positional words to the polarity is similar to the process of fairly distributing gains to several actors working in coalition (game theory). Therefore, we quote from the method of Shapley Value and finally gain PosCE to enhance the aspect-based representation for ABSA task. Furthermore, the PosCE can also be used for improving performances on multimodal ABSA task. Extensive experiments on both text and text-audio level using SemEval dataset show that the mainstream models advance performance in accuracy and F1 (increase 2.82% and 4.21% on average respectively) by applying our PosCE.
Perturbing Inputs for Fragile Interpretations in Deep Natural Language Processing
Authors: Sanchit Sinha, Hanjie Chen, Arshdeep Sekhon, Yangfeng Ji, Yanjun Qi
Date: 2021-08-11 00:00:00
Proceeding: EMNLP (BlackboxNLP) 2021 11
Tasks: Language Modeling, Language Modelling

Interpretability methods like Integrated Gradient and LIME are popular choices for explaining natural language model predictions with relative word importance scores. These interpretations need to be robust for trustworthy NLP applications in high-stake areas like medicine or finance. Our paper demonstrates how interpretations can be manipulated by making simple word perturbations on an input text. Via a small portion of word-level swaps, these adversarial perturbations aim to make the resulting text semantically and spatially similar to its seed input (therefore sharing similar interpretations). Simultaneously, the generated examples achieve the same prediction label as the seed yet are given a substantially different explanation by the interpretation methods. Our experiments generate fragile interpretations to attack two SOTA interpretation methods, across three popular Transformer models and on two different NLP datasets. We observe that the rank order correlation drops by over 20% when less than 10% of words are perturbed on average. Further, rank-order correlation keeps decreasing as more words get perturbed. Furthermore, we demonstrate that candidates generated from our method have good quality metrics.
Computing Diverse Sets of High Quality TSP Tours by EAX-Based Evolutionary Diversity Optimisation
Authors: Adel Nikfarjam, Jakob Bossek, Aneta Neumann, Frank Neumann
Date: 2021-08-11 00:00:00
Tasks: Diversity, Evolutionary Algorithms

Evolutionary algorithms based on edge assembly crossover (EAX) constitute some of the best performing incomplete solvers for the well-known traveling salesperson problem (TSP). Often, it is desirable to compute not just a single solution for a given problem, but a diverse set of high quality solutions from which a decision maker can choose one for implementation. Currently, there are only a few approaches for computing a diverse solution set for the TSP. Furthermore, almost all of them assume that the optimal solution is known. In this paper, we introduce evolutionary diversity optimisation (EDO) approaches for the TSP that find a diverse set of tours when the optimal tour is known or unknown. We show how to adopt EAX to not only find a high-quality solution but also to maximise the diversity of the population. The resulting EAX-based EDO approach, termed EAX-EDO is capable of obtaining diverse high-quality tours when the optimal solution for the TSP is known or unknown. A comparison to existing approaches shows that they are clearly outperformed by EAX-EDO.
Hardware realisation of nonlinear dynamical systems for and from biology
Authors: Hamid Soleimani
Date: 2021-08-10 00:00:00

The focus of this thesis is on the applications of nonlinear dynamical systems in bioengineering which are mainly used in large-scale and generally categorised into two groups: (1) dynamical systems from biology (2) dynamical systems for biology. The mathematical models describing the dynamical systems used in the above systems can be simulated with the use of powerful software such as MATLAB, however, for large-scale simulations software begins to collapse. Besides, computer-based simulations are not always suitable for interfacing with biological/physical systems where continuous monitoring with low power and area consumption might be required. To alleviate these issues, a few novel hardware techniques for both aforementioned groups are proposed and their hardware results compared and validated by software simulations.