pwc-archive/papers-with-abstracts huggingface dataset:
Deep Kronecker neural networks: A general framework for neural networks with adaptive activation functions
Authors: Ameya D. Jagtap, Yeonjong Shin, Kenji Kawaguchi, George Em Karniadakis
Date: 2021-05-20 00:00:00

We propose a new type of neural networks, Kronecker neural networks (KNNs), that form a general framework for neural networks with adaptive activation functions. KNNs employ the Kronecker product, which provides an efficient way of constructing a very wide network while keeping the number of parameters low. Our theoretical analysis reveals that under suitable conditions, KNNs induce a faster decay of the loss than that by the feed-forward networks. This is also empirically verified through a set of computational examples. Furthermore, under certain technical assumptions, we establish global convergence of gradient descent for KNNs. As a specific case, we propose the Rowdy activation function that is designed to get rid of any saturation region by injecting sinusoidal fluctuations, which include trainable parameters. The proposed Rowdy activation function can be employed in any neural network architecture like feed-forward neural networks, Recurrent neural networks, Convolutional neural networks etc. The effectiveness of KNNs with Rowdy activation is demonstrated through various computational experiments including function approximation using feed-forward neural networks, solution inference of partial differential equations using the physics-informed neural networks, and standard deep learning benchmark problems using convolutional and fully-connected neural networks.
Physics-informed neural networks (PINNs) for fluid mechanics: A review
Authors: Shengze Cai, Zhiping Mao, Zhicheng Wang, Minglang Yin, George Em Karniadakis
Date: 2021-05-20 00:00:00

Despite the significant progress over the last 50 years in simulating flow problems using numerical discretization of the Navier-Stokes equations (NSE), we still cannot incorporate seamlessly noisy data into existing algorithms, mesh-generation is complex, and we cannot tackle high-dimensional problems governed by parametrized NSE. Moreover, solving inverse flow problems is often prohibitively expensive and requires complex and expensive formulations and new computer codes. Here, we review flow physics-informed learning, integrating seamlessly data and mathematical models, and implementing them using physics-informed neural networks (PINNs). We demonstrate the effectiveness of PINNs for inverse problems related to three-dimensional wake flows, supersonic flows, and biomedical flows.
Contrastive Learning for Many-to-many Multilingual Neural Machine Translation
Authors: Xiao Pan, Mingxuan Wang, Liwei Wu, Lei LI
Date: 2021-05-20 00:00:00
Proceeding: ACL 2021 5
Tasks: Contrastive Learning, Data Augmentation, Machine Translation, Translation

Existing multilingual machine translation approaches mainly focus on English-centric directions, while the non-English directions still lag behind. In this work, we aim to build a many-to-many translation system with an emphasis on the quality of non-English language directions. Our intuition is based on the hypothesis that a universal cross-language representation leads to better multilingual translation performance. To this end, we propose mRASP2, a training method to obtain a single unified multilingual translation model. mRASP2 is empowered by two techniques: a) a contrastive learning scheme to close the gap among representations of different languages, and b) data augmentation on both multiple parallel and monolingual data to further align token representations. For English-centric directions, mRASP2 outperforms existing best unified model and achieves competitive or even better performance than the pre-trained and fine-tuned model mBART on tens of WMT's translation directions. For non-English directions, mRASP2 achieves an improvement of average 10+ BLEU compared with the multilingual Transformer baseline. Code, data and trained models are available at https://github.com/PANXiao1994/mRASP2.
A Preference Random Walk Algorithm for Link Prediction through Mutual Influence Nodes in Complex Networks
Authors: Kamal Berahmand, Elahe Nasiri, Saman Forouzandeh, Yuefeng Li
Date: 2021-05-20 00:00:00
Tasks: Link Prediction

Predicting links in complex networks has been one of the essential topics within the realm of data mining and science discovery over the past few years. This problem remains an attempt to identify future, deleted, and redundant links using the existing links in a graph. Local random walk is considered to be one of the most well-known algorithms in the category of quasi-local methods. It traverses the network using the traditional random walk with a limited number of steps, randomly selecting one adjacent node in each step among the nodes which have equal importance. Then this method uses the transition probability between node pairs to calculate the similarity between them. However, in most datasets, this method is not able to perform accurately in scoring remarkably similar nodes. In the present article, an efficient method is proposed for improving local random walk by encouraging random walk to move, in every step, towards the node which has a stronger influence. Therefore, the next node is selected according to the influence of the source node. To do so, using mutual information, the concept of the asymmetric mutual influence of nodes is presented. A comparison between the proposed method and other similarity-based methods (local, quasi-local, and global) has been performed, and results have been reported for 11 real-world networks. It had a higher prediction accuracy compared with other link prediction approaches.
Localization and Control of Magnetic Suture Needles in Cluttered Surgical Site with Blood and Tissue
Authors: Will Pryor, Yotam Barnoy, Suraj Raval, Xiaolong Liu, Lamar Mair, Daniel Lerner, Onder Erin, Gregory D. Hager, Yancy Diaz-Mercado, Axel Krieger
Date: 2021-05-20 00:00:00
Tasks: Position, Visual Localization

Real-time visual localization of needles is necessary for various surgical applications, including surgical automation and visual feedback. In this study we investigate localization and autonomous robotic control of needles in the context of our magneto-suturing system. Our system holds the potential for surgical manipulation with the benefit of minimal invasiveness and reduced patient side effects. However, the non-linear magnetic fields produce unintuitive forces and demand delicate position-based control that exceeds the capabilities of direct human manipulation. This makes automatic needle localization a necessity. Our localization method combines neural network-based segmentation and classical techniques, and we are able to consistently locate our needle with 0.73 mm RMS error in clean environments and 2.72 mm RMS error in challenging environments with blood and occlusion. The average localization RMS error is 2.16 mm for all environments we used in the experiments. We combine this localization method with our closed-loop feedback control system to demonstrate the further applicability of localization to autonomous control. Our needle is able to follow a running suture path in (1) no blood, no tissue; (2) heavy blood, no tissue; (3) no blood, with tissue; and (4) heavy blood, with tissue environments. The tip position tracking error ranges from 2.6 mm to 3.7 mm RMS, opening the door towards autonomous suturing tasks.
Decomposing reverse-mode automatic differentiation
Authors: Roy Frostig, Matthew J. Johnson, Dougal Maclaurin, Adam Paszke, Alexey Radul
Date: 2021-05-20 00:00:00

We decompose reverse-mode automatic differentiation into (forward-mode) linearization followed by transposition. Doing so isolates the essential difference between forward- and reverse-mode AD, and simplifies their joint implementation. In particular, once forward-mode AD rules are defined for every primitive operation in a source language, only linear primitives require an additional transposition rule in order to arrive at a complete reverse-mode AD implementation. This is how reverse-mode AD is written in JAX and Dex.
A Review of Autonomous Road Vehicle Integrated Approaches to an Emergency Obstacle Avoidance Maneuver
Authors: Evan Lowe, Levent Guvenç
Date: 2021-05-20 00:00:00
Tasks: Autonomous Vehicles, Decision Making

As passenger vehicle technologies have advanced, so have their capabilities to avoid obstacles, especially with developments in tires, suspensions, steering, as well as safety technologies like ABS, ESC, and more recently, ADAS systems. However, environments around passenger vehicles have also become more complex, and dangerous. There have previously been studies that outline driver tendencies and performance capabilities when attempting to avoid obstacles while driving passenger vehicles. Now that autonomous vehicles are being developed with obstacle avoidance capabilities, it is important to target performance that meets or exceeds that of human drivers. This manuscript highlights systems that are crucial for an emergency obstacle avoidance maneuver (EOAM) and identifies the state-of-the-art for each of the related systems, while considering the nuances of traveling at highway speeds. Some of the primary EOAM-related systems/areas that are discussed in this review are: general path planning methods, system hierarchies, decision-making, trajectory generation, and trajectory-tracking control methods. After concluding remarks, suggestions for future work which could lead to an ideal EOAM development, are discussed.
L1 Regression with Lewis Weights Subsampling
Authors: Aditya Parulekar, Advait Parulekar, Eric Price
Date: 2021-05-19 00:00:00
Tasks: regression

We consider the problem of finding an approximate solution to $\ell_1$ regression while only observing a small number of labels. Given an $n \times d$ unlabeled data matrix $X$, we must choose a small set of $m \ll n$ rows to observe the labels of, then output an estimate $\widehat{\beta}$ whose error on the original problem is within a $1 + \varepsilon$ factor of optimal. We show that sampling from $X$ according to its Lewis weights and outputting the empirical minimizer succeeds with probability $1-\delta$ for $m > O(\frac{1}{\varepsilon^2} d \log \frac{d}{\varepsilon \delta})$. This is analogous to the performance of sampling according to leverage scores for $\ell_2$ regression, but with exponentially better dependence on $\delta$. We also give a corresponding lower bound of $\Omega(\frac{d}{\varepsilon^2} + (d + \frac{1}{\varepsilon^2}) \log\frac{1}{\delta})$.
DistTune: Distributed Fine-Grained Adaptive Traffic Speed Prediction for Growing Transportation Networks
Authors: Ming-Chang Lee, Jia-Chun Lin, Ernst Gunnar Gran
Date: 2021-05-19 00:00:00
Tasks: Prediction

Over the past decade, many approaches have been introduced for traffic speed prediction. However, providing fine-grained, accurate, time-efficient, and adaptive traffic speed prediction for a growing transportation network where the size of the network keeps increasing and new traffic detectors are constantly deployed has not been well studied. To address this issue, this paper presents DistTune based on Long Short-Term Memory (LSTM) and the Nelder-Mead method. Whenever encountering an unprocessed detector, DistTune decides if it should customize an LSTM model for this detector by comparing the detector with other processed detectors in terms of the normalized traffic speed patterns they have observed. If similarity is found, DistTune directly shares an existing LSTM model with this detector to achieve time-efficient processing. Otherwise, DistTune customizes an LSTM model for the detector to achieve fine-grained prediction. To make DistTune even more time-efficient, DistTune performs on a cluster of computing nodes in parallel. To achieve adaptive traffic speed prediction, DistTune also provides LSTM re-customization for detectors that suffer from unsatisfactory prediction accuracy due to for instance traffic speed pattern change. Extensive experiments based on traffic data collected from freeway I5-N in California are conducted to evaluate the performance of DistTune. The results demonstrate that DistTune provides fine-grained, accurate, time-efficient, and adaptive traffic speed prediction for a growing transportation network.
Speech & Song Emotion Recognition Using Multilayer Perceptron and Standard Vector Machine
Authors: Behzad Javaheri
Date: 2021-05-19 00:00:00
Tasks: Data Augmentation, Emotion Recognition

Herein, we have compared the performance of SVM and MLP in emotion recognition using speech and song channels of the RAVDESS dataset. We have undertaken a journey to extract various audio features, identify optimal scaling strategy and hyperparameter for our models. To increase sample size, we have performed audio data augmentation and addressed data imbalance using SMOTE. Our data indicate that optimised SVM outperforms MLP with an accuracy of 82 compared to 75%. Following data augmentation, the performance of both algorithms was identical at ~79%, however, overfitting was evident for the SVM. Our final exploration indicated that the performance of both SVM and MLP were similar in which both resulted in lower accuracy for the speech channel compared to the song channel. Our findings suggest that both SVM and MLP are powerful classifiers for emotion recognition in a vocal-dependent manner.
Separation of Powers in Federated Learning
Authors: Pau-Chen Cheng, Kevin Eykholt, Zhongshu Gu, Hani Jamjoom, K. R. Jayaram, Enriquillo Valdez, Ashish Verma
Date: 2021-05-19 00:00:00
Tasks: Federated Learning

Federated Learning (FL) enables collaborative training among mutually distrusting parties. Model updates, rather than training data, are concentrated and fused in a central aggregation server. A key security challenge in FL is that an untrustworthy or compromised aggregation process might lead to unforeseeable information leakage. This challenge is especially acute due to recently demonstrated attacks that have reconstructed large fractions of training data from ostensibly "sanitized" model updates. In this paper, we introduce TRUDA, a new cross-silo FL system, employing a trustworthy and decentralized aggregation architecture to break down information concentration with regard to a single aggregator. Based on the unique computational properties of model-fusion algorithms, all exchanged model updates in TRUDA are disassembled at the parameter-granularity and re-stitched to random partitions designated for multiple TEE-protected aggregators. Thus, each aggregator only has a fragmentary and shuffled view of model updates and is oblivious to the model architecture. Our new security mechanisms can fundamentally mitigate training reconstruction attacks, while still preserving the final accuracy of trained models and keeping performance overheads low.
Balancing Robustness and Sensitivity using Feature Contrastive Learning
Authors: Seungyeon Kim, Daniel Glasner, Srikumar Ramalingam, Cho-Jui Hsieh, Kishore Papineni, Sanjiv Kumar
Date: 2021-05-19 00:00:00
Tasks: Contrastive Learning, Sensitivity

It is generally believed that robust training of extremely large networks is critical to their success in real-world applications. However, when taken to the extreme, methods that promote robustness can hurt the model's sensitivity to rare or underrepresented patterns. In this paper, we discuss this trade-off between sensitivity and robustness to natural (non-adversarial) perturbations by introducing two notions: contextual feature utility and contextual feature sensitivity. We propose Feature Contrastive Learning (FCL) that encourages a model to be more sensitive to the features that have higher contextual utility. Empirical results demonstrate that models trained with FCL achieve a better balance of robustness and sensitivity, leading to improved generalization in the presence of noise on both vision and NLP datasets.
Graph Sanitation with Application to Node Classification
Authors: Zhe Xu, Boxin Du, Hanghang Tong
Date: 2021-05-19 00:00:00
Tasks: Anomaly Detection, Bilevel Optimization, Classification, Clustering, Denoising, Graph Mining, Graph Neural Network, Imputation, Node Classification

The past decades have witnessed the prosperity of graph mining, with a multitude of sophisticated models and algorithms designed for various mining tasks, such as ranking, classification, clustering and anomaly detection. Generally speaking, the vast majority of the existing works aim to answer the following question, that is, given a graph, what is the best way to mine it? In this paper, we introduce the graph sanitation problem, to answer an orthogonal question. That is, given a mining task and an initial graph, what is the best way to improve the initially provided graph? By learning a better graph as part of the input of the mining model, it is expected to benefit graph mining in a variety of settings, ranging from denoising, imputation to defense. We formulate the graph sanitation problem as a bilevel optimization problem, and further instantiate it by semi-supervised node classification, together with an effective solver named GaSoliNe. Extensive experimental results demonstrate that the proposed method is (1) broadly applicable with respect to different graph neural network models and flexible graph modification strategies, (2) effective in improving the node classification accuracy on both the original and contaminated graphs in various perturbation scenarios. In particular, it brings up to 25% performance improvement over the existing robust graph neural network methods.
Using Machine Learning Techniques to Identify Key Risk Factors for Diabetes and Undiagnosed Diabetes
Authors: Avraham Adler
Date: 2021-05-19 00:00:00
Tasks: BIG-bench Machine Learning, Nutrition

This paper reviews a wide selection of machine learning models built to predict both the presence of diabetes and the presence of undiagnosed diabetes using eight years of National Health and Nutrition Examination Survey (NHANES) data. Models are tuned and compared via their Brier Scores. The most relevant variables of the best performing models are then compared. A Support Vector Machine with a linear kernel performed best for predicting diabetes, returning a Brier score of 0.0654 and an AUROC of 0.9235 on the test set. An elastic net regression performed best for predicting undiagnosed diabetes with a Brier score of 0.0294 and an AUROC of 0.9439 on the test set. Similar features appear prominently in the models for both sets of models. Blood osmolality, family history, the prevalance of various compounds, and hypertension are key indicators for all diabetes risk. For undiagnosed diabetes in particular, there are ethnicity or genetic components which arise as strong correlates as well.
User-Level Label Leakage from Gradients in Federated Learning
Authors: Aidmar Wainakh, Fabrizio Ventola, Till Müßig, Jens Keim, Carlos Garcia Cordero, Ephraim Zimmer, Tim Grube, Kristian Kersting, Max Mühlhäuser
Date: 2021-05-19 00:00:00
Tasks: Federated Learning

Federated learning enables multiple users to build a joint model by sharing their model updates (gradients), while their raw data remains local on their devices. In contrast to the common belief that this provides privacy benefits, we here add to the very recent results on privacy risks when sharing gradients. Specifically, we investigate Label Leakage from Gradients (LLG), a novel attack to extract the labels of the users' training data from their shared gradients. The attack exploits the direction and magnitude of gradients to determine the presence or absence of any label. LLG is simple yet effective, capable of leaking potential sensitive information represented by labels, and scales well to arbitrary batch sizes and multiple classes. We mathematically and empirically demonstrate the validity of the attack under different settings. Moreover, empirical results show that LLG successfully extracts labels with high accuracy at the early stages of model training. We also discuss different defense mechanisms against such leakage. Our findings suggest that gradient compression is a practical technique to mitigate the attack.
DeepDebug: Fixing Python Bugs Using Stack Traces, Backtranslation, and Code Skeletons
Authors: Dawn Drain, Colin B. Clement, Guillermo Serrato, Neel Sundaresan
Date: 2021-05-19 00:00:00
Tasks: Program Repair

The joint task of bug localization and program repair is an integral part of the software development process. In this work we present DeepDebug, an approach to automated debugging using large, pretrained transformers. We begin by training a bug-creation model on reversed commit data for the purpose of generating synthetic bugs. We apply these synthetic bugs toward two ends. First, we directly train a backtranslation model on all functions from 200K repositories. Next, we focus on 10K repositories for which we can execute tests, and create buggy versions of all functions in those repositories that are covered by passing tests. This provides us with rich debugging information such as stack traces and print statements, which we use to finetune our model which was pretrained on raw source code. Finally, we strengthen all our models by expanding the context window beyond the buggy function itself, and adding a skeleton consisting of that function's parent class, imports, signatures, docstrings, and method bodies, in order of priority. On the QuixBugs benchmark, we increase the total number of fixes found by over 50%, while also decreasing the false positive rate from 35% to 5% and decreasing the timeout from six hours to one minute. On our own benchmark of executable tests, our model fixes 68% of all bugs on its first attempt without using traces, and after adding traces it fixes 75% on first attempt. We will open-source our framework and validation set for evaluating on executable tests.
AI-Decision Support System Interface Using Cancer Related Data for Lung Cancer Prognosis
Authors: Asim Leblebici, Omer Gesoglu, Yasemin Basbinar
Date: 2021-05-19 00:00:00
Tasks: BIG-bench Machine Learning, Prognosis

Until the beginning of 2021, lung cancer is known to be the most common cancer in the world. The disease is common due to factors such as occupational exposure, smoking and environmental pollution. The early diagnosis and treatment of the disease is of great importance as well as the prevention of the causes that cause the disease. The study was planned to create a web interface that works with machine learning algorithms to predict prognosis using lung cancer clinical and gene expression in the GDC data portal.
Deep learning for solution and inversion of structural mechanics and vibrations
Authors: Ehsan Haghighat, Ali Can Bekar, Erdogan Madenci, Ruben Juanes
Date: 2021-05-18 00:00:00
Tasks: BIG-bench Machine Learning, Deep Learning

Deep learning has been the most popular machine learning method in the last few years. In this chapter, we present the application of deep learning and physics-informed neural networks concerning structural mechanics and vibration problems. Demonstration problems involve de-noising data, solution to time-dependent ordinary and partial differential equations, and characterizing the system's response for a given data.