pwc-archive/papers-with-abstracts huggingface dataset:
Learning to Solve an Order Fulfillment Problem in Milliseconds with Edge-Feature-Embedded Graph Attention
Authors: Jingwei Yang, Qingchun Hou, Xiaoqing Wang, Yang Wei, Yuming Deng, Hongyang Jia, Ning Zhang
Date: 2021-09-29 00:00:00
Tasks: Combinatorial Optimization, Graph Attention, Management

The order fulfillment problem is one of the fundamental combinatorial  optimization problems in supply chain management and it is  required to be solved in real-time for modern online retailing. Such a problem is computationally hard to address by exact mathematical programming methods.  In this paper, we propose a machine learning method to solve it in milliseconds by  formulating a tripartite graph and learning the best assignment policy through the proposed edge-feature-embedded graph attention mechanism. The edge-feature-embedded graph attention considers the high-dimensional edge features and accounts for the heterogeneous information, which are important characteristics of the studied  optimization problem. The model is also size-invariant for problem instances of any scale, and it can address cases that are completely unseen during training. Experiments show that our model substantially outperforms the baseline heuristic method in optimality. The online inference time is milliseconds, which is thousands of times faster than the exact mathematical programming methods.
LEARNING DISTRIBUTIONS GENERATED BY SINGLE-LAYER RELU NETWORKS IN THE PRESENCE OF ARBITRARY OUTLIERS
Authors: Saikiran Bulusu, Geethu Joseph, M. Cenk Gursoy, Pramod Varshney
Date: 2021-09-29 00:00:00

We consider a set of data samples such that a constant fraction of the samples are arbitrary outliers and the rest are the output samples of a single-layer neural network (NN) with rectified linear unit (ReLU) activation. The goal of this paper is to estimate the parameters (weight matrix and bias vector) of the NN assuming the bias vector to be non-negative. Our proposed method is a two-step algorithm. We first estimate the norms of the rows of the weight matrix and the bias vector using the gradient descent algorithm. Here, we also incorporate either the median or the trimmed mean based filters to mitigate the effect of the arbitrary outliers. Next, we estimate the angles between any two row vectors of the weight matrix. Combining the estimates of the norms and the angles, we obtain the final estimate of the weight matrix. Further, we prove that ${O}(\frac{1}{\epsilon p^4}\log\frac{d}{\delta})$ samples are sufficient for our algorithm to estimate the NN parameters within an error of $\epsilon$ with probability $1-\delta$ when the probability of a sample being uncorrupted is $p$ and the problem dimension is $d$. Our theoretical and simulation results provide insights on how the estimation of the NN parameters depends on the probability of a sample being uncorrupted, the number of samples, and the problem dimension.
ANOMALY DETECTION WITH FRAME-GROUP ATTENTION IN SURVEILLANCE VIDEOS
Authors: Jinsheng Xiao, Haowen Guo, Yuanxu Wu, Yunhua Chen, Honggang Xie
Date: 2021-09-29 00:00:00
Tasks: Anomaly Detection

The paper proposes an end-to-end abnormal behavior detection network to detect strenuous movements in slow moving crowds, such as running, bicycling, throwing from a height. The algorithm forms continuous video frames into a frame group and uses the frame-group feature extractor to obtain the spatio-temporal information. The implicit vector based attention mechanism will work on the extracted frame-group features to highlight the important features. We use fully connected layers to transform the space and reduce the computation. Finally, the group-pooling maps the processed frame-group features to the abnormal scores. The network input is flexible to cope with the form of video streams, and the network output is the abnormal score. The designed compound loss function will help the model improve the classification performance. This paper arranges several commonly used anomaly detection datasets and tests the algorithms on the integrated dataset. The experimental results show that the proposed algorithm has significant advantages in many objective metrics compared with other anomaly detection algorithms.
CONTEXT AUGMENTATION AND FEATURE REFINEMENT NETWORK FOR TINY OBJECT DETECTION
Authors: Jinsheng Xiao, Tao Zhao, Yuntao Yao, Qiuze Yu, Yunhua Chen
Date: 2021-09-29 00:00:00
Tasks: object-detection, Object Detection

Tiny objects are hard to detect due to their low resolution and small size. The poor detection performance of tiny objects is mainly caused by the limitation of network and the imbalance of training dataset. A new feature pyramid network is proposed to combine context augmentation and feature refinement. The features from multi-scale dilated convolution are fused and injected into feature pyramid network from top to bottom to supplement context information. The channel and spatial feature refinement mechanism is introduced to suppress the conflicting formation in multi-scale feature fusion and prevent tiny objects from being submerged in the conflict information. In addition, a data enhancement method called copy-reduce-paste is proposed, which can increase the contribution of tiny objects to loss during training, ensuring a more balanced training. Experimental results show that the mean average precision of target targets on the VOC dataset of the proposed network reaches 16.9% (IOU=0.5:0.95), which is 3.9% higher than YOLOV4, 7.7% higher than CenterNet, and 5.3% higher than RefineDet.
Fast fixed-backbone protein sequence and rotamer design
Authors: Namrata Anand, Tudor Achim
Date: 2021-09-29 00:00:00
Tasks: Graph Attention, Imitation Learning, Protein Design

Protein fixed-backbone sequence design is an important task in computational protein design, and being able to quickly and accurately modify or redesign side-chains is a useful subroutine in the context of functional design problems such as ligand binding site, enzyme, and binder design. We present a fast and accurate learned method for protein fixed-backbone sequence and rotamer design. We find that a graph attention model for joint rotamer and sequence prediction trained on-policy via imitation learning can produce a distributions of accurate sequences for target backbones. We show that this method generalizes to design sequences onto novel generated backbones.
Long-term Prediction of Nonlinear Time Series Using Autoencoder and Echo State Networks
Authors: Shahrokh Shahi, Flavio H. Fenton, Elizabeth M. Cherry
Date: 2021-09-29 00:00:00
Proceeding: ICLR 2022
Tasks: Prediction, Time Series, Time Series Analysis, Time Series Forecasting

Echo state networks (ESNs) provide a computationally efficient approach for multi-step-ahead prediction of complex time series, especially for highly nonlinear or chaotic cases, where the prediction superiority of commonly used deep-learning techniques does not carry over. However, due to the random nature of ESNs and the intrinsic sensitivity of such networks to the hyperparameters and the values of the untrained network parameters, finding an appropriate set of values is a challenging step in constructing such networks. Consequently, despite their efficiency, ESNs are rarely considered a practical forecasting technique in real-world applications. 

Building on recent results in long-term prediction of complex nonlinear time series, this work introduces and evaluates an integrated architecture in which a long short-term memory (LSTM) autoencoder is integrated into the ESN framework. In this approach, the autoencoder learns a compressed representation of the input nonlinear time series. Then, the trained encoder serves as a feature extraction component feeding the learned features into the recurrent reservoir in the echo state network (ESN).

The proposed approach is evaluated using synthetic and real-world experimental complex time series representing voltage recordings from cardiac cells, which have been shown to exhibit nonlinear and chaotic behavior. Not only do the computational results demonstrate improvements in prediction accuracy and robustness compared to mainstream data-driven approaches for chaotic time series forecasting, such as the baseline and physics-informed ESNs, but the representation provided by the feature extraction component in this technique also removes the requirement in related previous work for explicitly introducing additional exogenous time series to the network, which normally are not available in real-world applications.
Logarithmic landscape and power-law escape rate of SGD
Authors: Takashi Mori, Liu Ziyin, Kangqiao Liu, Masahito Ueda
Date: 2021-09-29 00:00:00

Stochastic gradient descent (SGD) undergoes complicated multiplicative noise for the mean-square loss. We use this property of the SGD noise to derive a stochastic differential equation (SDE) with simpler additive noise by performing a random time change. In the SDE, the loss gradient is replaced by the logarithmized loss gradient. By using this formalism, we obtain the escape rate formula from a local minimum, which is determined not by the loss barrier height $\Delta L=L(\theta^s)-L(\theta^*)$ between a minimum $\theta^*$ and a saddle $\theta^s$ but by the logarithmized loss barrier height $\Delta\log L=\log[L(\theta^s)/L(\theta^*)]$. Our escape-rate formula strongly depends on the typical magnitude $h^*$ and the number $n$ of the outlier eigenvalues of the Hessian. This result explains an empirical fact that SGD prefers flat minima with low effective dimensions, which gives an insight into implicit biases of SGD.
Breaking Down Questions for Outside-Knowledge VQA
Authors: Jialin Wu, Ray Mooney
Date: 2021-09-29 00:00:00
Tasks: Graph Neural Network, Question Answering, Visual Question Answering, Visual Question Answering (VQA)

While general Visual Question Answering (VQA) focuses on querying visual content within an image, there is a recent trend towards Knowledge-Based VQA (KB-VQA) where a system needs to link some aspects of the question to different types of knowledge beyond the image, such as commonsense concepts and factual information. To address this issue, we propose a novel approach that passes knowledge from various sources between different pieces of semantic content in the question. Questions are first segmented into several chunks, and each segment is used as a key to retrieve knowledge from ConceptNet and Wikipedia. Then, a graph neural network, taking advantage of the question's syntactic structure, integrates the knowledge for different segments to jointly predict the answer. Our experiments on the OK-VQA dataset show that our approach achieves new state-of-the-art results.
FResNet++: Augmenting deep residual surrogates with Fourier operators for accelerated PDE based flow and transport simulations
Authors: Janaki Vamaraju, Faruk O. Alpak, Jim W Jennings, Pandu Devarakota, Detlef Hohl
Date: 2021-09-29 00:00:00
Proceeding: ICLR 2022

Accurate numerical modeling of multi-phase flow and transport mechanisms is essential to study varied, complex physical phenomena. State-of-the-art complete physics-based solvers suffer from many computational challenges. High-fidelity data-driven surrogate models that solve the governing PDEs have the potential to optimize the time to solution and increase confidence in critical business and engineering decisions through better quantification of solution statistics. We leverage the recently proposed Fourier neural operators with quasi-linear time complexity to capture the spectral information from feature maps. Embedding Fourier layers within the residual blocks result in a highly effective structure that, while achieving competitive accuracy, also enables efficient training of deeper networks with a drastically reduced number of trainable parameters. FResNet++ also uses squeeze and excitation blocks, Atrous Spatial Pyramidal Pooling (ASPP), and attention blocks to increase its sensitivity to the relevant features, capture multi-scale information, and simulate 2D flow properties over long time horizons. Results show a speedup of at least six orders of magnitude compared to conventional numerical simulators. We demonstrate the ability of FResNet++ to generalize over multiple high-dimensional input parameter spaces with variable controls that model the complex interplay of gravity, capillary, and viscous forces. Our experiments validate that Fourier neural operators have the potential to be applied in various CNN-based architectures and can effectively substitute for repetitive physics-based forward simulations in the context of scenario testing and solving inverse problems.
Mutual Information Minimization Based Disentangled Learning Framework For Causal Effect Estimation
Authors: Mingyuan Cheng
Date: 2021-09-29 00:00:00
Tasks: Causal Inference, Representation Learning

Learning treatment effect from observational data is a fundamental problem in causal inference. Recently, disentangled representation learning methods, such as DR-CFR and DeR-CFR, have witnessed great success in treatment effect estimation, which aim to decompose covariates into three disjoint factors. However, we argue that these methods cannot identify underlying factors well, as they cannot obtain independent disentangled factors. Inspired by the success of mutual information minimization in disentangled representation learning, we propose a novel method called MimCE in this paper: Mutual Information Minimization based Disentangled Learning Framework for Causal Effect Estimation. MimCE mainly focuses on obtaining independent disentangled factors for treatment effect estimation and numerous experiments demonstrate that it performs better than the state-of-the-art methods both on the predictive performance and model stability.
PRNet: A Progressive Regression Network for No-Reference User-Generated-Content Video Quality Assessment
Authors: Yang YangR, Bo Jiang, Kailin Wu
Date: 2021-09-29 00:00:00
Tasks: regression, Video Quality Assessment, Visual Question Answering (VQA)

Non-professional video, commonly known as User Generated Content (UGC) has become very popular in today’s video sharing applications. However, objectively perceptual quality assessment of UGC-videos is still a challenge problem, which is arose from many reasons. First, the pristine sources of UGC-videos are not available, which makes the appropriate technique is the no-reference NR video quality assessment VQA (NR-VQA). Another factor leads the NR-UGC-VQA to a challenge is that subjective mean option scores (MOS) of all the UGC-datasets are not uniformly distributed. The largest UGC video dataset---YouTube-UGC still faces a problem that the database has right-skewed MOS distribution. In addition, authentic degradations occurred in the videos are not unique, therefore, not predicable. For example, an over- or under-exposure image/video, brightness and contrast static information is important for evaluation. Only employing verified priori statistic knowledge or generalized learning knowledge may not cover all possible distortions. To solve these problems, we introduce a novel NR-VQA framework---Progressive Regress Network (PRNet) in this paper. For the skewed MOS problem, a progressive regression model is proposed, which utilizes the coarse-to-fine strategy during the training process. This strategy can turn sparse subjective human rating scores into integers with denser samples, which can solve the in-balanced sample problem and make the training progress smoother. For the unpredictable distortions problem, a wide and deep model based on our PRNet is developed, which employs both low-level features generated from natural scene statistics (NSS) and high-level semantic features extracted by deep neural networks, to fuse memorizing priori knowledge and generalizing learning features. Our experimental results demonstrate that our proposed method PRNet achieves state-of-the-art performance in currently three main popular UGC-VQA datasets (KoNVid-1K, LIVE-VQC, and YouTube-UGC).
How does BERT address polysemy of Korean adverbial postpositions -ey, -eyse, and -(u)lo?
Authors: Seongmin Mun, Guillaume Desagulier, Gyu-Ho Shin
Date: 2021-09-29 00:00:00

The present study reports computational accounts of resolving word-level polysemy in a lesser-studied language—Korean. Postpositions, which are characterized as multiple form-function mapping and thus polysemous, pose a challenge to automatic analysis and model performance in identifying their functions. In this study, we devised a classification model by employing BERT and introduces a computational simulation that interactively demonstrates how a BERT model simulates human interpretation of word-level polysemy involving Korean adverbial postpositions -ey, -eyse, and -(u)lo. Results reveal that (i) there is an inverse relationship between the classification accuracy and the number of functions that each postposition manifests, (ii) the model performance is affected by the corpus size of each function, and (iii) the performance gradually improves as the epoch proceeds.
Unsupervised Artifact Detection for Whole Slide Images of Prostate Biopsies
Authors: Amit Suveer, Walter de Back, Nadieh Khalili, Yijiang Chen, Kristian Eurén, Andrew Janowczyk, Arto Järvinen
Date: 2021-09-29 00:00:00
Proceeding: ICLR 2022
Tasks: Artifact Detection, One-Class Classification, whole slide images

High-quality image digitisation of histological slides is essential for digital pathology to facilitate diagnosis and to develop reliable computer-aided assistance systems. Currently, image quality control (QC) to identify artefacts that result from slide preparation, staining, or scanning is mainly conducted manually, which is time-consuming and prone to errors. The few existing automated approaches are limited to specific artefacts, require manual parameter tuning, or depend on large amounts of annotated data for training.

Here, we propose an unsupervised and training-free method for artefact detection of digitised histological slides based on one-class classification. By assuming that deep representations from normal (in-distribution) tissue samples are dense and compact in feature space, artefacts can be identified by their distance to the mean in-distribution representation. The use of representations from convolutional neural networks pre-trained on natural images (ImageNet) avoids the need for the collection of annotated data and the costs of training. 

We demonstrate the efficacy of our approach for the detection of various types of artefacts (out-of-focus, pen marks, tissue folds, etc.) in an open dataset of prostate biopsy whole-slide images. We compare the proposed distance-based artefact detection with conventional task-specific artefact-detection methods as well as with approaches based on uncertainty estimation. Our method achieved a mean AUROC value of 0.82 on our test dataset consisting of 78 prostate biopsy whole-slide images annotated with five varieties of artefacts. Our proposed method can be utilised for slide-level QC as well as the identification of artefacts within a slide. It is a simple, efficient, yet robust method for automated QC of digitized histological slides.
Adaptive Inertia: Disentangling the Effects of Adaptive Learning Rate and Momentum
Authors: Zeke Xie, Xinrui Wang, Huishuai Zhang, Issei Sato, Masashi Sugiyama
Date: 2021-09-29 00:00:00

Adaptive Momentum Estimation (Adam), which combines Adaptive Learning Rate and Momentum, would be the most popular stochastic optimizer for accelerating the training of deep neural networks. However, empirically Adam often generalizes worse than Stochastic Gradient Descent (SGD). We unveil the mystery of this behavior in the diffusion theoretical framework. Specifically, we disentangle the effects of Adaptive Learning Rate and Momentum of the Adam dynamics on saddle-point escaping and flat minima selection. We prove that Adaptive Learning Rate can escape saddle points efficiently, but cannot select flat minima as SGD does. In contrast, Momentum provides a drift effect to help the training process pass through saddle points, and almost does not affect flat minima selection. This partly explains why SGD (with Momentum) generalizes better, while Adam generalizes worse but converges faster. Furthermore, motivated by the analysis, we design a novel adaptive optimization framework named Adaptive Inertia, which uses parameter-wise adaptive inertia to accelerate the training and provably favors flat minima as well as SGD. Our extensive experiments demonstrate that the proposed adaptive inertia method can generalize significantly better than SGD and conventional adaptive gradient methods.
What to expect of hardware metric predictors in NAS
Authors: Kevin Alexander Laube, Maximus Mutschler, Andreas Zell
Date: 2021-09-29 00:00:00
Tasks: Neural Architecture Search, Prediction

Modern Neural Architecture Search (NAS) focuses on finding the best performing architectures in hardware-aware settings; e.g., those with an optimal tradeoff of accuracy and latency. Due to many advantages of prediction models over live measurements, the search process is often guided by estimates of how well each considered network architecture performs on the desired metrics. Typical predic-
tion models range from operation-wise lookup tables over gradient-boosted trees and neural networks, with little known information on how they compare. We evaluate 18 different performance predictors on ten combinations of metrics, devices, network types, and training tasks, and find that MLP models are the most promising. We then simulate and evaluate how the guidance of such prediction models affects the subsequent architecture selection. Due to inaccurate predictions, the selected architectures are generally suboptimal, which we quantify as
an expected reduction in accuracy and hypervolume. We show that simply verifying the predictions of just the selected architectures can lead to substantially improved results. Under a time budget, we find it preferable to use a fast and inaccurate prediction model over accurate but slow live measurements.
A multi-domain splitting framework for time-varying graph structure
Authors: Zehua Yu, Xianwei Zheng, Zhulun Yang, Xutao Li
Date: 2021-09-29 00:00:00
Tasks: Anomaly Detection

The Graph Signal Processing (GSP) methods are widely used to solve structured data analysis problems, assuming that the data structure is fixed. In the recent GSP community, anomaly detection on datasets with the time-varying structure is an open challenge. To address the anomaly detection problem for datasets with a spatial-temporal structure, in this work, we propose a novel graph multi-domain splitting framework, called GMDS, by integrating the time, vertex, and frequency features to locate the anomalies. Firstly, by introducing the discrete wavelet transform into vertex function, we design a splitting approach for separating the graph sequences into several sub-sequences adaptively. Then, we specifically design an adjacency function in the vertex domain to generate the adjacency matrix adaptively. At last, by utilizing the learned graphs to the spectral graph wavelet transform, we design a module to extract vertices features in the frequency domain. To validate the effectiveness of our framework, we apply GMDS in the anomaly detection of actual traffic flow and urban datasets and compare its performances with acknowledged baselines. The experimental results show that our proposed framework outperforms all the baselines, which distinctly demonstrate the validity of GMDS.