pwc-archive/papers-with-abstracts huggingface dataset:
Improving Sentiment Classification Using 0-Shot Generated Labels for Custom Transformer Embeddings
Authors: Ryan Bluteau, Robin Gras
Date: 2021-09-29 00:00:00
Tasks: Classification, Sentiment Analysis, Sentiment Classification

We present an approach to improve sentiment classification for transformers (based on BERT and DistilBERT) using additional embeddings to represent emotion inputs. We used HuggingFace's 0-shot prediction pipeline to generate probabilities of whether emotions apply to a given sample. We generated 0-shot probabilities for 1.6 million samples from a sentiment classification dataset and a smaller sentiment airline dataset using 63 emotions. Then we added custom tokens to BERT's embeddings and tokenizers representing various levels of emotion for each predicted emotion. Finally, depending on the probability of each emotion, the respective custom token representing that level was prepended to the text input of the model to process and train for classification. We additionally test direct classification layer addition of emotion inputs and an ensemble of BERT and DistilBERT models both using emotion inputs achieving a modest increase in sentiment prediction accuracy. Our results show modest improvement in all cases over the original model for both BERT and DistilBERT tested with added emotion inputs generated from 0-shot pretrained models.
Lottery Ticket Structured Node Pruning for Tabular Datasets
Authors: Ryan Bluteau, Robin Gras, Mitchel Paulin, Zachary Innes
Date: 2021-09-29 00:00:00

In this paper we presented two pruning approaches on tabular neural networks based on the lottery ticket hypothesis that went beyond masking nodes by resizing the models accordingly. We showed top performing models in 6 of 8 datasets tested in terms of F1/RMSE. We also showed in 6 of 8 datasets a total reduction of over 85% of nodes and many over 98% reduced with minimal affect to accuracy. In one dataset the model reached a total size of one node per layer while still improving RMSE compared to the larger model used for pruning. We presented results for two approaches, iterative pruning using two styles, and oneshot pruning. Iterative pruning gradually reduces nodes in each layers based on norm pruning until we reach the smallest state, while oneshot will prune the model directly to the smallest state. We showed that the iterative approach will obtain the best result more consistently than oneshot.
A framework of deep neural networks via the solution operator of partial differential equations
Authors: Wenqi Tao, Zuoqiang Shi
Date: 2021-09-29 00:00:00
Tasks: Form

There is a close connection between deep neural networks (DNN) and partial differential equations (PDEs). Many DNN architectures can be modeled by PDEs and have been proposed in the literature. However, their neural network design space is restricted due to the specific form of PDEs, which prevents the design of more effective neural network structures. In this paper, we attempt to derive a general form of PDEs for the design of ResNet-like DNN. To achieve this goal, we first formulate DNN as an adjustment operator applied on the base classifier. Then based on several reasonable assumption, we show the adjustment operator for ResNet-like DNN is the solution operator of PDEs. To show the effectiveness for general form of PDEs, we show that several effective networks can be interpreted by our general form of PDEs and design a training method motivated by PDEs theory to train DNN models for better robustness and less chance of overfitting. Theoretically, we prove that the robustness of DNN trained with our method is certifiable and our training method reduces the generalization gap for DNN. Furthermore, we demonstrate that DNN trained with our method can achieve better generalization and is more resistant to adversarial perturbations than baseline model.
Deep Neural Networks on EEG signals to predict Attention Score using Gramian Angular Difference Field
Authors: Mahak Kothari, Shreyansh Joshi, Adarsh Nandanwar, Veeky Baths
Date: 2021-09-29 00:00:00
Tasks: Deep Learning, EEG, Electroencephalogram (EEG), Time Series Analysis

Auditory attention is a selective type of hearing in which people focus their attention intentionally on a specific source of a sound or spoken words whilst ignoring or inhibiting other auditory stimuli. In some sense, the auditory attention score of an individual shows the focus the person can have in auditory tasks. The recent advancements in deep learning and in the non-invasive technologies recording neural activity beg the question, can deep learning along with technologies such as electroencephalography (EEG) be used to predict the auditory attention score of an individual? In this paper, we focus on this very problem of estimating a person's auditory attention level based on their brain's electrical activity captured using 14-channeled EEG signals. More specifically, we deal with attention estimation as a regression problem. The work has been performed on the publicly available Phyaat dataset. The concept of Gramian Angular Difference Field (GADF) has been used to convert time-series EEG data into an image having 14 channels, enabling us to train various deep learning models such as 2D CNN, 3D CNN, and convolutional autoencoders. Their performances have been compared amongst themselves as well as with the work done previously. Amongst the different models we tried, 2D CNN gave the best performance. It outperformed the existing methods by a decent margin of 0.22 mean absolute error (MAE).
Generalization of Overparametrized Deep Neural Network Under Noisy Observations
Authors: Namjoon Suh, Hyunouk Ko, Xiaoming Huo
Date: 2021-09-29 00:00:00
Proceeding: ICLR 2022 4

We study the generalization properties of the overparameterized deep neural network (DNN) with Rectified Linear Unit (ReLU) activations.
Under the non-parametric regression framework, it is assumed that the ground-truth function is from a reproducing kernel Hilbert space (RKHS) induced by a neural tangent kernel (NTK) of ReLU DNN, and a dataset is given with the noises. Without a delicate adoption of early stopping, we prove that the overparametrized DNN trained by vanilla gradient descent does not recover the ground-truth function. It turns out that the estimated DNN's $L_{2}$ prediction error is bounded away from $0$. As a complement of the above result, we show that the $\ell_{2}$-regularized gradient descent enables the overparametrized DNN achieve the minimax optimal convergence rate of the $L_{2}$ prediction error, without early stopping. Notably, the rate we obtained is faster than $\mathcal{O}(n^{-1/2})$ known in the literature.
$$Research on fusion algorithm of multi-attribute decision making and reinforcement learning based on intuitionistic fuzzy number in wargame environment$$
Authors: Yuxiang Sun, Bo Yuan, Yufan Xue, Jiawei Zhou, Leonardo Stella, Xianzhong Zhou
Date: 2021-09-29 00:00:00
Tasks: Attribute, Decision Making, Management, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Intelligent games have seen an increasing interest within the research community on artificial intelligence. The article proposes an algorithm that combines the multi-attribute management and reinforcement learning methods, and that joined their effect on wargaming, it solves the problem of the agent’s low rate of winning against specific rules and its inability to quickly converge during intelligent wargame training. At the same time, this paper studied a multi-attribute decision making and reinforcement learning algorithm in a wargame simulation environment, yielding data on the conflict between red and blue sides. We calculate the weight of each attribute based on the intuitionistic fuzzy number weight calculations. And then we determine the threat posed by each opponent’s game agents . Using the red side reinforcement learning reward function, the AC framework is trained on the reward function, and an algorithm combining multi-attribute decision making with reinforcement learning is obtained. A simulation experiment confirms that the algorithm of multi-attribute decision making combined with reinforcement learning presented in this paper is significantly more intelligent than the pure reinforcement learning algorithm. By resolving the shortcomings of the agent’s neural network, coupled with sparse rewards in large-map combat games, this robust algorithm effectively reduces the difficulties of convergence. It is also the first time in this field that an algorithm design for intelligent wargaming combines multi-attribute decision making with reinforcement learning. Finally, another novelty of this research is the interdisciplinary, like designing intelligent wargames and improving reinforcement learning algorithms.
Robot Intent Recognition Method Based on State Grid Business Office
Authors: Lanfang Dong, Zhao Pu Hu, Hanchao Liu
Date: 2021-09-29 00:00:00
Tasks: Intent Detection, Intent Recognition, Model Compression

Artificial intelligence is currently in an era of change, not only changing the artificial intelligence technology itself, but also changing human society. It has become more and more common to use artificial intelligence as the core human-computer interaction technology to replace manpower. Intention recognition is an important part of the human-machine dialogue system, and deep learning technology is gradually being applied to the task of intent recognition. However, intent recognition based on deep learning often has problems such as low recognition accuracy and slow recognition speed. In response to these problems, this paper designs a BERT fine-tuning to improve the network structure based on the pre-training model and proposes new continuous pre-training goals. To improve the accuracy of intent recognition, a method based on multi-teacher model compression is proposed to compress the pre-training model, which reduces the time consumption of model inference.
UNCERTAINTY QUANTIFICATION USING VARIATIONAL INFERENCE FOR BIOMEDICAL IMAGE SEGMENTATION
Authors: Anonymous
Date: 2021-09-29 00:00:00
Proceeding: MICCAI Workshop COMPAY 2021 9
Tasks: Decoder, image-classification, Image Classification, Image Generation, Image Segmentation, Semantic Segmentation, Uncertainty Quantification, Variational Inference

Deep learning motivated by convolutional neural networks has been highly successful in a range of medical imaging problems like image classification, image
segmentation, image synthesis etc. However for validation and interpretability, not
only do we need the predictions made by the model but also how confident it is
while making those predictions. This is important in safety critical applications
for the people to accept it. In this work, we used an encoder decoder architecture
based on variational inference techniques for segmenting brain tumour images. We
evaluate our work on the publicly available BRATS dataset using Dice Similarity
Coefficient (DSC) and Intersection Over Union (IOU) as the evaluation metrics.
Our model is able to segment brain tumours while taking into account both aleatoric
uncertainty and epistemic uncertainty in a principled bayesian manner.
AA-PINN: ATTENTION AUGMENTED PHYSICS INFORMED NEURAL NETWORKS
Authors: Abhinav Sagar
Date: 2021-09-29 00:00:00

Physics Informed Neural Networks has been quite successful in modelling the complex nature of fluid flow. Computational Fluid Dynamics using parallel processing
algorithms on GPUs have considerably reduced the time to solve the Navier Stokes
Equations. CFD based approaches uses approximates to make the modelling easy
but it comes at the cost of decrease in accuracy. In this paper, we propose an
attention based network architecture named AA-PINN to model PDEs behind fluid
flow. We use a combination of channel and spatial attention module. We propose a
novel loss function which is more robust in handling the initial as well as boundary
conditions imposed. Using evaluation metrics like RMSE, divergence and thermal
kinetic energy, our network outperforms previous PINNs for modelling Navier
Stokes and Burgers Equation.
Low Entropy Deep Networks
Authors: Chris Subia-Waud, Srinandan Dasmahapatra
Date: 2021-09-29 00:00:00

The movement of data between processes and memory,  not arithmetic operations, dominate the energy costs of deep learning inference calculations.  This work focuses on reducing these data movement costs by reducing the number of unique weights in a network. The thinking goes that if the number of unique weights is kept small enough, then the entire network can be distributed and stored on processing elements (PEs) within accelerator designs,  and the data movement costs for weight reads substantially reduced.  To this end, we investigate the merits of a method, which we call Weight Fixing Networks (WFN). We design the approach to realise four model outcome objectives: i) very few unique weights, ii) low-entropy weight encodings,  iii) unique weight values which are amenable to energy-saving versions of hardware multiplication, and iv)  lossless task-performance.  Some of these goals are conflicting. To best balance these conflicts, we combine a few novel (and some well-trodden) tricks; a novel regularisation term, (i, ii) a view of clustering cost as relative distance change (i, ii, iv), and a focus on whole-network re-use of weights (i, iii).  Our Imagenet experiments demonstrate lossless compression using 56x fewer unique weights and a 1.9x  lower weight-space entropy than SOTA quantisation approaches.
Decouple and Reconstruct: Mining Discriminative Features for Cross-domain Object Detection
Authors: Jiawei Wang, Konghuai Shen, Shao Ming, Jun Yin, Ming Liu
Date: 2021-09-29 00:00:00
Tasks: Object, object-detection, Object Detection, Style Transfer

In recent years, a great progress has been witnessed for cross-domain object detection. Most state-of-the-art methods strive to handle the relation between local regions by calibrating cross-channel and spatial information to enable better alignment. They succeed in improving the generalization of the model, but implicitly drive networks to pay more attention on the shared attributes and ignore the domain-specific feature, which limits the performance of the algorithm. In order to search for the equilibrium between transferability and discriminability, we propose a novel adaptation framework for cross-domain object detection. Specifically, we adopt a style-aware feature fusion method and design two plug-and-play feature component regularization modules, which repositions the focus of the model on domain-specific features by restructuring the style and content of features. Our key insight is that while it is difficult to extract discriminative features in target domain, it is feasible to assign the underlying details to the model via feature style transfer. Without bells and whistles, our method significantly boosts the performance of existing Domain Adaptive Faster R-CNN detectors, and achieves state-of-the-art results on several benchmark datasets for cross-domain object detection.
On the Capacity and Superposition of Minima in Neural Network Loss Function Landscapes
Authors: Maximilian Paul Niroomand, John William Roger Morgan, Conor T Cafolla, David John Wales
Date: 2021-09-29 00:00:00

Minima of the loss function landscape of a neural network are locally optimal sets of
weights that extract and process information from the input data to make outcome predictions. 
In underparameterised networks, the capacity of the weights may be insufficient to fit all the relevant information. 
We demonstrate that different local minima specialise in certain aspects of the learning problem, and process the input
information differently. This effect can be exploited using a meta-network in
which the predictive power from multiple minima of the LFL is combined to produce a better
classifier. With this approach, we can increase the area under the receiver operating characteristic curve
(AUC) by around $20\%$ for a complex learning problem. 
We propose a theoretical basis for combining minima and show how a meta-network can
be trained to select the representative that is used for classification of a
specific data item. Finally, we present an analysis of symmetry-equivalent
solutions to machine learning problems, which provides a systematic means to improve the
efficiency of this approach.
An Optimally Weighted Echo State Neural Network for Highly Chaotic Time Series Modelling
Authors: Gianfranco Bino, Gabriel Bruno Benigno, Lyle Muller, Shantanu Basu
Date: 2021-09-29 00:00:00
Tasks: Time Series, Time Series Analysis

We demonstrate the development and implementation of a series of echo state neural networks in conjunction with optimal weighted averaging to produce robust, model-free predictions of highly chaotic time series. We deploy our model on simulated mass accretion data sets governing the formation of a protostar, accretion disk and gas envelope. Our methodology extends the parallel series approach of averaging all reservoir outputs by instead selecting an optimal set of weights for each realization representative of what we believe constructs the optimal output signal that best represents the data’s underlying temporal dynamics. The method is demonstrated by modelling hydrodynamic and stellar evolution simulations that are representative of highly chaotic systems.
Embedding Compression with Hashing for Efficient Representation Learning in Graph
Authors: Chin-Chia Michael Yeh, Mengting Gu, Yan Zheng, Huiyuan Chen, Javid Ebrahimi, Zhongfang Zhuang, Junpeng Wang, Liang Wang, Wei zhang
Date: 2021-09-29 00:00:00
Tasks: GPU, Representation Learning

Graph neural networks (GNNs) are deep learning models designed specifically for graph data, and they typically rely on node features as the input node representation to the first layer. When applying such type of networks on graph without node feature, one can extract simple graph-based node features (e.g., number of degrees) or learn the input node representation (i.e., embeddings) when training the network. While the latter approach, which trains node embeddings, more likely leads to better performance, the number of parameters associated with the embeddings grows linearly with the number of nodes. It is therefore impractical to train the input node embeddings together with GNNs within graphics processing unit (GPU) memory in an end-to-end fashion when dealing with industrial scale graph data. Inspired by the embedding compression methods developed for natural language processing (NLP) models, we develop a node embedding compression method where each node is compactly represented with a bit vector instead of a float-point vector. The parameters utilized in the compression method can be trained together with GNNs. We show that the proposed node embedding compression method achieves superior performance compared to the alternatives.
Manifold Micro-Surgery with Linearly Nearly Euclidean Metrics
Authors: Jun Chen, Tianxin Huang, Wenzhou Chen, Yong liu
Date: 2021-09-29 00:00:00

The Ricci flow is a method of manifold surgery, which can trim manifolds to more regular. However, in most cases, the Rich flow tends to develop singularities and lead to divergence of the solution. In this paper, we propose linearly nearly Euclidean metrics to assist manifold micro-surgery, which means that we prove the dynamical stability and convergence of such metrics under the Ricci-DeTurck flow. From the information geometry and mirror descent points of view, we give the approximation of the steepest descent gradient flow on the linearly nearly Euclidean manifold with dynamical stability. In practice, the regular shrinking or expanding of Ricci solitons with linearly nearly Euclidean metrics will provide a geometric optimization method for the solution on a manifold.
Revisiting the Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learning
Authors: Jian Hu, Siyang Jiang, Seth Austin Harding, Haibin Wu, Shih-wei Liao
Date: 2021-09-29 00:00:00
Tasks: Multi-agent Reinforcement Learning, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL), SMAC, SMAC+, Starcraft

QMIX, a popular MARL algorithm based on the monotonicity constraint, has been used as a baseline for the benchmark environments, such as Starcraft Multi-Agent Challenge (SMAC), Predator-Prey (PP). 
Recent variants of QMIX target relaxing the monotonicity constraint of QMIX to improve the expressive power of QMIX, allowing for performance improvement in SMAC. 
However, we find that such performance improvements of the variants are significantly affected by various implementation tricks. In this paper, we revisit the monotonicity constraint of QMIX,
(1) we design a novel model RMC to further investigate the monotonicity constraint; the results show that monotonicity constraint can improve sample efficiency in some purely cooperative tasks;
(2) we then re-evaluate the performance of QMIX and these variants by a grid hyperparameter search for the tricks; the results show QMIX achieves the best performance among them, achieving SOTA performance on SMAC and PP;
(3) we analyze the monotonic mixing network from a theoretical perspective and show that it can represent any tasks which can be interpreted as purely cooperative. These analyses demonstrate that relaxing the monotonicity constraint of the mixing network will not always improve the performance of QMIX, which breaks our previous impressions of the monotonicity constraints.
An improved model for the formation times of dark matter haloes
Authors: Carlo Giocoli, Jorge Moreno, Ravi K. Sheth, Giuseppe Tormen
Date: 2006-11-07 00:00:00

A dark matter halo is said to have formed when at least half its mass hass been assembled into a single progenitor. With this definition, it is possible to derive a simple but useful analytic estimate of the distribution of halo formation times. The standard estimate of this distribution depends on the shape of the conditional mass function--the distribution of progenitor masses of a halo as a function of time. If the spherical collapse model is used to estimate the progenitor mass function, then the formation times one infers systematically underestimate those seen in numerical simulations of hierarchical gravitational clustering. We provide estimates of halo formation which may be related to an ellipsoidal collapse model. These estimates provide a substantially better description of the simulations. We also provide an alternative derivation of the formation time distribution which is based on the assumption that haloes increase their mass through binary mergers only. Our results are useful for models which relate halo structure to halo formation.
The Neural Testbed: Evaluating Joint Predictions
Authors: Ian Osband, Zheng Wen, Seyed Mohammad Asghari, Vikranth Dwaracherla, Botao Hao, Morteza Ibrahimi, Dieterich Lawson, Xiuyuan Lu, Brendan O'Donoghue, Benjamin Van Roy
Date: 2021-10-09 00:00:00

Predictive distributions quantify uncertainties ignored by point estimates. This paper introduces The Neural Testbed: an open-source benchmark for controlled and principled evaluation of agents that generate such predictions. Crucially, the testbed assesses agents not only on the quality of their marginal predictions per input, but also on their joint predictions across many inputs. We evaluate a range of agents using a simple neural network data generating process. Our results indicate that some popular Bayesian deep learning agents do not fare well with joint predictions, even when they can produce accurate marginal predictions. We also show that the quality of joint predictions drives performance in downstream decision tasks. We find these results are robust across choice a wide range of generative models, and highlight the practical importance of joint predictions to the community.