pwc-archive/papers-with-abstracts huggingface dataset:
Variance-Based Feature Importance in Neural Networks
Authors: Cláudio Rebelo de Sá
Date: 2019-10-16 00:00:00
Tasks: Feature Importance

This paper proposes a new method to measure the relative importance of features in Artificial Neural Networks (ANN) models. Its underlying principle assumes that the more important a feature is, the more the weights, connected to the respective input neuron, will change during the training of the model. To capture this behavior, a running variance of every weight connected to the input layer is measured during training. For that, an adaptation of Welford’s online algorithm for computing the online variance is proposed. When the training is finished, for each input, the variances of the weights are combined with the final weights to obtain the measure of relative importance for each feature. This method was tested with shallow and deep neural network architectures on several well-known classification and regression problems. The results obtained confirm that this approach is making meaningful measurements. Moreover, results showed that the importance scores are highly correlated with the variable importance method from Random Forests (RF).
L2-Relaxation: With Applications to Forecast Combination and Portfolio Analysis
Authors: Zhentao Shi, Liangjun Su, Tian Xie
Date: 2020-10-19 00:00:00

This paper tackles forecast combination with many forecasts or minimum variance portfolio selection with many assets. A novel convex problem called L2-relaxation is proposed. In contrast to standard formulations, L2-relaxation minimizes the squared Euclidean norm of the weight vector subject to a set of relaxed linear inequality constraints. The magnitude of relaxation, controlled by a tuning parameter, balances the bias and variance. When the variance-covariance (VC) matrix of the individual forecast errors or financial assets exhibits latent group structures -- a block equicorrelation matrix plus a VC for idiosyncratic noises, the solution to L2-relaxation delivers roughly equal within-group weights. Optimality of the new method is established under the asymptotic framework when the number of the cross-sectional units $N$ potentially grows much faster than the time dimension $T$. Excellent finite sample performance of our method is demonstrated in Monte Carlo simulations. Its wide applicability is highlighted in three real data examples concerning empirical applications of microeconomics, macroeconomics, and finance.
Improved wide-angle, fisheye and omnidirectional camera calibration
Authors: Steffen Urban, Jens Leitloff, Stefan Hinz
Date: 2015-07-15 00:00:00
Tasks: Camera Calibration

In this paper an improved method for calibrating wide-angle, fisheye and omnidirectional imaging systems is presented. We extend the calibration procedure
proposed by Scaramuzza et al. by replacing the residual function and joint refinement of all parameters. In doing so, we achieve a more stable, robust and
accurate calibration (up to factor 7) and can reduce the number of necessary
calibration steps from five to three. After introducing the camera model and
highlighting the differences from the current calibration procedure, we perform
a comprehensive performance evaluation using several data sets and show the
impact of the proposed calibration procedure on the calibration results.
Sublimable materials facilitate the TEM sample preparation of oil-soluble nanomaterials
Authors: Yu-Hao Deng
Date: 2020-09-29 00:00:00

Sample preparation is significantly important to the high-resolution transmission electron microscopy (HRTEM) characterization of nanomaterials. However, many general organic solvents can dissolve the necessary organic polymer support layer in TEM grid, which causes it difficult to obtain high-quality samples of oil-soluble nanomaterials. In this study, a new sample preparation method for oil-soluble nanomaterials has been developed by using the sublimable material as a transition layer. Experiments also show that there is no damage to TEM grids and high-quality HRTEM images can be obtained via this method. This approach paves the way to applicable HRTEM sample preparation of oil-soluble nanomaterials.
CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characters
Authors: Hicham El Boukkouri, Olivier Ferret, Thomas Lavergne, Hiroshi Noji, Pierre Zweigenbaum, Junichi Tsujii
Date: 2020-10-20 00:00:00
Proceeding: COLING 2020 8
Tasks: Clinical Concept Extraction, Drug–drug Interaction Extraction, Natural Language Inference, Relation Extraction, Semantic Similarity

Due to the compelling improvements brought by BERT, many recent representation models adopted the Transformer architecture as their main building block, consequently inheriting the wordpiece tokenization system despite it not being intrinsically linked to the notion of Transformers. While this system is thought to achieve a good balance between the flexibility of characters and the efficiency of full words, using predefined wordpiece vocabularies from the general domain is not always suitable, especially when building models for specialized domains (e.g., the medical domain). Moreover, adopting a wordpiece tokenization shifts the focus from the word level to the subword level, making the models conceptually more complex and arguably less convenient in practice. For these reasons, we propose CharacterBERT, a new variant of BERT that drops the wordpiece system altogether and uses a Character-CNN module instead to represent entire words by consulting their characters. We show that this new model improves the performance of BERT on a variety of medical domain tasks while at the same time producing robust, word-level and open-vocabulary representations.
New Properties of the Data Distillation Method When Working With Tabular Data
Authors: Dmitry Medvedev, Alexander D'yakonov
Date: 2020-10-19 00:00:00

Data distillation is the problem of reducing the volume oftraining data while keeping only the necessary information. With thispaper, we deeper explore the new data distillation algorithm, previouslydesigned for image data. Our experiments with tabular data show thatthe model trained on distilled samples can outperform the model trainedon the original dataset. One of the problems of the considered algorithmis that produced data has poor generalization on models with differenthyperparameters. We show that using multiple architectures during distillation can help overcome this problem.
Orthrus: A Bimodal Learning Architecture for Malware Classification
Authors: Daniel Gibert, Carles Mateu, Jordi Planes
Date: 2020-09-28 00:00:00
Tasks: Classification, Deep Learning, Feature Engineering, General Classification, Malware Classification, Malware Detection

Malware detection and classification is a challenging problem and an active area of research. Traditional machine learning methods depend almost entirely on the ability to extract a set of discriminative features into which characterize malware. However, this feature engineering process is very time consuming. On the contrary, deep learning methods replace manual feature engineering by a system that performs both feature extraction and classification from raw data at once. Despite that, a major shortfall of these methods is their inhability to consider multiple disparate sources of information when performing classification, leading them to perform poorly when compared to multimodal approaches. In this work, we introduce Orthrus, a new bimodal approach to categorize malware into families based on deep learning. Orthrus combines two modalities of data: (1) the byte sequence representing the malware’s binary content, and (2) the assembly language instructions extracted from the assembly language source code of malware, and performs automatic feature learning and classification with a convolutional neural network. The idea is to benefit from multiple feature types to reflect malware’s characteristics. The experiments carried on the Microsoft Malware Classification Challenge dataset show that our proposed solution achieves higher classification performance than deep learning approaches in the literature and n-gram based methods.
A Concentration of Measure Framework to study convex problems and other implicit formulation problems in machine learning
Authors: Cosme Louart
Date: 2020-10-19 00:00:00
Tasks: regression

This paper provides a framework to show the concentration of solutions $Y^*$ to convex minimizing problem where the objective function $\phi(X)(Y)$ depends on some random vector $X$ satisfying concentration of measure hypotheses. More precisely, the convex problem translates into a contractive fixed point equation that ensure the transmission of the concentration from $X$ to $Y^*$. This result is of central interest to characterize many machine learning algorithms which are defined through implicit equations (e.g., logistic regression, lasso, boosting, etc.). Based on our framework, we provide precise estimations for the first moments of the solution $Y^*$, when $X= (x_1,\ldots, x_n)$ is a data matrix of independent columns and $\phi(X)(y)$ writes as a sum $\frac{1}{n}\sum_{i=1}^n h_i(x_i^TY)$. That allows to describe the behavior and performance (e.g., generalization error) of a wide variety of machine learning classifiers.
Energy-Efficient Edge-Facilitated Wireless Collaborative Computing using Map-Reduce
Authors: Antoine Paris, Hamed Mirghasemi, Ivan Stupia, Luc Vandendorpe
Date: 2019-03-06 00:00:00
Tasks: Distributed Computing

In this work, a heterogeneous set of wireless devices sharing a common access point collaborates to perform a set of tasks. Using the Map-Reduce distributed computing framework, the tasks are optimally distributed amongst the nodes with the objective of minimizing the total energy consumption of the nodes while satisfying a latency constraint. The derived optimal collaborative-computing scheme takes into account both the computing capabilities of the nodes and the strength of their communication links. Numerical simulations illustrate the benefits of the proposed optimal collaborative-computing scheme over a blind collaborative-computing scheme and the non-collaborative scenario, both in term of energy savings and achievable latency. The proposed optimal scheme also exhibits the interesting feature of allowing to trade energy for latency, and vice versa.
HYDRA: A multimodal deep learning framework for malware classification
Authors: Daniel Gibert, Carles Mateu, Jordi Planes
Date: 2020-05-12 00:00:00
Tasks: Classification, Deep Learning, Descriptive, Feature Engineering, General Classification, Malware Classification, Malware Detection, Multimodal Deep Learning

While traditional machine learning methods for malware detection largely depend on hand-designed features, which are based on experts’ knowledge of the domain, end-to-end learning approaches take the raw executable as input, and try to learn a set of descriptive features from it. Although the latter might behave badly in problems where there are not many data available or where the dataset is imbalanced. In this paper we present HYDRA, a novel framework to address the task of malware detection and classification by combining various types of features to discover the relationships between distinct modalities. Our approach learns from various sources to maximize the benefits of multiple feature types to reflect the characteristics of malware executables. We propose a baseline system that consists of both hand-engineered and end-to-end components to combine the benefits of feature engineering and deep learning so that malware characteristics are effectively represented. An extensive analysis of state-of-the-art methods on the Microsoft Malware Classification Challenge benchmark shows that the proposed solution achieves comparable results to gradient boosting methods in the literature and higher yield in comparison with deep learning approaches.
An End-to-End Deep Learning Architecture for Classification of Malware’s Binary Content
Authors: Daniel Gibert, Carles Mateu, Jordi Planes
Date: 2018-09-27 00:00:00
Tasks: Denoising, Descriptive, Feature Engineering, General Classification, Malware Classification, Malware Detection

In traditional machine learning techniques for malware detection and classification, significant efforts are expended on manually designing features based on expertise and domain-specific knowledge. These solutions perform feature engineering in order to extract features that provide an abstract view of the software program. Thus, the usefulness of the classifier is roughly dependent on the ability of the domain experts to extract a set of descriptive features. Instead, we introduce a file agnostic end-to-end deep learning approach for malware classification from raw byte sequences without extracting hand-crafted features. It consists of two key components: (1) a denoising autoencoder that learns a hidden representation of the malware’s binary content; and (2) a dilated residual network as classifier. The experiments show an impressive performance, achieving almost 99% of accuracy classifying malware into families.
A Hierarchical Convolutional Neural Network for Malware Classification
Authors: Daniel Gibert, Carles Mateu, Jordi Planes
Date: 2019-09-30 00:00:00
Tasks: Classification, General Classification, Malware Classification, Malware Detection

Malware detection and classification is a challenging problem and an active area of research. Particular challenges include how to best treat and preprocess malicious executables in order to feed machine learning algorithms. Novel approaches in the literature treat an executable as a sequence of bytes or as a sequence of assembly language instructions. However, in those approaches the hierarchical structure of programs is not taken into consideration. An executable exhibits various levels of spatial correlation. Adjacent code instructions are correlated spatially but that is not necessarily the case. Function calls and jump commands transfer the control of the program to a different point in the instruction stream. Furthermore, these discontinuities are maintained when treating the binary as a sequence of byte values. In addition, functions might be arranged randomly if addresses are correctly reorganized. To address these issues we propose a Hierarchical Convolutional Network (HCN) for malware classification. It has two levels of convolutional blocks applied at the mnemonic-level and at the function-level, enabling us to extract n-gram like features from both levels when constructing the malware representation. We validate our HCN method on the dataset released for the Microsoft Malware Classification Challenge, outperforming almost every deep learning method in the literature.
Kalibre: Knowledge-based Neural Surrogate Model Calibration for Data Center Digital Twins
Authors: Ruihang Wang, Xin Zhou, Linsen Dong, Yonggang Wen, Rui Tan, Li Chen, Guan Wang, Feng Zeng
Date: 2020-01-29 00:00:00
Tasks: Management

Computational fluid dynamics (CFD) model has been widely used for prototyping data centers. Evolving it to high-fidelity {\em digital twin} is desirable for the management and operations of large-scale data centers. Manually calibrating CFD model parameters to achieve twin-class fidelity by specially trained domain expert is tedious and labor-intensive. To reduce manual efforts, existing automatic calibration approaches developed for various computational models apply heuristics to search model configurations within an empirically defined parameter bound. However, in the context of CFD, each search step requires long-lasting CFD model's iterated solving, rendering these approaches impractical with increased model complexity. This paper presents Kalibre, a knowledge-based neural surrogate approach that performs CFD model calibration by iterating four key steps of i) training a neural surrogate model based on CFD-generated data, ii) finding the optimal parameters at the moment through neural surrogate retraining based on sensor-measured data, iii) configuring the found parameters back to the CFD model, and iv) validating the CFD model using sensor-measured data as the ground truth. Thus, the parameter search is offloaded to the neural surrogate which is ultra-faster than CFD model's iterated solving. To speed up the convergence of Kalibre, we integrate prior knowledge of the twinned data center's thermophysics into the neural surrogate design to improve its learning efficiency. With about five hours computation on a 32-core processor, Kalibre achieves mean absolute errors (MAEs) of $0.81^o$C and $0.75^o$C in calibrating two CFD models for two production data halls hosting thousands of servers each while requires fewer CFD solving processes than existing baseline approaches.
Scan‐specific robust artificial‐neural‐networks for k‐space interpolation (RAKI) reconstruction: Database‐free deep learning for fast imaging
Authors: Akçakaya, Mehmet
Date: 2018-09-18 00:00:00
Tasks: MRI Reconstruction

Purpose
To develop an improved k-space reconstruction method using scan-specific deep learning that is trained on autocalibration signal (ACS) data.

Theory
Robust Artificial-neural-networks for k-space Interpolation (RAKI) reconstruction trains convolutional neural networks on ACS data. This enables non-linear estimation of missing k-space lines from acquired k-space data with improved noise resilience, as opposed to conventional linear k-space interpolation-based methods, such as GRAPPA, which are based on linear convolutional kernels.

Methods
The training algorithm is implemented using a mean square error loss function over the target points in the ACS region, using a gradient descent algorithm. The neural network contains three layers of convolutional operators, with two of these including non-linear activation functions. The noise performance and reconstruction quality of the RAKI method was compared with GRAPPA in phantom, as well as in neurological and cardiac in vivo datasets.

Results
Phantom imaging shows that the proposed RAKI method outperforms GRAPPA at high (≥4) acceleration rates, both visually and quantitatively. Quantitative cardiac imaging shows improved noise resilience at high acceleration rates (rate 4: 23% and rate 5: 48%) over GRAPPA. The same trend of improved noise resilience is also observed in high-resolution brain imaging at high acceleration rates.

Conclusion
The RAKI method offers a training database-free deep learning approach for MRI reconstruction, with the potential to improve many existing reconstruction approaches, and is compatible with conventional data acquisition protocols.
SDF-SRN: Learning Signed Distance 3D Object Reconstruction from Static Images
Authors: Chen-Hsuan Lin, Chaoyang Wang, Simon Lucey
Date: 2020-10-20 00:00:00
Proceeding: NeurIPS 2020 12
Tasks: 3D Object Reconstruction, 3D Object Reconstruction From A Single Image, 3D Reconstruction, Object Reconstruction

Dense 3D object reconstruction from a single image has recently witnessed remarkable advances, but supervising neural networks with ground-truth 3D shapes is impractical due to the laborious process of creating paired image-shape datasets. Recent efforts have turned to learning 3D reconstruction without 3D supervision from RGB images with annotated 2D silhouettes, dramatically reducing the cost and effort of annotation. These techniques, however, remain impractical as they still require multi-view annotations of the same object instance during training. As a result, most experimental efforts to date have been limited to synthetic datasets. In this paper, we address this issue and propose SDF-SRN, an approach that requires only a single view of objects at training time, offering greater utility for real-world scenarios. SDF-SRN learns implicit 3D shape representations to handle arbitrary shape topologies that may exist in the datasets. To this end, we derive a novel differentiable rendering formulation for learning signed distance functions (SDF) from 2D silhouettes. Our method outperforms the state of the art under challenging single-view supervision settings on both synthetic and real-world datasets.
Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition
Authors: Yu Zhang, James Qin, Daniel S. Park, Wei Han, Chung-Cheng Chiu, Ruoming Pang, Quoc V. Le, Yonghui Wu
Date: 2020-10-20 00:00:00
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), speech-recognition, Speech Recognition

We employ a combination of recent developments in semi-supervised learning for automatic speech recognition to obtain state-of-the-art results on LibriSpeech utilizing the unlabeled audio of the Libri-Light dataset. More precisely, we carry out noisy student training with SpecAugment using giant Conformer models pre-trained using wav2vec 2.0 pre-training. By doing so, we are able to achieve word-error-rates (WERs) 1.4%/2.6% on the LibriSpeech test/test-other sets against the current state-of-the-art WERs 1.7%/3.3%.
Dual Averaging is Surprisingly Effective for Deep Learning Optimization
Authors: Samy Jelassi, Aaron Defazio
Date: 2020-10-20 00:00:00
Tasks: Deep Learning, Stochastic Optimization

First-order stochastic optimization methods are currently the most widely used class of methods for training deep neural networks. However, the choice of the optimizer has become an ad-hoc rule that can significantly affect the performance. For instance, SGD with momentum (SGD+M) is typically used in computer vision (CV) and Adam is used for training transformer models for Natural Language Processing (NLP). Using the wrong method can lead to significant performance degradation. Inspired by the dual averaging algorithm, we propose Modernized Dual Averaging (MDA), an optimizer that is able to perform as well as SGD+M in CV and as Adam in NLP. Our method is not adaptive and is significantly simpler than Adam. We show that MDA induces a decaying uncentered $L_2$-regularization compared to vanilla SGD+M and hypothesize that this may explain why it works on NLP problems where SGD+M fails.
Natural Language Inference with Mixed Effects
Authors: William Gantt, Benjamin Kane, Aaron Steven White
Date: 2020-10-20 00:00:00
Proceeding: Joint Conference on Lexical and Computational Semantics 2020
Tasks: Natural Language Inference

There is growing evidence that the prevalence of disagreement in the raw annotations used to construct natural language inference datasets makes the common practice of aggregating those annotations to a single label problematic. We propose a generic method that allows one to skip the aggregation step and train on the raw annotations directly without subjecting the model to unwanted noise that can arise from annotator response biases. We demonstrate that this method, which generalizes the notion of a \textit{mixed effects model} by incorporating \textit{annotator random effects} into any existing neural model, improves performance over models that do not incorporate such effects.
Optimal Subarchitecture Extraction For BERT
Authors: Adrian de Wynter, Daniel J. Perry
Date: 2020-10-20 00:00:00
Tasks: CPU, GPU, Natural Language Understanding, Neural Architecture Search

We extract an optimal subset of architectural parameters for the BERT architecture from Devlin et al. (2018) by applying recent breakthroughs in algorithms for neural architecture search. This optimal subset, which we refer to as "Bort", is demonstrably smaller, having an effective (that is, not counting the embedding layer) size of $5.5\%$ the original BERT-large architecture, and $16\%$ of the net size. Bort is also able to be pretrained in $288$ GPU hours, which is $1.2\%$ of the time required to pretrain the highest-performing BERT parametric architectural variant, RoBERTa-large (Liu et al., 2019), and about $33\%$ of that of the world-record, in GPU hours, required to train BERT-large on the same hardware. It is also $7.9$x faster on a CPU, as well as being better performing than other compressed variants of the architecture, and some of the non-compressed variants: it obtains performance improvements of between $0.3\%$ and $31\%$, absolute, with respect to BERT-large, on multiple public natural language understanding (NLU) benchmarks.