pwc-archive/papers-with-abstracts huggingface dataset:
Carbontracker: Tracking and Predicting the Carbon Footprint of Training Deep Learning Models
Authors: Lasse F. Wolff Anthony, Benjamin Kanding, Raghavendra Selvan
Date: 2020-07-06 00:00:00

Deep learning (DL) can achieve impressive results across a wide variety of tasks, but this often comes at the cost of training models for extensive periods on specialized hardware accelerators. This energy-intensive workload has seen immense growth in recent years. Machine learning (ML) may become a significant contributor to climate change if this exponential trend continues. If practitioners are aware of their energy and carbon footprint, then they may actively take steps to reduce it whenever possible. In this work, we present Carbontracker, a tool for tracking and predicting the energy and carbon footprint of training DL models. We propose that energy and carbon footprint of model development and training is reported alongside performance metrics using tools like Carbontracker. We hope this will promote responsible computing in ML and encourage research into energy-efficient deep neural networks.
Leveraging Class Hierarchies with Metric-Guided Prototype Learning
Authors: Vivien Sainte Fare Garnot, Loic Landrieu
Date: 2020-07-06 00:00:00
Tasks: General Classification, Semantic Segmentation, Time Series, Time Series Analysis, Time Series Classification

In many classification tasks, the set of target classes can be organized into a hierarchy. This structure induces a semantic distance between classes, and can be summarised under the form of a cost matrix, which defines a finite metric on the class set. In this paper, we propose to model the hierarchical class structure by integrating this metric in the supervision of a prototypical network. Our method relies on jointly learning a feature-extracting network and a set of class prototypes whose relative arrangement in the embedding space follows an hierarchical metric. We show that this approach allows for a consistent improvement of the error rate weighted by the cost matrix when compared to traditional methods and other prototype-based strategies. Furthermore, when the induced metric contains insight on the data structure, our method improves the overall precision as well. Experiments on four different public datasets - from agricultural time series classification to depth image semantic segmentation - validate our approach.
Machine Learning with the Sugeno Integral: The Case of Binary Classification
Authors: Sadegh Abbaszadeh, Eyke Hüllermeier
Date: 2020-07-06 00:00:00
Tasks: BIG-bench Machine Learning, Binary Classification, General Classification

In this paper, we elaborate on the use of the Sugeno integral in the context of machine learning. More specifically, we propose a method for binary classification, in which the Sugeno integral is used as an aggregation function that combines several local evaluations of an instance, pertaining to different features or measurements, into a single global evaluation. Due to the specific nature of the Sugeno integral, this approach is especially suitable for learning from ordinal data, that is, when measurements are taken from ordinal scales. This is a topic that has not received much attention in machine learning so far. The core of the learning problem itself consists of identifying the capacity underlying the Sugeno integral. To tackle this problem, we develop an algorithm based on linear programming. The algorithm also includes a suitable technique for transforming the original feature values into local evaluations (local utility scores), as well as a method for tuning a threshold on the global evaluation. To control the flexibility of the classifier and mitigate the problem of overfitting the training data, we generalize our approach toward $k$-maxitive capacities, where $k$ plays the role of a hyper-parameter of the learner. We present experimental studies, in which we compare our method with competing approaches on several benchmark data sets.
Robust Technique for Representative Volume Element Identification in Noisy Microtomography Images of Porous Materials Based on Pores Morphology and Their Spatial Distribution
Authors: Maxim Grigoriev, Anvar Khafizov, Vladislav Kokhan, Viktor Asadchikov
Date: 2020-07-06 00:00:00
Tasks: Denoising

Microtomography is a powerful method of materials investigation. It enables to obtain physical properties of porous media non-destructively that is useful in studies. One of the application ways is a calculation of porosity, pore sizes, surface area, and other parameters of metal-ceramic (cermet) membranes which are widely spread in the filtration industry. The microtomography approach is efficient because all of those parameters are calculated simultaneously in contrast to the conventional techniques. Nevertheless, the calculations on Micro-CT reconstructed images appear to be time-consuming, consequently representative volume element should be chosen to speed them up. This research sheds light on representative elementary volume identification without consideration of any physical parameters such as porosity, etc. Thus, the volume element could be found even in noised and grayscale images. The proposed method is flexible and does not overestimate the volume size in the case of anisotropic samples. The obtained volume element could be used for computations of the domain's physical characteristics if the image is filtered and binarized, or for selections of optimal filtering parameters for denoising procedure.
Continual Learning in Human Activity Recognition: an Empirical Analysis of Regularization
Authors: Saurav Jha, Martin Schiemer, Juan Ye
Date: 2020-07-06 00:00:00
Tasks: Activity Recognition, Continual Learning, Human Activity Recognition, Incremental Learning

Given the growing trend of continual learning techniques for deep neural networks focusing on the domain of computer vision, there is a need to identify which of these generalizes well to other tasks such as human activity recognition (HAR). As recent methods have mostly been composed of loss regularization terms and memory replay, we provide a constituent-wise analysis of some prominent task-incremental learning techniques employing these on HAR datasets. We find that most regularization approaches lack substantial effect and provide an intuition of when they fail. Thus, we make the case that the development of continual learning algorithms should be motivated by rather diverse task domains.
Labeling of Multilingual Breast MRI Reports
Authors: Chen-Han Tsai, Nahum Kiryati, Eli Konen, Miri Sklair-Levy, Arnaldo Mayer
Date: 2020-07-06 00:00:00

Medical reports are an essential medium in recording a patient's condition throughout a clinical trial. They contain valuable information that can be extracted to generate a large labeled dataset needed for the development of clinical tools. However, the majority of medical reports are stored in an unregularized format, and a trained human annotator (typically a doctor) must manually assess and label each case, resulting in an expensive and time consuming procedure. In this work, we present a framework for developing a multilingual breast MRI report classifier using a custom-built language representation called LAMBR. Our proposed method overcomes practical challenges faced in clinical settings, and we demonstrate improved performance in extracting labels from medical reports when compared with conventional approaches.
Determination of the most representative descriptor among a set of feature vectors for the same object
Authors: Dmitry Pozdnyakov
Date: 2020-07-06 00:00:00
Tasks: Face Recognition

On an example of solution of the face recognition problem the approach for estimation of the most representative descriptor among a set of feature vectors for the same face is considered in present study. The estimation is based on robust calculation of the mode-median mixture vector for the set as the descriptor by means of Welsch/Leclerc loss function application in case of very sparse filling of the feature space with feature vectors
Deep Contextual Embeddings for Address Classification in E-commerce
Authors: Shreyas Mangalgi, Lakshya Kumar, Ravindra Babu Tallamraju
Date: 2020-07-06 00:00:00
Tasks: Classification, General Classification, Language Modelling

E-commerce customers in developing nations like India tend to follow no fixed format while entering shipping addresses. Parsing such addresses is challenging because of a lack of inherent structure or hierarchy. It is imperative to understand the language of addresses, so that shipments can be routed without delays. In this paper, we propose a novel approach towards understanding customer addresses by deriving motivation from recent advances in Natural Language Processing (NLP). We also formulate different pre-processing steps for addresses using a combination of edit distance and phonetic algorithms. Then we approach the task of creating vector representations for addresses using Word2Vec with TF-IDF, Bi-LSTM and BERT based approaches. We compare these approaches with respect to sub-region classification task for North and South Indian cities. Through experiments, we demonstrate the effectiveness of generalized RoBERTa model, pre-trained over a large address corpus for language modelling task. Our proposed RoBERTa model achieves a classification accuracy of around 90% with minimal text preprocessing for sub-region classification task outperforming all other approaches. Once pre-trained, the RoBERTa model can be fine-tuned for various downstream tasks in supply chain like pincode suggestion and geo-coding. The model generalizes well for such tasks even with limited labelled data. To the best of our knowledge, this is the first of its kind research proposing a novel approach of understanding customer addresses in e-commerce domain by pre-training language models and fine-tuning them for different purposes.
Announcing CzEng 2.0 Parallel Corpus with over 2 Gigawords
Authors: Tom Kocmi, Martin Popel, Ondrej Bojar
Date: 2020-07-06 00:00:00

We present a new release of the Czech-English parallel corpus CzEng 2.0 consisting of over 2 billion words (2 "gigawords") in each language. The corpus contains document-level information and is filtered with several techniques to lower the amount of noise. In addition to the data in the previous version of CzEng, it contains new authentic and also high-quality synthetic parallel data. CzEng is freely available for research and educational purposes.
Massively Multilingual ASR: 50 Languages, 1 Model, 1 Billion Parameters
Authors: Vineel Pratap, Anuroop Sriram, Paden Tomasello, Awni Hannun, Vitaliy Liptchinsky, Gabriel Synnaeve, Ronan Collobert
Date: 2020-07-06 00:00:00
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), speech-recognition, Speech Recognition

We study training a single acoustic model for multiple languages with the aim of improving automatic speech recognition (ASR) performance on low-resource languages, and over-all simplifying deployment of ASR systems that support diverse languages. We perform an extensive benchmark on 51 languages, with varying amount of training data by language(from 100 hours to 1100 hours). We compare three variants of multilingual training from a single joint model without knowing the input language, to using this information, to multiple heads (one per language cluster). We show that multilingual training of ASR models on several languages can improve recognition performance, in particular, on low resource languages. We see 20.9%, 23% and 28.8% average WER relative reduction compared to monolingual baselines on joint model, joint model with language input and multi head model respectively. To our knowledge, this is the first work studying multilingual ASR at massive scale, with more than 50 languages and more than 16,000 hours of audio across them.
Consensus Multi-Agent Reinforcement Learning for Volt-VAR Control in Power Distribution Networks
Authors: Yuanqi Gao, Wei Wang, Nanpeng Yu
Date: 2020-07-06 00:00:00
Tasks: Deep Reinforcement Learning, Management, Multi-agent Reinforcement Learning, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Volt-VAR control (VVC) is a critical application in active distribution network management system to reduce network losses and improve voltage profile. To remove dependency on inaccurate and incomplete network models and enhance resiliency against communication or controller failure, we propose consensus multi-agent deep reinforcement learning algorithm to solve the VVC problem. The VVC problem is formulated as a networked multi-agent Markov decision process, which is solved using the maximum entropy reinforcement learning framework and a novel communication-efficient consensus strategy. The proposed algorithm allows individual agents to learn a group control policy using local rewards. Numerical studies on IEEE distribution test feeders show that our proposed algorithm matches the performance of single-agent reinforcement learning benchmark. In addition, the proposed algorithm is shown to be communication efficient and resilient.
Deep Learning for Apple Diseases: Classification and Identification
Authors: Asif Iqbal Khan, SMK Quadri, Saba Banday
Date: 2020-07-06 00:00:00
Tasks: Classification, Data Augmentation, Deep Learning, General Classification, image-classification, Image Classification, object-detection, Object Detection, Transfer Learning

Diseases and pests cause huge economic loss to the apple industry every year. The identification of various apple diseases is challenging for the farmers as the symptoms produced by different diseases may be very similar, and may be present simultaneously. This paper is an attempt to provide the timely and accurate detection and identification of apple diseases. In this study, we propose a deep learning based approach for identification and classification of apple diseases. The first part of the study is dataset creation which includes data collection and data labelling. Next, we train a Convolutional Neural Network (CNN) model on the prepared dataset for automatic classification of apple diseases. CNNs are end-to-end learning algorithms which perform automatic feature extraction and learn complex features directly from raw images, making them suitable for wide variety of tasks like image classification, object detection, segmentation etc. We applied transfer learning to initialize the parameters of the proposed deep model. Data augmentation techniques like rotation, translation, reflection and scaling were also applied to prevent overfitting. The proposed CNN model obtained encouraging results, reaching around 97.18% of accuracy on our prepared dataset. The results validate that the proposed method is effective in classifying various types of apple diseases and can be used as a practical tool by farmers.
Sharing Models or Coresets: A Study based on Membership Inference Attack
Authors: Hanlin Lu, Changchang Liu, Ting He, Shiqiang Wang, Kevin S. Chan
Date: 2020-07-06 00:00:00
Tasks: Federated Learning, Inference Attack, Membership Inference Attack

Distributed machine learning generally aims at training a global model based on distributed data without collecting all the data to a centralized location, where two different approaches have been proposed: collecting and aggregating local models (federated learning) and collecting and training over representative data summaries (coreset). While each approach preserves data privacy to some extent thanks to not sharing the raw data, the exact extent of protection is unclear under sophisticated attacks that try to infer the raw data from the shared information. We present the first comparison between the two approaches in terms of target model accuracy, communication cost, and data privacy, where the last is measured by the accuracy of a state-of-the-art attack strategy called the membership inference attack. Our experiments quantify the accuracy-privacy-cost tradeoff of each approach, and reveal a nontrivial comparison that can be used to guide the design of model training processes.
Estimating Generalization under Distribution Shifts via Domain-Invariant Representations
Authors: Ching-Yao Chuang, Antonio Torralba, Stefanie Jegelka
Date: 2020-07-06 00:00:00
Tasks: Domain Adaptation, Model Selection

When machine learning models are deployed on a test distribution different from the training distribution, they can perform poorly, but overestimate their performance. In this work, we aim to better estimate a model's performance under distribution shift, without supervision. To do so, we use a set of domain-invariant predictors as a proxy for the unknown, true target labels. Since the error of the resulting risk estimate depends on the target risk of the proxy model, we study generalization of domain-invariant representations and show that the complexity of the latent representation has a significant influence on the target risk. Empirically, our approach (1) enables self-tuning of domain adaptation models, and (2) accurately estimates the target error of given models under distribution shift. Other applications include model selection, deciding early stopping and error detection.
Predicting Porosity, Permeability, and Tortuosity of Porous Media from Images by Deep Learning
Authors: Krzysztof M. Graczyk, Maciej Matyka
Date: 2020-07-06 00:00:00
Tasks: Relation

Convolutional neural networks (CNN) are utilized to encode the relation between initial configurations of obstacles and three fundamental quantities in porous media: porosity ($\varphi$), permeability $k$, and tortuosity ($T$). The two-dimensional systems with obstacles are considered. The fluid flow through a porous medium is simulated with the lattice Boltzmann method. It is demonstrated that the CNNs are able to predict the porosity, permeability, and tortuosity with good accuracy. With the usage of the CNN models, the relation between $T$ and $\varphi$ has been reproduced and compared with the empirical estimate. The analysis has been performed for the systems with $\varphi \in (0.37,0.99)$ which covers five orders of magnitude span for permeability $k \in (0.78, 2.1\times 10^5)$ and tortuosity $T \in (1.03,2.74)$.
Distance-Geometric Graph Convolutional Network (DG-GCN) for Three-Dimensional (3D) Graphs
Authors: Daniel T. Chang
Date: 2020-07-06 00:00:00
Tasks: Translation

The distance-geometric graph representation adopts a unified scheme (distance) for representing the geometry of three-dimensional(3D) graphs. It is invariant to rotation and translation of the graph and it reflects pair-wise node interactions and their generally local nature. To facilitate the incorporation of geometry in deep learning on 3D graphs, we propose a message-passing graph convolutional network based on the distance-geometric graph representation: DG-GCN (distance-geometric graph convolution network). It utilizes continuous-filter convolutional layers, with filter-generating networks, that enable learning of filter weights from distances, thereby incorporating the geometry of 3D graphs in graph convolutions. Our results for the ESOL and FreeSolv datasets show major improvement over those of standard graph convolutions. They also show significant improvement over those of geometric graph convolutions employing edge weight / edge distance power laws. Our work demonstrates the utility and value of DG-GCN for end-to-end deep learning on 3D graphs, particularly molecular graphs.
Causal Feature Selection via Orthogonal Search
Authors: Ashkan Soleymani, Anant Raj, Stefan Bauer, Bernhard Schölkopf, Michel Besserve
Date: 2020-07-06 00:00:00
Tasks: Causal Discovery, feature selection

The problem of inferring the direct causal parents of a response variable among a large set of explanatory variables is of high practical importance in many disciplines. However, established approaches often scale at least exponentially with the number of explanatory variables, are difficult to extend to nonlinear relationships, and are difficult to extend to cyclic data. Inspired by {\em Debiased} machine learning methods, we study a one-vs.-the-rest feature selection approach to discover the direct causal parent of the response. We propose an algorithm that works for purely observational data while also offering theoretical guarantees, including the case of partially nonlinear relationships possibly under the presence of cycles. As it requires only one estimation for each variable, our approach is applicable even to large graphs. We demonstrate significant improvements compared to established approaches.
GateNet: Gating-Enhanced Deep Network for Click-Through Rate Prediction
Authors: Tongwen Huang, Qingyun She, Zhiqiang Wang, Junlin Zhang
Date: 2020-07-06 00:00:00
Tasks: Click-Through Rate Prediction, Recommendation Systems

Advertising and feed ranking are essential to many Internet companies such as Facebook. Among many real-world advertising and feed ranking systems, click through rate (CTR) prediction plays a central role. In recent years, many neural network based CTR models have been proposed and achieved success such as Factorization-Machine Supported Neural Networks, DeepFM and xDeepFM. Many of them contain two commonly used components: embedding layer and MLP hidden layers. On the other side, gating mechanism is also widely applied in many research fields such as computer vision(CV) and natural language processing(NLP). Some research has proved that gating mechanism improves the trainability of non-convex deep neural networks. Inspired by these observations, we propose a novel model named GateNet which introduces either the feature embedding gate or the hidden gate to the embedding layer or hidden layers of DNN CTR models, respectively. The feature embedding gate provides a learnable feature gating module to select salient latent information from the feature-level. The hidden gate helps the model to implicitly capture the high-order interaction more effectively. Extensive experiments conducted on three real-world datasets demonstrate its effectiveness to boost the performance of various state-of-the-art models such as FM, DeepFM and xDeepFM on all datasets.
Fictitious Play for Mean Field Games: Continuous Time Analysis and Applications
Authors: Sarah Perrin, Julien Perolat, Mathieu Laurière, Matthieu Geist, Romuald Elie, Olivier Pietquin
Date: 2020-07-05 00:00:00
Proceeding: NeurIPS 2020 12

In this paper, we deepen the analysis of continuous time Fictitious Play learning algorithm to the consideration of various finite state Mean Field Game settings (finite horizon, $\gamma$-discounted), allowing in particular for the introduction of an additional common noise. We first present a theoretical convergence analysis of the continuous time Fictitious Play process and prove that the induced exploitability decreases at a rate $O(\frac{1}{t})$. Such analysis emphasizes the use of exploitability as a relevant metric for evaluating the convergence towards a Nash equilibrium in the context of Mean Field Games. These theoretical contributions are supported by numerical experiments provided in either model-based or model-free settings. We provide hereby for the first time converging learning dynamics for Mean Field Games in the presence of common noise.