pwc-archive/papers-with-abstracts huggingface dataset:
Collaborative Teacher-Student Learning via Multiple Knowledge Transfer
Authors: Liyuan Sun, Jianping Gou, Baosheng Yu, Lan Du, DaCheng Tao
Date: 2021-01-21 00:00:00
Tasks: Knowledge Distillation, Model Compression, Self-Learning, Transfer Learning

Knowledge distillation (KD), as an efficient and effective model compression technique, has been receiving considerable attention in deep learning. The key to its success is to transfer knowledge from a large teacher network to a small student one. However, most of the existing knowledge distillation methods consider only one type of knowledge learned from either instance features or instance relations via a specific distillation strategy in teacher-student learning. There are few works that explore the idea of transferring different types of knowledge with different distillation strategies in a unified framework. Moreover, the frequently used offline distillation suffers from a limited learning capacity due to the fixed teacher-student architecture. In this paper we propose a collaborative teacher-student learning via multiple knowledge transfer (CTSL-MKT) that prompts both self-learning and collaborative learning. It allows multiple students learn knowledge from both individual instances and instance relations in a collaborative way. While learning from themselves with self-distillation, they can also guide each other via online distillation. The experiments and ablation studies on four image datasets demonstrate that the proposed CTSL-MKT significantly outperforms the state-of-the-art KD methods.
Exponential Moving Average Normalization for Self-supervised and Semi-supervised Learning
Authors: Zhaowei Cai, Avinash Ravichandran, Subhransu Maji, Charless Fowlkes, Zhuowen Tu, Stefano Soatto
Date: 2021-01-21 00:00:00
Proceeding: CVPR 2021 1
Tasks: Self-Supervised Learning, Semi-Supervised Image Classification

We present a plug-in replacement for batch normalization (BN) called exponential moving average normalization (EMAN), which improves the performance of existing student-teacher based self- and semi-supervised learning techniques. Unlike the standard BN, where the statistics are computed within each batch, EMAN, used in the teacher, updates its statistics by exponential moving average from the BN statistics of the student. This design reduces the intrinsic cross-sample dependency of BN and enhances the generalization of the teacher. EMAN improves strong baselines for self-supervised learning by 4-6/1-2 points and semi-supervised learning by about 7/2 points, when 1%/10% supervised labels are available on ImageNet. These improvements are consistent across methods, network architectures, training duration, and datasets, demonstrating the general effectiveness of this technique. The code is available at https://github.com/amazon-research/exponential-moving-average-normalization.
Differential Euler: Designing a Neural Network approximator to solve the Chaotic Three Body Problem
Authors: Pratyush Kumar, Aishwarya Das, Debayan Gupta
Date: 2021-01-21 00:00:00
Tasks: Experimental Design

The three body problem is a special case of the n body problem where one takes the initial positions and velocities of three point masses and attempts to predict their motion over time according to Newtonian laws of motion and universal gravitation. Though analytical solutions have been found for special cases, the general problem remains unsolved; the solutions that do exist are impractical. Fortunately, for many applications, we may not need to solve the problem completely, i.e., predicting with reasonable accuracy for some time steps, may be sufficient. Recently, Breen et al attempted to approximately solve the three body problem using a simple neural network. Although their methods appear to achieve some success in reducing the computational overhead, their model is extremely restricted, applying to a specialized 2D case. The authors do not provide explanations for critical decisions taken in their experimental design, no details on their model or architecture, and nor do they publish their code. Moreover, the model does not generalize well to unseen cases. In this paper, we propose a detailed experimental setup to determine the feasibility of using neural networks to solve the three body problem up to a certain number of time steps. We establish a benchmark on the dataset size and set an accuracy threshold to measure the viability of our results for practical applications. Then, we build our models according to the listed class of NNs using a dataset generated from standard numerical integrators. We gradually increase the complexity of our data set to determine whether NNs can learn a representation of the chaotic three body problem well enough to replace numerical integrators in real life scenarios.
Boost then Convolve: Gradient Boosting Meets Graph Neural Networks
Authors: Sergei Ivanov, Liudmila Prokhorenkova
Date: 2021-01-21 00:00:00
Proceeding: ICLR 2021 1
Tasks: Graph Representation Learning, Representation Learning

Graph neural networks (GNNs) are powerful models that have been successful in various graph representation learning tasks. Whereas gradient boosted decision trees (GBDT) often outperform other machine learning methods when faced with heterogeneous tabular data. But what approach should be used for graphs with tabular node features? Previous GNN models have mostly focused on networks with homogeneous sparse features and, as we show, are suboptimal in the heterogeneous setting. In this work, we propose a novel architecture that trains GBDT and GNN jointly to get the best of both worlds: the GBDT model deals with heterogeneous features, while GNN accounts for the graph structure. Our model benefits from end-to-end optimization by allowing new trees to fit the gradient updates of GNN. With an extensive experimental comparison to the leading GBDT and GNN models, we demonstrate a significant increase in performance on a variety of graphs with tabular features. The code is available: https://github.com/nd7141/bgnn.
Fidelity and Privacy of Synthetic Medical Data
Authors: Ofer Mendelevitch, Michael D. Lesh
Date: 2021-01-18 00:00:00
Tasks: De-identification

The digitization of medical records ushered in a new era of big data to clinical science, and with it the possibility that data could be shared, to multiply insights beyond what investigators could abstract from paper records. The need to share individual-level medical data to accelerate innovation in precision medicine continues to grow, and has never been more urgent, as scientists grapple with the COVID-19 pandemic. However, enthusiasm for the use of big data has been tempered by a fully appropriate concern for patient autonomy and privacy. That is, the ability to extract private or confidential information about an individual, in practice, renders it difficult to share data, since significant infrastructure and data governance must be established before data can be shared. Although HIPAA provided de-identification as an approved mechanism for data sharing, linkage attacks were identified as a major vulnerability. A variety of mechanisms have been established to avoid leaking private information, such as field suppression or abstraction, strictly limiting the amount of information that can be shared, or employing mathematical techniques such as differential privacy. Another approach, which we focus on here, is creating synthetic data that mimics the underlying data. For synthetic data to be a useful mechanism in support of medical innovation and a proxy for real-world evidence, one must demonstrate two properties of the synthetic dataset: (1) any analysis on the real data must be matched by analysis of the synthetic data (statistical fidelity) and (2) the synthetic data must preserve privacy, with minimal risk of re-identification (privacy guarantee). In this paper we propose a framework for quantifying the statistical fidelity and privacy preservation properties of synthetic datasets and demonstrate these metrics for synthetic data generated by Syntegra technology.
Free congruence: an exploration of expanded similarity measures for time series data
Authors: Lucas Cassiel Jacaruso
Date: 2021-01-17 00:00:00
Tasks: Dynamic Time Warping, Time Series, Time Series Analysis

Time series similarity measures are highly relevant in a wide range of emerging applications including training machine learning models, classification, and predictive modeling. Standard similarity measures for time series most often involve point-to-point distance measures including Euclidean distance and Dynamic Time Warping. Such similarity measures fundamentally require the fluctuation of values in the time series being compared to follow a corresponding order or cadence for similarity to be established. This paper is spurred by the exploration of a broader definition of similarity, namely one that takes into account the sheer numerical resemblance between sets of statistical properties for time series segments irrespectively of value labeling. Further, the presence of common pattern components between time series segments was examined even if they occur in a permuted order, which would not necessarily satisfy the criteria of more conventional point-to-point distance measures. Results were compared with those of Dynamic Time Warping on the same data for context. Surprisingly, the test for the numerical resemblance between sets of statistical properties established a stronger resemblance for pairings of decline years with greater statistical significance than Dynamic Time Warping on the particular data and sample size used.
Adversarial Machine Learning in Text Analysis and Generation
Authors: Izzat Alsmadi
Date: 2021-01-14 00:00:00
Tasks: BIG-bench Machine Learning

The research field of adversarial machine learning witnessed a significant interest in the last few years. A machine learner or model is secure if it can deliver main objectives with acceptable accuracy, efficiency, etc. while at the same time, it can resist different types and/or attempts of adversarial attacks. This paper focuses on studying aspects and research trends in adversarial machine learning specifically in text analysis and generation. The paper summarizes main research trends in the field such as GAN algorithms, models, types of attacks, and defense against those attacks.
Autocart -- spatially-aware regression trees for ecological and spatial modeling
Authors: Ethan Ancell, Brennan Bean
Date: 2021-01-20 00:00:00
Tasks: regression

Many ecological and spatial processes are complex in nature and are not accurately modeled by linear models. Regression trees promise to handle the high-order interactions that are present in ecological and spatial datasets, but fail to produce physically realistic characterizations of the underlying landscape. The "autocart" (autocorrelated regression trees) R package extends the functionality of previously proposed spatial regression tree methods through a spatially aware splitting function and novel adaptive inverse distance weighting method in each terminal node. The efficacy of these autocart models, including an autocart extension of random forest, is demonstrated on multiple datasets. This highlights the ability of autocart to model complex interactions between spatial variables while still providing physically realistic representations of the landscape.
The Diagnosis of Asthma using Hilbert-Huang Transform and Deep Learning on Lung Sounds
Authors: Gökhan Altan, Yakup Kutlu, Adnan Özhan Pekmezci, Serkan Nural
Date: 2021-01-20 00:00:00
Tasks: General Classification

Lung auscultation is the most effective and indispensable method for diagnosing various respiratory disorders by using the sounds from the airways during inspirium and exhalation using a stethoscope. In this study, the statistical features are calculated from intrinsic mode functions that are extracted by applying the HilbertHuang Transform to the lung sounds from 12 different auscultation regions on the chest and back. The classification of the lung sounds from asthma and healthy subjects is performed using Deep Belief Networks (DBN). The DBN classifier model with two hidden layers has been tested using 5-fold cross validation method. The proposed DBN separated lung sounds from asthmatic and healthy subjects with high classification performance rates of 84.61%, 85.83%, and 77.11% for overall accuracy, sensitivity, and selectivity, respectively using frequencytime analysis.
Text Line Segmentation for Challenging Handwritten Document Images Using Fully Convolutional Network
Authors: Berat Barakat, Ahmad Droby, Majeed Kassis, Jihad El-Sana
Date: 2021-01-20 00:00:00
Tasks: Segmentation

This paper presents a method for text line segmentation of challenging historical manuscript images. These manuscript images contain narrow interline spaces with touching components, interpenetrating vowel signs and inconsistent font types and sizes. In addition, they contain curved, multi-skewed and multi-directed side note lines within a complex page layout. Therefore, bounding polygon labeling would be very difficult and time consuming. Instead we rely on line masks that connect the components on the same text line. Then these line masks are predicted using a Fully Convolutional Network (FCN). In the literature, FCN has been successfully used for text line segmentation of regular handwritten document images. The present paper shows that FCN is useful with challenging manuscript images as well. Using a new evaluation metric that is sensitive to over segmentation as well as under segmentation, testing results on a publicly available challenging handwritten dataset are comparable with the results of a previous work on the same dataset.
Aesthetics, Personalization and Recommendation: A survey on Deep Learning in Fashion
Authors: Wei Gong, Laila Khalid
Date: 2021-01-20 00:00:00
Tasks: BIG-bench Machine Learning

Machine learning is completely changing the trends in the fashion industry. From big to small every brand is using machine learning techniques in order to improve their revenue, increase customers and stay ahead of the trend. People are into fashion and they want to know what looks best and how they can improve their style and elevate their personality. Using Deep learning technology and infusing it with Computer Vision techniques one can do so by utilizing Brain-inspired Deep Networks, and engaging into Neuroaesthetics, working with GANs and Training them, playing around with Unstructured Data,and infusing the transformer architecture are just some highlights which can be touched with the Fashion domain. Its all about designing a system that can tell us information regarding the fashion aspect that can come in handy with the ever growing demand. Personalization is a big factor that impacts the spending choices of customers.The survey also shows remarkable approaches that encroach the subject of achieving that by divulging deep into how visual data can be interpreted and leveraged into different models and approaches. Aesthetics play a vital role in clothing recommendation as users' decision depends largely on whether the clothing is in line with their aesthetics, however the conventional image features cannot portray this directly. For that the survey also highlights remarkable models like tensor factorization model, conditional random field model among others to cater the need to acknowledge aesthetics as an important factor in Apparel recommendation.These AI inspired deep models can pinpoint exactly which certain style resonates best with their customers and they can have an understanding of how the new designs will set in with the community. With AI and machine learning your businesses can stay ahead of the fashion trends.
Motif Identification using CNN-based Pairwise Subsequence Alignment Score Prediction
Authors: Ethan Jacob Moyer, Anup Das
Date: 2021-01-21 00:00:00

A common problem in bioinformatics is related to identifying gene regulatory regions marked by relatively high frequencies of motifs, or deoxyribonucleic acid sequences that often code for transcription and enhancer proteins. Predicting alignment scores between subsequence k-mers and a given motif enables the identification of candidate regulatory regions in a gene, which correspond to the transcription of these proteins. We propose a one-dimensional (1-D) Convolution Neural Network trained on k-mer formatted sequences interspaced with the given motif pattern to predict pairwise alignment scores between the consensus motif and subsequence k-mers. Our model consists of fifteen layers with three rounds of a one-dimensional convolution layer, a batch normalization layer, a dense layer, and a 1-D maximum pooling layer. We train the model using mean squared error loss on four different data sets each with a different motif pattern randomly inserted in DNA sequences: the first three data sets have zero, one, and two mutations applied on each inserted motif, and the fourth data set represents the inserted motif as a position-specific probability matrix. We use a novel proposed metric in order to evaluate the model's performance, $S_{\alpha}$, which is based on the Jaccard Index. We use 10-fold cross validation to evaluate out model. Using $S_{\alpha}$, we measure the accuracy of the model by identifying the 15 highest-scoring 15-mer indices of the predicted scores that agree with that of the actual scores within a selected $\alpha$ region. For the best performing data set, our results indicate on average 99.3% of the top 15 motifs were identified correctly within a one base pair stride ($\alpha = 1$) in the out of sample data. To the best of our knowledge, this is a novel approach that illustrates how data formatted in an intelligent way can be extrapolated using machine learning.
TDA-Net: Fusion of Persistent Homology and Deep Learning Features for COVID-19 Detection in Chest X-Ray Images
Authors: Mustafa Hajij, Ghada Zamzmi, Fawwaz Batayneh
Date: 2021-01-21 00:00:00
Tasks: Topological Data Analysis

Topological Data Analysis (TDA) has emerged recently as a robust tool to extract and compare the structure of datasets. TDA identifies features in data such as connected components and holes and assigns a quantitative measure to these features. Several studies reported that topological features extracted by TDA tools provide unique information about the data, discover new insights, and determine which feature is more related to the outcome. On the other hand, the overwhelming success of deep neural networks in learning patterns and relationships has been proven on a vast array of data applications, images in particular. To capture the characteristics of both powerful tools, we propose \textit{TDA-Net}, a novel ensemble network that fuses topological and deep features for the purpose of enhancing model generalizability and accuracy. We apply the proposed \textit{TDA-Net} to a critical application, which is the automated detection of COVID-19 from CXR images. The experimental results showed that the proposed network achieved excellent performance and suggests the applicability of our method in practice.
Blocked and Hierarchical Disentangled Representation From Information Theory Perspective
Authors: Ziwen Liu, Mingqiang Li, Congying Han
Date: 2021-01-21 00:00:00
Tasks: Attribute, Disentanglement, Representation Learning

We propose a novel and theoretical model, blocked and hierarchical variational autoencoder (BHiVAE), to get better-disentangled representation. It is well known that information theory has an excellent explanatory meaning for the network, so we start to solve the disentanglement problem from the perspective of information theory. BHiVAE mainly comes from the information bottleneck theory and information maximization principle. Our main idea is that (1) Neurons block not only one neuron node is used to represent attribute, which can contain enough information; (2) Create a hierarchical structure with different attributes on different layers, so that we can segment the information within each layer to ensure that the final representation is disentangled. Furthermore, we present supervised and unsupervised BHiVAE, respectively, where the difference is mainly reflected in the separation of information between different blocks. In supervised BHiVAE, we utilize the label information as the standard to separate blocks. In unsupervised BHiVAE, without extra information, we use the Total Correlation (TC) measure to achieve independence, and we design a new prior distribution of the latent space to guide the representation learning. It also exhibits excellent disentanglement results in experiments and superior classification accuracy in representation learning.
MoDL-QSM: Model-based Deep Learning for Quantitative Susceptibility Mapping
Authors: Ruimin Feng, Jiayi Zhao, He Wang, Baofeng Yang, Jie Feng, Yuting Shi, Ming Zhang, Chunlei Liu, Yuyao Zhang, Jie Zhuang, Hongjiang Wei
Date: 2021-01-21 00:00:00
Tasks: Deep Learning, SSIM

Quantitative susceptibility mapping (QSM) has demonstrated great potential in quantifying tissue susceptibility in various brain diseases. However, the intrinsic ill-posed inverse problem relating the tissue phase to the underlying susceptibility distribution affects the accuracy for quantifying tissue susceptibility. Recently, deep learning has shown promising results to improve accuracy by reducing the streaking artifacts. However, there exists a mismatch between the observed phase and the theoretical forward phase estimated by the susceptibility label. In this study, we proposed a model-based deep learning architecture that followed the STI (susceptibility tensor imaging) physical model, referred to as MoDL-QSM. Specifically, MoDL-QSM accounts for the relationship between STI-derived phase contrast induced by the susceptibility tensor terms (ki13,ki23,ki33) and the acquired single-orientation phase. The convolution neural networks are embedded into the physical model to learn a regularization term containing prior information. ki33 and phase induced by ki13 and ki23 terms were used as the labels for network training. Quantitative evaluation metrics (RSME, SSIM, and HFEN) were compared with recently developed deep learning QSM methods. The results showed that MoDL-QSM achieved superior performance, demonstrating its potential for future applications.
Noisy intermediate-scale quantum (NISQ) algorithms
Authors: Kishor Bharti, Alba Cervera-Lierta, Thi Ha Kyaw, Tobias Haug, Sumner Alperin-Lea, Abhinav Anand, Matthias Degroote, Hermanni Heimonen, Jakob S. Kottmann, Tim Menke, Wai-Keong Mok, Sukin Sim, Leong-Chuan Kwek, Alán Aspuru-Guzik
Date: 2021-01-21 00:00:00
Tasks: Benchmarking, Combinatorial Optimization

A universal fault-tolerant quantum computer that can solve efficiently problems such as integer factorization and unstructured database search requires millions of qubits with low error rates and long coherence times. While the experimental advancement towards realizing such devices will potentially take decades of research, noisy intermediate-scale quantum (NISQ) computers already exist. These computers are composed of hundreds of noisy qubits, i.e. qubits that are not error-corrected, and therefore perform imperfect operations in a limited coherence time. In the search for quantum advantage with these devices, algorithms have been proposed for applications in various disciplines spanning physics, machine learning, quantum chemistry and combinatorial optimization. The goal of such algorithms is to leverage the limited available resources to perform classically challenging tasks. In this review, we provide a thorough summary of NISQ computational paradigms and algorithms. We discuss the key structure of these algorithms, their limitations, and advantages. We additionally provide a comprehensive overview of various benchmarking and software tools useful for programming and testing NISQ devices.