pwc-archive/papers-with-abstracts huggingface dataset:
Fully Automatic Deep Learning Framework for Pancreatic Ductal Adenocarcinoma Detection on Computed Tomography
Authors: Natália Alves, Megan Schuurmans, Geke Litjens, Joeran S. Bosma, John Hermans, Henkjan Huisman
Date: 2021-11-30 00:00:00
Tasks: Anatomy, Deep Learning, Lesion Detection, Prognosis

Early detection improves prognosis in pancreatic ductal adenocarcinoma (PDAC) but is challenging as lesions are often small and poorly defined on contrast-enhanced computed tomography scans (CE-CT). Deep learning can facilitate PDAC diagnosis, however current models still fail to identify small (<2cm) lesions. In this study, state-of-the-art deep learning models were used to develop an automatic framework for PDAC detection, focusing on small lesions. Additionally, the impact of integrating surrounding anatomy was investigated. CE-CT scans from a cohort of 119 pathology-proven PDAC patients and a cohort of 123 patients without PDAC were used to train a nnUnet for automatic lesion detection and segmentation (nnUnet_T). Two additional nnUnets were trained to investigate the impact of anatomy integration: (1) segmenting the pancreas and tumor (nnUnet_TP), (2) segmenting the pancreas, tumor, and multiple surrounding anatomical structures (nnUnet_MS). An external, publicly available test set was used to compare the performance of the three networks. The nnUnet_MS achieved the best performance, with an area under the receiver operating characteristic curve of 0.91 for the whole test set and 0.88 for tumors <2cm, showing that state-of-the-art deep learning can detect small PDAC and benefits from anatomy information.
Assessment of Data Consistency through Cascades of Independently Recurrent Inference Machines for fast and robust accelerated MRI reconstruction
Authors: D. Karkalousos, S. Noteboom, H. E. Hulst, F. M. Vos, M. W. A. Caan
Date: 2021-11-30 00:00:00
Tasks: Denoising, Image Reconstruction, MRI Reconstruction

Machine Learning methods can learn how to reconstruct Magnetic Resonance Images and thereby accelerate acquisition, which is of paramount importance to the clinical workflow. Physics-informed networks incorporate the forward model of accelerated MRI reconstruction in the learning process. With increasing network complexity, robustness is not ensured when reconstructing data unseen during training. We aim to embed data consistency (DC) in deep networks while balancing the degree of network complexity. While doing so, we will assess whether either explicit or implicit enforcement of DC in varying network architectures is preferred to optimize performance. We propose a scheme called Cascades of Independently Recurrent Inference Machines (CIRIM) to assess DC through unrolled optimization. Herein we assess DC both implicitly by gradient descent and explicitly by a designed term. Extensive comparison of the CIRIM to CS as well as to other methods is performed: the E2EVN, CascadeNet, KIKINet, LPDNet, RIM, IRIM, and UNet. Models were trained and evaluated on T1-weighted and FLAIR contrast brain data, and T2-weighted knee data. Both 1D and 2D undersampling patterns were evaluated. Robustness was tested by reconstructing 7.5x prospectively undersampled 3D FLAIR MRI data of Multiple Sclerosis (MS) patients with white matter lesions. The CIRIM performed best when implicitly enforcing DC, while the E2EVN required an explicit DC formulation. In reconstructing MS patient data, prospectively acquired with a sampling pattern unseen during model training, the CIRIM maintained lesion contrast while efficiently denoising the images. The CIRIM showed highly promising generalization capabilities maintaining a very fair trade-off between reconstructed image quality and fast reconstruction times, which is crucial in the clinical workflow.
Gram Barcodes for Histopathology Tissue Texture Retrieval
Authors: Shalev Lifshitz, Abtin Riasatian, H. R. Tizhoosh
Date: 2021-11-28 00:00:00
Tasks: Image Retrieval, Retrieval

Recent advances in digital pathology have led to the need for Histopathology Image Retrieval (HIR) systems that search through databases of biopsy images to find similar cases to a given query image. These HIR systems allow pathologists to effortlessly and efficiently access thousands of previously diagnosed cases in order to exploit the knowledge in the corresponding pathology reports. Since HIR systems may have to deal with millions of gigapixel images, the extraction of compact and expressive image features must be available to allow for efficient and accurate retrieval. In this paper, we propose the application of Gram barcodes as image features for HIR systems. Unlike most feature generation schemes, Gram barcodes are based on high-order statistics that describe tissue texture by summarizing the correlations between different feature maps in layers of convolutional neural networks. We run HIR experiments on three public datasets using a pre-trained VGG19 network for Gram barcode generation and showcase highly competitive results.
Learning to Transfer for Traffic Forecasting via Multi-task Learning
Authors: Yichao Lu
Date: 2021-11-27 00:00:00
Tasks: Benchmarking, Domain Adaptation, Multi-Task Learning

Deep neural networks have demonstrated superior performance in short-term traffic forecasting. However, most existing traffic forecasting systems assume that the training and testing data are drawn from the same underlying distribution, which limits their practical applicability. The NeurIPS 2021 Traffic4cast challenge is the first of its kind dedicated to benchmarking the robustness of traffic forecasting models towards domain shifts in space and time. This technical report describes our solution to this challenge. In particular, we present a multi-task learning framework for temporal and spatio-temporal domain adaptation of traffic forecasting models. Experimental results demonstrate that our multi-task learning approach achieves strong empirical performance, outperforming a number of baseline domain adaptation methods, while remaining highly efficient. The source code for this technical report is available at https://github.com/YichaoLu/Traffic4cast2021.
The Exponentially Tilted Gaussian Prior for Variational Autoencoders
Authors: Griffin Floto, Stefan Kremer, Mihai Nica
Date: 2021-11-30 00:00:00
Tasks: Out-of-Distribution Detection

An important property for deep neural networks is the ability to perform robust out-of-distribution detection on previously unseen data. This property is essential for safety purposes when deploying models for real world applications. Recent studies show that probabilistic generative models can perform poorly on this task, which is surprising given that they seek to estimate the likelihood of training data. To alleviate this issue, we propose the exponentially tilted Gaussian prior distribution for the Variational Autoencoder (VAE) which pulls points onto the surface of a hyper-sphere in latent space. This achieves state-of-the art results on the area under the curve-receiver operator characteristics metric using just the log-likelihood that the VAE naturally assigns. Because this prior is a simple modification of the traditional VAE prior, it is faster and easier to implement than competitive methods.
2D-Motion Detection using SNNs with Graphene-Insulator-Graphene Memristive Synapses
Authors: Shubham Pande, Karthi Srinivasan, Suresh Balanethiram, Bhaswar Chakrabarti, Anjan Chakravorty
Date: 2021-11-30 00:00:00
Tasks: Motion Detection, Object

The event-driven nature of spiking neural networks makes them biologically plausible and more energy-efficient than artificial neural networks. In this work, we demonstrate motion detection of an object in a two-dimensional visual field. The network architecture presented here is biologically plausible and uses CMOS analog leaky integrate-and-fire neurons and ultra-low power multi-layer RRAM synapses. Detailed transistorlevel SPICE simulations show that the proposed structure can accurately and reliably detect complex motions of an object in a two-dimensional visual field.
High-Speed Light Focusing through Scattering Medium by Cooperatively Accelerated Genetic Algorithm
Authors: Shu Guo, Lin Pang
Date: 2021-11-29 00:00:00

We develop an accelerated Genetic Algorithm (GA) system constructed by the cooperation of field-programmable gate array (FPGA) and optimized parameters of the GA. We found the enhanced decay of mutation rate makes convergence of the GA much faster, enabling the parameter-induced acceleration of the GA. Furthermore, the accelerated configuration of the GA is programmed in FPGA to boost processing speed at the hardware level without external computation devices. This system has ability to focus light through scattering medium within 4 seconds with robust noise resistance and stable repetition performance, which could be further reduced to millisecond level with advanced board configuration. This study solves the long-term limitation of the GA, it promotes the applications of the GA in dynamic scattering mediums, with the capability to tackle wavefront shaping in biological material.
BrainScaleS Large Scale Spike Communication using Extoll
Authors: Tobias Thommes, Niels Buwen, Andreas Grübl, Eric Müller, Ulrich Brüning, Johannes Schemmel
Date: 2021-11-30 00:00:00

The BrainScaleS Neuromorphic Computing System is currently connected to a compute cluster via Gigabit-Ethernet network technology. This is convenient for the currently used experiment mode, where neuronal networks cover at most one wafer module. When modelling networks of larger size, as for example a full sized cortical microcircuit model, one has to think about connecting neurons across wafer modules to larger networks. This can be done, using the Extoll networking technology, which provides high bandwidth and low latencies, as well as a low overhead packet protocol format.
Nonparametric Topological Layers in Neural Networks
Authors: Dongfang Zhao
Date: 2021-11-27 00:00:00

Various topological techniques and tools have been applied to neural networks in terms of network complexity, explainability, and performance. One fundamental assumption of this line of research is the existence of a global (Euclidean) coordinate system upon which the topological layer is constructed. Despite promising results, such a \textit{topologization} method has yet to be widely adopted because the parametrization of a topologization layer takes a considerable amount of time and more importantly, lacks a theoretical foundation without which the performance of the neural network only achieves suboptimal performance. This paper proposes a learnable topological layer for neural networks without requiring a Euclidean space; Instead, the proposed construction requires nothing more than a general metric space except for an inner product, i.e., a Hilbert space. Accordingly, the according parametrization for the proposed topological layer is free of user-specified hyperparameters, which precludes the costly parametrization stage and the corresponding possibility of suboptimal networks.
Self-supervised Autoregressive Domain Adaptation for Time Series Data
Authors: Mohamed Ragab, Emadeldeen Eldele, Zhenghua Chen, Min Wu, Chee-Keong Kwoh, XiaoLi Li
Date: 2021-11-29 00:00:00
Tasks: Domain Adaptation, Self-Supervised Learning, Time Series, Time Series Analysis, Unsupervised Domain Adaptation

Unsupervised domain adaptation (UDA) has successfully addressed the domain shift problem for visual applications. Yet, these approaches may have limited performance for time series data due to the following reasons. First, they mainly rely on large-scale dataset (i.e., ImageNet) for the source pretraining, which is not applicable for time-series data. Second, they ignore the temporal dimension on the feature space of the source and target domains during the domain alignment step. Last, most of prior UDA methods can only align the global features without considering the fine-grained class distribution of the target domain. To address these limitations, we propose a Self-supervised Autoregressive Domain Adaptation (SLARDA) framework. In particular, we first design a self-supervised learning module that utilizes forecasting as an auxiliary task to improve the transferability of the source features. Second, we propose a novel autoregressive domain adaptation technique that incorporates temporal dependency of both source and target features during domain alignment. Finally, we develop an ensemble teacher model to align the class-wise distribution in the target domain via a confident pseudo labeling approach. Extensive experiments have been conducted on three real-world time series applications with 30 cross-domain scenarios. Results demonstrate that our proposed SLARDA method significantly outperforms the state-of-the-art approaches for time series domain adaptation.
Low-bit Quantization of Recurrent Neural Network Language Models Using Alternating Direction Methods of Multipliers
Authors: Junhao Xu, Xie Chen, Shoukang Hu, Jianwei Yu, Xunying Liu, Helen Meng
Date: 2021-11-29 00:00:00
Tasks: Quantization

The high memory consumption and computational costs of Recurrent neural network language models (RNNLMs) limit their wider application on resource constrained devices. In recent years, neural network quantization techniques that are capable of producing extremely low-bit compression, for example, binarized RNNLMs, are gaining increasing research interests. Directly training of quantized neural networks is difficult. By formulating quantized RNNLMs training as an optimization problem, this paper presents a novel method to train quantized RNNLMs from scratch using alternating direction methods of multipliers (ADMM). This method can also flexibly adjust the trade-off between the compression rate and model performance using tied low-bit quantization tables. Experiments on two tasks: Penn Treebank (PTB), and Switchboard (SWBD) suggest the proposed ADMM quantization achieved a model size compression factor of up to 31 times over the full precision baseline RNNLMs. Faster convergence of 5 times in model training over the baseline binarized RNNLM quantization was also obtained. Index Terms: Language models, Recurrent neural networks, Quantization, Alternating direction methods of multipliers.
p2pGNN: A Decentralized Graph Neural Network for Node Classification in Peer-to-Peer Networks
Authors: Emmanouil Krasanakis, Symeon Papadopoulos, Ioannis Kompatsiaris
Date: 2021-11-29 00:00:00
Tasks: Graph Neural Network, Node Classification

In this work, we aim to classify nodes of unstructured peer-to-peer networks with communication uncertainty, such as users of decentralized social networks. Graph Neural Networks (GNNs) are known to improve the accuracy of simple classifiers in centralized settings by leveraging naturally occurring network links, but graph convolutional layers are challenging to implement in decentralized settings when node neighbors are not constantly available. We address this problem by employing decoupled GNNs, where base classifier predictions and errors are diffused through graphs after training. For these, we deploy pre-trained and gossip-trained base classifiers and implement peer-to-peer graph diffusion under communication uncertainty. In particular, we develop an asynchronous decentralized formulation of diffusion that converges to centralized predictions in distribution and linearly with respect to communication rates. We experiment on three real-world graphs with node features and labels and simulate peer-to-peer networks with uniformly random communication frequencies; given a portion of known labels, our decentralized graph diffusion achieves comparable accuracy to centralized GNNs with minimal communication overhead (less than 3% of what gossip training already adds).
Evaluating Privacy-Preserving Machine Learning in Critical Infrastructures: A Case Study on Time-Series Classification
Authors: Dominique Mercier, Adriano Lucieri, Mohsin Munir, Andreas Dengel, Sheraz Ahmed
Date: 2021-11-29 00:00:00
Tasks: BIG-bench Machine Learning, Privacy Preserving, Time Series, Time Series Analysis, Time Series Classification

With the advent of machine learning in applications of critical infrastructure such as healthcare and energy, privacy is a growing concern in the minds of stakeholders. It is pivotal to ensure that neither the model nor the data can be used to extract sensitive information used by attackers against individuals or to harm whole societies through the exploitation of critical infrastructure. The applicability of machine learning in these domains is mostly limited due to a lack of trust regarding the transparency and the privacy constraints. Various safety-critical use cases (mostly relying on time-series data) are currently underrepresented in privacy-related considerations. By evaluating several privacy-preserving methods regarding their applicability on time-series data, we validated the inefficacy of encryption for deep learning, the strong dataset dependence of differential privacy, and the broad applicability of federated methods.
New PCA-based Category Encoder for Cybersecurity and Processing Data in IoT Devices
Authors: Hamed Farkhari, Joseanne Viana, Luis Miguel Campos, Pedro Sebastiao, Luis Bernardo
Date: 2021-11-29 00:00:00
Tasks: Binary Classification

Increasing the cardinality of categorical variables might decrease the overall performance of machine learning (ML) algorithms. This paper presents a novel computational preprocessing method to convert categorical to numerical variables ML algorithms. It uses a supervised binary classifier to extract additional context-related features from the categorical values. Up to two numerical variables per categorical variable are created, depending on the compression achieved by the Principal Component Analysis (PCA). The method requires two hyperparameters: a threshold related to the distribution of categories in the variables and the PCA representativeness. This paper applies the proposed approach to the well-known cybersecurity NSLKDD dataset to select and convert three categorical features to numerical features. After choosing the threshold parameter, we use conditional probabilities to convert the three categorical variables into six new numerical variables. After that, we feed these numerical variables to the PCA algorithm and select the whole or partial numbers of the Principal Components (PCs). Finally, by applying binary classification with ten different classifiers, we measure the performance of the new encoder and compare it with the other 17 well-known category encoders. The new technique achieves the highest performance related to accuracy and Area Under the Curve (AUC) on high cardinality categorical variables. Also, we define the harmonic average metrics to find the best trade-off between train and test performances and prevent underfitting and overfitting. Ultimately, the number of newly created numerical variables is minimal. This data reduction improves computational processing time in Internet of things (IoT) devices in future telecommunication networks.
Architecture Matters: Investigating the Influence of Differential Privacy on Neural Network Design
Authors: Felix Morsbach, Tobias Dehling, Ali Sunyaev
Date: 2021-11-29 00:00:00

One barrier to more widespread adoption of differentially private neural networks is the entailed accuracy loss. To address this issue, the relationship between neural network architectures and model accuracy under differential privacy constraints needs to be better understood. As a first step, we test whether extant knowledge on architecture design also holds in the differentially private setting. Our findings show that it does not; architectures that perform well without differential privacy, do not necessarily do so with differential privacy. Consequently, extant knowledge on neural network architecture design cannot be seamlessly translated into the differential privacy context. Future research is required to better understand the relationship between neural network architectures and model accuracy to enable better architecture design choices under differential privacy constraints.
Privacy-Preserving Serverless Edge Learning with Decentralized Small Data
Authors: Shih-Chun Lin, Chia-Hung Lin
Date: 2021-11-29 00:00:00
Tasks: Cloud Computing, Deep Learning, Privacy Preserving

In the last decade, data-driven algorithms outperformed traditional optimization-based algorithms in many research areas, such as computer vision, natural language processing, etc. However, extensive data usages bring a new challenge or even threat to deep learning algorithms, i.e., privacy-preserving. Distributed training strategies have recently become a promising approach to ensure data privacy when training deep models. This paper extends conventional serverless platforms with serverless edge learning architectures and provides an efficient distributed training framework from the networking perspective. This framework dynamically orchestrates available resources among heterogeneous physical units to efficiently fulfill deep learning objectives. The design jointly considers learning task requests and underlying infrastructure heterogeneity, including last-mile transmissions, computation abilities of mobile devices, edge and cloud computing centers, and devices battery status. Furthermore, to significantly reduce distributed training overheads, small-scale data training is proposed by integrating with a general, simple data classifier. This low-load enhancement can seamlessly work with various distributed deep models to improve communications and computation efficiencies during the training phase. Finally, open challenges and future research directions encourage the research community to develop efficient distributed deep learning techniques.
Classification of animal sounds in a hyperdiverse rainforest using Convolutional Neural Networks
Authors: Yuren Sun, Tatiana Midori Maeda, Claudia Solis-Lemus, Daniel Pimentel-Alarcon, Zuzana Burivalova
Date: 2021-11-29 00:00:00
Tasks: Data Augmentation, Management, Transfer Learning

To protect tropical forest biodiversity, we need to be able to detect it reliably, cheaply, and at scale. Automated species detection from passively recorded soundscapes via machine-learning approaches is a promising technique towards this goal, but it is constrained by the necessity of large training data sets. Using soundscapes from a tropical forest in Borneo and a Convolutional Neural Network model (CNN) created with transfer learning, we investigate i) the minimum viable training data set size for accurate prediction of call types ('sonotypes'), and ii) the extent to which data augmentation can overcome the issue of small training data sets. We found that even relatively high sample sizes (> 80 per call type) lead to mediocre accuracy, which however improves significantly with data augmentation, including at extremely small sample sizes, regardless of taxonomic group or call characteristics. Our results suggest that transfer learning and data augmentation can make the use of CNNs to classify species' vocalizations feasible even for small soundscape-based projects with many rare species. Retraining our open-source model requires only basic programming skills which makes it possible for individual conservation initiatives to match their local context, in order to enable more evidence-informed management of biodiversity.