pwc-archive/papers-with-abstracts huggingface dataset:
Model-based image adjustment for a successful pansharpening
Authors: Gintautas Palubinskas
Date: 2021-03-04 00:00:00
Tasks: model, Pansharpening

A new model-based image adjustment for the enhancement of multi-resolution image fusion or pansharpening is proposed. Such image adjustment is needed for most pansharpening methods using panchromatic band and/or intensity image (calculated as a weighted sum of multispectral bands) as an input. Due various reasons, e.g. calibration inaccuracies, usage of different sensors, input images for pansharpening: low resolution multispectral image or more precisely the calculated intensity image and high resolution panchromatic image may differ in values of their physical properties, e.g. radiances or reflectances depending on the processing level. But the same objects/classes in both images should exhibit similar values or more generally similar statistics. Similarity definition will depend on a particular application. For a successful fusion of data from two sensors the energy balance between radiances/reflectances of both sensors should hold. A virtual band is introduced to compensate for total energy disbalance in different sensors. Its estimation consists of several steps: first, weights for individual spectral bands are estimated in a low resolution scale, where both multispectral and panchromatic images (low pass filtered version) are available, then, the estimated virtual band is up-sampled to a high scale and, finally, high resolution panchromatic band is corrected by subtracting virtual band. This corrected panchromatic band is used instead of original panchromatic image in the following pansharpening. It is shown, for example, that the performance quality of component substitution based methods can be increased significantly.
BM3D vs 2-Layer ONN
Authors: Junaid Malik, Serkan Kiranyaz, Mehmet Yamac, Moncef Gabbouj
Date: 2021-03-04 00:00:00
Tasks: Denoising, Image Denoising

Despite their recent success on image denoising, the need for deep and complex architectures still hinders the practical usage of CNNs. Older but computationally more efficient methods such as BM3D remain a popular choice, especially in resource-constrained scenarios. In this study, we aim to find out whether compact neural networks can learn to produce competitive results as compared to BM3D for AWGN image denoising. To this end, we configure networks with only two hidden layers and employ different neuron models and layer widths for comparing the performance with BM3D across different AWGN noise levels. Our results conclusively show that the recently proposed self-organized variant of operational neural networks based on a generative neuron model (Self-ONNs) is not only a better choice as compared to CNNs, but also provide competitive results as compared to BM3D and even significantly surpass it for high noise levels.
Sub-pixel face landmarks using heatmaps and a bag of tricks
Authors: Samuel W. F. Earp, Aubin Samacoits, Sanjana Jain, Pavit Noinongyao, Siwa Boonpunmongkol
Date: 2021-03-04 00:00:00
Tasks: Face Alignment, Face Recognition, Position

Accurate face landmark localization is an essential part of face recognition, reconstruction and morphing. To accurately localize face landmarks, we present our heatmap regression approach. Each model consists of a MobileNetV2 backbone followed by several upscaling layers, with different tricks to optimize both performance and inference cost. We use five na\"ive face landmarks from a publicly available face detector to position and align the face instead of using the bounding box like traditional methods. Moreover, we show by adding random rotation, displacement and scaling -- after alignment -- that the model is more sensitive to the face position than orientation. We also show that it is possible to reduce the upscaling complexity by using a mixture of deconvolution and pixel-shuffle layers without impeding localization performance. We present our state-of-the-art face landmark localization model (ranking second on The 2nd Grand Challenge of 106-Point Facial Landmark Localization validation set). Finally, we test the effect on face recognition using these landmarks, using a publicly available model and benchmarks.
Neuromechanics-based Deep Reinforcement Learning of Neurostimulation Control in FES cycling
Authors: Nat Wannawas, Mahendran Subramanian, A. Aldo Faisal
Date: 2021-03-04 00:00:00
Tasks: Deep Reinforcement Learning, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Functional Electrical Stimulation (FES) can restore motion to a paralysed person's muscles. Yet, control stimulating many muscles to restore the practical function of entire limbs is an unsolved problem. Current neurostimulation engineering still relies on 20th Century control approaches and correspondingly shows only modest results that require daily tinkering to operate at all. Here, we present our state of the art Deep Reinforcement Learning (RL) developed for real time adaptive neurostimulation of paralysed legs for FES cycling. Core to our approach is the integration of a personalised neuromechanical component into our reinforcement learning framework that allows us to train the model efficiently without demanding extended training sessions with the patient and working out of the box. Our neuromechanical component includes merges musculoskeletal models of muscle and or tendon function and a multistate model of muscle fatigue, to render the neurostimulation responsive to a paraplegic's cyclist instantaneous muscle capacity. Our RL approach outperforms PID and Fuzzy Logic controllers in accuracy and performance. Crucially, our system learned to stimulate a cyclist's legs from ramping up speed at the start to maintaining a high cadence in steady state racing as the muscles fatigue. A part of our RL neurostimulation system has been successfully deployed at the Cybathlon 2020 bionic Olympics in the FES discipline with our paraplegic cyclist winning the Silver medal among 9 competing teams.
Self-supervised deep convolutional neural network for chest X-ray classification
Authors: Matej Gazda, Jakub Gazda, Jan Plavka, Peter Drotar
Date: 2021-03-04 00:00:00
Tasks: Diagnostic, General Classification, Medical Procedure, X-ray Classification

Chest radiography is a relatively cheap, widely available medical procedure that conveys key information for making diagnostic decisions. Chest X-rays are almost always used in the diagnosis of respiratory diseases such as pneumonia or the recent COVID-19. In this paper, we propose a self-supervised deep neural network that is pretrained on an unlabeled chest X-ray dataset. The learned representations are transferred to downstream task - the classification of respiratory diseases. The results obtained on four public datasets show that our approach yields competitive results without requiring large amounts of labeled training data.
A Neural Text-to-Speech Model Utilizing Broadcast Data Mixed with Background Music
Authors: Hanbin Bae, Jae-Sung Bae, Young-Sun Joo, Young-Ik Kim, Hoon-Young Cho
Date: 2021-03-04 00:00:00
Tasks: text-to-speech, Text to Speech

Recently, it has become easier to obtain speech data from various media such as the internet or YouTube, but directly utilizing them to train a neural text-to-speech (TTS) model is difficult. The proportion of clean speech is insufficient and the remainder includes background music. Even with the global style token (GST). Therefore, we propose the following method to successfully train an end-to-end TTS model with limited broadcast data. First, the background music is removed from the speech by introducing a music filter. Second, the GST-TTS model with an auxiliary quality classifier is trained with the filtered speech and a small amount of clean speech. In particular, the quality classifier makes the embedding vector of the GST layer focus on representing the speech quality (filtered or clean) of the input speech. The experimental results verified that the proposed method synthesized much more high-quality speech than conventional methods.
Detecting Spurious Correlations with Sanity Tests for Artificial Intelligence Guided Radiology Systems
Authors: Usman Mahmood, Robik Shrestha, David D. B. Bates, Lorenzo Mannelli, Giuseppe Corrias, Yusuf Erdi, Christopher Kanan
Date: 2021-03-04 00:00:00

Artificial intelligence (AI) has been successful at solving numerous problems in machine perception. In radiology, AI systems are rapidly evolving and show progress in guiding treatment decisions, diagnosing, localizing disease on medical images, and improving radiologists' efficiency. A critical component to deploying AI in radiology is to gain confidence in a developed system's efficacy and safety. The current gold standard approach is to conduct an analytical validation of performance on a generalization dataset from one or more institutions, followed by a clinical validation study of the system's efficacy during deployment. Clinical validation studies are time-consuming, and best practices dictate limited re-use of analytical validation data, so it is ideal to know ahead of time if a system is likely to fail analytical or clinical validation. In this paper, we describe a series of sanity tests to identify when a system performs well on development data for the wrong reasons. We illustrate the sanity tests' value by designing a deep learning system to classify pancreatic cancer seen in computed tomography scans.
PointGuard: Provably Robust 3D Point Cloud Classification
Authors: Hongbin Liu, Jinyuan Jia, Neil Zhenqiang Gong
Date: 2021-03-04 00:00:00
Proceeding: CVPR 2021 1
Tasks: 3D Point Cloud Classification, Autonomous Driving, Classification, General Classification, Point Cloud Classification, Robotic Grasping

3D point cloud classification has many safety-critical applications such as autonomous driving and robotic grasping. However, several studies showed that it is vulnerable to adversarial attacks. In particular, an attacker can make a classifier predict an incorrect label for a 3D point cloud via carefully modifying, adding, and/or deleting a small number of its points. Randomized smoothing is state-of-the-art technique to build certifiably robust 2D image classifiers. However, when applied to 3D point cloud classification, randomized smoothing can only certify robustness against adversarially modified points. In this work, we propose PointGuard, the first defense that has provable robustness guarantees against adversarially modified, added, and/or deleted points. Specifically, given a 3D point cloud and an arbitrary point cloud classifier, our PointGuard first creates multiple subsampled point clouds, each of which contains a random subset of the points in the original point cloud; then our PointGuard predicts the label of the original point cloud as the majority vote among the labels of the subsampled point clouds predicted by the point cloud classifier. Our first major theoretical contribution is that we show PointGuard provably predicts the same label for a 3D point cloud when the number of adversarially modified, added, and/or deleted points is bounded. Our second major theoretical contribution is that we prove the tightness of our derived bound when no assumptions on the point cloud classifier are made. Moreover, we design an efficient algorithm to compute our certified robustness guarantees. We also empirically evaluate PointGuard on ModelNet40 and ScanNet benchmark datasets.
Mobile Touchless Fingerprint Recognition: Implementation, Performance and Usability Aspects
Authors: Jannis Priesnitz, Rolf Huesmann, Christian Rathgeb, Nicolas Buchmann, Christoph Busch
Date: 2021-03-04 00:00:00

This work presents an automated touchless fingerprint recognition system for smartphones. We provide a comprehensive description of the entire recognition pipeline and discuss important requirements for a fully automated capturing system. Also, our implementation is made publicly available for research purposes. During a database acquisition, a total number of 1,360 touchless and touch-based samples of 29 subjects are captured in two different environmental situations. Experiments on the acquired database show a comparable performance of our touchless scheme and the touch-based baseline scheme under constrained environmental influences. A comparative usability study on both capturing device types indicates that the majority of subjects prefer the touchless capturing method. Based on our experimental results we analyze the impact of the current COVID-19 pandemic on fingerprint recognition systems. Finally, implementation aspects of touchless fingerprint recognition are summarized.
A Survey on Graph Structure Learning: Progress and Opportunities
Authors: Yanqiao Zhu, Weizhi Xu, Jinghao Zhang, Yuanqi Du, Jieyu Zhang, Qiang Liu, Carl Yang, Shu Wu
Date: 2021-03-04 00:00:00
Tasks: Graph structure learning, Survey

Graphs are widely used to describe real-world objects and their interactions. Graph Neural Networks (GNNs) as a de facto model for analyzing graphstructured data, are highly sensitive to the quality of the given graph structures. Therefore, noisy or incomplete graphs often lead to unsatisfactory representations and prevent us from fully understanding the mechanism underlying the system. In pursuit of an optimal graph structure for downstream tasks, recent studies have sparked an effort around the central theme of Graph Structure Learning (GSL), which aims to jointly learn an optimized graph structure and corresponding graph representations. In the presented survey, we broadly review recent progress in GSL methods. Specifically, we first formulate a general pipeline of GSL and review state-of-the-art methods classified by the way of modeling graph structures, followed by applications of GSL across domains. Finally, we point out some issues in current studies and discuss future directions.
CoTr: Efficiently Bridging CNN and Transformer for 3D Medical Image Segmentation
Authors: Yutong Xie, Jianpeng Zhang, Chunhua Shen, Yong Xia
Date: 2021-03-04 00:00:00
Tasks: Image Segmentation, Inductive Bias, Medical Image Segmentation, Organ Segmentation, Segmentation, Semantic Segmentation

Convolutional neural networks (CNNs) have been the de facto standard for nowadays 3D medical image segmentation. The convolutional operations used in these networks, however, inevitably have limitations in modeling the long-range dependency due to their inductive bias of locality and weight sharing. Although Transformer was born to address this issue, it suffers from extreme computational and spatial complexities in processing high-resolution 3D feature maps. In this paper, we propose a novel framework that efficiently bridges a {\bf Co}nvolutional neural network and a {\bf Tr}ansformer {\bf (CoTr)} for accurate 3D medical image segmentation. Under this framework, the CNN is constructed to extract feature representations and an efficient deformable Transformer (DeTrans) is built to model the long-range dependency on the extracted feature maps. Different from the vanilla Transformer which treats all image positions equally, our DeTrans pays attention only to a small set of key positions by introducing the deformable self-attention mechanism. Thus, the computational and spatial complexities of DeTrans have been greatly reduced, making it possible to process the multi-scale and high-resolution feature maps, which are usually of paramount importance for image segmentation. We conduct an extensive evaluation on the Multi-Atlas Labeling Beyond the Cranial Vault (BCV) dataset that covers 11 major human organs. The results indicate that our CoTr leads to a substantial performance improvement over other CNN-based, transformer-based, and hybrid methods on the 3D multi-organ segmentation task. Code is available at \def\UrlFont{\rm\small\ttfamily} \url{https://github.com/YtongXie/CoTr}
Lost in Pruning: The Effects of Pruning Neural Networks beyond Test Accuracy
Authors: Lucas Liebenwein, Cenk Baykal, Brandon Carter, David Gifford, Daniela Rus
Date: 2021-03-04 00:00:00
Tasks: Network Pruning

Neural network pruning is a popular technique used to reduce the inference costs of modern, potentially overparameterized, networks. Starting from a pre-trained network, the process is as follows: remove redundant parameters, retrain, and repeat while maintaining the same test accuracy. The result is a model that is a fraction of the size of the original with comparable predictive performance (test accuracy). Here, we reassess and evaluate whether the use of test accuracy alone in the terminating condition is sufficient to ensure that the resulting model performs well across a wide spectrum of "harder" metrics such as generalization to out-of-distribution data and resilience to noise. Across evaluations on varying architectures and data sets, we find that pruned networks effectively approximate the unpruned model, however, the prune ratio at which pruned networks achieve commensurate performance varies significantly across tasks. These results call into question the extent of \emph{genuine} overparameterization in deep learning and raise concerns about the practicability of deploying pruned networks, specifically in the context of safety-critical systems, unless they are widely evaluated beyond test accuracy to reliably predict their performance. Our code is available at https://github.com/lucaslie/torchprune.
The Transformer Network for the Traveling Salesman Problem
Authors: Xavier Bresson, Thomas Laurent
Date: 2021-03-04 00:00:00
Tasks: Traveling Salesman Problem

The Traveling Salesman Problem (TSP) is the most popular and most studied combinatorial problem, starting with von Neumann in 1951. It has driven the discovery of several optimization techniques such as cutting planes, branch-and-bound, local search, Lagrangian relaxation, and simulated annealing. The last five years have seen the emergence of promising techniques where (graph) neural networks have been capable to learn new combinatorial algorithms. The main question is whether deep learning can learn better heuristics from data, i.e. replacing human-engineered heuristics? This is appealing because developing algorithms to tackle efficiently NP-hard problems may require years of research, and many industry problems are combinatorial by nature. In this work, we propose to adapt the recent successful Transformer architecture originally developed for natural language processing to the combinatorial TSP. Training is done by reinforcement learning, hence without TSP training solutions, and decoding uses beam search. We report improved performances over recent learned heuristics with an optimal gap of 0.004% for TSP50 and 0.39% for TSP100.
Perceptual Image Restoration with High-Quality Priori and Degradation Learning
Authors: Chaoyi Han, Yiping Duan, Xiaoming Tao, Jianhua Lu
Date: 2021-03-04 00:00:00
Tasks: Image Quality Assessment, Image Restoration, No-Reference Image Quality Assessment, NR-IQA, Vocal Bursts Intensity Prediction

Perceptual image restoration seeks for high-fidelity images that most likely degrade to given images. For better visual quality, previous work proposed to search for solutions within the natural image manifold, by exploiting the latent space of a generative model. However, the quality of generated images are only guaranteed when latent embedding lies close to the prior distribution. In this work, we propose to restrict the feasible region within the prior manifold. This is accomplished with a non-parametric metric for two distributions: the Maximum Mean Discrepancy (MMD). Moreover, we model the degradation process directly as a conditional distribution. We show that our model performs well in measuring the similarity between restored and degraded images. Instead of optimizing the long criticized pixel-wise distance over degraded images, we rely on such model to find visual pleasing images with high probability. Our simultaneous restoration and enhancement framework generalizes well to real-world complicated degradation types. The experimental results on perceptual quality and no-reference image quality assessment (NR-IQA) demonstrate the superior performance of our method.
SpectralDefense: Detecting Adversarial Attacks on CNNs in the Fourier Domain
Authors: Paula Harder, Franz-Josef Pfreundt, Margret Keuper, Janis Keuper
Date: 2021-03-04 00:00:00
Tasks: Adversarial Attack

Despite the success of convolutional neural networks (CNNs) in many computer vision and image analysis tasks, they remain vulnerable against so-called adversarial attacks: Small, crafted perturbations in the input images can lead to false predictions. A possible defense is to detect adversarial examples. In this work, we show how analysis in the Fourier domain of input images and feature maps can be used to distinguish benign test samples from adversarial images. We propose two novel detection methods: Our first method employs the magnitude spectrum of the input images to detect an adversarial attack. This simple and robust classifier can successfully detect adversarial perturbations of three commonly used attack methods. The second method builds upon the first and additionally extracts the phase of Fourier coefficients of feature-maps at different layers of the network. With this extension, we are able to improve adversarial detection rates compared to state-of-the-art detectors on five different attack methods.
Optical Flow Estimation from a Single Motion-blurred Image
Authors: Dawit Mureja Argaw, Junsik Kim, Francois Rameau, Jae Won Cho, In So Kweon
Date: 2021-03-04 00:00:00
Tasks: Deblurring, Optical Flow Estimation, Semantic Segmentation

In most of computer vision applications, motion blur is regarded as an undesirable artifact. However, it has been shown that motion blur in an image may have practical interests in fundamental computer vision problems. In this work, we propose a novel framework to estimate optical flow from a single motion-blurred image in an end-to-end manner. We design our network with transformer networks to learn globally and locally varying motions from encoded features of a motion-blurred input, and decode left and right frame features without explicit frame supervision. A flow estimator network is then used to estimate optical flow from the decoded features in a coarse-to-fine manner. We qualitatively and quantitatively evaluate our model through a large set of experiments on synthetic and real motion-blur datasets. We also provide in-depth analysis of our model in connection with related approaches to highlight the effectiveness and favorability of our approach. Furthermore, we showcase the applicability of the flow estimated by our method on deblurring and moving object segmentation tasks.
Learning-based Adaptive Control using Contraction Theory
Authors: Hiroyasu Tsukamoto, Soon-Jo Chung, Jean-Jacques Slotine
Date: 2021-03-04 00:00:00

Adaptive control is subject to stability and performance issues when a learned model is used to enhance its performance. This paper thus presents a deep learning-based adaptive control framework for nonlinear systems with multiplicatively-separable parametrization, called adaptive Neural Contraction Metric (aNCM). The aNCM approximates real-time optimization for computing a differential Lyapunov function and a corresponding stabilizing adaptive control law by using a Deep Neural Network (DNN). The use of DNNs permits real-time implementation of the control law and broad applicability to a variety of nonlinear systems with parametric and nonparametric uncertainties. We show using contraction theory that the aNCM ensures exponential boundedness of the distance between the target and controlled trajectories in the presence of parametric uncertainties of the model, learning errors caused by aNCM approximation, and external disturbances. Its superiority to the existing robust and adaptive control methods is demonstrated using a cart-pole balancing model.