pwc-archive/papers-with-abstracts huggingface dataset:
Implicit Gradient Regularization
Authors: David G. T. Barrett, Benoit Dherin
Date: 2020-09-23 00:00:00
Proceeding: ICLR 2021 1

Gradient descent can be surprisingly good at optimizing deep neural networks without overfitting and without explicit regularization. We find that the discrete steps of gradient descent implicitly regularize models by penalizing gradient descent trajectories that have large loss gradients. We call this Implicit Gradient Regularization (IGR) and we use backward error analysis to calculate the size of this regularization. We confirm empirically that implicit gradient regularization biases gradient descent toward flat minima, where test errors are small and solutions are robust to noisy parameter perturbations. Furthermore, we demonstrate that the implicit gradient regularization term can be used as an explicit regularizer, allowing us to control this gradient regularization directly. More broadly, our work indicates that backward error analysis is a useful theoretical approach to the perennial question of how learning rate, model size, and parameter regularization interact to determine the properties of overparameterized models optimized with gradient descent.
Label-Efficient Multi-Task Segmentation using Contrastive Learning
Authors: Junichiro Iwasawa, Yuichiro Hirano, Yohei Sugawara
Date: 2020-09-23 00:00:00
Tasks: Contrastive Learning, Multi-Task Learning, Segmentation

Obtaining annotations for 3D medical images is expensive and time-consuming, despite its importance for automating segmentation tasks. Although multi-task learning is considered an effective method for training segmentation models using small amounts of annotated data, a systematic understanding of various subtasks is still lacking. In this study, we propose a multi-task segmentation model with a contrastive learning based subtask and compare its performance with other multi-task models, varying the number of labeled data for training. We further extend our model so that it can utilize unlabeled data through the regularization branch in a semi-supervised manner. We experimentally show that our proposed method outperforms other multi-task methods including the state-of-the-art fully supervised model when the amount of annotated data is limited.
2D-3D Geometric Fusion Network using Multi-Neighbourhood Graph Convolution for RGB-D Indoor Scene Classification
Authors: Albert Mosella-Montoro, Javier Ruiz-Hidalgo
Date: 2020-09-23 00:00:00
Tasks: General Classification, Scene Classification

Multi-modal fusion has been proved to help enhance the performance of scene classification tasks. This paper presents a 2D-3D Fusion stage that combines 3D Geometric Features with 2D Texture Features obtained by 2D Convolutional Neural Networks. To get a robust 3D Geometric embedding, a network that uses two novel layers is proposed. The first layer, Multi-Neighbourhood Graph Convolution, aims to learn a more robust geometric descriptor of the scene combining two different neighbourhoods: one in the Euclidean space and the other in the Feature space. The second proposed layer, Nearest Voxel Pooling, improves the performance of the well-known Voxel Pooling. Experimental results, using NYU-Depth-V2 and SUN RGB-D datasets, show that the proposed method outperforms the current state-of-the-art in RGB-D indoor scene classification task.
Hierarchical Pre-training for Sequence Labelling in Spoken Dialog
Authors: Emile Chapuis, Pierre Colombo, Matteo Manica, Matthieu Labeau, Chloe Clavel
Date: 2020-09-23 00:00:00
Proceeding: Findings of the Association for Computational Linguistics 2020
Tasks: Dialogue Act Classification, Emotion Recognition in Conversation, Text Classification

Sequence labelling tasks like Dialog Act and Emotion/Sentiment identification are a key component of spoken dialog systems. In this work, we propose a new approach to learn generic representations adapted to spoken dialog, which we evaluate on a new benchmark we call Sequence labellIng evaLuatIon benChmark fOr spoken laNguagE benchmark (\texttt{SILICONE}). \texttt{SILICONE} is model-agnostic and contains 10 different datasets of various sizes. We obtain our representations with a hierarchical encoder based on transformer architectures, for which we extend two well-known pre-training objectives. Pre-training is performed on OpenSubtitles: a large corpus of spoken dialog containing over $2.3$ billion of tokens. We demonstrate how hierarchical encoders achieve competitive results with consistently fewer parameters compared to state-of-the-art models and we show their importance for both pre-training and fine-tuning.
Information-Theoretic Visual Explanation for Black-Box Classifiers
Authors: Jihun Yi, Eunji Kim, Siwon Kim, Sungroh Yoon
Date: 2020-09-23 00:00:00

In this work, we attempt to explain the prediction of any black-box classifier from an information-theoretic perspective. For each input feature, we compare the classifier outputs with and without that feature using two information-theoretic metrics. Accordingly, we obtain two attribution maps--an information gain (IG) map and a point-wise mutual information (PMI) map. IG map provides a class-independent answer to "How informative is each pixel?", and PMI map offers a class-specific explanation of "How much does each pixel support a specific class?" Compared to existing methods, our method improves the correctness of the attribution maps in terms of a quantitative metric. We also provide a detailed analysis of an ImageNet classifier using the proposed method, and the code is available online.
Evolution of Part-of-Speech in Classical Chinese
Authors: Bai Li
Date: 2020-09-23 00:00:00
Tasks: Position

Classical Chinese is a language notable for its word class flexibility: the same word may often be used as a noun or a verb. Bisang (2008) claimed that Classical Chinese is a precategorical language, where the syntactic position of a word determines its part-of-speech category. In this paper, we apply entropy-based metrics to evaluate these claims on historical corpora. We further explore differences between nouns and verbs in Classical Chinese: using psycholinguistic norms, we find a positive correlation between concreteness and noun usage. Finally, we align character embeddings from Classical and Modern Chinese, and find that verbs undergo more semantic change than nouns.
The Scheduling Job-Set Optimization Problem: A Model-Based Diagnosis Approach
Authors: Patrick Rodler, Erich Teppan
Date: 2020-09-23 00:00:00
Tasks: Scheduling

A common issue for companies is that the volume of product orders may at times exceed the production capacity. We formally introduce two novel problems dealing with the question which orders to discard or postpone in order to meet certain (timeliness) goals, and try to approach them by means of model-based diagnosis. In thorough analyses, we identify many similarities of the introduced problems to diagnosis problems, but also reveal crucial idiosyncracies and outline ways to handle or leverage them. Finally, a proof-of-concept evaluation on industrial-scale problem instances from a well-known scheduling benchmark suite demonstrates that one of the two formalized problems can be well attacked by out-of-the-box model-based diagnosis tools.
Worst-Case-Aware Curriculum Learning for Zero and Few Shot Transfer
Authors: Sheng Zhang, Xin Zhang, Weiming Zhang, Anders Søgaard
Date: 2020-09-23 00:00:00
Tasks: Transfer Learning

Multi-task transfer learning based on pre-trained language encoders achieves state-of-the-art performance across a range of tasks. Standard approaches implicitly assume the tasks, for which we have training data, are equally representative of the tasks we are interested in, an assumption which is often hard to justify. This paper presents a more agnostic approach to multi-task transfer learning, which uses automated curriculum learning to minimize a new family of worst-case-aware losses across tasks. Not only do these losses lead to better performance on outlier tasks; they also lead to better performance in zero-shot and few-shot transfer settings.
Seq2Edits: Sequence Transduction Using Span-level Edit Operations
Authors: Felix Stahlberg, Shankar Kumar
Date: 2020-09-23 00:00:00
Proceeding: EMNLP 2020 11
Tasks: Grammatical Error Correction, Sentence, Sentence Fusion, TAG, Text Normalization, Text Simplification

We propose Seq2Edits, an open-vocabulary approach to sequence editing for natural language processing (NLP) tasks with a high degree of overlap between input and output texts. In this approach, each sequence-to-sequence transduction is represented as a sequence of edit operations, where each operation either replaces an entire source span with target tokens or keeps it unchanged. We evaluate our method on five NLP tasks (text normalization, sentence fusion, sentence splitting & rephrasing, text simplification, and grammatical error correction) and report competitive results across the board. For grammatical error correction, our method speeds up inference by up to 5.2x compared to full sequence models because inference time depends on the number of edits rather than the number of target tokens. For text normalization, sentence fusion, and grammatical error correction, our approach improves explainability by associating each edit operation with a human-readable tag.
Cosine Similarity of Multimodal Content Vectors for TV Programmes
Authors: Saba Nazir. Taner Cagali, Chris Newell, Mehrnoosh Sadrzadeh
Date: 2020-09-23 00:00:00
Tasks: Diversity

Multimodal information originates from a variety of sources: audiovisual files, textual descriptions, and metadata. We show how one can represent the content encoded by each individual source using vectors, how to combine the vectors via middle and late fusion techniques, and how to compute the semantic similarities between the contents. Our vectorial representations are built from spectral features and Bags of Audio Words, for audio, LSI topics and Doc2vec embeddings for subtitles, and the categorical features, for metadata. We implement our model on a dataset of BBC TV programmes and evaluate the fused representations to provide recommendations. The late fused similarity matrices significantly improve the precision and diversity of recommendations.
Using Under-trained Deep Ensembles to Learn Under Extreme Label Noise
Authors: Konstantinos Nikolaidis, Thomas Plagemann, Stein Kristiansen, Vera Goebel, Mohan Kankanhalli
Date: 2020-09-23 00:00:00
Tasks: Sleep apnea detection

Improper or erroneous labelling can pose a hindrance to reliable generalization for supervised learning. This can have negative consequences, especially for critical fields such as healthcare. We propose an effective new approach for learning under extreme label noise, based on under-trained deep ensembles. Each ensemble member is trained with a subset of the training data, to acquire a general overview of the decision boundary separation, without focusing on potentially erroneous details. The accumulated knowledge of the ensemble is combined to form new labels, that determine a better class separation than the original labels. A new model is trained with these labels to generalize reliably despite the label noise. We focus on a healthcare setting and extensively evaluate our approach on the task of sleep apnea detection. For comparison with related work, we additionally evaluate on the task of digit recognition. In our experiments, we observed performance improvement in accuracy from 6.7\% up-to 49.3\% for the task of digit classification and in kappa from 0.02 up-to 0.55 for the task of sleep apnea detection.
Anisotropic 3D Multi-Stream CNN for Accurate Prostate Segmentation from Multi-Planar MRI
Authors: Anneke Meyer, Grzegorz Chlebus, Marko Rak, Daniel Schindele, Martin Schostak, Bram van Ginneken, Andrea Schenk, Hans Meine, Horst K. Hahn, Andreas Schreiber, Christian Hansen
Date: 2020-09-23 00:00:00
Tasks: Hyperparameter Optimization, Segmentation

Background and Objective: Accurate and reliable segmentation of the prostate gland in MR images can support the clinical assessment of prostate cancer, as well as the planning and monitoring of focal and loco-regional therapeutic interventions. Despite the availability of multi-planar MR scans due to standardized protocols, the majority of segmentation approaches presented in the literature consider the axial scans only. Methods: We propose an anisotropic 3D multi-stream CNN architecture, which processes additional scan directions to produce a higher-resolution isotropic prostate segmentation. We investigate two variants of our architecture, which work on two (dual-plane) and three (triple-plane) image orientations, respectively. We compare them with the standard baseline (single-plane) used in literature, i.e., plain axial segmentation. To realize a fair comparison, we employ a hyperparameter optimization strategy to select optimal configurations for the individual approaches. Results: Training and evaluation on two datasets spanning multiple sites obtain statistical significant improvement over the plain axial segmentation ($p<0.05$ on the Dice similarity coefficient). The improvement can be observed especially at the base ($0.898$ single-plane vs. $0.906$ triple-plane) and apex ($0.888$ single-plane vs. $0.901$ dual-plane). Conclusion: This study indicates that models employing two or three scan directions are superior to plain axial segmentation. The knowledge of precise boundaries of the prostate is crucial for the conservation of risk structures. Thus, the proposed models have the potential to improve the outcome of prostate cancer diagnosis and therapies.
Multiple interaction learning with question-type prior knowledge for constraining answer search space in visual question answering
Authors: Tuong Do, Binh X. Nguyen, Huy Tran, Erman Tjiputra, Quang D. Tran, Thanh-Toan Do
Date: 2020-09-23 00:00:00
Tasks: Question Answering, Visual Question Answering, Visual Question Answering (VQA)

Different approaches have been proposed to Visual Question Answering (VQA). However, few works are aware of the behaviors of varying joint modality methods over question type prior knowledge extracted from data in constraining answer search space, of which information gives a reliable cue to reason about answers for questions asked in input images. In this paper, we propose a novel VQA model that utilizes the question-type prior information to improve VQA by leveraging the multiple interactions between different joint modality methods based on their behaviors in answering questions from different types. The solid experiments on two benchmark datasets, i.e., VQA 2.0 and TDIUC, indicate that the proposed method yields the best performance with the most competitive approaches.
Text Classification with Novelty Detection
Authors: Qi Qin, Wenpeng Hu, Bing Liu
Date: 2020-09-23 00:00:00
Tasks: Classification, General Classification, Novelty Detection, text-classification, Text Classification

This paper studies the problem of detecting novel or unexpected instances in text classification. In traditional text classification, the classes appeared in testing must have been seen in training. However, in many applications, this is not the case because in testing, we may see unexpected instances that are not from any of the training classes. In this paper, we propose a significantly more effective approach that converts the original problem to a pair-wise matching problem and then outputs how probable two instances belong to the same class. Under this approach, we present two models. The more effective model uses two embedding matrices of a pair of instances as two channels of a CNN. The output probabilities from such pairs are used to judge whether a test instance is from a seen class or is novel/unexpected. Experimental results show that the proposed method substantially outperforms the state-of-the-art baselines.
ANNdotNET -- deep learning tool on .NET Platform
Authors: Bahrudin Hrnjica
Date: 2020-09-23 00:00:00
Tasks: Deep Learning, Descriptive, GPU

ANNdotNET is an open source project for deep learning written in C# with ability to create, train, evaluate and export deep learning models. The project consists of the Graphical User Interface module capable to visually prepare data, fine tune hyper-parameters, design network architecture, evaluate and test trained models. The ANNdotNET introduces the Visual Network Designer, (VND) for visually design almost any sequential deep learning network. Beside VND, ANNdotNET implements Machine Learning Engine, (MLE) based on CNTK - deep learning framework, with ability to train and evaluate models on GPU. For model evaluation ANNdotNET contains rich set of visual and descriptive performance parameters, history of the training process and set of export/deployment options. The advantage of using ANNdotNET over the classic code based ML approach is more focus on deep learning network design and training process instead of focusing on coding and debugging. It is ideal for engineers not familiar with supported programming languages. The project is hosted at github.com/bhrnjica/anndotnet.
Efficient Incremental Modelling and Solving
Authors: Gökberk Koçak, Özgür Akgün, Nguyen Dang, Ian Miguel
Date: 2020-09-23 00:00:00

In various scenarios, a single phase of modelling and solving is either not sufficient or not feasible to solve the problem at hand. A standard approach to solving AI planning problems, for example, is to incrementally extend the planning horizon and solve the problem of trying to find a plan of a particular length. Indeed, any optimization problem can be solved as a sequence of decision problems in which the objective value is incrementally updated. Another example is constraint dominance programming (CDP), in which search is organized into a sequence of levels. The contribution of this work is to enable a native interaction between SAT solvers and the automated modelling system Savile Row to support efficient incremental modelling and solving. This allows adding new decision variables, posting new constraints and removing existing constraints (via assumptions) between incremental steps. Two additional benefits of the native coupling of modelling and solving are the ability to retain learned information between SAT solver calls and to enable SAT assumptions, further improving flexibility and efficiency. Experiments on one optimisation problem and five pattern mining tasks demonstrate that the native interaction between the modelling system and SAT solver consistently improves performance significantly.
Residual Embedding Similarity-Based Network Selection for Predicting Brain Network Evolution Trajectory from a Single Observation
Authors: Ahmet Serkan Goktas, Alaa Bessadok, Islem Rekik
Date: 2020-09-23 00:00:00

While existing predictive frameworks are able to handle Euclidean structured data (i.e, brain images), they might fail to generalize to geometric non-Euclidean data such as brain networks. Besides, these are rooted the sample selection step in using Euclidean or learned similarity measure between vectorized training and testing brain networks. Such sample connectomic representation might include irrelevant and redundant features that could mislead the training sample selection step. Undoubtedly, this fails to exploit and preserve the topology of the brain connectome. To overcome this major drawback, we propose Residual Embedding Similarity-Based Network selection (RESNets) for predicting brain network evolution trajectory from a single timepoint. RESNets first learns a compact geometric embedding of each training and testing sample using adversarial connectome embedding network. This nicely reduces the high-dimensionality of brain networks while preserving their topological properties via graph convolutional networks. Next, to compute the similarity between subjects, we introduce the concept of a connectional brain template (CBT), a fixed network reference, where we further represent each training and testing network as a deviation from the reference CBT in the embedding space. As such, we select the most similar training subjects to the testing subject at baseline by comparing their learned residual embeddings with respect to the pre-defined CBT. Once the best training samples are selected at baseline, we simply average their corresponding brain networks at follow-up timepoints to predict the evolution trajectory of the testing network. Our experiments on both healthy and disordered brain networks demonstrate the success of our proposed method in comparison to RESNets ablated versions and traditional approaches.
Robustification of Segmentation Models Against Adversarial Perturbations In Medical Imaging
Authors: Hanwool Park, Amirhossein Bayat, Mohammad Sabokrou, Jan S. Kirschke, Bjoern H. Menze
Date: 2020-09-23 00:00:00
Tasks: Deep Learning, Segmentation

This paper presents a novel yet efficient defense framework for segmentation models against adversarial attacks in medical imaging. In contrary to the defense methods against adversarial attacks for classification models which widely are investigated, such defense methods for segmentation models has been less explored. Our proposed method can be used for any deep learning models without revising the target deep learning models, as well as can be independent of adversarial attacks. Our framework consists of a frequency domain converter, a detector, and a reformer. The frequency domain converter helps the detector detects adversarial examples by using a frame domain of an image. The reformer helps target models to predict more precisely. We have experiments to empirically show that our proposed method has a better performance compared to the existing defense method.
Probabilistic Machine Learning for Healthcare
Authors: Irene Y. Chen, Shalmali Joshi, Marzyeh Ghassemi, Rajesh Ranganath
Date: 2020-09-23 00:00:00
Tasks: BIG-bench Machine Learning, reinforcement-learning, Reinforcement Learning (RL)

Machine learning can be used to make sense of healthcare data. Probabilistic machine learning models help provide a complete picture of observed data in healthcare. In this review, we examine how probabilistic machine learning can advance healthcare. We consider challenges in the predictive model building pipeline where probabilistic models can be beneficial including calibration and missing data. Beyond predictive models, we also investigate the utility of probabilistic machine learning models in phenotyping, in generative models for clinical use cases, and in reinforcement learning.
Multiplexed Illumination for Classifying Visually Similar Objects
Authors: Taihua Wang, Donald G. Dansereau
Date: 2020-09-23 00:00:00
Tasks: Classification, General Classification, Lesion Classification, Skin Lesion Classification

Distinguishing visually similar objects like forged/authentic bills and healthy/unhealthy plants is beyond the capabilities of even the most sophisticated classifiers. We propose the use of multiplexed illumination to extend the range of objects that can be successfully classified. We construct a compact RGB-IR light stage that images samples under different combinations of illuminant position and colour. We then develop a methodology for selecting illumination patterns and training a classifier using the resulting imagery. We use the light stage to model and synthetically relight training samples, and propose a greedy pattern selection scheme that exploits this ability to train in simulation. We then apply the trained patterns to carry out fast classification of new objects. We demonstrate the approach on visually similar artificial and real fruit samples, showing a marked improvement compared with fixed-illuminant approaches as well as a more conventional code selection scheme. This work allows fast classification of previously indistinguishable objects, with potential applications in forgery detection, quality control in agriculture and manufacturing, and skin lesion classification.