pwc-archive/papers-with-abstracts huggingface dataset:
Hybrid Supervised Reinforced Model for Dialogue Systems
Authors: Carlos Miranda, Yacine Kessaci
Date: 2020-11-04 00:00:00
Tasks: Decision Making, Dialogue Management, Management, model, Task-Oriented Dialogue Systems

This paper presents a recurrent hybrid model and training procedure for task-oriented dialogue systems based on Deep Recurrent Q-Networks (DRQN). The model copes with both tasks required for Dialogue Management: State Tracking and Decision Making. It is based on modeling Human-Machine interaction into a latent representation embedding an interaction context to guide the discussion. The model achieves greater performance, learning speed and robustness than a non-recurrent baseline. Moreover, results allow interpreting and validating the policy evolution and the latent representations information-wise.
Optimizing Transformer for Low-Resource Neural Machine Translation
Authors: Ali Araabi, Christof Monz
Date: 2020-11-04 00:00:00
Proceeding: COLING 2020 8
Tasks: Low Resource Neural Machine Translation, Low-Resource Neural Machine Translation, Machine Translation, Translation

Language pairs with limited amounts of parallel data, also known as low-resource languages, remain a challenge for neural machine translation. While the Transformer model has achieved significant improvements for many language pairs and has become the de facto mainstream architecture, its capability under low-resource conditions has not been fully investigated yet. Our experiments on different subsets of the IWSLT14 training data show that the effectiveness of Transformer under low-resource conditions is highly dependent on the hyper-parameter settings. Our experiments show that using an optimized Transformer for low-resource conditions improves the translation quality up to 7.3 BLEU points compared to using the Transformer default settings.
Indic-Transformers: An Analysis of Transformer Language Models for Indian Languages
Authors: Kushal Jain, Adwait Deshpande, Kumar Shridhar, Felix Laumann, Ayushman Dash
Date: 2020-11-04 00:00:00
Tasks: Classification, General Classification, Language Modeling, Language Modelling, Question Answering, text-classification, Text Classification, token-classification, Token Classification

Language models based on the Transformer architecture have achieved state-of-the-art performance on a wide range of NLP tasks such as text classification, question-answering, and token classification. However, this performance is usually tested and reported on high-resource languages, like English, French, Spanish, and German. Indian languages, on the other hand, are underrepresented in such benchmarks. Despite some Indian languages being included in training multilingual Transformer models, they have not been the primary focus of such work. In order to evaluate the performance on Indian languages specifically, we analyze these language models through extensive experiments on multiple downstream tasks in Hindi, Bengali, and Telugu language. Here, we compare the efficacy of fine-tuning model parameters of pre-trained models against that of training a language model from scratch. Moreover, we empirically argue against the strict dependency between the dataset size and model performance, but rather encourage task-specific model and method selection. We achieve state-of-the-art performance on Hindi and Bengali languages for text classification task. Finally, we present effective strategies for handling the modeling of Indian languages and we release our model checkpoints for the community : https://huggingface.co/neuralspace-reverie.
Investigating Novel Verb Learning in BERT: Selectional Preference Classes and Alternation-Based Syntactic Generalization
Authors: Tristan Thrush, Ethan Wilcox, Roger Levy
Date: 2020-11-04 00:00:00
Proceeding: EMNLP (BlackboxNLP) 2020 11
Tasks: Few-Shot Learning

Previous studies investigating the syntactic abilities of deep learning models have not targeted the relationship between the strength of the grammatical generalization and the amount of evidence to which the model is exposed during training. We address this issue by deploying a novel word-learning paradigm to test BERT's few-shot learning capabilities for two aspects of English verbs: alternations and classes of selectional preferences. For the former, we fine-tune BERT on a single frame in a verbal-alternation pair and ask whether the model expects the novel verb to occur in its sister frame. For the latter, we fine-tune BERT on an incomplete selectional network of verbal objects and ask whether it expects unattested but plausible verb/object pairs. We find that BERT makes robust grammatical generalizations after just one or two instances of a novel word in fine-tuning. For the verbal alternation tests, we find that the model displays behavior that is consistent with a transitivity bias: verbs seen few times are expected to take direct objects, but verbs seen with direct objects are not expected to occur intransitively.
Prosodic Representation Learning and Contextual Sampling for Neural Text-to-Speech
Authors: Sri Karlapati, Ammar Abbas, Zack Hodari, Alexis Moinet, Arnaud Joly, Penny Karanasou, Thomas Drugman
Date: 2020-11-04 00:00:00
Tasks: Graph Attention, Representation Learning, Sentence, Speech Synthesis, text-to-speech, Text to Speech

In this paper, we introduce Kathaka, a model trained with a novel two-stage training process for neural speech synthesis with contextually appropriate prosody. In Stage I, we learn a prosodic distribution at the sentence level from mel-spectrograms available during training. In Stage II, we propose a novel method to sample from this learnt prosodic distribution using the contextual information available in text. To do this, we use BERT on text, and graph-attention networks on parse trees extracted from text. We show a statistically significant relative improvement of $13.2\%$ in naturalness over a strong baseline when compared to recordings. We also conduct an ablation study on variations of our sampling technique, and show a statistically significant improvement over the baseline in each case.
PCP Theorems, SETH and More: Towards Proving Sub-linear Time Inapproximability
Authors: Hengzhao Ma, Jianzhong Li
Date: 2020-11-04 00:00:00

In this paper we propose the PCP-like theorem for sub-linear time inapproximability. Abboud et al. have devised the distributed PCP framework for sub-quadratic time inapproximability. We show that the distributed PCP theorem can be generalized for proving arbitrary polynomial time inapproximability, but fails in the linear case. We prove the sub-linear PCP theorem by adapting from an MA-protocol for the Set Containment problem, and show how to use the theorem to prove both existing and new inapproximability results, exhibiting the power of the sub-linear PCP theorem. Considering the emerging research works on sub-linear time algorithms, the sub-linear PCP theorem is important in guiding the research in sub-linear time approximation algorithms.
Physics-Informed Echo State Networks
Authors: Nguyen Anh Khoa Doan, Wolfgang Polifke, Luca Magri
Date: 2020-10-31 00:00:00

We propose a physics-informed Echo State Network (ESN) to predict the evolution of chaotic systems. Compared to conventional ESNs, the physics-informed ESNs are trained to solve supervised learning tasks while ensuring that their predictions do not violate physical laws. This is achieved by introducing an additional loss function during the training, which is based on the system's governing equations. The additional loss function penalizes non-physical predictions without the need of any additional training data. This approach is demonstrated on a chaotic Lorenz system and a truncation of the Charney-DeVore system. Compared to the conventional ESNs, the physics-informed ESNs improve the predictability horizon by about two Lyapunov times. This approach is also shown to be robust with regard to noise. The proposed framework shows the potential of using machine learning combined with prior physical knowledge to improve the time-accurate prediction of chaotic dynamical systems.
InferBench: Understanding Deep Learning Inference Serving with an Automatic Benchmarking System
Authors: Huaizheng Zhang, Yizheng Huang, Yonggang Wen, Jianxiong Yin, Kyle Guan
Date: 2020-11-04 00:00:00
Tasks: Benchmarking

Deep learning (DL) models have become core modules for many applications. However, deploying these models without careful performance benchmarking that considers both hardware and software's impact often leads to poor service and costly operational expenditure. To facilitate DL models' deployment, we implement an automatic and comprehensive benchmark system for DL developers. To accomplish benchmark-related tasks, the developers only need to prepare a configuration file consisting of a few lines of code. Our system, deployed to a leader server in DL clusters, will dispatch users' benchmark jobs to follower workers. Next, the corresponding requests, workload, and even models can be generated automatically by the system to conduct DL serving benchmarks. Finally, developers can leverage many analysis tools and models in our system to gain insights into the trade-offs of different system configurations. In addition, a two-tier scheduler is incorporated to avoid unnecessary interference and improve average job compilation time by up to 1.43x (equivalent of 30\% reduction). Our system design follows the best practice in DL clusters operations to expedite day-to-day DL service evaluation efforts by the developers. We conduct many benchmark experiments to provide in-depth and comprehensive evaluations. We believe these results are of great values as guidelines for DL service configuration and resource allocation.
Federated Knowledge Distillation
Authors: Hyowoon Seo, Jihong Park, Seungeun Oh, Mehdi Bennis, Seong-Lyun Kim
Date: 2020-11-04 00:00:00
Tasks: Federated Learning, Knowledge Distillation

Distributed learning frameworks often rely on exchanging model parameters across workers, instead of revealing their raw data. A prime example is federated learning that exchanges the gradients or weights of each neural network model. Under limited communication resources, however, such a method becomes extremely costly particularly for modern deep neural networks having a huge number of model parameters. In this regard, federated distillation (FD) is a compelling distributed learning solution that only exchanges the model outputs whose dimensions are commonly much smaller than the model sizes (e.g., 10 labels in the MNIST dataset). The goal of this chapter is to provide a deep understanding of FD while demonstrating its communication efficiency and applicability to a variety of tasks. To this end, towards demystifying the operational principle of FD, the first part of this chapter provides a novel asymptotic analysis for two foundational algorithms of FD, namely knowledge distillation (KD) and co-distillation (CD), by exploiting the theory of neural tangent kernel (NTK). Next, the second part elaborates on a baseline implementation of FD for a classification task, and illustrates its performance in terms of accuracy and communication efficiency compared to FL. Lastly, to demonstrate the applicability of FD to various distributed learning tasks and environments, the third part presents two selected applications, namely FD over asymmetric uplink-and-downlink wireless channels and FD for reinforcement learning.
Video Generative Adversarial Networks: A Review
Authors: Nuha Aldausari, Arcot Sowmya, Nadine Marcus, Gelareh Mohammadi
Date: 2020-11-04 00:00:00
Tasks: Anomaly Detection

With the increasing interest in the content creation field in multiple sectors such as media, education, and entertainment, there is an increasing trend in the papers that uses AI algorithms to generate content such as images, videos, audio, and text. Generative Adversarial Networks (GANs) in one of the promising models that synthesizes data samples that are similar to real data samples. While the variations of GANs models, in general, have been covered to some extent in several survey papers, to the best of our knowledge, this is among the first survey papers that reviews the state-of-the-art video GANs models. This paper first categorized GANs review papers into general GANs review papers, image GANs review papers, and special field GANs review papers such as anomaly detection, medical imaging, or cybersecurity. The paper then summarizes the main improvements in GANs frameworks that are not initially developed for the video domain but have been adopted in multiple video GANs variations. Then, a comprehensive review of video GANs models is provided under two main divisions according to the presence or non-presence of a condition. The conditional models then further grouped according to the type of condition into audio, text, video, and image. The paper is concluded by highlighting the main challenges and limitations of the current video GANs models. A comprehensive list of datasets, applied loss functions, and evaluation metrics is provided in the supplementary material.
Trustworthy AI
Authors: Richa Singh, Mayank Vatsa, Nalini Ratha
Date: 2020-11-02 00:00:00
Tasks: Fairness

Modern AI systems are reaping the advantage of novel learning methods. With their increasing usage, we are realizing the limitations and shortfalls of these systems. Brittleness to minor adversarial changes in the input data, ability to explain the decisions, address the bias in their training data, high opacity in terms of revealing the lineage of the system, how they were trained and tested, and under which parameters and conditions they can reliably guarantee a certain level of performance, are some of the most prominent limitations. Ensuring the privacy and security of the data, assigning appropriate credits to data sources, and delivering decent outputs are also required features of an AI system. We propose the tutorial on Trustworthy AI to address six critical issues in enhancing user and public trust in AI systems, namely: (i) bias and fairness, (ii) explainability, (iii) robust mitigation of adversarial attacks, (iv) improved privacy and security in model building, (v) being decent, and (vi) model attribution, including the right level of credit assignment to the data sources, model architectures, and transparency in lineage.
Convolutional Proximal Neural Networks and Plug-and-Play Algorithms
Authors: Johannes Hertrich, Sebastian Neumayer, Gabriele Steidl
Date: 2020-11-04 00:00:00
Tasks: Denoising

In this paper, we introduce convolutional proximal neural networks (cPNNs), which are by construction averaged operators. For filters of full length, we propose a stochastic gradient descent algorithm on a submanifold of the Stiefel manifold to train cPNNs. In case of filters with limited length, we design algorithms for minimizing functionals that approximate the orthogonality constraints imposed on the operators by penalizing the least squares distance to the identity operator. Then, we investigate how scaled cPNNs with a prescribed Lipschitz constant can be used for denoising signals and images, where the achieved quality depends on the Lipschitz constant. Finally, we apply cPNN based denoisers within a Plug-and-Play (PnP) framework and provide convergence results for the corresponding PnP forward-backward splitting algorithm based on an oracle construction.
"What We Can't Measure, We Can't Understand": Challenges to Demographic Data Procurement in the Pursuit of Fairness
Authors: McKane Andrus, Elena Spitzer, Jeffrey Brown, Alice Xiang
Date: 2020-10-30 00:00:00
Tasks: Fairness

As calls for fair and unbiased algorithmic systems increase, so too does the number of individuals working on algorithmic fairness in industry. However, these practitioners often do not have access to the demographic data they feel they need to detect bias in practice. Even with the growing variety of toolkits and strategies for working towards algorithmic fairness, they almost invariably require access to demographic attributes or proxies. We investigated this dilemma through semi-structured interviews with 38 practitioners and professionals either working in or adjacent to algorithmic fairness. Participants painted a complex picture of what demographic data availability and use look like on the ground, ranging from not having access to personal data of any kind to being legally required to collect and use demographic data for discrimination assessments. In many domains, demographic data collection raises a host of difficult questions, including how to balance privacy and fairness, how to define relevant social categories, how to ensure meaningful consent, and whether it is appropriate for private companies to infer someone's demographics. Our research suggests challenges that must be considered by businesses, regulators, researchers, and community groups in order to enable practitioners to address algorithmic bias in practice. Critically, we do not propose that the overall goal of future work should be to simply lower the barriers to collecting demographic data. Rather, our study surfaces a swath of normative questions about how, when, and whether this data should be procured, and, in cases where it is not, what should still be done to mitigate bias.
Surgical Data Science -- from Concepts toward Clinical Translation
Authors: Lena Maier-Hein, Matthias Eisenmann, Duygu Sarikaya, Keno März, Toby Collins, Anand Malpani, Johannes Fallert, Hubertus Feussner, Stamatia Giannarou, Pietro Mascagni, Hirenkumar Nakawala, Adrian Park, Carla Pugh, Danail Stoyanov, Swaroop S. Vedula, Kevin Cleary, Gabor Fichtinger, Germain Forestier, Bernard Gibaud, Teodor Grantcharov, Makoto Hashizume, Doreen Heckmann-Nötzel, Hannes G. Kenngott, Ron Kikinis, Lars Mündermann, Nassir Navab, Sinan Onogur, Raphael Sznitman, Russell H. Taylor, Minu D. Tizabi, Martin Wagner, Gregory D. Hager, Thomas Neumuth, Nicolas Padoy, Justin Collins, Ines Gockel, Jan Goedeke, Daniel A. Hashimoto, Luc Joyeux, Kyle Lam, Daniel R. Leff, Amin Madani, Hani J. Marcus, Ozanan Meireles, Alexander Seitel, Dogu Teber, Frank Ückert, Beat P. Müller-Stich, Pierre Jannin, Stefanie Speidel
Date: 2020-10-30 00:00:00
Tasks: Translation

Recent developments in data science in general and machine learning in particular have transformed the way experts envision the future of surgery. Surgical Data Science (SDS) is a new research field that aims to improve the quality of interventional healthcare through the capture, organization, analysis and modeling of data. While an increasing number of data-driven approaches and clinical applications have been studied in the fields of radiological and clinical data science, translational success stories are still lacking in surgery. In this publication, we shed light on the underlying reasons and provide a roadmap for future advances in the field. Based on an international workshop involving leading researchers in the field of SDS, we review current practice, key achievements and initiatives as well as available standards and tools for a number of topics relevant to the field, namely (1) infrastructure for data acquisition, storage and access in the presence of regulatory constraints, (2) data annotation and sharing and (3) data analytics. We further complement this technical perspective with (4) a review of currently available SDS products and the translational progress from academia and (5) a roadmap for faster clinical translation and exploitation of the full potential of SDS, based on an international multi-round Delphi process.
Fast Data-Driven Learning of MRI Sampling Pattern for Large Scale Problems
Authors: Marcelo V. W. Zibetti, Gabor T. Herman, Ravinder R. Regatte
Date: 2020-11-04 00:00:00
Tasks: Anatomy

Purpose: A fast data-driven optimization approach, named bias-accelerated subset selection (BASS), is proposed for learning efficacious sampling patterns (SPs) with the purpose of reducing scan time in large-dimensional parallel MRI. Methods: BASS is applicable when Cartesian fully-sampled k-space data of specific anatomy is available for training and the reconstruction method is specified, learning which k-space points are more relevant for the specific anatomy and reconstruction in recovering the non-sampled points. BASS was tested with four reconstruction methods for parallel MRI based on low-rankness and sparsity that allow a free choice of the SP. Two datasets were tested, one of the brain images for high-resolution imaging and another of knee images for quantitative mapping of the cartilage. Results: BASS, with its low computational cost and fast convergence, obtained SPs 100 times faster than the current best greedy approaches. Reconstruction quality increased up to 45\% with our learned SP over that provided by variable density and Poisson disk SPs, considering the same scan time. Optionally, the scan time can be nearly halved without loss of reconstruction quality. Conclusion: Compared with current approaches, BASS can be used to rapidly learn effective SPs for various reconstruction methods, using larger SP and larger datasets. This enables a better selection of efficacious sampling-reconstruction pairs for specific MRI problems.
Single channel voice separation for unknown number of speakers under reverberant and noisy settings
Authors: Shlomo E. Chazan, Lior Wolf, Eliya Nachmani, Yossi Adi
Date: 2020-11-04 00:00:00
Tasks: Classification, General Classification

We present a unified network for voice separation of an unknown number of speakers. The proposed approach is composed of several separation heads optimized together with a speaker classification branch. The separation is carried out in the time domain, together with parameter sharing between all separation heads. The classification branch estimates the number of speakers while each head is specialized in separating a different number of speakers. We evaluate the proposed model under both clean and noisy reverberant set-tings. Results suggest that the proposed approach is superior to the baseline model by a significant margin. Additionally, we present a new noisy and reverberant dataset of up to five different speakers speaking simultaneously.
Fault Detection for Covered Conductors With High-Frequency Voltage Signals: From Local Patterns to Global Features
Authors: Kunjin Chen, Tomáš Vantuch, Yu Zhang, Jun Hu, Jinliang He
Date: 2020-11-01 00:00:00
Tasks: Clustering, Diversity, Fault Detection

The detection and characterization of partial discharge (PD) are crucial for the insulation diagnosis of overhead lines with covered conductors. With the release of a large dataset containing thousands of naturally obtained high-frequency voltage signals, data-driven analysis of fault-related PD patterns on an unprecedented scale becomes viable. The high diversity of PD patterns and background noise interferences motivates us to design an innovative pulse shape characterization method based on clustering techniques, which can dynamically identify a set of representative PD-related pulses. Capitalizing on those pulses as referential patterns, we construct insightful features and develop a novel machine learning model with a superior detection performance for early-stage covered conductor faults. The presented model outperforms the winning model in a Kaggle competition and provides the state-of-the-art solution to detect real-time disturbances in the field.