pwc-archive/papers-with-abstracts huggingface dataset:
I-nteract 2.0: A Cyber-Physical System to Design 3D Models using Mixed Reality Technologies and Deep Learning for Additive Manufacturing
Authors: Ammar Malik, Hugo Lhachemi, Robert Shorten
Date: 2020-10-21 00:00:00
Tasks: Mixed Reality

I-nteract is a cyber-physical system that enables real-time interaction with both virtual and real artifacts to design 3D models for additive manufacturing by leveraging on mixed reality technologies. This paper presents novel advances in the development of the interaction platform I-nteract to generate 3D models using both constructive solid geometry and artificial intelligence. The system also enables the user to adjust the dimensions of the 3D models with respect to their physical workspace. The effectiveness of the system is demonstrated by generating 3D models of furniture (e.g., chairs and tables) and fitting them into the physical space in a mixed reality environment.
Deep Neural Networks Are Congestion Games: From Loss Landscape to Wardrop Equilibrium and Beyond
Authors: Nina Vesseron, Ievgen Redko, Charlotte Laclau
Date: 2020-10-21 00:00:00

The theoretical analysis of deep neural networks (DNN) is arguably among the most challenging research directions in machine learning (ML) right now, as it requires from scientists to lay novel statistical learning foundations to explain their behaviour in practice. While some success has been achieved recently in this endeavour, the question on whether DNNs can be analyzed using the tools from other scientific fields outside the ML community has not received the attention it may well have deserved. In this paper, we explore the interplay between DNNs and game theory (GT), and show how one can benefit from the classic readily available results from the latter when analyzing the former. In particular, we consider the widely studied class of congestion games, and illustrate their intrinsic relatedness to both linear and non-linear DNNs and to the properties of their loss surface. Beyond retrieving the state-of-the-art results from the literature, we argue that our work provides a very promising novel tool for analyzing the DNNs and support this claim by proposing concrete open problems that can advance significantly our understanding of DNNs when solved.
LT3 at SemEval-2020 Task 9: Cross-lingual Embeddings for Sentiment Analysis of Hinglish Social Media Text
Authors: Pranaydeep Singh, Els Lefever
Date: 2020-10-21 00:00:00
Proceeding: SEMEVAL 2020
Tasks: Sentiment Analysis, Word Embeddings

This paper describes our contribution to the SemEval-2020 Task 9 on Sentiment Analysis for Code-mixed Social Media Text. We investigated two approaches to solve the task of Hinglish sentiment analysis. The first approach uses cross-lingual embeddings resulting from projecting Hinglish and pre-trained English FastText word embeddings in the same space. The second approach incorporates pre-trained English embeddings that are incrementally retrained with a set of Hinglish tweets. The results show that the second approach performs best, with an F1-score of 70.52% on the held-out test data.
Token Drop mechanism for Neural Machine Translation
Authors: Huaao Zhang, Shigui Qiu, Xiangyu Duan, Min Zhang
Date: 2020-10-21 00:00:00
Proceeding: COLING 2020 8
Tasks: Machine Translation, NMT, Translation

Neural machine translation with millions of parameters is vulnerable to unfamiliar inputs. We propose Token Drop to improve generalization and avoid overfitting for the NMT model. Similar to word dropout, whereas we replace dropped token with a special token instead of setting zero to words. We further introduce two self-supervised objectives: Replaced Token Detection and Dropped Token Prediction. Our method aims to force model generating target translation with less information, in this way the model can learn textual representation better. Experiments on Chinese-English and English-Romanian benchmark demonstrate the effectiveness of our approach and our model achieves significant improvements over a strong Transformer baseline.
Complex data labeling with deep learning methods: Lessons from fisheries acoustics
Authors: J. M. A. Sarr, T. Brochier, P. Brehmer, Y. Perrot, A. Bah, A. Sarré, M. A. Jeyid, M. Sidibeh, S. El Ayoub
Date: 2020-10-21 00:00:00
Tasks: Deep Learning

Quantitative and qualitative analysis of acoustic backscattered signals from the seabed bottom to the sea surface is used worldwide for fish stocks assessment and marine ecosystem monitoring. Huge amounts of raw data are collected yet require tedious expert labeling. This paper focuses on a case study where the ground truth labels are non-obvious: echograms labeling, which is time-consuming and critical for the quality of fisheries and ecological analysis. We investigate how these tasks can benefit from supervised learning algorithms and demonstrate that convolutional neural networks trained with non-stationary datasets can be used to stress parts of a new dataset needing human expert correction. Further development of this approach paves the way toward a standardization of the labeling process in fisheries acoustics and is a good case study for non-obvious data labeling processes.
What is Wrong with Continual Learning in Medical Image Segmentation?
Authors: Camila Gonzalez, Nick Lemke, Georgios Sakas, Anirban Mukhopadhyay
Date: 2020-10-21 00:00:00
Tasks: Continual Learning, Diagnostic, Hippocampus, image-classification, Image Classification, Image Segmentation, Medical Image Segmentation, Segmentation, Semantic Segmentation

Continual learning protocols are attracting increasing attention from the medical imaging community. In continual environments, datasets acquired under different conditions arrive sequentially; and each is only available for a limited period of time. Given the inherent privacy risks associated with medical data, this setup reflects the reality of deployment for deep learning diagnostic radiology systems. Many techniques exist to learn continuously for image classification, and several have been adapted to semantic segmentation. Yet most struggle to accumulate knowledge in a meaningful manner. Instead, they focus on preventing the problem of catastrophic forgetting, even when this reduces model plasticity and thereon burdens the training process. This puts into question whether the additional overhead of knowledge preservation is worth it - particularly for medical image segmentation, where computation requirements are already high - or if maintaining separate models would be a better solution. We propose UNEG, a simple and widely applicable multi-model benchmark that maintains separate segmentation and autoencoder networks for each training stage. The autoencoder is built from the same architecture as the segmentation network, which in our case is a full-resolution nnU-Net, to bypass any additional design decisions. During inference, the reconstruction error is used to select the most appropriate segmenter for each test image. Open this concept, we develop a fair evaluation scheme for different continual learning settings that moves beyond the prevention of catastrophic forgetting. Our results across three regions of interest (prostate, hippocampus, and right ventricle) show that UNEG outperforms several continual learning methods, reinforcing the need for strong baselines in continual learning research.
Controllable Text Simplification with Explicit Paraphrasing
Authors: Mounica Maddela, Fernando Alva-Manchego, Wei Xu
Date: 2020-10-21 00:00:00
Proceeding: NAACL 2021 4
Tasks: Data Augmentation, Text Simplification

Text Simplification improves the readability of sentences through several rewriting transformations, such as lexical paraphrasing, deletion, and splitting. Current simplification systems are predominantly sequence-to-sequence models that are trained end-to-end to perform all these operations simultaneously. However, such systems limit themselves to mostly deleting words and cannot easily adapt to the requirements of different target audiences. In this paper, we propose a novel hybrid approach that leverages linguistically-motivated rules for splitting and deletion, and couples them with a neural paraphrasing model to produce varied rewriting styles. We introduce a new data augmentation method to improve the paraphrasing capability of our model. Through automatic and manual evaluations, we show that our proposed model establishes a new state-of-the-art for the task, paraphrasing more often than the existing systems, and can control the degree of each simplification operation applied to the input texts.
Unsupervised Multiple Choices Question Answering: Start Learning from Basic Knowledge
Authors: Chi-Liang Liu, Hung-Yi Lee
Date: 2020-10-21 00:00:00
Proceeding: EMNLP (MRQA) 2021 11
Tasks: Question Answering

In this paper, we study the possibility of almost unsupervised Multiple Choices Question Answering (MCQA). Starting from very basic knowledge, MCQA model knows that some choices have higher probabilities of being correct than the others. The information, though very noisy, guides the training of an MCQA model. The proposed method is shown to outperform the baseline approaches on RACE and even comparable with some supervised learning approaches on MC500.
Optimal Off-Policy Evaluation from Multiple Logging Policies
Authors: Nathan Kallus, Yuta Saito, Masatoshi Uehara
Date: 2020-10-21 00:00:00
Tasks: Off-policy evaluation

We study off-policy evaluation (OPE) from multiple logging policies, each generating a dataset of fixed size, i.e., stratified sampling. Previous work noted that in this setting the ordering of the variances of different importance sampling estimators is instance-dependent, which brings up a dilemma as to which importance sampling weights to use. In this paper, we resolve this dilemma by finding the OPE estimator for multiple loggers with minimum variance for any instance, i.e., the efficient one. In particular, we establish the efficiency bound under stratified sampling and propose an estimator achieving this bound when given consistent $q$-estimates. To guard against misspecification of $q$-functions, we also provide a way to choose the control variate in a hypothesis class to minimize variance. Extensive experiments demonstrate the benefits of our methods' efficiently leveraging of the stratified sampling of off-policy data from multiple loggers.
Is Retriever Merely an Approximator of Reader?
Authors: Sohee Yang, Minjoon Seo
Date: 2020-10-21 00:00:00
Tasks: Open-Domain Question Answering, Question Answering

The state of the art in open-domain question answering (QA) relies on an efficient retriever that drastically reduces the search space for the expensive reader. A rather overlooked question in the community is the relationship between the retriever and the reader, and in particular, if the whole purpose of the retriever is just a fast approximation for the reader. Our empirical evidence indicates that the answer is no, and that the reader and the retriever are complementary to each other even in terms of accuracy only. We make a careful conjecture that the architectural constraint of the retriever, which has been originally intended for enabling approximate search, seems to also make the model more robust in large-scale search. We then propose to distill the reader into the retriever so that the retriever absorbs the strength of the reader while keeping its own benefit. Experimental results show that our method can enhance the document recall rate as well as the end-to-end QA accuracy of off-the-shelf retrievers in open-domain QA tasks.
Open-Domain Frame Semantic Parsing Using Transformers
Authors: Aditya Kalyanpur, Or Biran, Tom Breloff, Jennifer Chu-Carroll, Ariel Diertani, Owen Rambow, Mark Sammons
Date: 2020-10-21 00:00:00
Tasks: Decoder, Multi-Task Learning, Semantic Parsing

Frame semantic parsing is a complex problem which includes multiple underlying subtasks. Recent approaches have employed joint learning of subtasks (such as predicate and argument detection), and multi-task learning of related tasks (such as syntactic and semantic parsing). In this paper, we explore multi-task learning of all subtasks with transformer-based models. We show that a purely generative encoder-decoder architecture handily beats the previous state of the art in FrameNet 1.7 parsing, and that a mixed decoding multi-task approach achieves even better performance. Finally, we show that the multi-task model also outperforms recent state of the art systems for PropBank SRL parsing on the CoNLL 2012 benchmark.
GFL: A Decentralized Federated Learning Framework Based On Blockchain
Authors: Yifan Hu, YuHang Zhou, Jun Xiao, Chao Wu
Date: 2020-10-21 00:00:00
Tasks: Data Poisoning, Federated Learning

Federated learning(FL) is a rapidly growing field and many centralized and decentralized FL frameworks have been proposed. However, it is of great challenge for current FL frameworks to improve communication performance and maintain the security and robustness under malicious node attacks. In this paper, we propose Galaxy Federated Learning Framework(GFL), a decentralized FL framework based on blockchain. GFL introduces the consistent hashing algorithm to improve communication performance and proposes a novel ring decentralized FL algorithm(RDFL) to improve decentralized FL performance and bandwidth utilization. In addition, GFL introduces InterPlanetary File System(IPFS) and blockchain to further improve communication efficiency and FL security. Our experiments show that GFL improves communication performance and decentralized FL performance under the data poisoning of malicious nodes and non-independent and identically distributed(Non-IID) datasets.
On Random Subset Generalization Error Bounds and the Stochastic Gradient Langevin Dynamics Algorithm
Authors: Borja Rodríguez-Gálvez, Germán Bassi, Ragnar Thobaben, Mikael Skoglund
Date: 2020-10-21 00:00:00

In this work, we unify several expected generalization error bounds based on random subsets using the framework developed by Hellstr\"om and Durisi [1]. First, we recover the bounds based on the individual sample mutual information from Bu et al. [2] and on a random subset of the dataset from Negrea et al. [3]. Then, we introduce their new, analogous bounds in the randomized subsample setting from Steinke and Zakynthinou [4], and we identify some limitations of the framework. Finally, we extend the bounds from Haghifam et al. [5] for Langevin dynamics to stochastic gradient Langevin dynamics and we refine them for loss functions with potentially large gradient norms.
The Effect of the Rooney Rule on Implicit Bias in the Long Term
Authors: L. Elisa Celis, Chris Hays, Anay Mehrotra, Nisheeth K. Vishnoi
Date: 2020-10-21 00:00:00

A robust body of evidence demonstrates the adverse effects of implicit bias in various contexts--from hiring to health care. The Rooney Rule is an intervention developed to counter implicit bias and has been implemented in the private and public sectors. The Rooney Rule requires that a selection panel include at least one candidate from an underrepresented group in their shortlist of candidates. Recently, Kleinberg and Raghavan proposed a model of implicit bias and studied the effectiveness of the Rooney Rule when applied to a single selection decision. However, selection decisions often occur repeatedly over time. Further, it has been observed that, given consistent counterstereotypical feedback, implicit biases against underrepresented candidates can change. We consider a model of how a selection panel's implicit bias changes over time given their hiring decisions either with or without the Rooney Rule in place. Our main result is that, when the panel is constrained by the Rooney Rule, their implicit bias roughly reduces at a rate that is the inverse of the size of the shortlist--independent of the number of candidates, whereas without the Rooney Rule, the rate is inversely proportional to the number of candidates. Thus, when the number of candidates is much larger than the size of the shortlist, the Rooney Rule enables a faster reduction in implicit bias, providing an additional reason in favor of using it as a strategy to mitigate implicit bias. Towards empirically evaluating the long-term effect of the Rooney Rule in repeated selection decisions, we conduct an iterative candidate selection experiment on Amazon MTurk. We observe that, indeed, decision-makers subject to the Rooney Rule select more minority candidates in addition to those required by the rule itself than they would if no rule is in effect, and do so without considerably decreasing the utility of candidates selected.
Certified Distributional Robustness on Smoothed Classifiers
Authors: Jungang Yang, Liyao Xiang, Ruidong Chen, Yukun Wang, Wei Wang, Xinbing Wang
Date: 2020-10-21 00:00:00

The robustness of deep neural networks (DNNs) against adversarial example attacks has raised wide attention. For smoothed classifiers, we propose the worst-case adversarial loss over input distributions as a robustness certificate. Compared with previous certificates, our certificate better describes the empirical performance of the smoothed classifiers. By exploiting duality and the smoothness property, we provide an easy-to-compute upper bound as a surrogate for the certificate. We adopt a noisy adversarial learning procedure to minimize the surrogate loss to improve model robustness. We show that our training method provides a theoretically tighter bound over the distributional robust base classifiers. Experiments on a variety of datasets further demonstrate superior robustness performance of our method over the state-of-the-art certified or heuristic methods.
Highly-scalable stochastic neuron based on Ovonic Threshold Switch (OTS) and its applications in Restricted Boltzmann Machine (RBM)
Authors: Seong-il Im, Hyejin Lee, Jaesang Lee, Jae-Seung Jeong, Joon Young Kwak, Keunsu Kim, Jeong Ho Cho, Hyunsu Ju, Suyoun Lee
Date: 2020-10-21 00:00:00

Interest in Restricted Boltzmann Machine (RBM) is growing as a generative stochastic artificial neural network to implement a novel energy-efficient machine-learning (ML) technique. For a hardware implementation of the RBM, an essential building block is a reliable stochastic binary neuron device that generates random spikes following the Boltzmann distribution. Here, we propose a highly-scalable stochastic neuron device based on Ovonic Threshold Switch (OTS) which utilizes the random emission and capture process of traps as the source of stochasticity. The switching probability is well described by the Boltzmann distribution, which can be controlled by operating parameters. As a candidate for a true random number generator (TRNG), it passes 15 among the 16 tests of the National Institute of Standards and Technology (NIST) Statistical Test Suite (Special Publication 800-22). In addition, the recognition task of handwritten digits (MNIST) is demonstrated using a simulated RBM network consisting of the proposed device with a maximum recognition accuracy of 86.07 %. Furthermore, reconstruction of images is successfully demonstrated using images contaminated with noises, resulting in images with the noise removed. These results show the promising properties of OTS-based stochastic neuron devices for applications in RBM systems.
Amnesiac Machine Learning
Authors: Laura Graves, Vineel Nagisetty, Vijay Ganesh
Date: 2020-10-21 00:00:00
Tasks: BIG-bench Machine Learning

The Right to be Forgotten is part of the recently enacted General Data Protection Regulation (GDPR) law that affects any data holder that has data on European Union residents. It gives EU residents the ability to request deletion of their personal data, including training records used to train machine learning models. Unfortunately, Deep Neural Network models are vulnerable to information leaking attacks such as model inversion attacks which extract class information from a trained model and membership inference attacks which determine the presence of an example in a model's training data. If a malicious party can mount an attack and learn private information that was meant to be removed, then it implies that the model owner has not properly protected their user's rights and their models may not be compliant with the GDPR law. In this paper, we present two efficient methods that address this question of how a model owner or data holder may delete personal data from models in such a way that they may not be vulnerable to model inversion and membership inference attacks while maintaining model efficacy. We start by presenting a real-world threat model that shows that simply removing training data is insufficient to protect users. We follow that up with two data removal methods, namely Unlearning and Amnesiac Unlearning, that enable model owners to protect themselves against such attacks while being compliant with regulations. We provide extensive empirical analysis that show that these methods are indeed efficient, safe to apply, effectively remove learned information about sensitive data from trained models while maintaining model efficacy.
Synthetic Expressions are Better Than Real for Learning to Detect Facial Actions
Authors: Koichiro Niinuma, Itir Onal Ertugrul, Jeffrey F Cohn, László A Jeni
Date: 2020-10-21 00:00:00
Tasks: Facial expression generation

Critical obstacles in training classifiers to detect facial actions are the limited sizes of annotated video databases and the relatively low frequencies of occurrence of many actions. To address these problems, we propose an approach that makes use of facial expression generation. Our approach reconstructs the 3D shape of the face from each video frame, aligns the 3D mesh to a canonical view, and then trains a GAN-based network to synthesize novel images with facial action units of interest. To evaluate this approach, a deep neural network was trained on two separate datasets: One network was trained on video of synthesized facial expressions generated from FERA17; the other network was trained on unaltered video from the same database. Both networks used the same train and validation partitions and were tested on the test partition of actual video from FERA17. The network trained on synthesized facial expressions outperformed the one trained on actual facial expressions and surpassed current state-of-the-art approaches.
Regularised Least-Squares Regression with Infinite-Dimensional Output Space
Authors: Junhyunng Park, Krikamol Muandet
Date: 2020-10-21 00:00:00
Tasks: Learning Theory, regression

This short technical report presents some learning theory results on vector-valued reproducing kernel Hilbert space (RKHS) regression, where the input space is allowed to be non-compact and the output space is a (possibly infinite-dimensional) Hilbert space. Our approach is based on the integral operator technique using spectral theory for non-compact operators. We place a particular emphasis on obtaining results with as few assumptions as possible; as such we only use Chebyshev's inequality, and no effort is made to obtain the best rates or constants.
Progressive Batching for Efficient Non-linear Least Squares
Authors: Huu Le, Christopher Zach, Edward Rosten, Oliver J. Woodford
Date: 2020-10-21 00:00:00
Tasks: Stochastic Optimization

Non-linear least squares solvers are used across a broad range of offline and real-time model fitting problems. Most improvements of the basic Gauss-Newton algorithm tackle convergence guarantees or leverage the sparsity of the underlying problem structure for computational speedup. With the success of deep learning methods leveraging large datasets, stochastic optimization methods received recently a lot of attention. Our work borrows ideas from both stochastic machine learning and statistics, and we present an approach for non-linear least-squares that guarantees convergence while at the same time significantly reduces the required amount of computation. Empirical results show that our proposed method achieves competitive convergence rates compared to traditional second-order approaches on common computer vision problems, such as image alignment and essential matrix estimation, with very large numbers of residuals.