pwc-archive/papers-with-abstracts huggingface dataset:
Dataset Geography: Mapping Language Data to Language Users
Authors: Fahim Faisal, Yinkai Wang, Antonios Anastasopoulos
Date: 2021-12-07 00:00:00
Proceeding: ACL 2022 5
Tasks: Diversity

As language technologies become more ubiquitous, there are increasing efforts towards expanding the language diversity and coverage of natural language processing (NLP) systems. Arguably, the most important factor influencing the quality of modern NLP systems is data availability. In this work, we study the geographical representativeness of NLP datasets, aiming to quantify if and by how much do NLP datasets match the expected needs of the language speakers. In doing so, we use entity recognition and linking systems, also making important observations about their cross-lingual consistency and giving suggestions for more robust evaluation. Last, we explore some geographical and economic factors that may explain the observed dataset distributions. Code and data are available here: https://github.com/ffaisal93/dataset_geography. Additional visualizations are available here: https://nlp.cs.gmu.edu/project/datasetmaps/.
Improving Neural Cross-Lingual Summarization via Employing Optimal Transport Distance for Knowledge Distillation
Authors: Thong Nguyen, Luu Anh Tuan
Date: 2021-12-07 00:00:00
Tasks: Knowledge Distillation, Multi-Task Learning

Current state-of-the-art cross-lingual summarization models employ multi-task learning paradigm, which works on a shared vocabulary module and relies on the self-attention mechanism to attend among tokens in two languages. However, correlation learned by self-attention is often loose and implicit, inefficient in capturing crucial cross-lingual representations between languages. The matter worsens when performing on languages with separate morphological or structural features, making the cross-lingual alignment more challenging, resulting in the performance drop. To overcome this problem, we propose a novel Knowledge-Distillation-based framework for Cross-Lingual Summarization, seeking to explicitly construct cross-lingual correlation by distilling the knowledge of the monolingual summarization teacher into the cross-lingual summarization student. Since the representations of the teacher and the student lie on two different vector spaces, we further propose a Knowledge Distillation loss using Sinkhorn Divergence, an Optimal-Transport distance, to estimate the discrepancy between those teacher and student representations. Due to the intuitively geometric nature of Sinkhorn Divergence, the student model can productively learn to align its produced cross-lingual hidden states with monolingual hidden states, hence leading to a strong correlation between distant languages. Experiments on cross-lingual summarization datasets in pairs of distant languages demonstrate that our method outperforms state-of-the-art models under both high and low-resourced settings.
Deep Surrogate Assisted MAP-Elites for Automated Hearthstone Deckbuilding
Authors: Yulun Zhang, Matthew C. Fontaine, Amy K. Hoover, Stefanos Nikolaidis
Date: 2021-12-07 00:00:00
Tasks: Diversity

We study the problem of efficiently generating high-quality and diverse content in games. Previous work on automated deckbuilding in Hearthstone shows that the quality diversity algorithm MAP-Elites can generate a collection of high-performing decks with diverse strategic gameplay. However, MAP-Elites requires a large number of expensive evaluations to discover a diverse collection of decks. We propose assisting MAP-Elites with a deep surrogate model trained online to predict game outcomes with respect to candidate decks. MAP-Elites discovers a diverse dataset to improve the surrogate model accuracy, while the surrogate model helps guide MAP-Elites towards promising new content. In a Hearthstone deckbuilding case study, we show that our approach improves the sample efficiency of MAP-Elites and outperforms a model trained offline with random decks, as well as a linear surrogate model baseline, setting a new state-of-the-art for quality diversity approaches in automated Hearthstone deckbuilding. We include the source code for all the experiments at: https://github.com/icaros-usc/EvoStone2.
Machine Learning in the Search for New Fundamental Physics
Authors: Georgia Karagiorgi, Gregor Kasieczka, Scott Kravitz, Benjamin Nachman, David Shih
Date: 2021-12-07 00:00:00
Tasks: BIG-bench Machine Learning

Machine learning plays a crucial role in enhancing and accelerating the search for new fundamental physics. We review the state of machine learning methods and applications for new physics searches in the context of terrestrial high energy physics experiments, including the Large Hadron Collider, rare event searches, and neutrino experiments. While machine learning has a long history in these fields, the deep learning revolution (early 2010s) has yielded a qualitative shift in terms of the scope and ambition of research. These modern machine learning developments are the focus of the present review.
Interpolating between BSDEs and PINNs: deep learning for elliptic and parabolic boundary value problems
Authors: Nikolas Nüsken, Lorenz Richter
Date: 2021-12-07 00:00:00

Solving high-dimensional partial differential equations is a recurrent challenge in economics, science and engineering. In recent years, a great number of computational approaches have been developed, most of them relying on a combination of Monte Carlo sampling and deep learning based approximation. For elliptic and parabolic problems, existing methods can broadly be classified into those resting on reformulations in terms of $\textit{backward stochastic differential equations}$ (BSDEs) and those aiming to minimize a regression-type $L^2$-error ($\textit{physics-informed neural networks}$, PINNs). In this paper, we review the literature and suggest a methodology based on the novel $\textit{diffusion loss}$ that interpolates between BSDEs and PINNs. Our contribution opens the door towards a unified understanding of numerical approaches for high-dimensional PDEs, as well as for implementations that combine the strengths of BSDEs and PINNs. The diffusion loss furthermore bears close similarities to $\textit{(least squares) temporal difference}$ objectives found in reinforcement learning. We also discuss eigenvalue problems and perform extensive numerical studies, including calculations of the ground state for nonlinear Schr\"odinger operators and committor functions relevant in molecular dynamics.
A Bayesian take on option pricing with Gaussian processes
Authors: Martin Tegner, Stephen Roberts
Date: 2021-12-07 00:00:00
Tasks: Bayesian Inference, Gaussian Processes

Local volatility is a versatile option pricing model due to its state dependent diffusion coefficient. Calibration is, however, non-trivial as it involves both proposing a hypothesis model of the latent function and a method for fitting it to data. In this paper we present novel Bayesian inference with Gaussian process priors. We obtain a rich representation of the local volatility function with a probabilistic notion of uncertainty attached to the calibrate. We propose an inference algorithm and apply our approach to S&P 500 market data.
Mesh-Based Solutions for Nonparametric Penalized Regression
Authors: Brayan Ortiz, Noah Simon
Date: 2021-12-07 00:00:00
Tasks: regression

It is often of interest to estimate regression functions non-parametrically. Penalized regression (PR) is one statistically-effective, well-studied solution to this problem. Unfortunately, in many cases, finding exact solutions to PR problems is computationally intractable. In this manuscript, we propose a mesh-based approximate solution (MBS) for those scenarios. MBS transforms the complicated functional minimization of NPR, to a finite parameter, discrete convex minimization; and allows us to leverage the tools of modern convex optimization. We show applications of MBS in a number of explicit examples (including both uni- and multi-variate regression), and explore how the number of parameters must increase with our sample-size in order for MBS to maintain the rate-optimality of NPR. We also give an efficient algorithm to minimize the MBS objective while effectively leveraging the sparsity inherent in MBS.
A Sensitivity Analysis of the MSMARCO Passage Collection
Authors: Joel Mackenzie, Matthias Petri, Alistair Moffat
Date: 2021-12-06 00:00:00
Tasks: Passage Retrieval, Retrieval, Sensitivity

The recent MSMARCO passage retrieval collection has allowed researchers to develop highly tuned retrieval systems. One aspect of this data set that makes it distinctive compared to traditional corpora is that most of the topics only have a single answer passage marked relevant. Here we carry out a "what if" sensitivity study, asking whether a set of systems would still have the same relative performance if more passages per topic were deemed to be "relevant", exploring several mechanisms for identifying sets of passages to be so categorized. Our results show that, in general, while run scores can vary markedly if additional plausible passages are presumed to be relevant, the derived system ordering is relatively insensitive to additional relevance, providing support for the methodology that was used at the time the MSMARCO passage collection was created.
T-DFNN: An Incremental Learning Algorithm for Intrusion Detection Systems
Authors: Mahendra Data, Masayoshi Aritsugi
Date: 2021-11-15 00:00:00
Proceeding: IEEE Access 2021 11
Tasks: Incremental Learning, Intrusion Detection

Machine learning has recently become a popular algorithm in building reliable intrusion detection systems (IDSs). However, most of the models are static and trained using datasets containing all targeted intrusions. If new intrusions emerge, these trained models must be retrained using old and new datasets to classify all intrusions accurately. In real-world situations, new threats continuously appear. Therefore, machine learning algorithms used for IDSs should have the ability to learn incrementally when these new intrusions emerge. To solve this issue, we propose T-DFNN. T-DFNN is an algorithm capable of learning new intrusions incrementally as they emerge. A T-DFNN model is composed of multiple deep feedforward neural network (DFNN) models connected in a tree-like structure. We examined our proposed algorithm using CICIDS2017, an open and widely used network intrusion dataset covering benign traffic and the most common network intrusions. The experimental results showed that the T-DFNN algorithm can incrementally learn new intrusions and reduce the catastrophic forgetting effect. The macro average of the F1-score of the T-DFNN model was over 0.85 for every retraining process. In addition, our proposed T-DFNN model has some advantages in several aspects compared to other models. Compared to the DFNN and Hoeffding tree models trained with a dataset containing only the latest targeted intrusions, our proposed T-DFNN model has higher F1-scores. Moreover, our proposed T-DFNN model has significantly shorter training times than a DFNN model trained using a dataset containing all targeted intrusions. Even though several factors can affect the duration of the training process, the T-DFNN algorithm shows promising results in solving the problem of ever-evolving network intrusion variants.
Small-Footprint Wake Up Word Recognition in Noisy Environments Employing Competing-Words-Based Feature
Authors: Ki-Mu Yoon, Wooil Kim
Date: 2020-12-21 00:00:00
Proceeding: MDPI Electronics 2020 12
Tasks: Data Augmentation, Room Impulse Response (RIR)

This paper proposes a small-footprint wake-up-word (WUW) recognition system for real noisy environments by employing the competing-words-based feature. Competing-words-based features are generated using a ResNet-based deep neural network with small parameters using the competing-words dataset. The competing-words dataset consists of the most acoustically similar and dissimilar words to the WUW used for our system. The obtained features are used as input to the classification network, which is developed using the convolutional neural network (CNN) model. To obtain sufficient data for training, data augmentation is performed by using a room impulse response filter and adding sound signals of various television shows as background noise, which simulates an actual living room environment. The experimental results demonstrate that the proposed WUW recognition system outperforms the baselines that employ CNN and ResNet models. The proposed system shows 1.31% in equal error rate and 1.40% false rejection rate at a 1.0% false alarm rate, which are 29.57% and 50.00% relative improvements compared to the ResNet system, respectively. The number of parameters used for the proposed system is reduced by 83.53% compared to the ResNet system. These results prove that the proposed system with the competing-words-based feature is highly effective at improving WUW recognition performance in noisy environments with a smaller footprint.
Secure learning-based MPC via garbled circuit
Authors: K. Tjell, N. Schlüter, P. Binfet, M. Schulze Darup
Date: 2021-12-07 00:00:00
Tasks: Model Predictive Control

Encrypted control seeks confidential controller evaluation in cloud-based or networked systems. Many existing approaches build on homomorphic encryption (HE) that allow simple mathematical operations to be carried out on encrypted data. Unfortunately, HE is computationally demanding and many control laws (in particular non-polynomial ones) cannot be efficiently implemented with this technology. We show in this paper that secure two-party computation using garbled circuits provides a powerful alternative to HE for encrypted control. More precisely, we present a novel scheme that allows to efficiently implement (non-polynomial) max-out neural networks with one hidden layer in a secure fashion. These networks are of special interest for control since they allow, in principle, to exactly describe piecewise affine control laws resulting from, e.g., linear model predictive control (MPC). However, exact fits require high-dimensional preactivations of the neurons. Fortunately, we illustrate that even low-dimensional learning-based approximations are sufficiently accurate for linear MPC. In addition, these approximations can be securely evaluated using garbled circuit in less than 100~ms for our numerical example. Hence, our approach opens new opportunities for applying encrypted control.
Constrained Resource Allocation Problems in Communications: An Information-assisted Approach
Authors: I. Zakir Ahmed, Hamid Sadjadpour, Shahram Yousefi
Date: 2021-12-07 00:00:00
Tasks: Combinatorial Optimization

We consider a class of resource allocation problems given a set of unconditional constraints whose objective function satisfies Bellman's optimality principle. Such problems are ubiquitous in wireless communication, signal processing, and networking. These constrained combinatorial optimization problems are, in general, NP-Hard. This paper proposes two algorithms to solve this class of problems using a dynamic programming framework assisted by an information-theoretic measure. We demonstrate that the proposed algorithms ensure optimal solutions under carefully chosen conditions and use significantly reduced computational resources. We substantiate our claims by solving the power-constrained bit allocation problem in 5G massive Multiple-Input Multiple-Output receivers using the proposed approach.
Convergence Guarantees for Deep Epsilon Greedy Policy Learning
Authors: Michael Rawson, Radu Balan
Date: 2021-12-02 00:00:00
Tasks: reinforcement-learning, Reinforcement Learning (RL)

Policy learning is a quickly growing area. As robotics and computers control day-to-day life, their error rate needs to be minimized and controlled. There are many policy learning methods and bandit methods with provable error rates that accompany them. We show an error or regret bound and convergence of the Deep Epsilon Greedy method which chooses actions with a neural network's prediction. We also show that Epsilon Greedy method regret upper bound is minimized with cubic root exploration. In experiments with the real-world dataset MNIST, we construct a nonlinear reinforcement learning problem. We witness how with either high or low noise, some methods do and some do not converge which agrees with our proof of convergence.
Is Complexity Important for Philosophy of Mind?
Authors: Kristina Šekrst, Sandro Skansi
Date: 2021-11-02 00:00:00
Tasks: Philosophy

Computational complexity has often been ignored in philosophy of mind, in philosophical artificial intelligence studies. The purpose of this paper is threefold. First and foremost, to show the importance of complexity rather than computability in philosophical and AI problems. Second, to rephrase the notion of computability in terms of solvability, i.e. treating computability as non-sufficient for establishing intelligence. The Church-Turing thesis is therefore revisited and rephrased in order to capture the ontological background of spatial and temporal complexity. Third, to emphasize ontological differences between different time complexities, which seem to provide a solid base towards better understanding of artificial intelligence in general.
Training end-to-end speech-to-text models on mobile phones
Authors: Zitha S, Raghavendra Rao Suresh, Pooja Rao, T. V. Prabhakar
Date: 2021-12-07 00:00:00
Tasks: CPU, Speech-to-Text

Training the state-of-the-art speech-to-text (STT) models in mobile devices is challenging due to its limited resources relative to a server environment. In addition, these models are trained on generic datasets that are not exhaustive in capturing user-specific characteristics. Recently, on-device personalization techniques have been making strides in mitigating the problem. Although many current works have already explored the effectiveness of on-device personalization, the majority of their findings are limited to simulation settings or a specific smartphone. In this paper, we develop and provide a detailed explanation of our framework to train end-to-end models in mobile phones. To make it simple, we considered a model based on connectionist temporal classification (CTC) loss. We evaluated the framework on various mobile phones from different brands and reported the results. We provide enough evidence that fine-tuning the models and choosing the right hyperparameter values is a trade-off between the lowest WER achievable, training time on-device, and memory consumption. Hence, this is vital for a successful deployment of on-device training onto a resource-limited environment like mobile phones. We use training sets from speakers with different accents and record a 7.6% decrease in average word error rate (WER). We also report the associated computational cost measurements with respect to time, memory usage, and cpu utilization in mobile phones in real-time.
Danna-Sep: Unite to separate them all
Authors: Chin-Yun Yu, Kin-Wai Cheuk
Date: 2021-12-07 00:00:00
Tasks: All, Music Source Separation

Deep learning-based music source separation has gained a lot of interest in the last decades. Most of the existing methods operate with either spectrograms or waveforms. Spectrogram based models learn suitable masks for separating magnitude spectrogram into different sources, and waveform-based models directly generate waveforms of individual sources. The two types of models have complementary strengths; the former is superior given harmonic sources such as vocals, while the latter demonstrates better results for percussion and bass instruments. In this work, we improved upon the state-of-the-art (SoTA) models and successfully combined the best of both worlds. The backbones of the proposed framework, dubbed Danna-Sep, are two spectrogram-based models including a modified X-UMX and U-Net, and an enhanced Demucs as the waveform-based model. Given an input of mixture, we linearly combined respective outputs from the three models to obtain the final result. We showed in the experiments that, despite its simplicity, Danna-Sep surpassed the SoTA models by a large margin in terms of Source-to-Distortion Ratio.
Robust Speech Representation Learning via Flow-based Embedding Regularization
Authors: Woo Hyun Kang, Jahangir Alam, Abderrahim Fathan
Date: 2021-12-07 00:00:00
Tasks: Deep Learning, Language Identification, Representation Learning, Speaker Verification, Speech Representation Learning

Over the recent years, various deep learning-based methods were proposed for extracting a fixed-dimensional embedding vector from speech signals. Although the deep learning-based embedding extraction methods have shown good performance in numerous tasks including speaker verification, language identification and anti-spoofing, their performance is limited when it comes to mismatched conditions due to the variability within them unrelated to the main task. In order to alleviate this problem, we propose a novel training strategy that regularizes the embedding network to have minimum information about the nuisance attributes. To achieve this, our proposed method directly incorporates the information bottleneck scheme into the training process, where the mutual information is estimated using the main task classifier and an auxiliary normalizing flow network. The proposed method was evaluated on different speech processing tasks and showed improvement over the standard training strategy in all experimentation.
Learning nonlinear feedforward: a Gaussian Process Approach Applied to a Printer with Friction
Authors: Max van Meer, Maurice Poot, Jim Portegies, Tom Oomen
Date: 2021-12-07 00:00:00
Tasks: Friction

Feedforward control is essential to achieving good tracking performance in positioning systems. The aim of this paper is to develop an identification strategy for inverse models of systems with nonlinear dynamics of unknown structure using input-output data, which directly delivers feedforward signals for a-priori unknown tasks. To this end, inverse systems are regarded as noncausal nonlinear finite impulse response (NFIR) systems and modeled as a Gaussian Process with a stationary kernel function that imposes properties such as smoothness and periodicity. The approach is validated experimentally on a consumer printer with friction and shown to lead to improved tracking performance with respect to linear feedforward.
Data-Driven Controllability Analysis and Stabilization for Linear Descriptor Systems
Authors: Jiabao He, Xuan Zhang, Feng Xu, Junbo Tan, Xueqian Wang
Date: 2021-12-07 00:00:00

For a parameter-unknown linear descriptor system, this paper proposes data-driven methods to testify the system's type and controllability and then to stabilize it. First, a data-based condition is developed to identify whether this unknown system is a descriptor system or is equivalent to a normal system. Furthermore, various controllability concepts are testified by replacing the descriptor system's matrices with data. Finally, a data-based decomposing method is proposed to transfer the nominal system into its slow-fast subsystems' form, so that a state feedback controller for the slow subsystem can be obtained from persistently exciting input and state sequences. Meanwhile, due to the equivalent stabilizability between the nominal system and its slow subsystem, a state feedback controller which stabilizes the nominal system is also obtained. A simulation example is provided to illustrate the effectiveness of those methods.