pwc-archive/papers-with-abstracts huggingface dataset:
Delayed rejection Hamiltonian Monte Carlo for sampling multiscale distributions
Authors: Chirag Modi, Alex Barnett, Bob Carpenter
Date: 2021-10-01 00:00:00

The efficiency of Hamiltonian Monte Carlo (HMC) can suffer when sampling a distribution with a wide range of length scales, because the small step sizes needed for stability in high-curvature regions are inefficient elsewhere. To address this we present a delayed rejection variant: if an initial HMC trajectory is rejected, we make one or more subsequent proposals each using a step size geometrically smaller than the last. We extend the standard delayed rejection framework by allowing the probability of a retry to depend on the probability of accepting the previous proposal. We test the scheme in several sampling tasks, including multiscale model distributions such as Neal's funnel, and statistical applications. Delayed rejection enables up to five-fold performance gains over optimally-tuned HMC, as measured by effective sample size per gradient evaluation. Even for simpler distributions, delayed rejection provides increased robustness to step size misspecification. Along the way, we provide an accessible but rigorous review of detailed balance for HMC.
Inexact bilevel stochastic gradient methods for constrained and unconstrained lower-level problems
Authors: Tommaso Giovannelli, Griffin Dean Kent, Luis Nunes Vicente
Date: 2021-10-01 00:00:00
Tasks: BIG-bench Machine Learning, Bilevel Optimization, Continual Learning, Neural Architecture Search, Stochastic Optimization

Two-level stochastic optimization formulations have become instrumental in a number of machine learning contexts such as continual learning, neural architecture search, adversarial learning, and hyperparameter tuning. Practical stochastic bilevel optimization problems become challenging in optimization or learning scenarios where the number of variables is high or there are constraints. In this paper, we introduce a bilevel stochastic gradient method for bilevel problems with nonlinear and possibly nonconvex lower-level constraints. We also present a comprehensive convergence theory that addresses both the lower-level unconstrained and constrained cases and covers all inexact calculations of the adjoint gradient (also called hypergradient), such as the inexact solution of the lower-level problem, inexact computation of the adjoint formula (due to the inexact solution of the adjoint equation or use of a truncated Neumann series), and noisy estimates of the gradients, Hessians, and Jacobians involved. To promote the use of bilevel optimization in large-scale learning, we have developed new low-rank practical bilevel stochastic gradient methods (BSG-N-FD and~BSG-1) that do not require second-order derivatives and, in the lower-level unconstrained case, dismiss any matrix-vector products.
STRONG: Synchronous and asynchronous RObust Network localization, under Non-Gaussian noise
Authors: Claudia Soares, João Gomes
Date: 2021-10-01 00:00:00

Real-world network applications must cope with failing nodes, malicious attacks, or nodes facing corrupted data - data classified as outliers. Our work addresses these concerns in the scope of the sensor network localization problem where, despite the abundance of technical literature, prior research seldom considered outlier data. We propose robust, fast, and distributed network localization algorithms, resilient to high-power noise, but also precise under regular Gaussian noise. We use a Huber M-estimator, thus obtaining a robust (but nonconvex) optimization problem. We convexify and change the problem representation, to allow for distributed robust localization algorithms: a synchronous distributed method that has optimal convergence rate and an asynchronous one with proven convergence guarantees. A major highlight of our contribution lies on the fact that we pay no price for provable distributed computation neither in accuracy, nor in communication cost or convergence speed. Simulations showcase the superior performance of our algorithms, both in the presence of outliers and under regular Gaussian noise: our method exceeds the accuracy of alternative approaches, distributed and centralized, even under heavy additive and multiplicative outlier noise.
Classification of Time-Series Data Using Boosted Decision Trees
Authors: Erfan Aasi, Cristian Ioan Vasile, Mahroo Bahreinian, Calin Belta
Date: 2021-10-01 00:00:00
Tasks: Autonomous Driving, Classification, Self-Driving Cars, Time Series, Time Series Analysis

Time-series data classification is central to the analysis and control of autonomous systems, such as robots and self-driving cars. Temporal logic-based learning algorithms have been proposed recently as classifiers of such data. However, current frameworks are either inaccurate for real-world applications, such as autonomous driving, or they generate long and complicated formulae that lack interpretability. To address these limitations, we introduce a novel learning method, called Boosted Concise Decision Trees (BCDTs), to generate binary classifiers that are represented as Signal Temporal Logic (STL) formulae. Our algorithm leverages an ensemble of Concise Decision Trees (CDTs) to improve the classification performance, where each CDT is a decision tree that is empowered by a set of techniques to generate simpler formulae and improve interpretability. The effectiveness and classification performance of our algorithm are evaluated on naval surveillance and urban-driving case studies.
Label Propagation Through Optimal Transport
Authors: Mourad El Hamri, Younès Bennani, Issam Falih
Date: 2021-10-01 00:00:00

In this paper, we tackle the transductive semi-supervised learning problem that aims to obtain label predictions for the given unlabeled data points according to Vapnik's principle. Our proposed approach is based on optimal transport, a mathematical theory that has been successfully used to address various machine learning problems, and is starting to attract renewed interest in semi-supervised learning community. The proposed approach, Optimal Transport Propagation (OTP), performs in an incremental process, label propagation through the edges of a complete bipartite edge-weighted graph, whose affinity matrix is constructed from the optimal transport plan between empirical measures defined on labeled and unlabeled data. OTP ensures a high degree of predictions certitude by controlling the propagation process using a certainty score based on Shannon's entropy. We also provide a convergence analysis of our algorithm. Experiments task show the superiority of the proposed approach over the state-of-the-art. We make our code publicly available.
Extended dynamic mode decomposition with dictionary learning using neural ordinary differential equations
Authors: Hiroaki Terao, Sho Shirasaka, Hideyuki Suzuki
Date: 2021-10-01 00:00:00
Tasks: Dictionary Learning

Nonlinear phenomena can be analyzed via linear techniques using operator-theoretic approaches. Data-driven method called the extended dynamic mode decomposition (EDMD) and its variants, which approximate the Koopman operator associated with the nonlinear phenomena, have been rapidly developing by incorporating machine learning methods. Neural ordinary differential equations (NODEs), which are a neural network equipped with a continuum of layers, and have high parameter and memory efficiencies, have been proposed. In this paper, we propose an algorithm to perform EDMD using NODEs. NODEs are used to find a parameter-efficient dictionary which provides a good finite-dimensional approximation of the Koopman operator. We show the superiority of the parameter efficiency of the proposed method through numerical experiments.
Artificial Neural Network and its Application Research Progress in Distillation
Authors: Jing Sun, Qi Tang
Date: 2021-10-01 00:00:00
Tasks: Self-Learning

Artificial neural networks learn various rules and algorithms to form different ways of processing information, and have been widely used in various chemical processes. Among them, with the development of rectification technology, its production scale continues to expand, and its calculation requirements are also more stringent, because the artificial neural network has the advantages of self-learning, associative storage and high-speed search for optimized solutions, it can make high-precision simulation predictions for rectification operations, so it is widely used in the chemical field of rectification. This article gives a basic overview of artificial neural networks, and introduces the application research of artificial neural networks in distillation at home and abroad.
Machine Learning with Knowledge Constraints for Process Optimization of Open-Air Perovskite Solar Cell Manufacturing
Authors: Zhe Liu, Nicholas Rolston, Austin C. Flick, Thomas W. Colburn, Zekun Ren, Reinhold H. Dauskardt, Tonio Buonassisi
Date: 2021-10-01 00:00:00
Tasks: Bayesian Optimization, Benchmarking, Transfer Learning

Perovskite photovoltaics (PV) have achieved rapid development in the past decade in terms of power conversion efficiency of small-area lab-scale devices; however, successful commercialization still requires further development of low-cost, scalable, and high-throughput manufacturing techniques. One of the critical challenges of developing a new fabrication technique is the high-dimensional parameter space for optimization, but machine learning (ML) can readily be used to accelerate perovskite PV scaling. Herein, we present an ML-guided framework of sequential learning for manufacturing process optimization. We apply our methodology to the Rapid Spray Plasma Processing (RSPP) technique for perovskite thin films in ambient conditions. With a limited experimental budget of screening 100 process conditions, we demonstrated an efficiency improvement to 18.5% as the best-in-our-lab device fabricated by RSPP, and we also experimentally found 10 unique process conditions to produce the top-performing devices of more than 17% efficiency, which is 5 times higher rate of success than the control experiments with pseudo-random Latin hypercube sampling. Our model is enabled by three innovations: (a) flexible knowledge transfer between experimental processes by incorporating data from prior experimental data as a probabilistic constraint; (b) incorporation of both subjective human observations and ML insights when selecting next experiments; (c) adaptive strategy of locating the region of interest using Bayesian optimization first, and then conducting local exploration for high-efficiency devices. Furthermore, in virtual benchmarking, our framework achieves faster improvements with limited experimental budgets than traditional design-of-experiments methods (e.g., one-variable-at-a-time sampling).
Neural Network Verification in Control
Authors: Michael Everett
Date: 2021-09-30 00:00:00
Tasks: Deep Reinforcement Learning, Reinforcement Learning (RL)

Learning-based methods could provide solutions to many of the long-standing challenges in control. However, the neural networks (NNs) commonly used in modern learning approaches present substantial challenges for analyzing the resulting control systems' safety properties. Fortunately, a new body of literature could provide tractable methods for analysis and verification of these high dimensional, highly nonlinear representations. This tutorial first introduces and unifies recent techniques (many of which originated in the computer vision and machine learning communities) for verifying robustness properties of NNs. The techniques are then extended to provide formal guarantees of neural feedback loops (e.g., closed-loop system with NN control policy). The provided tools are shown to enable closed-loop reachability analysis and robust deep reinforcement learning.
Paradigm Shift Through the Integration of Physical Methodology and Data Science
Authors: Takashi Miyamoto
Date: 2021-09-30 00:00:00

Data science methodologies, which have undergone significant developments recently, provide flexible representational performance and fast computational means to address the challenges faced by traditional scientific methodologies while revealing unprecedented challenges such as the interpretability of computations and the demand for extrapolative predictions on the amount of data. Methods that integrate traditional physical and data science methodologies are new methods of mathematical analysis that complement both methodologies and are being studied in various scientific fields. This paper highlights the significance and importance of such integrated methods from the viewpoint of scientific theory. Additionally, a comprehensive survey of specific methods and applications are conducted, and the current state of the art in relevant research fields are summarized.
Mulberry Leaf Yield Prediction Using Machine Learning Techniques
Authors: Srikantaiah K C, Deeksha A
Date: 2021-09-30 00:00:00
Tasks: BIG-bench Machine Learning, Prediction, regression

Soil nutrients are essential for the growth of healthy crops. India produces a humungous quantity of Mulberry leaves which in turn produces the raw silk. Since the climatic conditions in India is favourable, Mulberry is grown throughout the year. Majority of the farmers hardly pay attention to the nature of soil and abiotic factors due to which leaves become malnutritious and thus when they are consumed by the silkworm, desired quality end-product, raw silk, will not be produced. It is beneficial for the farmers to know the amount of yield that their land can produce so that they can plan in advance. In this paper, different Machine Learning techniques are used in predicting the yield of the Mulberry crops based on the soil parameters. Three advanced machine-learning models are selected and compared, namely, Multiple linear regression, Ridge regression and Random Forest Regression (RF). The experimental results show that Random Forest Regression outperforms other algorithms.
Prediction of IPL Match Outcome Using Machine Learning Techniques
Authors: Srikantaiah K C, Aryan Khetan, Baibhav Kumar, Divy Tolani, Harshal Patel
Date: 2021-09-30 00:00:00
Tasks: BIG-bench Machine Learning, Prediction

India's most popular sport is cricket and is played across all over the nation in different formats like T20, ODI, and Test. The Indian Premier League (IPL) is a national cricket match where players are drawn from regional teams of India, National Team and also from international team. Many factors like live streaming, radio, TV broadcast made this league as popular among cricket fans. The prediction of the outcome of the IPL matches is very important for online traders and sponsors. We can predict the match between two teams based on various factors like team composition, batting and bowling averages of each player in the team, and the team's success in their previous matches, in addition to traditional factors such as toss, venue, and day-night, the probability of winning by batting first at a specified match venue against a specific team. In this paper, we have proposed a model for predicting outcome of the IPL matches using Machine learning Algorithms namely SVM, Random Forest Classifier (RFC), Logistic Regression and K-Nearest Neighbor. Experimental results showed that the Random Forest algorithm outperforms other algorithms with an accuracy of 88.10%.
Encoder Adaptation of Dense Passage Retrieval for Open-Domain Question Answering
Authors: Minghan Li, Jimmy Lin
Date: 2021-10-04 00:00:00
Tasks: Domain Adaptation, Open-Domain Question Answering, Passage Retrieval, Question Answering, Retrieval

One key feature of dense passage retrievers (DPR) is the use of separate question and passage encoder in a bi-encoder design. Previous work on generalization of DPR mainly focus on testing both encoders in tandem on out-of-distribution (OOD) question-answering (QA) tasks, which is also known as domain adaptation. However, it is still unknown how DPR's individual question/passage encoder affects generalization. Specifically, in this paper, we want to know how an in-distribution (IND) question/passage encoder would generalize if paired with an OOD passage/question encoder from another domain. We refer to this challenge as \textit{encoder adaptation}. To answer this question, we inspect different combinations of DPR's question and passage encoder learned from five benchmark QA datasets on both in-domain and out-of-domain questions. We find that the passage encoder has more influence on the lower bound of generalization while the question encoder seems to affect the upper bound in general. For example, applying an OOD passage encoder usually hurts the retrieval accuracy while an OOD question encoder sometimes even improves the accuracy.
Towards Theme Detection in Personal Finance Questions
Authors: John Xi Qiu, Adam Faulkner, Aysu Ezen Can
Date: 2021-10-04 00:00:00
Tasks: Semantic Parsing, Sentence

Banking call centers receive millions of calls annually, with much of the information in these calls unavailable to analysts interested in tracking new and emerging call center trends. In this study we present an approach to call center theme detection that captures the occurrence of multiple themes in a question, using a publicly available corpus of StackExchange personal finance questions, labeled by users with topic tags, as a testbed. To capture the occurrence of multiple themes in a single question, the approach encodes and clusters at the sentence- rather than question-level. We also present a comparison of state-of-the-art sentence encoding models, including the SBERT family of sentence encoders. We frame our evaluation as a multiclass classification task and show that a simple combination of the original sentence text, Universal Sentence Encoder, and KMeans outperforms more sophisticated techniques that involve semantic parsing, SBERT-family models, and HDBSCAN. Our highest performing approach achieves a Micro-F1 of 0.46 for this task and we show that the resulting clusters, even when slightly noisy, contain sentences that are topically consistent with the label associated with the cluster.
A Proposed Conceptual Framework for a Representational Approach to Information Retrieval
Authors: Jimmy Lin
Date: 2021-10-04 00:00:00
Tasks: Information Retrieval, Retrieval, Sentence, Sentence Similarity, Text Retrieval

This paper outlines a conceptual framework for understanding recent developments in information retrieval and natural language processing that attempts to integrate dense and sparse retrieval methods. I propose a representational approach that breaks the core text retrieval problem into a logical scoring model and a physical retrieval model. The scoring model is defined in terms of encoders, which map queries and documents into a representational space, and a comparison function that computes query-document scores. The physical retrieval model defines how a system produces the top-$k$ scoring documents from an arbitrarily large corpus with respect to a query. The scoring model can be further analyzed along two dimensions: dense vs. sparse representations and supervised (learned) vs. unsupervised approaches. I show that many recently proposed retrieval methods, including multi-stage ranking designs, can be seen as different parameterizations in this framework, and that a unified view suggests a number of open research questions, providing a roadmap for future work. As a bonus, this conceptual framework establishes connections to sentence similarity tasks in natural language processing and information access "technologies" prior to the dawn of computing.
Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristics
Authors: Prajjwal Bhargava, Aleksandr Drozd, Anna Rogers
Date: 2021-10-04 00:00:00
Proceeding: EMNLP (insights) 2021 11

Much of recent progress in NLU was shown to be due to models' learning dataset-specific heuristics. We conduct a case study of generalization in NLI (from MNLI to the adversarially constructed HANS dataset) in a range of BERT-based architectures (adapters, Siamese Transformers, HEX debiasing), as well as with subsampling the data and increasing the model size. We report 2 successful and 3 unsuccessful strategies, all providing insights into how Transformer-based models learn to generalize.
JuriBERT: A Masked-Language Model Adaptation for French Legal Text
Authors: Stella Douka, Hadi Abdine, Michalis Vazirgiannis, Rajaa El Hamdani, David Restrepo Amariles
Date: 2021-10-04 00:00:00
Proceeding: EMNLP (NLLP) 2021 11
Tasks: Language Modeling, Language Modelling

Language models have proven to be very useful when adapted to specific domains. Nonetheless, little research has been done on the adaptation of domain-specific BERT models in the French language. In this paper, we focus on creating a language model adapted to French legal text with the goal of helping law professionals. We conclude that some specific tasks do not benefit from generic language models pre-trained on large amounts of data. We explore the use of smaller architectures in domain-specific sub-languages and their benefits for French legal text. We prove that domain-specific pre-trained models can perform better than their equivalent generalised ones in the legal domain. Finally, we release JuriBERT, a new set of BERT models adapted to the French legal domain.
Building a Noisy Audio Dataset to Evaluate Machine Learning Approaches for Automatic Speech Recognition Systems
Authors: Julio Cesar Duarte, Sérgio Colcher
Date: 2021-10-04 00:00:00
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), Data Augmentation, speech-recognition, Speech Recognition

Automatic speech recognition systems are part of people's daily lives, embedded in personal assistants and mobile phones, helping as a facilitator for human-machine interaction while allowing access to information in a practically intuitive way. Such systems are usually implemented using machine learning techniques, especially with deep neural networks. Even with its high performance in the task of transcribing text from speech, few works address the issue of its recognition in noisy environments and, usually, the datasets used do not contain noisy audio examples, while only mitigating this issue using data augmentation techniques. This work aims to present the process of building a dataset of noisy audios, in a specific case of degenerated audios due to interference, commonly present in radio transmissions. Additionally, we present initial results of a classifier that uses such data for evaluation, indicating the benefits of using this dataset in the recognizer's training process. Such recognizer achieves an average result of 0.4116 in terms of character error rate in the noisy set (SNR = 30).
Protagonists' Tagger in Literary Domain -- New Datasets and a Method for Person Entity Linkage
Authors: Weronika Łajewska, Anna Wróblewska
Date: 2021-10-04 00:00:00
Tasks: Entity Disambiguation, named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER), NER, Text Matching

Semantic annotation of long texts, such as novels, remains an open challenge in Natural Language Processing (NLP). This research investigates the problem of detecting person entities and assigning them unique identities, i.e., recognizing people (especially main characters) in novels. We prepared a method for person entity linkage (named entity recognition and disambiguation) and new testing datasets. The datasets comprise 1,300 sentences from 13 classic novels of different genres that a novel reader had manually annotated. Our process of identifying literary characters in a text, implemented in protagonistTagger, comprises two stages: (1) named entity recognition (NER) of persons, (2) named entity disambiguation (NED) - matching each recognized person with the literary character's full name, based on approximate text matching. The protagonistTagger achieves both precision and recall of above 83% on the prepared testing sets. Finally, we gathered a corpus of 13 full-text novels tagged with protagonistTagger that comprises more than 35,000 mentions of literary characters.