pwc-archive/papers-with-abstracts huggingface dataset:
Transfer training from smaller language model
Authors: Han Zhang
Date: 2021-04-23 00:00:00
Tasks: Language Modeling, Language Modelling, Large Language Model, model, valid

Large language models have led to state-of-the-art accuracies across a range of tasks. However,training large language model needs massive computing resource, as more and more open source pre-training models are available, it is worthy to study how to take full advantage of available model. We find a method to save training time and resource cost by changing the small well-trained model to large model. We initialize a larger target model from a smaller source model by copy weight values from source model and padding with zeros or small initialization values on it to make the source and target model have approximate outputs, which is valid due to block matrix multiplication and residual connection in transformer structure. We test the target model on several data sets and find it is still comparable with the source model. When we continue training the target model, the training loss can start from a smaller value.
Earnings-21: A Practical Benchmark for ASR in the Wild
Authors: Miguel Del Rio, Natalie Delworth, Ryan Westerman, Michelle Huang, Nishchal Bhandari, Joseph Palakapilly, Quinten McNamara, Joshua Dong, Piotr Zelasko, Miguel Jette
Date: 2021-04-22 00:00:00
Tasks: named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER), NER

Commonly used speech corpora inadequately challenge academic and commercial ASR systems. In particular, speech corpora lack metadata needed for detailed analysis and WER measurement. In response, we present Earnings-21, a 39-hour corpus of earnings calls containing entity-dense speech from nine different financial sectors. This corpus is intended to benchmark ASR systems in the wild with special attention towards named entity recognition. We benchmark four commercial ASR models, two internal models built with open-source tools, and an open-source LibriSpeech model and discuss their differences in performance on Earnings-21. Using our recently released fstalign tool, we provide a candid analysis of each model's recognition capabilities under different partitions. Our analysis finds that ASR accuracy for certain NER categories is poor, presenting a significant impediment to transcript comprehension and usage. Earnings-21 bridges academic and commercial ASR system evaluation and enables further research on entity modeling and WER on real world audio.
Correlated Dynamics in Marketing Sensitivities
Authors: Ryan Dew, Yuhao Fan
Date: 2021-04-23 00:00:00
Tasks: Gaussian Processes, Marketing

Understanding individual customers' sensitivities to prices, promotions, brands, and other marketing mix elements is fundamental to a wide swath of marketing problems. An important but understudied aspect of this problem is the dynamic nature of these sensitivities, which change over time and vary across individuals. Prior work has developed methods for capturing such dynamic heterogeneity within product categories, but neglected the possibility of correlated dynamics across categories. In this work, we introduce a framework to capture such correlated dynamics using a hierarchical dynamic factor model, where individual preference parameters are influenced by common cross-category dynamic latent factors, estimated through Bayesian nonparametric Gaussian processes. We apply our model to grocery purchase data, and find that a surprising degree of dynamic heterogeneity can be accounted for by only a few global trends. We also characterize the patterns in how consumers' sensitivities evolve across categories. Managerially, the proposed framework not only enhances predictive accuracy by leveraging cross-category data, but enables more precise estimation of quantities of interest, like price elasticity.
Transitional Conditional Independence
Authors: Patrick Forré
Date: 2021-04-23 00:00:00

We develope the framework of transitional conditional independence. For this we introduce transition probability spaces and transitional random variables. These constructions will generalize, strengthen and unify previous notions of (conditional) random variables and non-stochastic variables, (extended) stochastic conditional independence and some form of functional conditional independence. Transitional conditional independence is asymmetric in general and it will be shown that it satisfies all desired relevance relations in terms of left and right versions of the separoid rules, except symmetry, on standard, analytic and universal measurable spaces. As a preparation we prove a disintegration theorem for transition probabilities, i.e. the existence and essential uniqueness of (regular) conditional Markov kernels, on those spaces. Transitional conditional independence will be able to express classical statistical concepts like sufficiency, adequacy and ancillarity. As an application, we will then show how transitional conditional independence can be used to prove a directed global Markov property for causal graphical models that allow for non-stochastic input variables in strong generality. This will then also allow us to show the main rules of causal/do-calculus, relating observational and interventional distributions, in such measure theoretic generality.
Learning to reflect: A unifying approach for data-driven stochastic control strategies
Authors: Sören Christensen, Claudia Strauch, Lukas Trottner
Date: 2021-04-23 00:00:00

Stochastic optimal control problems have a long tradition in applied probability, with the questions addressed being of high relevance in a multitude of fields. Even though theoretical solutions are well understood in many scenarios, their practicability suffers from the assumption of known dynamics of the underlying stochastic process, raising the statistical challenge of developing purely data-driven strategies. For the mathematically separated classes of continuous diffusion processes and L\'evy processes, we show that developing efficient strategies for related singular stochastic control problems can essentially be reduced to finding rate-optimal estimators with respect to the sup-norm risk of objects associated to the invariant distribution of ergodic processes which determine the theoretical solution of the control problem. From a statistical perspective, we exploit the exponential $\beta$-mixing property as the common factor of both scenarios to drive the convergence analysis, indicating that relying on general stability properties of Markov processes is a sufficiently powerful and flexible approach to treat complex applications requiring statistical methods. We show moreover that in the L\'evy case $-$ even though per se jump processes are more difficult to handle both in statistics and control theory $-$ a fully data-driven strategy with regret of significantly better order than in the diffusion case can be constructed.
Decentralized Federated Averaging
Authors: Tao Sun, Dongsheng Li, Bao Wang
Date: 2021-04-23 00:00:00

Federated averaging (FedAvg) is a communication efficient algorithm for the distributed training with an enormous number of clients. In FedAvg, clients keep their data locally for privacy protection; a central parameter server is used to communicate between clients. This central server distributes the parameters to each client and collects the updated parameters from clients. FedAvg is mostly studied in centralized fashions, which requires massive communication between server and clients in each communication. Moreover, attacking the central server can break the whole system's privacy. In this paper, we study the decentralized FedAvg with momentum (DFedAvgM), which is implemented on clients that are connected by an undirected graph. In DFedAvgM, all clients perform stochastic gradient descent with momentum and communicate with their neighbors only. To further reduce the communication cost, we also consider the quantized DFedAvgM. We prove convergence of the (quantized) DFedAvgM under trivial assumptions; the convergence rate can be improved when the loss function satisfies the P{\L} property. Finally, we numerically verify the efficacy of DFedAvgM.
A Dimension-Insensitive Algorithm for Stochastic Zeroth-Order Optimization
Authors: Hongcheng Liu, Yu Yang
Date: 2021-04-22 00:00:00

This paper concerns a convex, stochastic zeroth-order optimization (S-ZOO) problem. The objective is to minimize the expectation of a cost function whose gradient is not directly accessible. For this problem, traditional optimization algorithms mostly yield query complexities that grow polynomially with dimensionality (the number of decision variables). Consequently, these methods may not perform well in solving massive-dimensional problems arising in many modern applications. Although more recent methods can be provably dimension-insensitive, almost all of them require arguably more stringent conditions such as everywhere sparse or compressible gradient. In this paper, we propose a sparsity-inducing stochastic gradient-free (SI-SGF) algorithm, which provably yields a dimension-free (up to a logarithmic term) query complexity in both convex and strongly convex cases. Such insensitivity to the dimensionality growth is proven, for the first time, to be achievable when neither gradient sparsity nor gradient compressibility is satisfied. Our numerical results demonstrate a consistency between our theoretical prediction and the empirical performance.
Structuring and presenting data for testing of automotive electronics to reduce effort during decision making
Authors: Martin Tran, Ashish M Husain
Date: 2021-04-23 00:00:00
Tasks: Decision Making

Automotive engineering is recognized as a combination of software and mechanical engineering due to the ever-increasing number of software-based components in vehicles. Since vehicles have become more sophisticated than before to ensure robustness, testing of automotive electronics is performed in high volume, producing immense test-related data. This study investigates how unstructured and decentralized test-related data from testing of automotive electronics creates issues in decision making during the testing and analysis process of test artifacts by performing an exploratory case-study at one of the leading automotive companies, Volvo Cars. From the findings of the exploratory study, a prototype was designed to improve the data and information structure and presentation for test analysis and diagnostics for automotive electronics. The prototype's results showed that providing better data and information structure significantly increases the efficiency and reduces the workload for testers when conducting test analysis and diagnostics. Testers showed a decrease in task load for tasks related to testing due to better information structure, presentation, correctness and accessibility. Hence, the improvements aided the testers to arrive at decisions regarding root cause analysis of failed tests efficiently. The findings of this study can assist automotive companies in systematically investigating and improving the testing process of automotive electronics in regards to managing and structuring test-related data. Keywords: Testing, Automotive Electronics, Electronic Control Unit, ECU, Unstructured Data.
Tablext: A Combined Neural Network And Heuristic Based Table Extractor
Authors: Zach Colter, Morteza Fayazi, Zineb Benameur-El, Serafina Kamp, Shuyan Yu, Ronald Dreslinski
Date: 2021-04-22 00:00:00
Tasks: object-detection, Object Detection, Optical Character Recognition (OCR), Table Extraction

A significant portion of the data available today is found within tables. Therefore, it is necessary to use automated table extraction to obtain thorough results when data-mining. Today's popular state-of-the-art methods for table extraction struggle to adequately extract tables with machine-readable text and structural data. To make matters worse, many tables do not have machine-readable data, such as tables saved as images, making most extraction methods completely ineffective. In order to address these issues, a novel, general format table extractor tool, Tablext, is proposed. This tool uses a combination of computer vision techniques and machine learning methods to efficiently and effectively identify and extract data from tables. Tablext begins by using a custom Convolutional Neural Network (CNN) to identify and separate all potential tables. The identification process is optimized by combining the custom CNN with the YOLO object detection network. Then, the high-level structure of each table is identified with computer vision methods. This high-level, structural meta-data is used by another CNN to identify exact cell locations. As a final step, Optical Characters Recognition (OCR) is performed on every individual cell to extract their content without needing machine-readable text. This multi-stage algorithm allows for the neural networks to focus on completing complex tasks, while letting image processing methods efficiently complete the simpler ones. This leads to the proposed approach to be general-purpose enough to handle a large batch of tables regardless of their internal encodings or their layout complexity. Additionally, it becomes accurate enough to outperform competing state-of-the-art table extractors on the ICDAR 2013 table dataset.
A field guide to cultivating computational biology
Authors: Anne E Carpenter, Casey S Greene, Piero Carnici, Benilton S Carvalho, Michiel de Hoon, Stacey Finley, Kim-Anh Le Cao, Jerry SH Lee, Luigi Marchionni, Suzanne Sindi, Fabian J Theis, Gregory P Way, Jean YH Yang, Elana J Fertig
Date: 2021-04-23 00:00:00

Biomedical research centers can empower basic discovery and novel therapeutic strategies by leveraging their large-scale datasets from experiments and patients. This data, together with new technologies to create and analyze it, has ushered in an era of data-driven discovery which requires moving beyond the traditional individual, single-discipline investigator research model. This interdisciplinary niche is where computational biology thrives. It has matured over the past three decades and made major contributions to scientific knowledge and human health, yet researchers in the field often languish in career advancement, publication, and grant review. We propose solutions for individual scientists, institutions, journal publishers, funding agencies, and educators.
Antenna Efficiency in Massive MIMO Detection
Authors: Ruichen Jiang, Ya-Feng Liu
Date: 2021-04-23 00:00:00

In this paper, we consider the multi-user detection problem in a multiple-input multiple-output (MIMO) system, where the number of receive antennas at the base station (BS) grows infinitely large. We propose a new performance metric, called antenna efficiency, to characterize how fast the vector error probability (VEP) decreases as the number of receive antennas increases in the large system limit. We analyze the optimal maximum-likelihood (ML) detector and the simple zero-forcing (ZF) detector and prove that their antenna efficiency admits a simple closed form, which quantifies the impacts of the user-to-antenna ratio, the signal-to-noise ratio (SNR), and the constellation set on the VEP. Numerical results show that our analysis can well describe the empirical detection error performance in a realistic massive MIMO system.
Deep Multi-Stage CSI Acquisition for Reconfigurable Intelligent Surface Aided MIMO Systems
Authors: Shen Gao, Peihao Dong, Zhiwen Pan, Geoffrey Ye Li
Date: 2021-04-23 00:00:00
Tasks: Deep Learning

This article aims to reduce huge pilot overhead when estimating the reconfigurable intelligent surface (RIS) relayed wireless channel. Motivated by the compelling grasp of deep learning in tackling nonlinear mapping problems, the proposed approach only activates a part of RIS elements and utilizes the corresponding cascaded channel estimate to predict another part. Through a synthetic deep neural network (DNN), the direct channel and active cascaded channel are first estimated sequentially, followed by the channel prediction for the inactive RIS elements. A three-stage training strategy is developed for this synthetic DNN. From simulation results, the proposed deep learning based approach is effective in reducing the pilot overhead and guaranteeing the reliable estimation accuracy.
Federated Double Deep Q-learning for Joint Delay and Energy Minimization in IoT networks
Authors: Sheyda Zarandi, Hina Tabassum
Date: 2021-04-02 00:00:00
Tasks: Deep Reinforcement Learning, Federated Learning, Q-Learning, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

In this paper, we propose a federated deep reinforcement learning framework to solve a multi-objective optimization problem, where we consider minimizing the expected long-term task completion delay and energy consumption of IoT devices. This is done by optimizing offloading decisions, computation resource allocation, and transmit power allocation. Since the formulated problem is a mixed-integer non-linear programming (MINLP), we first cast our problem as a multi-agent distributed deep reinforcement learning (DRL) problem and address it using double deep Q-network (DDQN), where the actions are offloading decisions. The immediate cost of each agent is calculated through solving either the transmit power optimization or local computation resource optimization, based on the selected offloading decisions (actions). Then, to enhance the learning speed of IoT devices (agents), we incorporate federated learning (FDL) at the end of each episode. FDL enhances the scalability of the proposed DRL framework, creates a context for cooperation between agents, and minimizes their privacy concerns. Our numerical results demonstrate the efficacy of our proposed federated DDQN framework in terms of learning speed compared to federated deep Q network (DQN) and non-federated DDQN algorithms. In addition, we investigate the impact of batch size, network layers, DDQN target network update frequency on the learning speed of the FDL.
Output Regulation of Linear Stochastic Systems
Authors: Alberto Mellone, Giordano Scarciotti
Date: 2021-04-23 00:00:00

We address the output regulation problem for a general class of linear stochastic systems. Specifically, we formulate and solve the ideal full-information and output-feedback problems, obtaining perfect, but non-causal, asymptotic regulation. A characterisation of the problem solvability is deduced. We point out that the ideal problems cannot be solved in practice because they unrealistically require that the Brownian motion affecting the system is available for feedback. Drawing from the ideal solution, we formulate and solve approximate versions of the full-information and output-feedback problems, which do not yield perfect asymptotic tracking but can be solved in a realistic scenario. These solutions rely on two key ideas: first we introduce a discrete-time a-posteriori estimator of the variations of the Brownian motion obtained causally by sampling the system state or output; second we introduce a hybrid state observer and a hybrid regulator scheme which employ the estimated Brownian variations. The approximate solution tends to the ideal as the sampling period tends to zero. The proposed theory is validated by the regulation of a circuit subject to electromagnetic noise.
An inexact-penalty method for GNE seeking in games with dynamic agents
Authors: Andrew R. Romano, Lacra Pavel
Date: 2021-04-23 00:00:00

We consider a network of autonomous agents whose outputs are actions in a game with coupled constraints. In such network scenarios, agents seeking to minimize coupled cost functions using distributed information while satisfying the coupled constraints. Current methods consider the small class of multi-integrator agents using primal-dual methods. These methods can only ensure constraint satisfaction in steady-state. In contrast, we propose an inexact penalty method using a barrier function for nonlinear agents with equilibrium-independent passive dynamics. We show that these dynamics converge to an epsilon-GNE while satisfying the constraints for all time, not only in steady-state. We develop these dynamics in both the full-information and partial-information settings. In the partial-information setting, dynamic estimates of the others' actions are used to make decisions and are updated through local communication. Applications to optical networks and velocity synchronization of flexible robots are provided.
Passive soft-reset controllers for nonlinear systems
Authors: Justin H. Le, Andrew R. Teel
Date: 2021-04-23 00:00:00

Soft-reset controllers are introduced as a way to approximate hard-reset controllers. The focus is on implementing reset controllers that are (strictly) passive and on analyzing their interconnection with passive plants. A passive hard-reset controller that has a strongly convex energy function can be approximated as a soft-reset controller. A hard-reset controller is a hybrid system whereas a soft-reset controller corresponds to a differential inclusion, living entirely in the continuous-time domain. This feature may make soft-reset controllers easier to understand and implement. A soft-reset controller contains a parameter that can be adjusted to better approximate the action of the hard-reset controller. Closed-loop asymptotic stability is established for the interconnection of a passive soft-reset controller with a passive plant, under appropriate detectability assumptions. Several examples are used to illustrate the efficacy of soft-reset controllers.
Backup Control Barrier Functions: Formulation and Comparative Study
Authors: Yuxiao Chen, Mrdjan Jankovic, Mario Santillo, Aaron D. Ames
Date: 2021-04-22 00:00:00
Tasks: Math

The backup control barrier function (CBF) was recently proposed as a tractable formulation that guarantees the feasibility of the CBF quadratic programming (QP) via an implicitly defined control invariant set. The control invariant set is based on a fixed backup policy and evaluated online by forward integrating the dynamics under the backup policy. This paper is intended as a tutorial of the backup CBF approach and a comparative study to some benchmarks. First, the backup CBF approach is presented step by step with the underlying math explained in detail. Second, we prove that the backup CBF always has a relative degree 1 under mild assumptions. Third, the backup CBF approach is compared with benchmarks such as Hamilton Jacobi PDE and Sum-of-Squares on the computation of control invariant sets, which shows that one can obtain a control invariant set close to the maximum control invariant set under a good backup policy for many practical problems.
Integrated Framework of Vehicle Dynamics, Instabilities, Energy Models, and Sparse Flow Smoothing Controllers
Authors: Jonathan W. Lee, George Gunter, Rabie Ramadan, Sulaiman Almatrudi, Paige Arnold, John Aquino, William Barbour, Rahul Bhadani, Joy Carpio, Fang-Chieh Chou, Marsalis Gibson, Xiaoqian Gong, Amaury Hayat, Nour Khoudari, Abdul Rahman Kreidieh, Maya Kumar, Nathan Lichtlé, Sean McQuade, Brian Nguyen, Megan Ross, Sydney Truong, Eugene Vinitsky, Yibo Zhao, Jonathan Sprinkle, Benedetto Piccoli, Alexandre M. Bayen, Daniel B. Work, Benjamin Seibold
Date: 2021-04-22 00:00:00

This work presents an integrated framework of: vehicle dynamics models, with a particular attention to instabilities and traffic waves; vehicle energy models, with particular attention to accurate energy values for strongly unsteady driving profiles; and sparse Lagrangian controls via automated vehicles, with a focus on controls that can be executed via existing technology such as adaptive cruise control systems. This framework serves as a key building block in developing control strategies for human-in-the-loop traffic flow smoothing on real highways. In this contribution, we outline the fundamental merits of integrating vehicle dynamics and energy modeling into a single framework, and we demonstrate the energy impact of sparse flow smoothing controllers via simulation results.