pwc-archive/papers-with-abstracts huggingface dataset:
Reducing the Long Tail Losses in Scientific Emulations with Active Learning
Authors: Yi Heng Lim, Muhammad Firmansyah Kasim
Date: 2021-11-15 00:00:00
Tasks: Active Learning, Deep Learning

Deep-learning-based models are increasingly used to emulate scientific simulations to accelerate scientific research. However, accurate, supervised deep learning models require huge amount of labelled data, and that often becomes the bottleneck in employing neural networks. In this work, we leveraged an active learning approach called core-set selection to actively select data, per a pre-defined budget, to be labelled for training. To further improve the model performance and reduce the training costs, we also warm started the training using a shrink-and-perturb trick. We tested on two case studies in different fields, namely galaxy halo occupation distribution modelling in astrophysics and x-ray emission spectroscopy in plasma physics, and the results are promising: we achieved competitive overall performance compared to using a random sampling baseline, and more importantly, successfully reduced the larger absolute losses, i.e. the long tail in the loss distribution, at virtually no overhead costs.
Towards Lightweight Controllable Audio Synthesis with Conditional Implicit Neural Representations
Authors: Jan Zuiderveld, Marco Federici, Erik J. Bekkers
Date: 2021-11-14 00:00:00
Tasks: Audio Synthesis

The high temporal resolution of audio and our perceptual sensitivity to small irregularities in waveforms make synthesizing at high sampling rates a complex and computationally intensive task, prohibiting real-time, controllable synthesis within many approaches. In this work we aim to shed light on the potential of Conditional Implicit Neural Representations (CINRs) as lightweight backbones in generative frameworks for audio synthesis. Our experiments show that small Periodic Conditional INRs (PCINRs) learn faster and generally produce quantitatively better audio reconstructions than Transposed Convolutional Neural Networks with equal parameter counts. However, their performance is very sensitive to activation scaling hyperparameters. When learning to represent more uniform sets, PCINRs tend to introduce artificial high-frequency components in reconstructions. We validate this noise can be minimized by applying standard weight regularization during training or decreasing the compositional depth of PCINRs, and suggest directions for future research.
Uncertainty quantification and inverse modeling for subsurface flow in 3D heterogeneous formations using a theory-guided convolutional encoder-decoder network
Authors: Rui Xu, Dongxiao Zhang, Nanzhe Wang
Date: 2021-11-14 00:00:00
Tasks: Computational Efficiency, Decoder, Image-to-Image Regression, Uncertainty Quantification

We build surrogate models for dynamic 3D subsurface single-phase flow problems with multiple vertical producing wells. The surrogate model provides efficient pressure estimation of the entire formation at any timestep given a stochastic permeability field, arbitrary well locations and penetration lengths, and a timestep matrix as inputs. The well production rate or bottom hole pressure can then be determined based on Peaceman's formula. The original surrogate modeling task is transformed into an image-to-image regression problem using a convolutional encoder-decoder neural network architecture. The residual of the governing flow equation in its discretized form is incorporated into the loss function to impose theoretical guidance on the model training process. As a result, the accuracy and generalization ability of the trained surrogate models are significantly improved compared to fully data-driven models. They are also shown to have flexible extrapolation ability to permeability fields with different statistics. The surrogate models are used to conduct uncertainty quantification considering a stochastic permeability field, as well as to infer unknown permeability information based on limited well production data and observation data of formation properties. Results are shown to be in good agreement with traditional numerical simulation tools, but computational efficiency is dramatically improved.
An Energy Consumption Model for Electrical Vehicle Networks via Extended Federated-learning
Authors: Shiliang Zhang
Date: 2021-11-13 00:00:00
Tasks: Anomaly Detection, Federated Learning

Electrical vehicle (EV) raises to promote an eco-sustainable society. Nevertheless, the "range anxiety" of EV hinders its wider acceptance among customers. This paper proposes a novel solution to range anxiety based on a federated-learning model, which is capable of estimating battery consumption and providing energy-efficient route planning for vehicle networks. Specifically, the new approach extends the federated-learning structure with two components: anomaly detection and sharing policy. The first component identifies preventing factors in model learning, while the second component offers guidelines for information sharing amongst vehicle networks when the sharing is necessary to preserve learning efficiency. The two components collaborate to enhance learning robustness against data heterogeneities in networks. Numerical experiments are conducted, and the results show that compared with considered solutions, the proposed approach could provide higher accuracy of battery-consumption estimation for vehicles under heterogeneous data distributions, without increasing the time complexity or transmitting raw data among vehicle networks.
Multiset Signal Processing and Electronics
Authors: Luciano da F. Costa
Date: 2021-11-13 00:00:00

Multisets are an intuitive extension of the traditional concept of sets that allow repetition of elements, with the number of times each element appears being understood as the respective multiplicity. Recent generalizations of multisets to real-valued functions, accounting for possibly negative values, have paved the way to a number of interesting implications and applications, including respective implementations as electronic systems. The basic multiset operations include the set complementation (sign change), intersection (minimum between two values), union (maximum between two values), difference and sum (identical to the algebraic counterparts). When applied to functions or signals, the sign and conjoint sign functions are also required. Given that signals are functions, it becomes possible to effectively translate the multiset and multifunction operations to analog electronics, which is the objective of the present work. It is proposed that effective multiset operations capable of high performance self and cross-correlation can be obtained with relative simplicity in either discrete or integrated circuits. The problem of switching noise is also briefly discussed. The present results have great potential for applications and related developments in analog and digital electronics, as well as for pattern recognition, signal processing, and deep learning.
Multiset Neurons
Authors: Luciano da F. Costa
Date: 2021-11-13 00:00:00
Tasks: Image Segmentation, Semantic Segmentation

The present work reports a comparative performance of artificial neurons obtained in terms of the real-valued Jaccard and coincidence similarity indices and respectively derived functionals. The interiority index and classic cross-correlation are also included for comparison purposes. After presenting the basic concepts related to real-valued multisets and the adopted similarity metrics, including the generalization of the real-valued Jaccard and coincidence indices to higher orders, we proceed to studying the response of a single neuron, not taking into account the output non-linearity (e.g.~sigmoid), respectively to the detection of gaussian two-dimensional stimulus in presence of displacement, magnification, intensity variation, noise and interference from additional patterns. It is shown that the real-valued Jaccard and coincidence approaches are substantially more robust and effective than the interiority index and the classic cross-correlation. The coincidence-based neurons are shown to have the best overall performance respectively to the considered type of data and perturbations. The potential of the multiset neurons is further illustrated with respect to the challenging problem of image segmentation, leading to impressive cost/benefit performance. The reported concepts, methods, and results, have substantial implications not only for pattern recognition and machine learning, but also regarding neurobiology and neuroscience.
Spatial machine-learning model diagnostics: a model-agnostic distance-based approach
Authors: Alexander Brenning
Date: 2021-11-13 00:00:00
Tasks: BIG-bench Machine Learning, Diagnostic, Land Cover Classification, model, Prediction

While significant progress has been made towards explaining black-box machine-learning (ML) models, there is still a distinct lack of diagnostic tools that elucidate the spatial behaviour of ML models in terms of predictive skill and variable importance. This contribution proposes spatial prediction error profiles (SPEPs) and spatial variable importance profiles (SVIPs) as novel model-agnostic assessment and interpretation tools for spatial prediction models with a focus on prediction distance. Their suitability is demonstrated in two case studies representing a regionalization task in an environmental-science context, and a classification task from remotely-sensed land cover classification. In these case studies, the SPEPs and SVIPs of geostatistical methods, linear models, random forest, and hybrid algorithms show striking differences but also relevant similarities. Limitations of related cross-validation techniques are outlined, and the case is made that modelers should focus their model assessment and interpretation on the intended spatial prediction horizon. The range of autocorrelation, in contrast, is not a suitable criterion for defining spatial cross-validation test sets. The novel diagnostic tools enrich the toolkit of spatial data science, and may improve ML model interpretation, selection, and design.
A Shallow U-Net Architecture for Reliably Predicting Blood Pressure (BP) from Photoplethysmogram (PPG) and Electrocardiogram (ECG) Signals
Authors: Sakib Mahmud, Nabil Ibtehaz, Amith Khandakar, Anas Tahir, Tawsifur Rahman, Khandaker Reajul Islam, Md Shafayet Hossain, M. Sohel Rahman, Mohammad Tariqul Islam, Muhammad E. H. Chowdhury
Date: 2021-11-12 00:00:00

Cardiovascular diseases are the most common causes of death around the world. To detect and treat heart-related diseases, continuous Blood Pressure (BP) monitoring along with many other parameters are required. Several invasive and non-invasive methods have been developed for this purpose. Most existing methods used in the hospitals for continuous monitoring of BP are invasive. On the contrary, cuff-based BP monitoring methods, which can predict Systolic Blood Pressure (SBP) and Diastolic Blood Pressure (DBP), cannot be used for continuous monitoring. Several studies attempted to predict BP from non-invasively collectible signals such as Photoplethysmogram (PPG) and Electrocardiogram (ECG), which can be used for continuous monitoring. In this study, we explored the applicability of autoencoders in predicting BP from PPG and ECG signals. The investigation was carried out on 12,000 instances of 942 patients of the MIMIC-II dataset and it was found that a very shallow, one-dimensional autoencoder can extract the relevant features to predict the SBP and DBP with the state-of-the-art performance on a very large dataset. Independent test set from a portion of the MIMIC-II dataset provides an MAE of 2.333 and 0.713 for SBP and DBP, respectively. On an external dataset of forty subjects, the model trained on the MIMIC-II dataset, provides an MAE of 2.728 and 1.166 for SBP and DBP, respectively. For both the cases, the results met British Hypertension Society (BHS) Grade A and surpassed the studies from the current literature.
PySINDy: A comprehensive Python package for robust sparse system identification
Authors: Alan A. Kaptanoglu, Brian M. de Silva, Urban Fasel, Kadierdan Kaheman, Andy J. Goldschmidt, Jared L. Callaham, Charles B. Delahunt, Zachary G. Nicolaou, Kathleen Champion, Jean-Christophe Loiseau, J. Nathan Kutz, Steven L. Brunton
Date: 2021-11-12 00:00:00
Tasks: Model Discovery, regression

Automated data-driven modeling, the process of directly discovering the governing equations of a system from data, is increasingly being used across the scientific community. PySINDy is a Python package that provides tools for applying the sparse identification of nonlinear dynamics (SINDy) approach to data-driven model discovery. In this major update to PySINDy, we implement several advanced features that enable the discovery of more general differential equations from noisy and limited data. The library of candidate terms is extended for the identification of actuated systems, partial differential equations (PDEs), and implicit differential equations. Robust formulations, including the integral form of SINDy and ensembling techniques, are also implemented to improve performance for real-world data. Finally, we provide a range of new optimization algorithms, including several sparse regression techniques and algorithms to enforce and promote inequality constraints and stability. Together, these updates enable entirely new SINDy model discovery capabilities that have not been reported in the literature, such as constrained PDE identification and ensembling with different sparse regression optimizers.
Document AI: Benchmarks, Models and Applications
Authors: Lei Cui, Yiheng Xu, Tengchao Lv, Furu Wei
Date: 2021-11-16 00:00:00
Tasks: Deep Learning, Document AI, document-image-classification, Document Image Classification, Document Layout Analysis, image-classification, Image Classification, Question Answering, Visual Question Answering, Visual Question Answering (VQA)

Document AI, or Document Intelligence, is a relatively new research topic that refers to the techniques for automatically reading, understanding, and analyzing business documents. It is an important research direction for natural language processing and computer vision. In recent years, the popularity of deep learning technology has greatly advanced the development of Document AI, such as document layout analysis, visual information extraction, document visual question answering, document image classification, etc. This paper briefly reviews some of the representative models, tasks, and benchmark datasets. Furthermore, we also introduce early-stage heuristic rule-based document analysis, statistical machine learning algorithms, and deep learning approaches especially pre-training methods. Finally, we look into future directions for Document AI research.
Coral: An Approach for Conversational Agents in Mental Health Applications
Authors: Harsh Sakhrani, Saloni Parekh, Shubham Mahajan
Date: 2021-11-16 00:00:00
Tasks: Chatbot

It may be difficult for some individuals to open up and share their thoughts and feelings in front of a mental health expert. For those who are more at ease with a virtual agent, conversational agents can serve as an intermediate step in the right direction. The conversational agent must therefore be empathetic and able to conduct free-flowing conversations. To this effect, we present an approach for creating a generative empathetic open-domain chatbot that can be used for mental health applications. We leverage large scale pre-training and empathetic conversational data to make the responses more empathetic in nature and a multi-turn dialogue arrangement to maintain context. Our models achieve state-of-the-art results on the Empathetic Dialogues test set.
The role of attraction-repulsion dynamics in simulating the emergence of inflectional class systems
Authors: Erich R. Round, Sacha Beniamine, Louise Esher
Date: 2021-11-16 00:00:00
Tasks: Diversity

Dynamic models of paradigm change can elucidate how the simplest of processes may lead to unexpected outcomes, and thereby can reveal new potential explanations for observed linguistic phenomena. Ackerman & Malouf (2015) present a model in which inflectional systems reduce in disorder through the action of an attraction-only dynamic, in which lexemes only ever grow more similar to one another over time. Here we emphasise that: (1) Attraction-only models cannot evolve the structured diversity which characterises true inflectional systems, because they inevitably remove all variation; and (2) Models with both attraction and repulsion enable the emergence of systems that are strikingly reminiscent of morphomic structure such as inflection classes. Thus, just one small ingredient -- change based on dissimilarity -- separates models that tend inexorably to uniformity, and which therefore are implausible for inflectional morphology, from those which evolve stable, morphome-like structure. These models have the potential to alter how we attempt to account for morphological complexity.
Multi-Vector Models with Textual Guidance for Fine-Grained Scientific Document Similarity
Authors: Sheshera Mysore, Arman Cohan, Tom Hope
Date: 2021-11-16 00:00:00
Proceeding: NAACL 2022 7
Tasks: Sentence

We present a new scientific document similarity model based on matching fine-grained aspects of texts. To train our model, we exploit a naturally-occurring source of supervision: sentences in the full-text of papers that cite multiple papers together (co-citations). Such co-citations not only reflect close paper relatedness, but also provide textual descriptions of how the co-cited papers are related. This novel form of textual supervision is used for learning to match aspects across papers. We develop multi-vector representations where vectors correspond to sentence-level aspects of documents, and present two methods for aspect matching: (1) A fast method that only matches single aspects, and (2) a method that makes sparse multiple matches with an Optimal Transport mechanism that computes an Earth Mover's Distance between aspects. Our approach improves performance on document similarity tasks in four datasets. Further, our fast single-match method achieves competitive results, paving the way for applying fine-grained similarity to large scientific corpora. Code, data, and models available at: https://github.com/allenai/aspire
Few-Shot Self-Rationalization with Natural Language Prompts
Authors: Ana Marasović, Iz Beltagy, Doug Downey, Matthew E. Peters
Date: 2021-11-16 00:00:00
Proceeding: Findings (NAACL) 2022 7

Self-rationalization models that predict task labels and generate free-text elaborations for their predictions could enable more intuitive interaction with NLP systems. These models are, however, currently trained with a large amount of human-written free-text explanations for each task which hinders their broader usage. We propose to study a more realistic setting of self-rationalization using few training examples. We present FEB -- a standardized collection of four existing English-language datasets and associated metrics. We identify the right prompting approach by extensively exploring natural language prompts on FEB. Then, by using this prompt and scaling the model size, we demonstrate that making progress on few-shot self-rationalization is possible. We show there is still ample room for improvement in this task: the average plausibility of generated explanations assessed by human annotators is at most 51% (with GPT-3), while plausibility of human explanations is 76%. We hope that FEB and our proposed approach will spur the community to take on the few-shot self-rationalization challenge.
Meeting Summarization with Pre-training and Clustering Methods
Authors: Andras Huebner, Wei Ji, Xiang Xiao
Date: 2021-11-16 00:00:00
Tasks: Clustering, Language Modeling, Language Modelling, Meeting Summarization, News Summarization

Automatic meeting summarization is becoming increasingly popular these days. The ability to automatically summarize meetings and to extract key information could greatly increase the efficiency of our work and life. In this paper, we experiment with different approaches to improve the performance of query-based meeting summarization. We started with HMNet\cite{hmnet}, a hierarchical network that employs both a word-level transformer and a turn-level transformer, as the baseline. We explore the effectiveness of pre-training the model with a large news-summarization dataset. We investigate adding the embeddings of queries as a part of the input vectors for query-based summarization. Furthermore, we experiment with extending the locate-then-summarize approach of QMSum\cite{qmsum} with an intermediate clustering step. Lastly, we compare the performance of our baseline models with BART, a state-of-the-art language model that is effective for summarization. We achieved improved performance by adding query embeddings to the input of the model, by using BART as an alternative language model, and by using clustering methods to extract key information at utterance level before feeding the text into summarization models.
Attention-based Multi-hypothesis Fusion for Speech Summarization
Authors: Takatomo Kano, Atsunori Ogawa, Marc Delcroix, Shinji Watanabe
Date: 2021-11-16 00:00:00
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), speech-recognition, Speech Recognition, Text Summarization

Speech summarization, which generates a text summary from speech, can be achieved by combining automatic speech recognition (ASR) and text summarization (TS). With this cascade approach, we can exploit state-of-the-art models and large training datasets for both subtasks, i.e., Transformer for ASR and Bidirectional Encoder Representations from Transformers (BERT) for TS. However, ASR errors directly affect the quality of the output summary in the cascade approach. We propose a cascade speech summarization model that is robust to ASR errors and that exploits multiple hypotheses generated by ASR to attenuate the effect of ASR errors on the summary. We investigate several schemes to combine ASR hypotheses. First, we propose using the sum of sub-word embedding vectors weighted by their posterior values provided by an ASR system as an input to a BERT-based TS system. Then, we introduce a more general scheme that uses an attention-based fusion module added to a pre-trained BERT module to align and combine several ASR hypotheses. Finally, we perform speech summarization experiments on the How2 dataset and a newly assembled TED-based dataset that we will release with this paper. These experiments show that retraining the BERT-based TS system with these schemes can improve summarization performance and that the attention-based fusion module is particularly effective.
CoCA-MDD: A Coupled Cross-Attention based Framework for Streaming Mispronunciation Detection and Diagnosis
Authors: Nianzu Zheng, Liqun Deng, Wenyong Huang, Yu Ting Yeung, Baohua Xu, Yuanyuan Guo, Yasheng Wang, Xiao Chen, Xin Jiang, Qun Liu
Date: 2021-11-16 00:00:00
Tasks: Multi-Task Learning, Phone-level pronunciation scoring

Mispronunciation detection and diagnosis (MDD) is a popular research focus in computer-aided pronunciation training (CAPT) systems. End-to-end (e2e) approaches are becoming dominant in MDD. However an e2e MDD model usually requires entire speech utterances as input context, which leads to significant time latency especially for long paragraphs. We propose a streaming e2e MDD model called CoCA-MDD. We utilize conv-transformer structure to encode input speech in a streaming manner. A coupled cross-attention (CoCA) mechanism is proposed to integrate frame-level acoustic features with encoded reference linguistic features. CoCA also enables our model to perform mispronunciation classification with whole utterances. The proposed model allows system fusion between the streaming output and mispronunciation classification output for further performance enhancement. We evaluate CoCA-MDD on publicly available corpora. CoCA-MDD achieves F1 scores of 57.03% and 60.78% for streaming and fusion modes respectively on L2-ARCTIC. For phone-level pronunciation scoring, CoCA-MDD achieves 0.58 Pearson correlation coefficient (PCC) value on SpeechOcean762.