pwc-archive/papers-with-abstracts huggingface dataset:
Temporal Dependencies in Feature Importance for Time Series Predictions
Authors: Kin Kwan Leung, Clayton Rooke, Jonathan Smith, Saba Zuberi, Maksims Volkovs
Date: 2021-07-29 00:00:00
Tasks: Feature Importance, Time Series, Time Series Analysis, Time Series Prediction

Time series data introduces two key challenges for explainability methods: firstly, observations of the same feature over subsequent time steps are not independent, and secondly, the same feature can have varying importance to model predictions over time. In this paper, we propose Windowed Feature Importance in Time (WinIT), a feature removal based explainability approach to address these issues. Unlike existing feature removal explanation methods, WinIT explicitly accounts for the temporal dependence between different observations of the same feature in the construction of its importance score. Furthermore, WinIT captures the varying importance of a feature over time, by summarizing its importance over a window of past time steps. We conduct an extensive empirical study on synthetic and real-world data, compare against a wide range of leading explainability methods, and explore the impact of various evaluation strategies. Our results show that WinIT achieves significant gains over existing methods, with more consistent performance across different evaluation metrics. The code for our work is publicly available at \url{https://github.com/layer6ai-labs/WinIT}.
Distributed Identification of Contracting and/or Monotone Network Dynamics
Authors: Max Revay, Jack Umenberger, Ian R. Manchester
Date: 2021-07-29 00:00:00

This paper proposes methods for identification of large-scale networked systems with guarantees that the resulting model will be contracting -- a strong form of nonlinear stability -- and/or monotone, i.e. order relations between states are preserved. The main challenges that we address are: simultaneously searching for model parameters and a certificate of stability, and scalability to networks with hundreds or thousands of nodes. We propose a model set that admits convex constraints for stability and monotonicity, and has a separable structure that allows distributed identification via the alternating directions method of multipliers (ADMM). The performance and scalability of the approach is illustrated on a variety of linear and non-linear case studies, including a nonlinear traffic network with a 200-dimensional state space.
Self-Supervised Transformer for Sparse and Irregularly Sampled Multivariate Clinical Time-Series
Authors: Sindhu Tipirneni, Chandan K. Reddy
Date: 2021-07-29 00:00:00
Tasks: Imputation, Mortality Prediction, Time Series, Time Series Analysis, Time Series Forecasting, Triplet

Multivariate time-series data are frequently observed in critical care settings and are typically characterized by sparsity (missing information) and irregular time intervals. Existing approaches for learning representations in this domain handle these challenges by either aggregation or imputation of values, which in-turn suppresses the fine-grained information and adds undesirable noise/overhead into the machine learning model. To tackle this problem, we propose a Self-supervised Transformer for Time-Series (STraTS) model which overcomes these pitfalls by treating time-series as a set of observation triplets instead of using the standard dense matrix representation. It employs a novel Continuous Value Embedding technique to encode continuous time and variable values without the need for discretization. It is composed of a Transformer component with multi-head attention layers which enable it to learn contextual triplet embeddings while avoiding the problems of recurrence and vanishing gradients that occur in recurrent architectures. In addition, to tackle the problem of limited availability of labeled data (which is typically observed in many healthcare applications), STraTS utilizes self-supervision by leveraging unlabeled data to learn better representations by using time-series forecasting as an auxiliary proxy task. Experiments on real-world multivariate clinical time-series benchmark datasets demonstrate that STraTS has better prediction performance than state-of-the-art methods for mortality prediction, especially when labeled data is limited. Finally, we also present an interpretable version of STraTS which can identify important measurements in the time-series data. Our data preprocessing and model implementation codes are available at https://github.com/sindhura97/STraTS.
Deciphering Cryptic Behavior in Bimetallic Transition Metal Complexes with Machine Learning
Authors: Michael G. Taylor, Aditya Nandy, Connie C. Lu, Heather J. Kulik
Date: 2021-07-29 00:00:00
Tasks: BIG-bench Machine Learning, regression

The rational tailoring of transition metal complexes is necessary to address outstanding challenges in energy utilization and storage. Heterobimetallic transition metal complexes that exhibit metal-metal bonding in stacked "double decker" ligand structures are an emerging, attractive platform for catalysis, but their properties are challenging to predict prior to laborious synthetic efforts. We demonstrate an alternative, data-driven approach to uncovering structure-property relationships for rational bimetallic complex design. We tailor graph-based representations of the metal-local environment for these heterobimetallic complexes for use in training of multiple linear regression and kernel ridge regression (KRR) models. Focusing on oxidation potentials, we obtain a set of 28 experimentally characterized complexes to develop a multiple linear regression model. On this training set, we achieve good accuracy (mean absolute error, MAE, of 0.25 V) and preserve transferability to unseen experimental data with a new ligand structure. We trained a KRR model on a subset of 330 structurally characterized heterobimetallics to predict the degree of metal-metal bonding. This KRR model predicts relative metal-metal bond lengths in the test set to within 5%, and analysis of key features reveals the fundamental atomic contributions (e.g., the valence electron configuration) that most strongly influence the behavior of complexes. Our work provides guidance for rational bimetallic design, suggesting that properties including the formal shortness ratio should be transferable from one period to another.
Quantifying Uncertainty for Machine Learning Based Diagnostic
Authors: Owen Convery, Lewis Smith, Yarin Gal, Adi Hanuka
Date: 2021-07-29 00:00:00
Tasks: BIG-bench Machine Learning, Deep Learning, Diagnostic, Prediction, quantile regression, Uncertainty Quantification

Virtual Diagnostic (VD) is a deep learning tool that can be used to predict a diagnostic output. VDs are especially useful in systems where measuring the output is invasive, limited, costly or runs the risk of damaging the output. Given a prediction, it is necessary to relay how reliable that prediction is. This is known as 'uncertainty quantification' of a prediction. In this paper, we use ensemble methods and quantile regression neural networks to explore different ways of creating and analyzing prediction's uncertainty on experimental data from the Linac Coherent Light Source at SLAC. We aim to accurately and confidently predict the current profile or longitudinal phase space images of the electron beam. The ability to make informed decisions under uncertainty is crucial for reliable deployment of deep learning tools on safety-critical systems as particle accelerators.
Modeling and Optimizing Laser-Induced Graphene
Authors: Lars Kotthoff, Sourin Dey, Vivek Jain, Alexander Tyrrell, Hud Wahab, Patrick Johnson
Date: 2021-07-29 00:00:00

A lot of technological advances depend on next-generation materials, such as graphene, which enables a raft of new applications, for example better electronics. Manufacturing such materials is often difficult; in particular, producing graphene at scale is an open problem. We provide a series of datasets that describe the optimization of the production of laser-induced graphene, an established manufacturing method that has shown great promise. We pose three challenges based on the datasets we provide -- modeling the behavior of laser-induced graphene production with respect to parameters of the production process, transferring models and knowledge between different precursor materials, and optimizing the outcome of the transformation over the space of possible production parameters. We present illustrative results, along with the code used to generate them, as a starting point for interested users. The data we provide represents an important real-world application of machine learning; to the best of our knowledge, no similar datasets are available.
EEG multipurpose eye blink detector using convolutional neural network
Authors: Amanda Ferrari Iaquinta, Ana Carolina de Sousa Silva, Aldrumont Ferraz Júnior, Jessica Monique de Toledo, Gustavo Voltani von Atzingen
Date: 2021-07-29 00:00:00
Tasks: EEG, Electroencephalogram (EEG)

The electrical signal emitted by the eyes movement produces a very strong artifact on EEG signaldue to its close proximity to the sensors and abundance of occurrence. In the context of detectingeye blink artifacts in EEG waveforms for further removal and signal purification, multiple strategieswhere proposed in the literature. Most commonly applied methods require the use of a large numberof electrodes, complex equipment for sampling and processing data. The goal of this work is to createa reliable and user independent algorithm for detecting and removing eye blink in EEG signals usingCNN (convolutional neural network). For training and validation, three sets of public EEG data wereused. All three sets contain samples obtained while the recruited subjects performed assigned tasksthat included blink voluntarily in specific moments, watch a video and read an article. The modelused in this study was able to have an embracing understanding of all the features that distinguish atrivial EEG signal from a signal contaminated with eye blink artifacts without being overfitted byspecific features that only occurred in the situations when the signals were registered.
Developing Open Source Educational Resources for Machine Learning and Data Science
Authors: Ludwig Bothmann, Sven Strickroth, Giuseppe Casalicchio, David Rügamer, Marius Lindauer, Fabian Scheipl, Bernd Bischl
Date: 2021-07-28 00:00:00
Tasks: BIG-bench Machine Learning

Education should not be a privilege but a common good. It should be openly accessible to everyone, with as few barriers as possible; even more so for key technologies such as Machine Learning (ML) and Data Science (DS). Open Educational Resources (OER) are a crucial factor for greater educational equity. In this paper, we describe the specific requirements for OER in ML and DS and argue that it is especially important for these fields to make source files publicly available, leading to Open Source Educational Resources (OSER). We present our view on the collaborative development of OSER, the challenges this poses, and first steps towards their solutions. We outline how OSER can be used for blended learning scenarios and share our experiences in university education. Finally, we discuss additional challenges such as credit assignment or granting certificates.
Towards Universality in Multilingual Text Rewriting
Authors: Xavier Garcia, Noah Constant, Mandy Guo, Orhan Firat
Date: 2021-07-30 00:00:00
Tasks: Translation

In this work, we take the first steps towards building a universal rewriter: a model capable of rewriting text in any language to exhibit a wide variety of attributes, including styles and languages, while preserving as much of the original semantics as possible. In addition to obtaining state-of-the-art results on unsupervised translation, we also demonstrate the ability to do zero-shot sentiment transfer in non-English languages using only English exemplars for sentiment. We then show that our model is able to modify multiple attributes at once, for example adjusting both language and sentiment jointly. Finally, we show that our model is capable of performing zero-shot formality-sensitive translation.
Automatic Claim Review for Climate Science via Explanation Generation
Authors: Shraey Bhatia, Jey Han Lau, Timothy Baldwin
Date: 2021-07-30 00:00:00
Tasks: Decoder, Explanation Generation, Fact Checking, Open-Domain Question Answering, Question Answering

There is unison is the scientific community about human induced climate change. Despite this, we see the web awash with claims around climate change scepticism, thus driving the need for fact checking them but at the same time providing an explanation and justification for the fact check. Scientists and experts have been trying to address it by providing manually written feedback for these claims. In this paper, we try to aid them by automating generating explanation for a predicted veracity label for a claim by deploying the approach used in open domain question answering of a fusion in decoder augmented with retrieved supporting passages from an external knowledge. We experiment with different knowledge sources, retrievers, retriever depths and demonstrate that even a small number of high quality manually written explanations can help us in generating good explanations.
Investigating Disagreement in the Scientific Literature
Authors: Wout S. Lamers, Kevin Boyack, Vincent Larivière, Cassidy R. Sugimoto, Nees Jan van Eck, Ludo Waltman, Dakota Murray
Date: 2021-07-30 00:00:00
Tasks: Articles, text-classification, Text Classification

Disagreement is essential to scientific progress. However, the extent of disagreement in science, its evolution over time, and the fields in which it happens, remains poorly understood. Leveraging a massive collection of English-language scientific texts, we develop a cue-phrase based approach to identify instances of disagreement citations across more than four million scientific articles. Using this method, we construct an indicator of disagreement across scientific fields over the 2000-2015 period. In contrast with black-box text classification methods, our framework is transparent and easily interpretable. We reveal a disciplinary spectrum of disagreement, with higher disagreement in the social sciences and lower disagreement in physics and mathematics. However, detailed disciplinary analysis demonstrates heterogeneity across sub-fields, revealing the importance of local disciplinary cultures and epistemic characteristics of disagreement. Paper-level analysis reveals notable episodes of disagreement in science, and illustrates how methodological artifacts can confound analyses of scientific texts. These findings contribute to a broader understanding of disagreement and establish a foundation for future research to understanding key processes underlying scientific progress.
An automated domain-independent text reading, interpreting and extracting approach for reviewing the scientific literature
Authors: Amauri J Paula
Date: 2021-07-30 00:00:00
Tasks: Articles, named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER), NER, Part-Of-Speech Tagging, POS, text annotation

It is presented here a machine learning-based (ML) natural language processing (NLP) approach capable to automatically recognize and extract categorical and numerical parameters from a corpus of articles. The approach (named a.RIX) operates with a concomitant/interchangeable use of ML models such as neuron networks (NNs), latent semantic analysis (LSA), naive-Bayes classifiers (NBC), and a pattern recognition model using regular expression (REGEX). A corpus of 7,873 scientific articles dealing with natural products (NPs) was used to demonstrate the efficiency of the a.RIX engine. The engine automatically extracts categorical and numerical parameters such as (i) the plant species from which active molecules are extracted, (ii) the microorganisms species for which active molecules can act against, and (iii) the values of minimum inhibitory concentration (MIC) against these microorganisms. The parameters are extracted without part-of-speech tagging (POS) and named entity recognition (NER) approaches (i.e. without the need of text annotation), and the models training is performed with unsupervised approaches. In this way, a.RIX can be essentially used on articles from any scientific field. Finally, it can potentially make obsolete the current article reviewing process in some areas, especially those in which machine learning models capture texts structure, text semantics, and latent knowledge.
USC: An Open-Source Uzbek Speech Corpus and Initial Speech Recognition Experiments
Authors: Muhammadjon Musaev, Saida Mussakhojayeva, Ilyos Khujayorov, Yerbolat Khassanov, Mannon Ochilov, Huseyin Atakan Varol
Date: 2021-07-30 00:00:00
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), speech-recognition, Speech Recognition

We present a freely available speech corpus for the Uzbek language and report preliminary automatic speech recognition (ASR) results using both the deep neural network hidden Markov model (DNN-HMM) and end-to-end (E2E) architectures. The Uzbek speech corpus (USC) comprises 958 different speakers with a total of 105 hours of transcribed audio recordings. To the best of our knowledge, this is the first open-source Uzbek speech corpus dedicated to the ASR task. To ensure high quality, the USC has been manually checked by native speakers. We first describe the design and development procedures of the USC, and then explain the conducted ASR experiments in detail. The experimental results demonstrate promising results for the applicability of the USC for ASR. Specifically, 18.1% and 17.4% word error rates were achieved on the validation and test sets, respectively. To enable experiment reproducibility, we share the USC dataset, pre-trained models, and training recipes in our GitHub repository.
WiC = TSV = WSD: On the Equivalence of Three Semantic Tasks
Authors: Bradley Hauer, Grzegorz Kondrak
Date: 2021-07-29 00:00:00
Proceeding: NAACL 2022 7
Tasks: Word Sense Disambiguation

The Word-in-Context (WiC) task has attracted considerable attention in the NLP community, as demonstrated by the popularity of the recent MCL-WiC SemEval shared task. Systems and lexical resources from word sense disambiguation (WSD) are often used for the WiC task and WiC dataset construction. In this paper, we establish the exact relationship between WiC and WSD, as well as the related task of target sense verification (TSV). Building upon a novel hypothesis on the equivalence of sense and meaning distinctions, we demonstrate through the application of tools from theoretical computer science that these three semantic classification problems can be pairwise reduced to each other, and therefore are equivalent. The results of experiments that involve systems and datasets for both WiC and WSD provide strong empirical evidence that our problem reductions work in practice.
IIITG-ADBU@HASOC-Dravidian-CodeMix-FIRE2020: Offensive Content Detection in Code-Mixed Dravidian Text
Authors: Arup Baruah, Kaushik Amar Das, Ferdous Ahmed Barbhuiya, Kuntal Dey
Date: 2021-07-29 00:00:00

This paper presents the results obtained by our SVM and XLM-RoBERTa based classifiers in the shared task Dravidian-CodeMix-HASOC 2020. The SVM classifier trained using TF-IDF features of character and word n-grams performed the best on the code-mixed Malayalam text. It obtained a weighted F1 score of 0.95 (1st Rank) and 0.76 (3rd Rank) on the YouTube and Twitter dataset respectively. The XLM-RoBERTa based classifier performed the best on the code-mixed Tamil text. It obtained a weighted F1 score of 0.87 (3rd Rank) on the code-mixed Tamil Twitter dataset.
MDQE: A More Accurate Direct Pretraining for Machine Translation Quality Estimation
Authors: Lei Lin
Date: 2021-07-24 00:00:00
Tasks: Machine Translation, Translation

It is expensive to evaluate the results of Machine Translation(MT), which usually requires manual translation as a reference. Machine Translation Quality Estimation (QE) is a task of predicting the quality of machine translations without relying on any reference. Recently, the emergence of predictor-estimator framework which trains the predictor as a feature extractor and estimator as a QE predictor, and pre-trained language models(PLM) have achieved promising QE performance. However, we argue that there are still gaps between the predictor and the estimator in both data quality and training objectives, which preclude QE models from benefiting from a large number of parallel corpora more directly. Based on previous related work that have alleviated gaps to some extent, we propose a novel framework that provides a more accurate direct pretraining for QE tasks. In this framework, a generator is trained to produce pseudo data that is closer to the real QE data, and a estimator is pretrained on these data with novel objectives that are the same as the QE task. Experiments on widely used benchmarks show that our proposed framework outperforms existing methods, without using any pretraining models such as BERT.
Text Classification and Clustering with Annealing Soft Nearest Neighbor Loss
Authors: Abien Fred Agarap
Date: 2021-07-23 00:00:00
Tasks: Classification, Clustering, Disentanglement, Representation Learning, text-classification, Text Classification, Text Clustering

We define disentanglement as how far class-different data points from each other are, relative to the distances among class-similar data points. When maximizing disentanglement during representation learning, we obtain a transformed feature representation where the class memberships of the data points are preserved. If the class memberships of the data points are preserved, we would have a feature representation space in which a nearest neighbour classifier or a clustering algorithm would perform well. We take advantage of this method to learn better natural language representation, and employ it on text classification and text clustering tasks. Through disentanglement, we obtain text representations with better-defined clusters and improve text classification performance. Our approach had a test classification accuracy of as high as 90.11% and test clustering accuracy of 88% on the AG News dataset, outperforming our baseline models -- without any other training tricks or regularization.
Multi-stage Pre-training over Simplified Multimodal Pre-training Models
Authors: Tongtong Liu, Fangxiang Feng, Xiaojie Wang
Date: 2021-07-22 00:00:00
Proceeding: ACL 2021 5
Tasks: Image-text Retrieval, Retrieval, Sentence, Text Retrieval

Multimodal pre-training models, such as LXMERT, have achieved excellent results in downstream tasks. However, current pre-trained models require large amounts of training data and have huge model sizes, which make them difficult to apply in low-resource situations. How to obtain similar or even better performance than a larger model under the premise of less pre-training data and smaller model size has become an important problem. In this paper, we propose a new Multi-stage Pre-training (MSP) method, which uses information at different granularities from word, phrase to sentence in both texts and images to pre-train the model in stages. We also design several different pre-training tasks suitable for the information granularity in different stage in order to efficiently capture the diverse knowledge from a limited corpus. We take a Simplified LXMERT (LXMERT- S), which has only 45.9% parameters of the original LXMERT model and 11.76% of the original pre-training data as the testbed of our MSP method. Experimental results show that our method achieves comparable performance to the original LXMERT model in all downstream tasks, and even outperforms the original model in Image-Text Retrieval task.