pwc-archive/papers-with-abstracts huggingface dataset:
Aspect Based Sentiment Analysis Using Spectral Temporal Graph Neural Network
Authors: Abir Chakraborty
Date: 2022-02-14 00:00:00
Proceeding: ICON 2021 12
Tasks: Aspect-Based Sentiment Analysis, Aspect-Based Sentiment Analysis (ABSA), Graph Attention, Graph Neural Network, Natural Language Understanding, Negation, Sentiment Analysis

The objective of Aspect Based Sentiment Analysis is to capture the sentiment of reviewers associated with different aspects. However, complexity of the review sentences, presence of double negation and specific usage of words found in different domains make it difficult to predict the sentiment accurately and overall a challenging natural language understanding task. While recurrent neural network, attention mechanism and more recently, graph attention based models are prevalent, in this paper we propose graph Fourier transform based network with features created in the spectral domain. While this approach has found considerable success in the forecasting domain, it has not been explored earlier for any natural language processing task. The method relies on creating and learning an underlying graph from the raw data and thereby using the adjacency matrix to shift to the graph Fourier domain. Subsequently, Fourier transform is used to switch to the frequency (spectral) domain where new features are created. These series of transformation proved to be extremely efficient in learning the right representation as we have found that our model achieves the best result on both the SemEval-2014 datasets, i.e., "Laptop" and "Restaurants" domain. Our proposed model also found competitive results on the two other recently proposed datasets from the e-commerce domain.
Trace norm regularization for multi-task learning with scarce data
Authors: Etienne Boursier, Mikhail Konobeev, Nicolas Flammarion
Date: 2022-02-14 00:00:00
Tasks: Meta-Learning, Multi-Task Learning

Multi-task learning leverages structural similarities between multiple tasks to learn despite very few samples. Motivated by the recent success of neural networks applied to data-scarce tasks, we consider a linear low-dimensional shared representation model. Despite an extensive literature, existing theoretical results either guarantee weak estimation rates or require a large number of samples per task. This work provides the first estimation error bound for the trace norm regularized estimator when the number of samples per task is small. The advantages of trace norm regularization for learning data-scarce tasks extend to meta-learning and are confirmed empirically on synthetic datasets.
Attention-based Deep Neural Networks for Battery Discharge Capacity Forecasting
Authors: Yadong Zhang, Chenye Zou, Xin Chen
Date: 2022-02-14 00:00:00
Tasks: Capacity Estimation, Management

Battery discharge capacity forecasting is critically essential for the applications of lithium-ion batteries. The capacity degeneration can be treated as the memory of the initial battery state of charge from the data point of view. The streaming sensor data collected by battery management systems (BMS) reflect the usable battery capacity degradation rates under various operational working conditions. The battery capacity in different cycles can be measured with the temporal patterns extracted from the streaming sensor data based on the attention mechanism. The attention-based similarity regarding the first cycle can describe the battery capacity degradation in the following cycles. The deep degradation network (DDN) is developed with the attention mechanism to measure similarity and predict battery capacity. The DDN model can extract the degeneration-related temporal patterns from the streaming sensor data and perform the battery capacity prediction efficiently online in real-time. Based on the MIT-Stanford open-access battery aging dataset, the root-mean-square error of the capacity estimation is 1.3 mAh. The mean absolute percentage error of the proposed DDN model is 0.06{\%}. The DDN model also performance well in the Oxford Battery Degradation Dataset with dynamic load profiles. Therefore, the high accuracy and strong robustness of the proposed algorithm are verified.
A Data-Centric Approach to Generate Invariants for a Smart Grid Using Machine Learning
Authors: Danish Hudani, Muhammad Haseeb, Muhammad Taufiq, Muhammad Azmi Umer, Nandha Kumar Kandasamy
Date: 2022-02-14 00:00:00

Cyber-Physical Systems (CPS) have gained popularity due to the increased requirements on their uninterrupted connectivity and process automation. Due to their connectivity over the network including intranet and internet, dependence on sensitive data, heterogeneous nature, and large-scale deployment, they are highly vulnerable to cyber-attacks. Cyber-attacks are performed by creating anomalies in the normal operation of the systems with a goal either to disrupt the operation or destroy the system completely. The study proposed here focuses on detecting those anomalies which could be the cause of cyber-attacks. This is achieved by deriving the rules that govern the physical behavior of a process within a plant. These rules are called Invariants. We have proposed a Data-Centric approach (DaC) to generate such invariants. The entire study was conducted using the operational data of a functional smart power grid which is also a living lab.
Versatile Dueling Bandits: Best-of-both-World Analyses for Online Learning from Preferences
Authors: Aadirupa Saha, Pierre Gaillard
Date: 2022-02-14 00:00:00
Tasks: Multi-Armed Bandits

We study the problem of $K$-armed dueling bandit for both stochastic and adversarial environments, where the goal of the learner is to aggregate information through relative preferences of pair of decisions points queried in an online sequential manner. We first propose a novel reduction from any (general) dueling bandits to multi-armed bandits and despite the simplicity, it allows us to improve many existing results in dueling bandits. In particular, \emph{we give the first best-of-both world result for the dueling bandits regret minimization problem} -- a unified framework that is guaranteed to perform optimally for both stochastic and adversarial preferences simultaneously. Moreover, our algorithm is also the first to achieve an optimal $O(\sum_{i = 1}^K \frac{\log T}{\Delta_i})$ regret bound against the Condorcet-winner benchmark, which scales optimally both in terms of the arm-size $K$ and the instance-specific suboptimality gaps $\{\Delta_i\}_{i = 1}^K$. This resolves the long-standing problem of designing an instancewise gap-dependent order optimal regret algorithm for dueling bandits (with matching lower bounds up to small constant factors). We further justify the robustness of our proposed algorithm by proving its optimal regret rate under adversarially corrupted preferences -- this outperforms the existing state-of-the-art corrupted dueling results by a large margin. In summary, we believe our reduction idea will find a broader scope in solving a diverse class of dueling bandits setting, which are otherwise studied separately from multi-armed bandits with often more complex solutions and worse guarantees. The efficacy of our proposed algorithms is empirically corroborated against the existing dueling bandit methods.
CodeFill: Multi-token Code Completion by Jointly Learning from Structure and Naming Sequences
Authors: Maliheh Izadi, Roberta Gismondi, Georgios Gousios
Date: 2022-02-14 00:00:00
Tasks: Code Completion, Language Modelling, Multi-Task Learning

Code completion is an essential feature of IDEs, yet current autocompleters are restricted to either grammar-based or NLP-based single token completions. Both approaches have significant drawbacks: grammar-based autocompletion is restricted in dynamically-typed language environments, whereas NLP-based autocompleters struggle to understand the semantics of the programming language and the developer's code context. In this work, we present CodeFill, a language model for autocompletion that combines learned structure and naming information. Using a parallel Transformer architecture and multi-task learning, CodeFill consumes sequences of source code token names and their equivalent AST token types. Uniquely, CodeFill is trained both for single-token and multi-token (statement) prediction, which enables it to learn long-range dependencies among grammatical and naming elements. We train CodeFill on two datasets, consisting of 29M and 425M lines of code, respectively. To make the evaluation more realistic, we develop a method to automatically infer points in the source code at which completion matters. We compare CodeFill against four baselines and two state-of-the-art models, GPT-C and TravTrans+.CodeFill surpasses all baselines in single token prediction (MRR: 70.9% vs. 66.2% and 67.8%) and outperforms the state of the art for multi-token prediction (ROUGE-L: 63.7% vs. 52.4% and 59.2%, for n=4 tokens). We publicly release our source code and datasets.
Partially Fake Audio Detection by Self-attention-based Fake Span Discovery
Authors: Haibin Wu, Heng-Cheng Kuo, Naijun Zheng, Kuo-Hsuan Hung, Hung-Yi Lee, Yu Tsao, Hsin-Min Wang, Helen Meng
Date: 2022-02-14 00:00:00
Tasks: Open-Ended Question Answering, Question Answering, Speech Synthesis, Voice Conversion

The past few years have witnessed the significant advances of speech synthesis and voice conversion technologies. However, such technologies can undermine the robustness of broadly implemented biometric identification models and can be harnessed by in-the-wild attackers for illegal uses. The ASVspoof challenge mainly focuses on synthesized audios by advanced speech synthesis and voice conversion models, and replay attacks. Recently, the first Audio Deep Synthesis Detection challenge (ADD 2022) extends the attack scenarios into more aspects. Also ADD 2022 is the first challenge to propose the partially fake audio detection task. Such brand new attacks are dangerous and how to tackle such attacks remains an open question. Thus, we propose a novel framework by introducing the question-answering (fake span discovery) strategy with the self-attention mechanism to detect partially fake audios. The proposed fake span detection module tasks the anti-spoofing model to predict the start and end positions of the fake clip within the partially fake audio, address the model's attention into discovering the fake spans rather than other shortcuts with less generalization, and finally equips the model with the discrimination capacity between real and partially fake audios. Our submission ranked second in the partially fake audio detection track of ADD 2022.
The Impact of Batch Learning in Stochastic Linear Bandits
Authors: Danil Provodin, Pratik Gajane, Mykola Pechenizkiy, Maurits Kaptein
Date: 2022-02-14 00:00:00

We consider a special case of bandit problems, named batched bandits, in which an agent observes batches of responses over a certain time period. Unlike previous work, we consider a more practically relevant batch-centric scenario of batch learning. That is to say, we provide a policy-agnostic regret analysis and demonstrate upper and lower bounds for the regret of a candidate policy. Our main theoretical results show that the impact of batch learning is a multiplicative factor of batch size relative to the regret of online behavior. Primarily, we study two settings of the stochastic linear bandits: bandits with finitely and infinitely many arms. While the regret bounds are the same for both settings, the former setting results hold under milder assumptions. Also, we provide a more robust result for the 2-armed bandit problem as an important insight. Finally, we demonstrate the consistency of theoretical results by conducting empirical experiments and reflect on optimal batch size choice.
Out of Thin Air: Is Zero-Shot Cross-Lingual Keyword Detection Better Than Unsupervised?
Authors: Boshko Koloski, Senja Pollak, Blaž Škrlj, Matej Martinc
Date: 2022-02-14 00:00:00
Proceeding: LREC 2022 6
Tasks: Keyword Extraction, Pretrained Multilingual Language Models

Keyword extraction is the task of retrieving words that are essential to the content of a given document. Researchers proposed various approaches to tackle this problem. At the top-most level, approaches are divided into ones that require training - supervised and ones that do not - unsupervised. In this study, we are interested in settings, where for a language under investigation, no training data is available. More specifically, we explore whether pretrained multilingual language models can be employed for zero-shot cross-lingual keyword extraction on low-resource languages with limited or no available labeled training data and whether they outperform state-of-the-art unsupervised keyword extractors. The comparison is conducted on six news article datasets covering two high-resource languages, English and Russian, and four low-resource languages, Croatian, Estonian, Latvian, and Slovenian. We find that the pretrained models fine-tuned on a multilingual corpus covering languages that do not appear in the test set (i.e. in a zero-shot setting), consistently outscore unsupervised models in all six languages.
Continuous-time stochastic gradient descent for optimizing over the stationary distribution of stochastic differential equations
Authors: Ziheng Wang, Justin Sirignano
Date: 2022-02-14 00:00:00

We develop a new continuous-time stochastic gradient descent method for optimizing over the stationary distribution of stochastic differential equation (SDE) models. The algorithm continuously updates the SDE model's parameters using an estimate for the gradient of the stationary distribution. The gradient estimate is simultaneously updated using forward propagation of the SDE state derivatives, asymptotically converging to the direction of steepest descent. We rigorously prove convergence of the online forward propagation algorithm for linear SDE models (i.e., the multi-dimensional Ornstein-Uhlenbeck process) and present its numerical results for nonlinear examples. The proof requires analysis of the fluctuations of the parameter evolution around the direction of steepest descent. Bounds on the fluctuations are challenging to obtain due to the online nature of the algorithm (e.g., the stationary distribution will continuously change as the parameters change). We prove bounds for the solutions of a new class of Poisson partial differential equations (PDEs), which are then used to analyze the parameter fluctuations in the algorithm. Our algorithm is applicable to a range of mathematical finance applications involving statistical calibration of SDE models and stochastic optimal control for long time horizons where ergodicity of the data and stochastic process is a suitable modeling framework. Numerical examples explore these potential applications, including learning a neural network control for high-dimensional optimal control of SDEs and training stochastic point process models of limit order book events.
What is Next when Sequential Prediction Meets Implicitly Hard Interaction?
Authors: Kaixi Hu, Lin Li, Qing Xie, Jianquan Liu, Xiaohui Tao
Date: 2022-02-14 00:00:00

Hard interaction learning between source sequences and their next targets is challenging, which exists in a myriad of sequential prediction tasks. During the training process, most existing methods focus on explicitly hard interactions caused by wrong responses. However, a model might conduct correct responses by capturing a subset of learnable patterns, which results in implicitly hard interactions with some unlearned patterns. As such, its generalization performance is weakened. The problem gets more serious in sequential prediction due to the interference of substantial similar candidate targets. To this end, we propose a Hardness Aware Interaction Learning framework (HAIL) that mainly consists of two base sequential learning networks and mutual exclusivity distillation (MED). The base networks are initialized differently to learn distinctive view patterns, thus gaining different training experiences. The experiences in the form of the unlikelihood of correct responses are drawn from each other by MED, which provides mutual exclusivity knowledge to figure out implicitly hard interactions. Moreover, we deduce that the unlikelihood essentially introduces additional gradients to push the pattern learning of correct responses. Our framework can be easily extended to more peer base networks. Evaluation is conducted on four datasets covering cyber and physical spaces. The experimental results demonstrate that our framework outperforms several state-of-the-art methods in terms of top-k based metrics.
A Differential Entropy Estimator for Training Neural Networks
Authors: Georg Pichler, Pierre Colombo, Malik Boudiaf, Günther Koliander, Pablo Piantanida
Date: 2022-02-14 00:00:00
Tasks: Domain Adaptation

Mutual Information (MI) has been widely used as a loss regularizer for training neural networks. This has been particularly effective when learn disentangled or compressed representations of high dimensional data. However, differential entropy (DE), another fundamental measure of information, has not found widespread use in neural network training. Although DE offers a potentially wider range of applications than MI, off-the-shelf DE estimators are either non differentiable, computationally intractable or fail to adapt to changes in the underlying distribution. These drawbacks prevent them from being used as regularizers in neural networks training. To address shortcomings in previously proposed estimators for DE, here we introduce KNIFE, a fully parameterized, differentiable kernel-based estimator of DE. The flexibility of our approach also allows us to construct KNIFE-based estimators for conditional (on either discrete or continuous variables) DE, as well as MI. We empirically validate our method on high-dimensional synthetic data and further apply it to guide the training of neural networks for real-world tasks. Our experiments on a large variety of tasks, including visual domain adaptation, textual fair classification, and textual fine-tuning demonstrate the effectiveness of KNIFE-based estimation. Code can be found at https://github.com/g-pichler/knife.
An Application of Online Learning to Spacecraft Memory Dump Optimization
Authors: Tommaso Cesari, Jonathan Pergoli, Michele Maestrini, Pierluigi Di Lizia
Date: 2022-02-14 00:00:00

In this paper, we present a real-world application of online learning with expert advice to the field of Space Operations, testing our theory on real-life data coming from the Copernicus Sentinel-6 satellite. We show that in Spacecraft Memory Dump Optimization, a lightweight Follow-The-Leader algorithm leads to an increase in performance of over $60\%$ when compared to traditional techniques.
Toward Unsupervised Test Scenario Extraction for Automated Driving Systems from Urban Naturalistic Road Traffic Data
Authors: Nico Weber, Christoph Thiem, Ulrich Konigorski
Date: 2022-02-14 00:00:00
Tasks: Clustering, Dimensionality Reduction

Scenario-based testing is a promising approach to solve the challenge of proving the safe behavior of vehicles equipped with automated driving systems. Since an infinite number of concrete scenarios can theoretically occur in real-world road traffic, the extraction of scenarios relevant in terms of the safety-related behavior of these systems is a key aspect for their successful verification and validation. Therefore, a method for extracting multimodal urban traffic scenarios from naturalistic road traffic data in an unsupervised manner, minimizing the amount of (potentially biased) prior expert knowledge, is proposed. Rather than an (elaborate) rule-based assignment by extracting concrete scenarios into predefined functional scenarios, the presented method deploys an unsupervised machine learning pipeline. The approach allows exploring the unknown nature of the data and their interpretation as test scenarios that experts could not have anticipated. The method is evaluated for naturalistic road traffic data at urban intersections from the inD and the Silicon Valley Intersections datasets. For this purpose, it is analyzed with which clustering approach (K-Means, hierarchical clustering, and DBSCAN) the scenario extraction method performs best (referring to an elaborate rule-based implementation). Subsequently, using hierarchical clustering the results show both a jump in overall accuracy of around 20% when moving from 4 to 5 clusters and a saturation effect starting at 41 clusters with an overall accuracy of 84%. These observations can be a valuable contribution in the context of the trade-off between the number of functional scenarios (i.e., clustering accuracy) and testing effort. Possible reasons for the observed accuracy variations of different clusters, each with a fixed total number of given clusters, are discussed.
Deep Monte Carlo Quantile Regression for Quantifying Aleatoric Uncertainty in Physics-informed Temperature Field Reconstruction
Authors: Xiaohu Zheng, Wen Yao, Zhiqiang Gong, Yunyang Zhang, Xiaoyu Zhao, Tingsong Jiang
Date: 2022-02-14 00:00:00
Tasks: Image-to-Image Regression, quantile regression, regression

For the temperature field reconstruction (TFR), a complex image-to-image regression problem, the convolutional neural network (CNN) is a powerful surrogate model due to the convolutional layer's good image feature extraction ability. However, a lot of labeled data is needed to train CNN, and the common CNN can not quantify the aleatoric uncertainty caused by data noise. In actual engineering, the noiseless and labeled training data is hardly obtained for the TFR. To solve these two problems, this paper proposes a deep Monte Carlo quantile regression (Deep MC-QR) method for reconstructing the temperature field and quantifying aleatoric uncertainty caused by data noise. On the one hand, the Deep MC-QR method uses physical knowledge to guide the training of CNN. Thereby, the Deep MC-QR method can reconstruct an accurate TFR surrogate model without any labeled training data. On the other hand, the Deep MC-QR method constructs a quantile level image for each input in each training epoch. Then, the trained CNN model can quantify aleatoric uncertainty by quantile level image sampling during the prediction stage. Finally, the effectiveness of the proposed Deep MC-QR method is validated by many experiments, and the influence of data noise on TFR is analyzed.
Statistical Inference for the Dynamic Time Warping Distance, with Application to Abnormal Time-Series Detection
Authors: Vo Nguyen Le Duy, Ichiro Takeuchi
Date: 2022-02-14 00:00:00
Tasks: Decision Making, Dynamic Time Warping, Time Series, Time Series Analysis, valid

We study statistical inference on the similarity/distance between two time-series under uncertain environment by considering a statistical hypothesis test on the distance obtained from Dynamic Time Warping (DTW) algorithm. The sampling distribution of the DTW distance is too difficult to derive because it is obtained based on the solution of the DTW algorithm, which is complicated. To circumvent this difficulty, we propose to employ the conditional selective inference framework, which enables us to derive a valid inference method on the DTW distance. To our knowledge, this is the first method that can provide a valid p-value to quantify the statistical significance of the DTW distance, which is helpful for high-stake decision making such as abnormal time-series detection problems. We evaluate the performance of the proposed inference method on both synthetic and real-world datasets.