pwc-archive/papers-with-abstracts huggingface dataset:
Interpretable Latent Variables in Deep State Space Models
Authors: Haoxuan Wu, David S. Matteson, Martin T. Wells
Date: 2022-03-03 00:00:00
Tasks: Decoder, State Space Models, Time Series, Time Series Analysis

We introduce a new version of deep state-space models (DSSMs) that combines a recurrent neural network with a state-space framework to forecast time series data. The model estimates the observed series as functions of latent variables that evolve non-linearly through time. Due to the complexity and non-linearity inherent in DSSMs, previous works on DSSMs typically produced latent variables that are very difficult to interpret. Our paper focus on producing interpretable latent parameters with two key modifications. First, we simplify the predictive decoder by restricting the response variables to be a linear transformation of the latent variables plus some noise. Second, we utilize shrinkage priors on the latent variables to reduce redundancy and improve robustness. These changes make the latent variables much easier to understand and allow us to interpret the resulting latent variables as random effects in a linear mixed model. We show through two public benchmark datasets the resulting model improves forecasting performances.
Symmetry Structured Convolutional Neural Networks
Authors: Kehelwala Dewage Gayan Maduranga, Vasily Zadorozhnyy, Qiang Ye
Date: 2022-03-03 00:00:00
Tasks: Sequential Recommendation

We consider Convolutional Neural Networks (CNNs) with 2D structured features that are symmetric in the spatial dimensions. Such networks arise in modeling pairwise relationships for a sequential recommendation problem, as well as secondary structure inference problems of RNA and protein sequences. We develop a CNN architecture that generates and preserves the symmetry structure in the network's convolutional layers. We present parameterizations for the convolutional kernels that produce update rules to maintain symmetry throughout the training. We apply this architecture to the sequential recommendation problem, the RNA secondary structure inference problem, and the protein contact map prediction problem, showing that the symmetric structured networks produce improved results using fewer numbers of machine parameters.
Compressed Predictive Information Coding
Authors: Rui Meng, Tianyi Luo, Kristofer Bouchard
Date: 2022-03-03 00:00:00
Tasks: Mutual Information Estimation

Unsupervised learning plays an important role in many fields, such as artificial intelligence, machine learning, and neuroscience. Compared to static data, methods for extracting low-dimensional structure for dynamic data are lagging. We developed a novel information-theoretic framework, Compressed Predictive Information Coding (CPIC), to extract useful representations from dynamic data. CPIC selectively projects the past (input) into a linear subspace that is predictive about the compressed data projected from the future (output). The key insight of our framework is to learn representations by minimizing the compression complexity and maximizing the predictive information in latent space. We derive variational bounds of the CPIC loss which induces the latent space to capture information that is maximally predictive. Our variational bounds are tractable by leveraging bounds of mutual information. We find that introducing stochasticity in the encoder robustly contributes to better representation. Furthermore, variational approaches perform better in mutual information estimation compared with estimates under a Gaussian assumption. We demonstrate that CPIC is able to recover the latent space of noisy dynamical systems with low signal-to-noise ratios, and extracts features predictive of exogenous variables in neuroscience data.
Automatic Detection and Segmentation of Postoperative Cerebellar Damage Based on Normalization
Authors: Silu Zhang, Stuart McAfee, Zoltan Patay, Matthew Scoggins
Date: 2022-03-03 00:00:00

Surgical resection is a common procedure in the treatment of pediatric posterior fossa tumors. However, surgical damage is often unavoidable and its association with postoperative complications is not well understood. A reliable localization and measure of cerebellar damage is fundamental to study the relationship between the damaged cerebellar regions and postoperative neurological outcomes. Existing cerebellum normalization methods are not reliable on postoperative scans, therefore current approaches to measure surgical damage rely on manual labelling. In this work, we develop a robust algorithm to automatically detect and measure cerebellum damage due to surgery using postoperative 3D T1 magnetic resonance imaging. In our proposed approach, normal brain tissues are first segmented using a Bayesian algorithm customized for postoperative scans. Next, the cerebellum is isolated by nonlinear registration of a whole brain template to the native space. The isolated cerebellum is then normalized into the spatially unbiased atlas (SUIT) space using anatomical information derived from the previous step. Finally, the damage is detected in the atlas space by comparing the normalized cerebellum and the SUIT template. We evaluated our damage detection tool on postoperative scans of 153 patients diagnosed with medulloblastoma based on inspection by human expects. We also designed a simulation to test the proposed approach without human intervention. Our results show that the proposed approach has superior performance on various scenarios.
Data-Efficient and Interpretable Tabular Anomaly Detection
Authors: Chun-Hao Chang, Jinsung Yoon, Sercan Arik, Madeleine Udell, Tomas Pfister
Date: 2022-03-03 00:00:00
Tasks: Additive models, Anomaly Detection

Anomaly detection (AD) plays an important role in numerous applications. We focus on two understudied aspects of AD that are critical for integration into real-world applications. First, most AD methods cannot incorporate labeled data that are often available in practice in small quantities and can be crucial to achieve high AD accuracy. Second, most AD methods are not interpretable, a bottleneck that prevents stakeholders from understanding the reason behind the anomalies. In this paper, we propose a novel AD framework that adapts a white-box model class, Generalized Additive Models, to detect anomalies using a partial identification objective which naturally handles noisy or heterogeneous features. In addition, the proposed framework, DIAD, can incorporate a small amount of labeled data to further boost anomaly detection performances in semi-supervised settings. We demonstrate the superiority of our framework compared to previous work in both unsupervised and semi-supervised settings using diverse tabular datasets. For example, under 5 labeled anomalies DIAD improves from 86.2\% to 89.4\% AUC by learning AD from unlabeled data. We also present insightful interpretations that explain why DIAD deems certain samples as anomalies.
Provable and Efficient Continual Representation Learning
Authors: Yingcong Li, Mingchen Li, M. Salman Asif, Samet Oymak
Date: 2022-03-03 00:00:00
Tasks: Continual Learning, Diversity, Representation Learning

In continual learning (CL), the goal is to design models that can learn a sequence of tasks without catastrophic forgetting. While there is a rich set of techniques for CL, relatively little understanding exists on how representations built by previous tasks benefit new tasks that are added to the network. To address this, we study the problem of continual representation learning (CRL) where we learn an evolving representation as new tasks arrive. Focusing on zero-forgetting methods where tasks are embedded in subnetworks (e.g., PackNet), we first provide experiments demonstrating CRL can significantly boost sample efficiency when learning new tasks. To explain this, we establish theoretical guarantees for CRL by providing sample complexity and generalization error bounds for new tasks by formalizing the statistical benefits of previously-learned representations. Our analysis and experiments also highlight the importance of the order in which we learn the tasks. Specifically, we show that CL benefits if the initial tasks have large sample size and high "representation diversity". Diversity ensures that adding new tasks incurs small representation mismatch and can be learned with few samples while training only few additional nonzero weights. Finally, we ask whether one can ensure each task subnetwork to be efficient during inference time while retaining the benefits of representation learning. To this end, we propose an inference-efficient variation of PackNet called Efficient Sparse PackNet (ESPN) which employs joint channel & weight pruning. ESPN embeds tasks in channel-sparse subnets requiring up to 80% less FLOPs to compute while approximately retaining accuracy and is very competitive with a variety of baselines. In summary, this work takes a step towards data and compute-efficient CL with a representation learning perspective. GitHub page: https://github.com/ucr-optml/CtRL
Nonlinear predictive models computation in ADPCM schemes
Authors: Marcos Faundez-Zanuy
Date: 2022-03-03 00:00:00

Recently several papers have been published on nonlinear prediction applied to speech coding. At ICASSP98 we presented a system based on an ADPCM scheme with a nonlinear predictor based on a neural net. The most critical parameter was the training procedure in order to achieve good generalization capability and robustness against mismatch between training and testing conditions. In this paper, we propose several new approaches that improve the performance of the original system in up to 1.2dB of SEGSNR (using bayesian regularization). The variance of the SEGSNR between frames is also minimized, so the new scheme produces a more stable quality of the output.
Zero-shot Transfer Learning within a Heterogeneous Graph via Knowledge Transfer Networks
Authors: Minji Yoon, John Palowitch, Dustin Zelle, Ziniu Hu, Ruslan Salakhutdinov, Bryan Perozzi
Date: 2022-03-03 00:00:00
Tasks: Domain Adaptation, Graph Learning, Representation Learning, Transfer Learning

Data continuously emitted from industrial ecosystems such as social or e-commerce platforms are commonly represented as heterogeneous graphs (HG) composed of multiple node/edge types. State-of-the-art graph learning methods for HGs known as heterogeneous graph neural networks (HGNNs) are applied to learn deep context-informed node representations. However, many HG datasets from industrial applications suffer from label imbalance between node types. As there is no direct way to learn using labels rooted at different node types, HGNNs have been applied to only a few node types with abundant labels. We propose a zero-shot transfer learning module for HGNNs called a Knowledge Transfer Network (KTN) that transfers knowledge from label-abundant node types to zero-labeled node types through rich relational information given in the HG. KTN is derived from the theoretical relationship, which we introduce in this work, between distinct feature extractors for each node type given in an HGNN model. KTN improves performance of 6 different types of HGNN models by up to 960% for inference on zero-labeled node types and outperforms state-of-the-art transfer learning baselines by up to 73% across 18 different transfer learning tasks on HGs.
On consistency of constrained spectral clustering under representation-aware stochastic block model
Authors: Shubham Gupta, Ambedkar Dukkipati
Date: 2022-03-03 00:00:00
Tasks: Clustering, Computational Efficiency, Stochastic Block Model

Spectral clustering is widely used in practice due to its flexibility, computational efficiency, and well-understood theoretical performance guarantees. Recently, spectral clustering has been studied to find balanced clusters under population-level constraints. These constraints are specified by additional information available in the form of auxiliary categorical node attributes. In this paper, we consider a scenario where these attributes may not be observable, but manifest as latent features of an auxiliary graph. Motivated by this, we study constrained spectral clustering with the aim of finding balanced clusters in a given \textit{similarity graph} $\mathcal{G}$, such that each individual is adequately represented with respect to an auxiliary graph $\mathcal{R}$ (we refer to this as representation graph). We propose an individual-level balancing constraint that formalizes this idea. Our work leads to an interesting stochastic block model that not only plants the given partitions in $\mathcal{G}$ but also plants the auxiliary information encoded in the representation graph $\mathcal{R}$. We develop unnormalized and normalized variants of spectral clustering in this setting. These algorithms use $\mathcal{R}$ to find clusters in $\mathcal{G}$ that approximately satisfy the proposed constraint. We also establish the first statistical consistency result for constrained spectral clustering under individual-level constraints for graphs sampled from the above-mentioned variant of the stochastic block model. Our experimental results corroborate our theoretical findings.
Enhanced physics-constrained deep neural networks for modeling vanadium redox flow battery
Authors: Qizhi He, Yucheng Fu, Panos Stinis, Alexandre Tartakovsky
Date: 2022-03-03 00:00:00
Tasks: Prediction

Numerical modeling and simulation have become indispensable tools for advancing a comprehensive understanding of the underlying mechanisms and cost-effective process optimization and control of flow batteries. In this study, we propose an enhanced version of the physics-constrained deep neural network (PCDNN) approach [1] to provide high-accuracy voltage predictions in the vanadium redox flow batteries (VRFBs). The purpose of the PCDNN approach is to enforce the physics-based zero-dimensional (0D) VRFB model in a neural network to assure model generalization for various battery operation conditions. Limited by the simplifications of the 0D model, the PCDNN cannot capture sharp voltage changes in the extreme SOC regions. To improve the accuracy of voltage prediction at extreme ranges, we introduce a second (enhanced) DNN to mitigate the prediction errors carried from the 0D model itself and call the resulting approach enhanced PCDNN (ePCDNN). By comparing the model prediction with experimental data, we demonstrate that the ePCDNN approach can accurately capture the voltage response throughout the charge--discharge cycle, including the tail region of the voltage discharge curve. Compared to the standard PCDNN, the prediction accuracy of the ePCDNN is significantly improved. The loss function for training the ePCDNN is designed to be flexible by adjusting the weights of the physics-constrained DNN and the enhanced DNN. This allows the ePCDNN framework to be transferable to battery systems with variable physical model fidelity.
Intrinsically-Motivated Reinforcement Learning: A Brief Introduction
Authors: Mingqi Yuan
Date: 2022-03-03 00:00:00
Tasks: Autonomous Driving, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL), Starcraft

Reinforcement learning (RL) is one of the three basic paradigms of machine learning. It has demonstrated impressive performance in many complex tasks like Go and StarCraft, which is increasingly involved in smart manufacturing and autonomous driving. However, RL consistently suffers from the exploration-exploitation dilemma. In this paper, we investigated the problem of improving exploration in RL and introduced the intrinsically-motivated RL. In sharp contrast to the classic exploration strategies, intrinsically-motivated RL utilizes the intrinsic learning motivation to provide sustainable exploration incentives. We carefully classified the existing intrinsic reward methods and analyzed their practical drawbacks. Moreover, we proposed a new intrinsic reward method via R\'enyi state entropy maximization, which overcomes the drawbacks of the preceding methods and provides powerful exploration incentives. Finally, extensive simulation demonstrated that the proposed module achieve superior performance with higher efficiency and robustness.
Comparison of LSTM autoencoder based deep learning enabled Bayesian inference using two time series reconstruction approaches
Authors: Saumik Dana
Date: 2022-03-03 00:00:00
Tasks: Bayesian Inference, Time Series, Time Series Analysis

In this work, we use a combination of Bayesian inference, Markov chain Monte Carlo and deep learning in the form of LSTM autoencoders to build and test a framework to provide robust estimates of injection rate from ground surface data in coupled flow and geomechanics problems. We use LSTM autoencoders to reconstruct the displacement time series for grid points on the top surface of a faulting due to water injection problem. We then deploy this LSTM autoencoder based model instead of the high fidelity model in the Bayesian inference framework to estimate injection rate from displacement input.
A photonic chip-based machine learning approach for the prediction of molecular properties
Authors: HUI ZHANG, Jonathan Wei Zhong Lau, Lingxiao Wan, Liang Shi, Hong Cai, Xianshu Luo, Patrick Lo, Chee-Kong Lee, Leong-Chuan Kwek, Ai Qun Liu
Date: 2022-03-03 00:00:00
Tasks: BIG-bench Machine Learning, Computational chemistry, Drug Discovery, Molecular Property Prediction

Machine learning methods have revolutionized the discovery process of new molecules and materials. However, the intensive training process of neural networks for molecules with ever-increasing complexity has resulted in exponential growth in computation cost, leading to long simulation time and high energy consumption. Photonic chip technology offers an alternative platform for implementing neural networks with faster data processing and lower energy usage compared to digital computers. Photonics technology is naturally capable of implementing complex-valued neural networks at no additional hardware cost. Here, we demonstrate the capability of photonic neural networks for predicting the quantum mechanical properties of molecules. To the best of our knowledge, this work is the first to harness photonic technology for machine learning applications in computational chemistry and molecular sciences, such as drug discovery and materials design. We further show that multiple properties can be learned simultaneously in a photonic chip via a multi-task regression learning algorithm, which is also the first of its kind as well, as most previous works focus on implementing a network in the classification task.
From Simultaneous to Streaming Machine Translation by Leveraging Streaming History
Authors: Javier Iranzo-Sánchez, Jorge Civera, Alfons Juan
Date: 2022-03-04 00:00:00
Proceeding: ACL 2022 5
Tasks: Machine Translation, Sentence, Translation

Simultaneous Machine Translation is the task of incrementally translating an input sentence before it is fully available. Currently, simultaneous translation is carried out by translating each sentence independently of the previously translated text. More generally, Streaming MT can be understood as an extension of Simultaneous MT to the incremental translation of a continuous input text stream. In this work, a state-of-the-art simultaneous sentence-level MT system is extended to the streaming setup by leveraging the streaming history. Extensive empirical results are reported on IWSLT Translation Tasks, showing that leveraging the streaming history leads to significant quality gains. In particular, the proposed system proves to compare favorably to the best performing systems.
Continuous Rating as Reliable Human Evaluation of Simultaneous Speech Translation
Authors: Dávid Javorský, Dominik Macháček, Ondřej Bojar
Date: 2022-03-04 00:00:00
Tasks: Machine Translation, Translation

Simultaneous speech translation (SST) can be evaluated on simulated online events where human evaluators watch subtitled videos and continuously express their satisfaction by pressing buttons (so called Continuous Rating). Continuous Rating is easy to collect, but little is known about its reliability, or relation to comprehension of foreign language document by SST users. In this paper, we contrast Continuous Rating with factual questionnaires on judges with different levels of source language knowledge. Our results show that Continuous Rating is easy and reliable SST quality assessment if the judges have at least limited knowledge of the source language. Our study indicates users' preferences on subtitle layout and presentation style and, most importantly, provides a significant evidence that users with advanced source language knowledge prefer low latency over fewer re-translations.
Beyond Plain Toxic: Detection of Inappropriate Statements on Flammable Topics for the Russian Language
Authors: Nikolay Babakov, Varvara Logacheva, Alexander Panchenko
Date: 2022-03-04 00:00:00
Tasks: Chatbot, Cultural Vocal Bursts Intensity Prediction

Toxicity on the Internet, such as hate speech, offenses towards particular users or groups of people, or the use of obscene words, is an acknowledged problem. However, there also exist other types of inappropriate messages which are usually not viewed as toxic, e.g. as they do not contain explicit offences. Such messages can contain covered toxicity or generalizations, incite harmful actions (crime, suicide, drug use), provoke "heated" discussions. Such messages are often related to particular sensitive topics, e.g. on politics, sexual minorities, social injustice which more often than other topics, e.g. cars or computing, yield toxic emotional reactions. At the same time, clearly not all messages within such flammable topics are inappropriate. Towards this end, in this work, we present two text collections labelled according to binary notion of inapropriateness and a multinomial notion of sensitive topic. Assuming that the notion of inappropriateness is common among people of the same culture, we base our approach on human intuitive understanding of what is not acceptable and harmful. To objectivise the notion of inappropriateness, we define it in a data-driven way though crowdsourcing. Namely we run a large-scale annotation study asking workers if a given chatbot textual statement could harm reputation of a company created it. Acceptably high values of inter-annotator agreement suggest that the notion of inappropriateness exists and can be uniformly understood by different people. To define the notion of sensitive topics in an objective way we use on guidelines suggested commonly by specialists of legal and PR department of a large public company as potentially harmful.
IISERB Brains at SemEval 2022 Task 6: A Deep-learning Framework to Identify Intended Sarcasm in English
Authors: Tanuj Singh Shekhawat, Manoj Kumar, Udaybhan Rathore, Aditya Joshi, Jasabanta Patro
Date: 2022-03-04 00:00:00

This paper describes the system architectures and the models submitted by our team "IISERBBrains" to SemEval 2022 Task 6 competition. We contested for all three sub-tasks floated for the English dataset. On the leader-board, wegot19th rank out of43 teams for sub-taskA, the 8th rank out of22 teams for sub-task B,and13th rank out of 16 teams for sub-taskC. Apart from the submitted results and models, we also report the other models and results that we obtained through our experiments after organizers published the gold labels of their evaluation data