pwc-archive/papers-with-abstracts huggingface dataset:
Language-Universal Adapter Learning with Knowledge Distillation for End-to-End Multilingual Speech Recognition
Authors: Zhijie Shen, Wu Guo, Bin Gu
Date: 2023-02-28 00:00:00
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), Knowledge Distillation, speech-recognition, Speech Recognition

In this paper, we propose a language-universal adapter learning framework based on a pre-trained model for end-to-end multilingual automatic speech recognition (ASR). For acoustic modeling, the wav2vec 2.0 pre-trained model is fine-tuned by inserting language-specific and language-universal adapters. An online knowledge distillation is then used to enable the language-universal adapters to learn both language-specific and universal features. The linguistic information confusion is also reduced by leveraging language identifiers (LIDs). With LIDs we perform a position-wise modification on the multi-head attention outputs. In the inference procedure, the language-specific adapters are removed while the language-universal adapters are kept activated. The proposed method improves the recognition accuracy and addresses the linear increase of the number of adapters' parameters with the number of languages in common multilingual ASR systems. Experiments on the BABEL dataset confirm the effectiveness of the proposed framework. Compared to the conventional multilingual model, a 3.3% absolute error rate reduction is achieved. The code is available at: https://github.com/shen9712/UniversalAdapterLearning.
Adversarial Examples Exist in Two-Layer ReLU Networks for Low Dimensional Linear Subspaces
Date: 2023-03-01 00:00:00
Proceeding: NeurIPS 2023 11

Despite a great deal of research, it is still not well-understood why trained neural networks are highly vulnerable to adversarial examples. In this work we focus on two-layer neural networks trained using data which lie on a low dimensional linear subspace. We show that standard gradient methods lead to non-robust neural networks, namely, networks which have large gradients in directions orthogonal to the data subspace, and are susceptible to small adversarial $L_2$-perturbations in these directions. Moreover, we show that decreasing the initialization scale of the training algorithm, or adding $L_2$ regularization, can make the trained network more robust to adversarial perturbations orthogonal to the data.
Communication Trade-offs in Federated Learning of Spiking Neural Networks
Authors: Soumi Chaki, David Weinberg, Ayca Özcelikkale
Date: 2023-02-27 00:00:00
Tasks: Federated Learning

Spiking Neural Networks (SNNs) are biologically inspired alternatives to conventional Artificial Neural Networks (ANNs). Despite promising preliminary results, the trade-offs in the training of SNNs in a distributed scheme are not well understood. Here, we consider SNNs in a federated learning setting where a high-quality global model is created by aggregating multiple local models from the clients without sharing any data. We investigate federated learning for training multiple SNNs at clients when two mechanisms reduce the uplink communication cost: i) random masking of the model updates sent from the clients to the server; and ii) client dropouts where some clients do not send their updates to the server. We evaluated the performance of the SNNs using a subset of the Spiking Heidelberg digits (SHD) dataset. The results show that a trade-off between the random masking and the client drop probabilities is crucial to obtain a satisfactory performance for a fixed number of clients.
Evolving Deep Neural Network by Customized Moth Flame Optimization Algorithm for Underwater Targets Recognition
Authors: Mohammad Khishe, Mokhtar Mohammadi, Tarik A. Rashid, Hoger Mahmud, Seyedali Mirjalili
Date: 2023-02-16 00:00:00

This chapter proposes using the Moth Flame Optimization (MFO) algorithm for finetuning a Deep Neural Network to recognize different underwater sonar datasets. Same as other models evolved by metaheuristic algorithms, premature convergence, trapping in local minima, and failure to converge in a reasonable time are three defects MFO confronts in solving problems with high-dimension search space. Spiral flying is the key component of the MFO as it determines how the moths adjust their positions in relation to flames; thereby, the shape of spiral motions can regulate the transition behavior between the exploration and exploitation phases. Therefore, this chapter investigates the efficiency of seven spiral motions with different curvatures and slopes in the performance of the MFO, especially for underwater target classification tasks. To assess the performance of the customized model, in addition to benchmark Sejnowski & Gorman's dataset, two experimental sonar datasets, i.e., the passive sonar and active datasets, are exploited. The results of MFO and its modifications are compared to four novel nature-inspired algorithms, including Heap-Based Optimizer (HBO), Chimp Optimization Algorithm (ChOA), Ant Lion Optimization (ALO), Stochastic Fractals Search (SFS), as well as the classic Particle Swarm Optimization (PSO). The results confirm that the customized MFO shows better performance than the other state-of-the-art models so that the classification rates are increased 1.5979, 0.9985, and 2.0879 for Sejnowski & Gorman, passive, and active datasets, respectively. The results also approve that time complexity is not significantly increased by using different spiral motions.
Stability and Machine Learning Applications of Persistent Homology Using the Delaunay-Rips Complex
Authors: Amish Mishra, Francis C. Motta
Date: 2023-03-02 00:00:00

In this paper we define, implement, and investigate a simplicial complex construction for computing persistent homology of Euclidean point cloud data, which we call the Delaunay-Rips complex (DR). Assigning the Vietoris-Rips weights to simplices, DR experiences speed-up in the persistence calculations by only considering simplices that appear in the Delaunay triangulation of the point cloud. We document and compare a Python implementation of DR with other simplicial complex constructions for generating persistence diagrams. By imposing sufficient conditions on point cloud data, we are able to theoretically justify the stability of the persistence diagrams produced using DR. When the Delaunay triangulation of the point cloud changes under perturbations of the points, we prove that DR-produced persistence diagrams exhibit instability. Since we cannot guarantee that real-world data will satisfy our stability conditions, we demonstrate the practical robustness of DR for persistent homology in comparison with other simplicial complexes in machine learning applications. We find in our experiments that using DR for an ML-TDA pipeline performs comparatively well as using other simplicial complex constructions.
Design-based conformal prediction
Authors: Jerzy Wieczorek
Date: 2023-03-02 00:00:00
Tasks: Conformal Prediction, Prediction, Prediction Intervals, Survey

Conformal prediction is an assumption-lean approach to generating distribution-free prediction intervals or sets, for nearly arbitrary predictive models, with guaranteed finite-sample coverage. Conformal methods are an active research topic in statistics and machine learning, but only recently have they been extended to non-exchangeable data. In this paper, we invite survey methodologists to begin using and contributing to conformal methods. We introduce how conformal prediction can be applied to data from several common complex sample survey designs, under a framework of design-based inference for a finite population, and we point out gaps where survey methodologists could fruitfully apply their expertise. Our simulations empirically bear out the theoretical guarantees of finite-sample coverage, and our real-data example demonstrates how conformal prediction can be applied to complex sample survey data in practice.
Discrete-time Competing-Risks Regression with or without Penalization
Authors: Tomer Meir, Malka Gorfine
Date: 2023-03-02 00:00:00
Tasks: regression, Survival Analysis

Many studies employ the analysis of time-to-event data that incorporates competing risks and right censoring. Most methods and software packages are geared towards analyzing data that comes from a continuous failure time distribution. However, failure-time data may sometimes be discrete either because time is inherently discrete or due to imprecise measurement. This paper introduces a new estimation procedure for discrete-time survival analysis with competing events. The proposed approach offers a major key advantage over existing procedures and allows for straightforward integration and application of widely used regularized regression and screening-features methods. We illustrate the benefits of our proposed approach by a comprehensive simulation study. Additionally, we showcase the utility of the proposed procedure by estimating a survival model for the length of stay of patients hospitalized in the intensive care unit, considering three competing events: discharge to home, transfer to another medical facility, and in-hospital death. A Python package, PyDTS, is available for applying the proposed method with additional features.
Continuous-Time Functional Diffusion Processes
Authors: Giulio Franzese, Giulio Corallo, Simone Rossi, Markus Heinonen, Maurizio Filippone, Pietro Michiardi
Date: 2023-03-01 00:00:00
Proceeding: NeurIPS 2023 11
Tasks: Image Generation

We introduce Functional Diffusion Processes (FDPs), which generalize score-based diffusion models to infinite-dimensional function spaces. FDPs require a new mathematical framework to describe the forward and backward dynamics, and several extensions to derive practical training objectives. These include infinite-dimensional versions of Girsanov theorem, in order to be able to compute an ELBO, and of the sampling theorem, in order to guarantee that functional evaluations in a countable set of points are equivalent to infinite-dimensional functions. We use FDPs to build a new breed of generative models in function spaces, which do not require specialized network architectures, and that can work with any kind of continuous data. Our results on real data show that FDPs achieve high-quality image generation, using a simple MLP architecture with orders of magnitude fewer parameters than existing diffusion models.
Mean-Square Analysis of Discretized Itô Diffusions for Heavy-tailed Sampling
Authors: Ye He, Tyler Farghly, Krishnakumar Balasubramanian, Murat A. Erdogdu
Date: 2023-03-01 00:00:00

We analyze the complexity of sampling from a class of heavy-tailed distributions by discretizing a natural class of It\^o diffusions associated with weighted Poincar\'e inequalities. Based on a mean-square analysis, we establish the iteration complexity for obtaining a sample whose distribution is $\epsilon$ close to the target distribution in the Wasserstein-2 metric. In this paper, our results take the mean-square analysis to its limits, i.e., we invariably only require that the target density has finite variance, the minimal requirement for a mean-square analysis. To obtain explicit estimates, we compute upper bounds on certain moments associated with heavy-tailed targets under various assumptions. We also provide similar iteration complexity results for the case where only function evaluations of the unnormalized target density are available by estimating the gradients using a Gaussian smoothing technique. We provide illustrative examples based on the multivariate $t$-distribution.
Non asymptotic analysis of Adaptive stochastic gradient algorithms and applications
Authors: Antoine Godichon-Baggioni, Pierre Tarrago
Date: 2023-03-01 00:00:00
Tasks: regression, Stochastic Optimization

In stochastic optimization, a common tool to deal sequentially with large sample is to consider the well-known stochastic gradient algorithm. Nevertheless, since the stepsequence is the same for each direction, this can lead to bad results in practice in case of ill-conditionned problem. To overcome this, adaptive gradient algorithms such that Adagrad or Stochastic Newton algorithms should be prefered. This paper is devoted to the non asymptotic analyis of these adaptive gradient algorithms for strongly convex objective. All the theoretical results will be adapted to linear regression and regularized generalized linear model for both Adagrad and Stochastic Newton algorithms.
Retrieval for Extremely Long Queries and Documents with RPRS: a Highly Efficient and Effective Transformer-based Re-Ranker
Authors: Arian Askari, Suzan Verberne, Amin Abolghasemi, Wessel Kraaij, Gabriella Pasi
Date: 2023-03-02 00:00:00
Tasks: Information Retrieval, Retrieval

Retrieval with extremely long queries and documents is a well-known and challenging task in information retrieval and is commonly known as Query-by-Document (QBD) retrieval. Specifically designed Transformer models that can handle long input sequences have not shown high effectiveness in QBD tasks in previous work. We propose a Re-Ranker based on the novel Proportional Relevance Score (RPRS) to compute the relevance score between a query and the top-k candidate documents. Our extensive evaluation shows RPRS obtains significantly better results than the state-of-the-art models on five different datasets. Furthermore, RPRS is highly efficient since all documents can be pre-processed, embedded, and indexed before query time which gives our re-ranker the advantage of having a complexity of O(N) where N is the total number of sentences in the query and candidate documents. Furthermore, our method solves the problem of the low-resource training in QBD retrieval tasks as it does not need large amounts of training data, and has only three parameters with a limited range that can be optimized with a grid search even if a small amount of labeled data is available. Our detailed analysis shows that RPRS benefits from covering the full length of candidate documents and queries.
Effective Visualization and Analysis of Recommender Systems
Authors: Hao Wang
Date: 2023-03-02 00:00:00
Tasks: Diversity, Recommendation Systems

Recommender system exists everywhere in the business world. From Goodreads to TikTok, customers of internet products become more addicted to the products thanks to the technology. Industrial practitioners focus on increasing the technical accuracy of recommender systems while at same time balancing other factors such as diversity and serendipity. In spite of the length of the research and development history of recommender systems, there has been little discussion on how to take advantage of visualization techniques to facilitate the algorithmic design of the technology. In this paper, we use a series of data analysis and visualization techniques such as Takens Embedding, Determinantal Point Process and Social Network Analysis to help people develop effective recommender systems by predicting intermediate computational cost and output performance. Our work is pioneering in the field, as to our limited knowledge, there have been few publications (if any) on visualization of recommender systems.
Heterogeneous Graph Contrastive Learning for Recommendation
Authors: Mengru Chen, Chao Huang, Lianghao Xia, Wei Wei, Yong Xu, Ronghua Luo
Date: 2023-03-02 00:00:00
Tasks: Contrastive Learning, Recommendation Systems, Relational Reasoning, Self-Supervised Learning, Transfer Learning

Graph Neural Networks (GNNs) have become powerful tools in modeling graph-structured data in recommender systems. However, real-life recommendation scenarios usually involve heterogeneous relationships (e.g., social-aware user influence, knowledge-aware item dependency) which contains fruitful information to enhance the user preference learning. In this paper, we study the problem of heterogeneous graph-enhanced relational learning for recommendation. Recently, contrastive self-supervised learning has become successful in recommendation. In light of this, we propose a Heterogeneous Graph Contrastive Learning (HGCL), which is able to incorporate heterogeneous relational semantics into the user-item interaction modeling with contrastive learning-enhanced knowledge transfer across different views. However, the influence of heterogeneous side information on interactions may vary by users and items. To move this idea forward, we enhance our heterogeneous graph contrastive learning with meta networks to allow the personalized knowledge transformer with adaptive contrastive augmentation. The experimental results on three real-world datasets demonstrate the superiority of HGCL over state-of-the-art recommendation methods. Through ablation study, key components in HGCL method are validated to benefit the recommendation performance improvement. The source code of the model implementation is available at the link https://github.com/HKUDS/HGCL.
On the Role of Reviewer Expertise in Temporal Review Helpfulness Prediction
Authors: Mir Tafseer Nayeem, Davood Rafiei
Date: 2023-02-22 00:00:00

Helpful reviews have been essential for the success of e-commerce services, as they help customers make quick purchase decisions and benefit the merchants in their sales. While many reviews are informative, others provide little value and may contain spam, excessive appraisal, or unexpected biases. With the large volume of reviews and their uneven quality, the problem of detecting helpful reviews has drawn much attention lately. Existing methods for identifying helpful reviews primarily focus on review text and ignore the two key factors of (1) who post the reviews and (2) when the reviews are posted. Moreover, the helpfulness votes suffer from scarcity for less popular products and recently submitted (a.k.a., cold-start) reviews. To address these challenges, we introduce a dataset and develop a model that integrates the reviewer's expertise, derived from the past review history of the reviewers, and the temporal dynamics of the reviews to automatically assess review helpfulness. We conduct experiments on our dataset to demonstrate the effectiveness of incorporating these factors and report improved results compared to several well-established baselines.
A System of ODEs for Representing Trends of CGM Signals
Authors: Giulia Elena Aliffi, Giovanni Nastasi, Vittorio Romano, Dario Pitocco, Alessandro Rizzi, Elvin J. Moore, Andrea De Gaetano
Date: 2023-02-27 00:00:00
Tasks: Density Estimation

Diabetes Mellitus is a metabolic disorder which may result in severe and potentially fatal complications if not well-treated and monitored. In this study, a quantitative analysis of the data collected using CGM (Continuous Glucose Monitoring) devices from eight subjects with type 2 diabetes in good metabolic control at the University Polyclinic Agostino Gemelli, Catholic University of the Sacred Heart, was carried out. In particular, a system of ordinary differential equations whose state variables are affected by a sequence of stochastic perturbations was proposed and used to extract more informative inferences from the patients' data. For this work, Matlab and R programs were used to find the most appropriate values of the parameters (according to the Akaike Information Criterion (AIC) and the Bayesian Information Criterion (BIC)) for each patient. Fitting was carried out by Particle Swarm Optimization to minimize the ordinary least squares error between the observed CGM data and the data from the ODE model. Goodness of fit tests were made in order to assess which probability distribution was best suitable for representing the waiting times computed from the model parameters. Finally, both parametric and non-parametric density estimation of the frequency histograms associated with the variability of the glucose elimination rate from blood were conducted and their representative parameters assessed from the data. The results show that the chosen models succeed in capturing most of the glucose fluctuations for almost every patient.
Subset-Based Instance Optimality in Private Estimation
Authors: Travis Dick, Alex Kulesza, Ziteng Sun, Ananda Theertha Suresh
Date: 2023-03-01 00:00:00

We propose a new definition of instance optimality for differentially private estimation algorithms. Our definition requires an optimal algorithm to compete, simultaneously for every dataset $D$, with the best private benchmark algorithm that (a) knows $D$ in advance and (b) is evaluated by its worst-case performance on large subsets of $D$. That is, the benchmark algorithm need not perform well when potentially extreme points are added to $D$; it only has to handle the removal of a small number of real data points that already exist. This makes our benchmark significantly stronger than those proposed in prior work. We nevertheless show, for real-valued datasets, how to construct private algorithms that achieve our notion of instance optimality when estimating a broad class of dataset properties, including means, quantiles, and $\ell_p$-norm minimizers. For means in particular, we provide a detailed analysis and show that our algorithm simultaneously matches or exceeds the asymptotic performance of existing algorithms under a range of distributional assumptions.
Poster: Sponge ML Model Attacks of Mobile Apps
Authors: Souvik Paul, Nicolas Kourtellis
Date: 2023-03-01 00:00:00
Tasks: Attribute, Federated Learning, Model Poisoning

Machine Learning (ML)-powered apps are used in pervasive devices such as phones, tablets, smartwatches and IoT devices. Recent advances in collaborative, distributed ML such as Federated Learning (FL) attempt to solve privacy concerns of users and data owners, and thus used by tech industry leaders such as Google, Facebook and Apple. However, FL systems and models are still vulnerable to adversarial membership and attribute inferences and model poisoning attacks, especially in FL-as-a-Service ecosystems recently proposed, which can enable attackers to access multiple ML-powered apps. In this work, we focus on the recently proposed Sponge attack: It is designed to soak up energy consumed while executing inference (not training) of ML model, without hampering the classifier's performance. Recent work has shown sponge attacks on ASCI-enabled GPUs can potentially escalate the power consumption and inference time. For the first time, in this work, we investigate this attack in the mobile setting and measure the effect it can have on ML models running inside apps on mobile devices.
Explainable Artificial Intelligence and Cybersecurity: A Systematic Literature Review
Authors: Carlos Mendes, Tatiane Nogueira Rios
Date: 2023-02-27 00:00:00
Tasks: Explainable artificial intelligence, Explainable Artificial Intelligence (XAI), Systematic Literature Review

Cybersecurity vendors consistently apply AI (Artificial Intelligence) to their solutions and many cybersecurity domains can benefit from AI technology. However, black-box AI techniques present some difficulties in comprehension and adoption by its operators, given that their decisions are not always humanly understandable (as is usually the case with deep neural networks, for example). Since it aims to make the operation of AI algorithms more interpretable for its users and developers, XAI (eXplainable Artificial Intelligence) can be used to address this issue. Through a systematic literature review, this work seeks to investigate the current research scenario on XAI applied to cybersecurity, aiming to discover which XAI techniques have been applied in cybersecurity, and which areas of cybersecurity have already benefited from this technology.