pwc-archive/papers-with-abstracts huggingface dataset:
Role of Bias Terms in Dot-Product Attention
Authors: Mahdi Namazifar, Devamanyu Hazarika, Dilek Hakkani-Tur
Date: 2023-02-16 00:00:00
Tasks: Language Modeling, Language Modelling, Natural Language Understanding, Text Generation

Dot-product attention is a core module in the present generation of neural network models, particularly transformers, and is being leveraged across numerous areas such as natural language processing and computer vision. This attention module is comprised of three linear transformations, namely query, key, and value linear transformations, each of which has a bias term. In this work, we study the role of these bias terms, and mathematically show that the bias term of the key linear transformation is redundant and could be omitted without any impact on the attention module. Moreover, we argue that the bias term of the value linear transformation has a more prominent role than that of the bias term of the query linear transformation. We empirically verify these findings through multiple experiments on language modeling, natural language understanding, and natural language generation tasks.
Adaptive Axonal Delays in feedforward spiking neural networks for accurate spoken word recognition
Authors: Pengfei Sun, Ehsan Eqlimi, Yansong Chua, Paul Devos, Dick Botteldooren
Date: 2023-02-16 00:00:00
Tasks: Audio Classification, Automatic Speech Recognition, Automatic Speech Recognition (ASR), speech-recognition, Speech Recognition

Spiking neural networks (SNN) are a promising research avenue for building accurate and efficient automatic speech recognition systems. Recent advances in audio-to-spike encoding and training algorithms enable SNN to be applied in practical tasks. Biologically-inspired SNN communicates using sparse asynchronous events. Therefore, spike-timing is critical to SNN performance. In this aspect, most works focus on training synaptic weights and few have considered delays in event transmission, namely axonal delay. In this work, we consider a learnable axonal delay capped at a maximum value, which can be adapted according to the axonal delay distribution in each network layer. We show that our proposed method achieves the best classification results reported on the SHD dataset (92.45%) and NTIDIGITS dataset (95.09%). Our work illustrates the potential of training axonal delays for tasks with complex temporal structures.
Universality laws for Gaussian mixtures in generalized linear models
Date: 2023-02-17 00:00:00
Proceeding: NeurIPS 2023 11

Let $(x_{i}, y_{i})_{i=1,\dots,n}$ denote independent samples from a general mixture distribution $\sum_{c\in\mathcal{C}}\rho_{c}P_{c}^{x}$, and consider the hypothesis class of generalized linear models $\hat{y} = F(\Theta^{\top}x)$. In this work, we investigate the asymptotic joint statistics of the family of generalized linear estimators $(\Theta_{1}, \dots, \Theta_{M})$ obtained either from (a) minimizing an empirical risk $\hat{R}_{n}(\Theta;X,y)$ or (b) sampling from the associated Gibbs measure $\exp(-\beta n \hat{R}_{n}(\Theta;X,y))$. Our main contribution is to characterize under which conditions the asymptotic joint statistics of this family depends (on a weak sense) only on the means and covariances of the class conditional features distribution $P_{c}^{x}$. In particular, this allow us to prove the universality of different quantities of interest, such as the training and generalization errors, redeeming a recent line of work in high-dimensional statistics working under the Gaussian mixture hypothesis. Finally, we discuss the applications of our results to different machine learning tasks of interest, such as ensembling and uncertainty
Multi-Behavior Graph Neural Networks for Recommender System
Authors: Lianghao Xia, Chao Huang, Yong Xu, Peng Dai, Liefeng Bo
Date: 2023-02-17 00:00:00
Tasks: Collaborative Filtering, Graph Neural Network, Recommendation Systems, TAG

Recommender systems have been demonstrated to be effective to meet user's personalized interests for many online services (e.g., E-commerce and online advertising platforms). Recent years have witnessed the emerging success of many deep learning-based recommendation models for augmenting collaborative filtering architectures with various neural network architectures, such as multi-layer perceptron and autoencoder. However, the majority of them model the user-item relationship with single type of interaction, while overlooking the diversity of user behaviors on interacting with items, which can be click, add-to-cart, tag-as-favorite and purchase. Such various types of interaction behaviors have great potential in providing rich information for understanding the user preferences. In this paper, we pay special attention on user-item relationships with the exploration of multi-typed user behaviors. Technically, we contribute a new multi-behavior graph neural network (MBRec), which specially accounts for diverse interaction patterns as well as the underlying cross-type behavior inter-dependencies. In the MBRec framework, we develop a graph-structured learning framework to perform expressive modeling of high-order connectivity in behavior-aware user-item interaction graph. After that, a mutual relation encoder is proposed to adaptively uncover complex relational structures and make aggregations across layer-specific behavior representations. Through comprehensive evaluation on real-world datasets, the advantages of our MBRec method have been validated under different experimental settings. Further analysis verifies the positive effects of incorporating the multi-behavioral context into the recommendation paradigm. Additionally, the conducted case studies offer insights into the interpretability of user multi-behavior representations.
Tensor Networks Meet Neural Networks: A Survey and Future Perspectives
Authors: Maolin Wang, Yu Pan, Zenglin Xu, Xiangli Yang, Guangxi Li, Andrzej Cichocki
Date: 2023-01-22 00:00:00
Tasks: Survey, Tensor Networks

Tensor networks (TNs) and neural networks (NNs) are two fundamental data modeling approaches. TNs were introduced to solve the curse of dimensionality in large-scale tensors by converting an exponential number of dimensions to polynomial complexity. As a result, they have attracted significant attention in the fields of quantum physics and machine learning. Meanwhile, NNs have displayed exceptional performance in various applications, e.g., computer vision, natural language processing, and robotics research. Interestingly, although these two types of networks originate from different observations, they are inherently linked through the common multilinearity structure underlying both TNs and NNs, thereby motivating a significant number of intellectual developments regarding combinations of TNs and NNs. In this paper, we refer to these combinations as tensorial neural networks (TNNs), and present an introduction to TNNs in three primary aspects: network compression, information fusion, and quantum circuit simulation. Furthermore, this survey also explores methods for improving TNNs, examines flexible toolboxes for implementing TNNs, and documents TNN development while highlighting potential future directions. To the best of our knowledge, this is the first comprehensive survey that bridges the connections among NNs, TNs, and quantum circuits. We provide a curated list of TNNs at \url{https://github.com/tnbar/awesome-tensorial-neural-networks}.
Revisiting the Assumption of Latent Separability for Backdoor Defenses
Authors: Xiangyu Qi, Tinghao Xie, Tinghao_Xie1, Yiming Li, Saeed Mahloujifar, Prateek Mittal
Date: 2023-02-01 00:00:00
Proceeding: ICLR 2023 2

Recent studies revealed that deep learning is susceptible to backdoor poisoning attacks. An adversary can embed a hidden backdoor into a model to manipulate its predictions by only modifying a few training data, without controlling the training process. Currently, a tangible signature has been widely observed across a diverse set of backdoor poisoning attacks --- models trained on a poisoned dataset tend to learn separable latent representations for poison and clean samples. This latent separation is so pervasive that a family of backdoor defenses directly take it as a default assumption (dubbed latent separability assumption), based on which to identify poison samples via cluster analysis in the latent space. An intriguing question consequently follows: is the latent separation unavoidable for backdoor poisoning attacks? This question is central to understanding whether the assumption of latent separability provides a reliable foundation for defending against backdoor poisoning attacks. In this paper, we design adaptive backdoor poisoning attacks to present counter-examples against this assumption. Our methods include two key components: (1) a set of trigger-planted samples correctly labeled to their semantic classes (other than the target class) that can regularize backdoor learning; (2) asymmetric trigger planting strategies that help to boost attack success rate (ASR) as well as to diversify latent representations of poison samples. Extensive experiments on benchmark datasets verify the effectiveness of our adaptive attacks in bypassing existing latent separation based backdoor defenses. Moreover, our attacks still maintain a high attack success rate with negligible clean accuracy drop. Our studies call for defense designers to take caution when leveraging latent separation as an assumption in their defenses.
Deep Generative Mixture Model for Robust Imbalance Classification
Authors: Xinyue Wang, Liping Jing, Yilin Lyu, Mingzhe Guo, Jiaqi Wang, Huafeng Liu, Jian Yu, and Tieyong Zeng
Date: 2023-03-01 00:00:00
Proceeding: IEEE Transactions on Pattern Analysis and Machine Intelligence 2023 3
Tasks: Classification, imbalanced classification, model

Discovering hidden pattern from imbalanced data is a critical issue in various real-world applications. Existing classification methods usually suffer from the limitation of data especially for minority classes, and result in unstable prediction and low performance. In this paper, a deep generative classifier is proposed to mitigate this issue via both model perturbation and data perturbation. Specially, the proposed generative classifier is derived from a deep latent variable model where two variables are involved. One variable is to capture the essential information of the original data, denoted as latent codes, which are represented by a probability distribution rather than a single fixed value. The learnt distribution aims to enforce the uncertainty of model and implement model perturbation, thus, lead to stable predictions. The other variable is a prior to latent codes so that the codes are restricted to lie on components in Gaussian Mixture Model. As a confounder affecting generative processes of data (feature/label), the latent variables are supposed to capture the discriminative latent distribution and implement data perturbation. Extensive experiments have been conducted on widely-used real imbalanced image datasets. Experimental results demonstrate the superiority of our proposed model by comparing with popular imbalanced classification baselines on imbalance classification task.
Low Complexity Hybrid Beamforming for mmWave Full-Duplex Integrated Access and Backhaul
Authors: Elyes Balti, Chris Dick, Brian L. Evans
Date: 2022-09-05 00:00:00
Tasks: Benchmarking

We consider an integrated access and backhaul (IAB) node operating in full-duplex (FD) mode. We analyze simultaneous transmission from the New Radio gNB to the IAB node on the backhaul uplink, IAB node to a user equipment (UE) on the access downlink, and IAB transmitter to the IAB receiver on the self-interference (SI) channel. Our contributions include (1) a low complexity algorithm to jointly design the hybrid analog/digital beamformers for all three nodes to maximize the sum spectral efficiency of the access and backhaul links by canceling SI and maximizing received power; (2) derivation of all-digital beamforming and spectral efficiency upper bound for use in benchmarking; and (3) simulations to compare full vs. half duplex modes, hybrid vs. all-digital beamforming algorithms, proposed hybrid vs. conventional beamforming algorithms, and spectral efficiency upper bound. In simulations, the proposed algorithm shows significant reduction in SI power and increase in sum spectral efficiency.
Exploratory Font Selection Using Crowdsourced Attributes
Authors: Peter O'Donovan, Jānis Lībeks, Aseem Agarwala, Aaron Hertzmann
Date: 2014-07-27 00:00:00
Proceeding: ACM Transactions on Graphics 2014 7
Tasks: Attribute, Descriptive

This paper presents interfaces for exploring large collections of fonts for design tasks. Existing interfaces typically list fonts in a long, alphabetically-sorted menu that can be challenging and frustrating to explore. We instead propose three interfaces for font selection. First, we organize fonts using high-level descriptive attributes, such as "dramatic" or "legible." Second, we organize fonts in a tree-based hierarchical menu based on perceptual similarity. Third, we display fonts that are most similar to a user's currently-selected font. These tools are complementary; a user may search for "graceful" fonts, select a reasonable one, and then refine the results from a list of fonts similar to the selection. To enable these tools, we use crowdsourcing to gather font attribute data, and then train models to predict attribute values for new fonts. We use attributes to help learn a font similarity metric using crowdsourced comparisons. We evaluate the interfaces against a conventional list interface and find that our interfaces are preferred to the baseline. Our interfaces also produce better results in two real-world tasks: finding the nearest match to a target font, and font selection for graphic designs.
Tensorized Optical Multimodal Fusion Network
Authors: Yequan Zhao, Xian Xiao, Geza Kurczveil, Raymond G. Beausoleil, Zheng Zhang
Date: 2023-02-17 00:00:00

We propose the first tensorized optical multimodal fusion network architecture with a self-attention mechanism and low-rank tensor fusion. Simulation results show $51.3 \times$ less hardware requirement and $3.7\times 10^{13}$ MAC/J energy efficiency.
DSE Stock Price Prediction using Hidden Markov Model
Authors: Raihan Tanvir, MD Tanvir Rouf Shawon, Md. Golam Rabiul Alam
Date: 2023-01-26 00:00:00
Tasks: model, Prediction, Stock Price Prediction

Stock market forecasting is a classic problem that has been thoroughly investigated using machine learning and artificial neural network based tools and techniques. Interesting aspects of this problem include its time reliance as well as its volatility and other complex relationships. To combine them, hidden markov models (HMMs) have been utilized to anticipate the price of stocks. We demonstrated the Maximum A Posteriori (MAP) HMM method for predicting stock prices for the next day based on previous data. An HMM is trained by analyzing the fractional change in the stock price as well as the intraday high and low values. It is then utilized to produce a MAP estimate across all possible stock prices for the next day. The approach demonstrated in our work is quite generalized and can be used to predict the stock price for any company, given that the HMM is trained on the dataset of that company's stocks dataset. We evaluated the accuracy of our models using some extensively used accuracy metrics for regression problems and came up with a satisfactory outcome.
Modelling and Kron reduction of power flow networks in directed graphs
Authors: Ruohan Wang, Zhiyong Sun
Date: 2023-02-17 00:00:00

Electrical grids are large-sized complex systems that require strong computing power for monitoring and analysis. Kron reduction is a general reduction method in graph theory and is often used for electrical circuit simplification. In this paper, we propose a novel formulation of the weighted Laplacian matrix for directed graphs. The proposed matrix is proved to be strictly equivalent to the conventionally formulated Laplacian matrix and is verified to well model a lossless DC power flow network in directed graphs. We as well present significant properties of the proposed weighted Laplacian and conditions of Kron reduction in directed graphs and in lossless DC power flow networks. The reduction method is verified via simulation models of IEEE-3, IEEE-5, IEEE-9, IEEE-14, and IEEE RTS-96 test systems.
Welfare Distribution in Two-sided Random Matching Markets
Authors: Itai Ashlagi, Mark Braverman, Geng Zhao
Date: 2023-02-16 00:00:00
Tasks: Vocal Bursts Valence Prediction

We study the welfare structure in two-sided large random matching markets. In the model, each agent has a latent personal score for every agent on the other side of the market and her preferences follow a logit model based on these scores. Under a contiguity condition, we provide a tight description of stable outcomes. First, we identify an intrinsic fitness for each agent that represents her relative competitiveness in the market, independent of the realized stable outcome. The intrinsic fitness values correspond to scaling coefficients needed to make a latent mutual matrix bi-stochastic, where the latent scores can be interpreted as a-priori probabilities of a pair being matched. Second, in every stable (or even approximately stable) matching, the welfare or the ranks of the agents on each side of the market, when scaled by their intrinsic fitness, have an approximately exponential empirical distribution. Moreover, the average welfare of agents on one side of the market is sufficient to determine the average on the other side. Overall, each agent's welfare is determined by a global parameter, her intrinsic fitness, and an extrinsic factor with exponential distribution across the population.
OTFS Signaling for SCMA With Coordinated Multi-Point Vehicle Communications
Authors: Yao Ge, Qinwen Deng, David González G., Yong Liang Guan, Zhi Ding
Date: 2023-02-17 00:00:00
Tasks: Diversity

This paper investigates an uplink coordinated multi-point (CoMP) coverage scenario, in which multiple mobile users are grouped for sparse code multiple access (SCMA), and served by the remote radio head (RRH) in front of them and the RRH behind them simultaneously. We apply orthogonal time frequency space (OTFS) modulation for each user to exploit the degrees of freedom arising from both the delay and Doppler domains. As the signals received by the RRHs in front of and behind the users experience respectively positive and negative Doppler frequency shifts, our proposed OTFS-based SCMA (OBSCMA) with CoMP system can effectively harvest extra Doppler and spatial diversity for better performance. Based on maximum likelihood (ML) detector, we analyze the single-user average bit error rate (ABER) bound as the benchmark of the ABER performance for our proposed OBSCMA with CoMP system. We also develop a customized Gaussian approximation with expectation propagation (GAEP) algorithm for multi-user detection and propose efficient algorithm structures for centralized and decentralized detectors. Our proposed OBSCMA with CoMP system leads to stronger performance than the existing solutions. The proposed centralized and decentralized detectors exhibit effective reception and robustness under channel state information uncertainty.
OTFS -- Predictability in the Delay-Doppler Domain and its Value to Communication and Radar Sensing
Authors: Saif Khan Mohammed, Ronny Hadani, Ananthanarayanan Chockalingam, Robert Calderbank
Date: 2023-02-17 00:00:00
Tasks: Relation

In our first paper [2] we explained why the Zak-OTFS input-output (I/O) relation is predictable and non-fading when the delay and Doppler periods are greater than the effective channel delay and Doppler spreads, a condition which we refer to as the crystallization condition. We argued that a communication system should operate within the crystalline regime. It is well known that it is possible to identify a linear time varying (LTV) channel if and only if it is under-spread. The crystallization condition is more restrictive than the under-spread condition, so identification is always possible. In the crystalline regime, we show that Zak-OTFS pilot sequences minimize the complexity of identifying the effective DD domain channel filter. We demonstrate that the filter taps can simply be read off from the response to a single Zak-OTFS pilot. In general, we provide an explicit formula for reconstructing the Zak-OTFS I/O relation from a finite number of received pilot symbols in the delay-Doppler (DD) domain. This reconstruction formula makes it possible to study predictability of the Zak-OTFS I/O relation for a sampled system that operates under finite duration and bandwidth constraints. We analyze reconstruction accuracy for different choices of the delay and Doppler periods, and of the pulse shaping filter. Reconstruction accuracy is high when the crystallization condition is satisfied, implying that it is possible to learn directly the I/O relation without needing to estimate the underlying channel. This opens up the possibility of a model-free mode of operation, which is especially useful when a traditional model-dependent mode of operation (reliant on estimation of the underlying physical channel) is out of reach (for example, when the channel comprises of unresolvable paths, or exhibits a continuous delay-Doppler profile such as in presence of acceleration). Our study clarifies the
OTFS -- A Mathematical Foundation for Communication and Radar Sensing in the Delay-Doppler Domain
Authors: Saif Khan Mohammed, Ronny Hadani, Ananthanarayanan Chockalingam, Robert Calderbank
Date: 2023-02-17 00:00:00
Tasks: Self-Driving Cars

Orthogonal time frequency space (OTFS) is a framework for communication and active sensing that processes signals in the delay-Doppler (DD) domain. This paper explores three key features of the OTFS framework, and explains their value to applications. The first feature is a compact and sparse DD domain parameterization of the wireless channel, where the parameters map directly to physical attributes of the reflectors that comprise the scattering environment, and as a consequence these parameters evolve predictably. The second feature is a novel waveform / modulation technique, matched to the DD channel model, that embeds information symbols in the DD domain. The relation between channel inputs and outputs is localized, non-fading and predictable, even in the presence of significant delay and Doppler spread, and as a consequence the channel can be efficiently acquired and equalized. By avoiding fading, the post equalization SNR remains constant across all information symbols in a packet, so that bit error performance is superior to contemporary multi-carrier waveforms. Further, the OTFS carrier waveform is a localized pulse in the DD domain, making it possible to separate reflectors along both delay and Doppler simultaneously, and to achieve a high-resolution delay-Doppler radar image of the environment. In other words, the DD parameterization provides a common mathematical framework for communication and radar. This is the third feature of the OTFS framework, and it is ideally suited to intelligent transportation systems involving self-driving cars and unmanned ground/aerial vehicles which are self/network controlled. The OTFS waveform is able to support stable and superior performance over a wide range of user speeds.