pwc-archive/papers-with-abstracts huggingface dataset:
Sentences as connection paths: A neural language architecture of sentence structure in the brain
Authors: Frank van der Velde
Date: 2022-05-19 00:00:00
Tasks: Question Answering, Sentence

This article presents a neural language architecture of sentence structure in the brain, in which sentences are temporal connection paths that interconnect neural structures underlying their words. Words remain 'in-situ', hence they are always content-addressable. Arbitrary and novel sentences (with novel words) can be created with 'neural blackboards' for words and sentences. Hence, the unlimited productivity of natural language can be achieved with a 'fixed' small world like network structure. The article focuses on the neural blackboard for sentences. The architecture uses only one 'connection matrix' for binding all structural relations between words in sentences. Its ability to represent arbitrary (English) sentences is discussed in detail, based on a comprehensive analysis of them. The architecture simulates intra-cranial brain activity observed during sentence processing and fMRI observations related to sentence complexity and ambiguity. The simulations indicate that the observed effects relate to global control over the architecture, not to the sentence structures involved, which predicts higher activity differences related to complexity and ambiguity with higher comprehension capacity. Other aspects discussed are the 'intrinsic' sentence structures provided by connection paths and their relation to scope and inflection, the use of a dependency parser for control of binding, long-distance dependencies and gaps, question answering, ambiguity resolution based on backward processing without explicit backtracking, garden paths, and performance difficulties related to embeddings.
Programming molecular systems to emulate a learning spiking neuron
Authors: Jakub Fil, Neil Dalchau, Dominique Chu
Date: 2022-05-09 00:00:00

Hebbian theory seeks to explain how the neurons in the brain adapt to stimuli, to enable learning. An interesting feature of Hebbian learning is that it is an unsupervised method and as such, does not require feedback, making it suitable in contexts where systems have to learn autonomously. This paper explores how molecular systems can be designed to show such proto-intelligent behaviours, and proposes the first chemical reaction network (CRN) that can exhibit autonomous Hebbian learning across arbitrarily many input channels. The system emulates a spiking neuron, and we demonstrate that it can learn statistical biases of incoming inputs. The basic CRN is a minimal, thermodynamically plausible set of micro-reversible chemical equations that can be analysed with respect to their energy requirements. However, to explore how such chemical systems might be engineered de novo, we also propose an extended version based on enzyme-driven compartmentalised reactions. Finally, we also show how a purely DNA system, built upon the paradigm of DNA strand displacement, can realise neuronal dynamics. Our analysis provides a compelling blueprint for exploring autonomous learning in biological settings, bringing us closer to realising real synthetic biological intelligence.
Unifying Summary Statistic Selection for Approximate Bayesian Computation
Authors: Till Hoffmann, Jukka-Pekka Onnela
Date: 2022-06-06 00:00:00
Tasks: Bayesian Inference, Dimensionality Reduction, Model Selection

Extracting low-dimensional summary statistics from large datasets is essential for efficient (likelihood-free) inference. We characterize different classes of summaries and demonstrate their importance for correctly analysing dimensionality reduction algorithms. We demonstrate that minimizing the expected posterior entropy (EPE) under the prior predictive distribution of the model subsumes many existing methods. They are equivalent to or are special or limiting cases of minimizing the EPE. We offer a unifying framework for obtaining informative summaries, provide concrete recommendations for practitioners, and propose a practical method to obtain high-fidelity summaries whose utility we demonstrate for both benchmark and practical examples.
Constructing unbiased gradient estimators with finite variance for conditional stochastic optimization
Authors: Takashi Goda, Wataru Kitade
Date: 2022-06-04 00:00:00
Tasks: Stochastic Optimization

We study stochastic gradient descent for solving conditional stochastic optimization problems, in which an objective to be minimized is given by a parametric nested expectation with an outer expectation taken with respect to one random variable and an inner conditional expectation with respect to the other random variable. The gradient of such a parametric nested expectation is again expressed as a nested expectation, which makes it hard for the standard nested Monte Carlo estimator to be unbiased. In this paper, we show under some conditions that a multilevel Monte Carlo gradient estimator is unbiased and has finite variance and finite expected computational cost, so that the standard theory from stochastic optimization for a parametric (non-nested) expectation directly applies. We also discuss a special case for which yet another unbiased gradient estimator with finite variance and cost can be constructed.
A Neural Corpus Indexer for Document Retrieval
Authors: Yujing Wang, Yingyan Hou, Haonan Wang, Ziming Miao, Shibin Wu, Hao Sun, Qi Chen, Yuqing Xia, Chengmin Chi, Guoshuai Zhao, Zheng Liu, Xing Xie, Hao Allen Sun, Weiwei Deng, Qi Zhang, Mao Yang
Date: 2022-06-06 00:00:00
Tasks: Decoder, Retrieval, TriviaQA

Current state-of-the-art document retrieval solutions mainly follow an index-retrieve paradigm, where the index is hard to be directly optimized for the final retrieval target. In this paper, we aim to show that an end-to-end deep neural network unifying training and indexing stages can significantly improve the recall performance of traditional methods. To this end, we propose Neural Corpus Indexer (NCI), a sequence-to-sequence network that generates relevant document identifiers directly for a designated query. To optimize the recall performance of NCI, we invent a prefix-aware weight-adaptive decoder architecture, and leverage tailored techniques including query generation, semantic document identifiers, and consistency-based regularization. Empirical studies demonstrated the superiority of NCI on two commonly used academic benchmarks, achieving +21.4% and +16.8% relative enhancement for Recall@1 on NQ320k dataset and R-Precision on TriviaQA dataset, respectively, compared to the best baseline method.
Offline Evaluation of Ranked Lists using Parametric Estimation of Propensities
Authors: Vishwa Vinay, Manoj Kilaru, David Arbour
Date: 2022-06-06 00:00:00
Tasks: Learning-To-Rank, Recommendation Systems

Search engines and recommendation systems attempt to continually improve the quality of the experience they afford to their users. Refining the ranker that produces the lists displayed in response to user requests is an important component of this process. A common practice is for the service providers to make changes (e.g. new ranking features, different ranking models) and A/B test them on a fraction of their users to establish the value of the change. An alternative approach estimates the effectiveness of the proposed changes offline, utilising previously collected clickthrough data on the old ranker to posit what the user behaviour on ranked lists produced by the new ranker would have been. A majority of offline evaluation approaches invoke the well studied inverse propensity weighting to adjust for biases inherent in logged data. In this paper, we propose the use of parametric estimates for these propensities. Specifically, by leveraging well known learning-to-rank methods as subroutines, we show how accurate offline evaluation can be achieved when the new rankings to be evaluated differ from the logged ones.
ID-Agnostic User Behavior Pre-training for Sequential Recommendation
Authors: Shanlei Mu, Yupeng Hou, Wayne Xin Zhao, Yaliang Li, Bolin Ding
Date: 2022-06-06 00:00:00
Tasks: Attribute, Language Modeling, Language Modelling, Sequential Recommendation

Recently, sequential recommendation has emerged as a widely studied topic. Existing researches mainly design effective neural architectures to model user behavior sequences based on item IDs. However, this kind of approach highly relies on user-item interaction data and neglects the attribute- or characteristic-level correlations among similar items preferred by a user. In light of these issues, we propose IDA-SR, which stands for ID-Agnostic User Behavior Pre-training approach for Sequential Recommendation. Instead of explicitly learning representations for item IDs, IDA-SR directly learns item representations from rich text information. To bridge the gap between text semantics and sequential user behaviors, we utilize the pre-trained language model as text encoder, and conduct a pre-training architecture on the sequential user behaviors. In this way, item text can be directly utilized for sequential recommendation without relying on item IDs. Extensive experiments show that the proposed approach can achieve comparable results when only using ID-agnostic item representations, and performs better than baselines by a large margin when fine-tuned with ID information.
Learning Binarized Graph Representations with Multi-faceted Quantization Reinforcement for Top-K Recommendation
Authors: Yankai Chen, Huifeng Guo, Yingxue Zhang, Chen Ma, Ruiming Tang, Jingjie Li, Irwin King
Date: 2022-06-05 00:00:00
Tasks: Binarization, Informativeness, Quantization, Recommendation Systems, Representation Learning

Learning vectorized embeddings is at the core of various recommender systems for user-item matching. To perform efficient online inference, representation quantization, aiming to embed the latent features by a compact sequence of discrete numbers, recently shows the promising potentiality in optimizing both memory and computation overheads. However, existing work merely focuses on numerical quantization whilst ignoring the concomitant information loss issue, which, consequently, leads to conspicuous performance degradation. In this paper, we propose a novel quantization framework to learn Binarized Graph Representations for Top-K Recommendation (BiGeaR). BiGeaR introduces multi-faceted quantization reinforcement at the pre-, mid-, and post-stage of binarized representation learning, which substantially retains the representation informativeness against embedding binarization. In addition to saving the memory footprint, BiGeaR further develops solid online inference acceleration with bitwise operations, providing alternative flexibility for the realistic deployment. The empirical results over five large real-world benchmarks show that BiGeaR achieves about 22%~40% performance improvement over the state-of-the-art quantization-based recommender system, and recovers about 95%~102% of the performance capability of the best full-precision counterpart with over 8x time and space reduction.
Soft Retargeting Network for Click Through Rate Prediction
Authors: Xiaochen Li, Xin Song, Pengjia Yuan, Xialong Liu, Yu Zhang
Date: 2022-06-04 00:00:00
Tasks: Click-Through Rate Prediction, Graph Embedding, Prediction

The study of user interest models has received a great deal of attention in click through rate (CTR) prediction recently. These models aim at capturing user interest from different perspectives, including user interest evolution, session interest, multiple interests, etc. In this paper, we focus on a new type of user interest, i.e., user retargeting interest. User retargeting interest is defined as user's click interest on target items the same as or similar to historical click items. We propose a novel soft retargeting network (SRN) to model this specific interest. Specifically, we first calculate the similarity between target item and each historical item with the help of graph embedding. Then we learn to aggregate the similarity weights to measure the extent of user's click interest on target item. Furthermore, we model the evolution of user retargeting interest. Experimental results on public datasets and industrial dataset demonstrate that our model achieves significant improvements over state-of-the-art models.
Prospective Preference Enhanced Mixed Attentive Model for Session-based Recommendation
Authors: Bo Peng, Chang-Yu Tai, Srinivasan Parthasarathy, Xia Ning
Date: 2022-06-04 00:00:00
Tasks: Position, Session-Based Recommendations

Session-based recommendation aims to generate recommendations for the next item of users' interest based on a given session. In this manuscript, we develop prospective preference enhanced mixed attentive model (P2MAM) to generate session-based recommendations using two important factors: temporal patterns and estimates of users' prospective preferences. Unlike existing methods, P2MAM models the temporal patterns using a light-weight while effective position-sensitive attention mechanism. In P2MAM, we also leverage the estimate of users' prospective preferences to signify important items, and generate better recommendations. Our experimental results demonstrate that P2MAM models significantly outperform the state-of-the-art methods in six benchmark datasets, with an improvement as much as 19.2%. In addition, our run-time performance comparison demonstrates that during testing, P2MAM models are much more efficient than the best baseline method, with a significant average speedup of 47.7 folds.
DCAN: Diversified News Recommendation with Coverage-Attentive Networks
Authors: Hao Shi, Zi-Jiao Wang, Lan-Ru Zhai
Date: 2022-06-02 00:00:00
Tasks: Diversity, News Recommendation

Self-attention based models are widely used in news recommendation tasks. However, previous Attention architecture does not constrain repeated information in the user's historical behavior, which limits the power of hidden representation and leads to some problems such as information redundancy and filter bubbles. To solve this problem, we propose a personalized news recommendation model called DCAN.It captures multi-grained user-news matching signals through news encoders and user encoders. We keep updating a coverage vector to track the history of news attention and augment the vector in 4 types of ways. Then we fed the augmented Coverage vector into the Multi-headed Self-attention model to help adjust the future attention and added the Coverage regulation to the loss function(CRL), which enabled the recommendation system to consider more about differentiated information. Extensive experiments on Microsoft News Recommendation Dataset (MIND) show that our model significantly improve the diversity of news recommendations with minimal sacrifice in accuracy.
ResAct: Reinforcing Long-term Engagement in Sequential Recommendation with Residual Actor
Authors: Wanqi Xue, Qingpeng Cai, Ruohan Zhan, Dong Zheng, Peng Jiang, Kun Gai, Bo An
Date: 2022-06-01 00:00:00
Tasks: Reinforcement Learning (RL), Sequential Recommendation

Long-term engagement is preferred over immediate engagement in sequential recommendation as it directly affects product operational metrics such as daily active users (DAUs) and dwell time. Meanwhile, reinforcement learning (RL) is widely regarded as a promising framework for optimizing long-term engagement in sequential recommendation. However, due to expensive online interactions, it is very difficult for RL algorithms to perform state-action value estimation, exploration and feature extraction when optimizing long-term engagement. In this paper, we propose ResAct which seeks a policy that is close to, but better than, the online-serving policy. In this way, we can collect sufficient data near the learned policy so that state-action values can be properly estimated, and there is no need to perform online exploration. ResAct optimizes the policy by first reconstructing the online behaviors and then improving it via a Residual Actor. To extract long-term information, ResAct utilizes two information-theoretical regularizers to confirm the expressiveness and conciseness of features. We conduct experiments on a benchmark dataset and a large-scale industrial dataset which consists of tens of millions of recommendation requests. Experimental results show that our method significantly outperforms the state-of-the-art baselines in various long-term engagement optimization tasks.
Improving Ads-Profitability Using Traffic-Fingerprints
Authors: Adam Gabriel Dobrakowski, Andrzej Pacuk, Piotr Sankowski, Marcin Mucha, Paweł Brach
Date: 2022-05-31 00:00:00
Tasks: Blocking, Clustering

This paper introduces the concept of traffic-fingerprints, i.e., normalized 24-dimensional vectors representing a distribution of daily traffic on a web page. Using k-means clustering we show that similarity of traffic-fingerprints is related to the similarity of profitability time patterns for ads shown on these pages. In other words, these fingerprints are correlated with the conversions rates, thus allowing us to argue about conversion rates on pages with negligible traffic. By blocking or unblocking whole clusters of pages we were able to increase the revenue of online campaigns by more than 50%.
Contemporary Recommendation Systems on Big Data and Their Applications: A Survey
Authors: Ziyuan Xia, Anchen Sun, Jingyi Xu, Yuanzhe Peng, Rui Ma, Minghui Cheng
Date: 2022-05-31 00:00:00
Tasks: Collaborative Filtering, Recommendation Systems, Survey

This survey paper conducts a comprehensive analysis of the evolution and contemporary landscape of recommendation systems, which have been extensively incorporated across a myriad of web applications. It delves into the progression of personalized recommendation methodologies tailored for online products or services, organizing the array of recommendation techniques into four main categories: content-based, collaborative filtering, knowledge-based, and hybrid approaches, each designed to cater to specific contexts. The document provides an in-depth review of both the historical underpinnings and the cutting-edge innovations in the domain of recommendation systems, with a special focus on implementations leveraging big data analytics. The paper also highlights the utilization of prominent datasets such as MovieLens, Amazon Reviews, Netflix Prize, Last.fm, and Yelp in evaluating recommendation algorithms. It further outlines and explores the predominant challenges encountered in the current generation of recommendation systems, including issues related to data sparsity, scalability, and the imperative for diversified recommendation outputs. The survey underscores these challenges as promising directions for subsequent research endeavors within the discipline. Additionally, the paper examines various real-life applications driven by recommendation systems, addressing the hurdles involved in seamlessly integrating these systems into everyday life. Ultimately, the survey underscores how the advancements in recommendation systems, propelled by big data technologies, have the potential to significantly enhance real-world experiences.
Towards Fair Federated Recommendation Learning: Characterizing the Inter-Dependence of System and Data Heterogeneity
Authors: Kiwan Maeng, Haiyu Lu, Luca Melis, John Nguyen, Mike Rabbat, Carole-Jean Wu
Date: 2022-05-30 00:00:00
Tasks: Fairness, Federated Learning, Model Optimization, Recommendation Systems

Federated learning (FL) is an effective mechanism for data privacy in recommender systems by running machine learning model training on-device. While prior FL optimizations tackled the data and system heterogeneity challenges faced by FL, they assume the two are independent of each other. This fundamental assumption is not reflective of real-world, large-scale recommender systems -- data and system heterogeneity are tightly intertwined. This paper takes a data-driven approach to show the inter-dependence of data and system heterogeneity in real-world data and quantifies its impact on the overall model quality and fairness. We design a framework, RF^2, to model the inter-dependence and evaluate its impact on state-of-the-art model optimization techniques for federated recommendation tasks. We demonstrate that the impact on fairness can be severe under realistic heterogeneity scenarios, by up to 15.8--41x compared to a simple setup assumed in most (if not all) prior work. It means when realistic system-induced data heterogeneity is not properly modeled, the fairness impact of an optimization can be downplayed by up to 41x. The result shows that modeling realistic system-induced data heterogeneity is essential to achieving fair federated recommendation learning. We plan to open-source RF^2 to enable future design and evaluation of FL innovations.
Package Recommendation with Intra- and Inter-Package Attention Networks
Authors: Chen Li, Yuanfu Lu, Wei Wang
Date: 2021-07-11 00:00:00
Proceeding: SIGIR 2021 7
Tasks: Recommendation Systems

With the booming of online social networks in the mobile internet,
an emerging recommendation scenario has played a vital role in
information acquisition for user, where users are no longer recommended with a single item or item list, but a combination of
heterogeneous and diverse objects (called a package, e.g., a package
including news, publisher, and friends viewing the news). Different
from the conventional recommendation where users are recommended with the item itself, in package recommendation, users
would show great interests on the explicitly displayed objects that
could have a significant influence on the user behaviors. However,
to the best of our knowledge, few effort has been made for package recommendation and existing approaches can hardly model
the complex interactions of diverse objects in a package. Thus, in
this paper, we make a first study on package recommendation and
propose an Intra- and inter-package attention network for Package Recommendation (IPRec). Specifically, for package modeling,
an intra-package attention network is put forward to capture the
object-level intention of user interacting with the package, while
an inter-package attention network acts as a package-level information encoder that captures collaborative features of neighboring packages. In addition, to capture users preference representation,
we present a user preference learner equipped with a fine-grained
feature aggregation network and coarse-grained package aggregation network. Extensive experiments on three real-world datasets
demonstrate that IPRec significantly outperforms the state of the
arts. Moreover, the model analysis demonstrates the interpretability
of our IPRec and the characteristics of user behaviors. Codes and
datasets can be obtained at https://github.com/LeeChenChen/IPRec.
DPSRGAN: Dilation Patch Super-Resolution Generative Adversarial Networks
Authors: Kapil Mirchandani, Kushal Chordiya
Date: 2021-05-10 00:00:00
Proceeding: 6th International Conference for Convergence in Technology (I2CT) 2021 5
Tasks: Image Super-Resolution, Super-Resolution

Single Image Super-Resolution (SISR) has proven itself as a highly challenging and ill-posed problem. Multiple methods have been applied to this problem in the past, with varying degrees of success. Recently, methods using deep learning such as Generative Adversarial Networks (GAN) and Variational Auto-Encoders (VAE) in particular have proven to be extremely effective. However, most of the present methods either create a blurry output, lacking fine details, or use extremely heavy models to achieve better results. We introduce a novel, lightweight GAN architecture for 4× super-resolution of images, which builds on previous methods, showing high quality of features both qualitatively and quantitatively. To achieve this, we use dilated convolutions in our generator architecture, a Markovian discriminator, a modified loss function and a training process more typical of a conditional GAN (cGAN). For testing our results qualitatively, we use Mean Opinion Score (MOS). The obtained MOS show the effectiveness of our model at generating visually superior images. Our code is available at https://www.github.com/kushalchordiya216/Super-Resolution.