pwc-archive/papers-with-abstracts huggingface dataset:
Hardware Aware Evolutionary Neural Architecture Search using Representation Similarity Metric
Authors: Nilotpal Sinha, Abd El Rahman Shabayek, Anis Kacem, Peyman Rostami, Carl Shneider, Djamila Aouada
Date: 2023-11-07 00:00:00
Tasks: Hardware Aware Neural Architecture Search, Neural Architecture Search

Hardware-aware Neural Architecture Search (HW-NAS) is a technique used to automatically design the architecture of a neural network for a specific task and target hardware. However, evaluating the performance of candidate architectures is a key challenge in HW-NAS, as it requires significant computational resources. To address this challenge, we propose an efficient hardware-aware evolution-based NAS approach called HW-EvRSNAS. Our approach re-frames the neural architecture search problem as finding an architecture with performance similar to that of a reference model for a target hardware, while adhering to a cost constraint for that hardware. This is achieved through a representation similarity metric known as Representation Mutual Information (RMI) employed as a proxy performance evaluator. It measures the mutual information between the hidden layer representations of a reference model and those of sampled architectures using a single training batch. We also use a penalty term that penalizes the search process in proportion to how far an architecture's hardware cost is from the desired hardware cost threshold. This resulted in a significantly reduced search time compared to the literature that reached up to 8000x speedups resulting in lower CO2 emissions. The proposed approach is evaluated on two different search spaces while using lower computational resources. Furthermore, our approach is thoroughly examined on six different edge devices under various hardware cost constraints.
Hilbert's projective metric for functions of bounded growth and exponential convergence of Sinkhorn's algorithm
Authors: Stephan Eckstein
Date: 2023-11-07 00:00:00

Motivated by the entropic optimal transport problem in unbounded settings, we study versions of Hilbert's projective metric for spaces of integrable functions of bounded growth. These versions of Hilbert's metric originate from cones which are relaxations of the cone of all non-negative functions, in the sense that they include all functions having non-negative integral values when multiplied with certain test functions. We show that kernel integral operators are contractions with respect to suitable specifications of such metrics even for kernels which are not bounded away from zero, provided that the decay to zero of the kernel is controlled. As an application to entropic optimal transport, we show exponential convergence of Sinkhorn's algorithm in settings where the marginal distributions have sufficiently light tails compared to the growth of the cost function.
On the Impact of Overparameterization on the Training of a Shallow Neural Network in High Dimensions
Authors: Simon Martin, Francis Bach, Giulio Biroli
Date: 2023-11-07 00:00:00

We study the training dynamics of a shallow neural network with quadratic activation functions and quadratic cost in a teacher-student setup. In line with previous works on the same neural architecture, the optimization is performed following the gradient flow on the population risk, where the average over data points is replaced by the expectation over their distribution, assumed to be Gaussian.We first derive convergence properties for the gradient flow and quantify the overparameterization that is necessary to achieve a strong signal recovery. Then, assuming that the teachers and the students at initialization form independent orthonormal families, we derive a high-dimensional limit for the flow and show that the minimal overparameterization is sufficient for strong recovery. We verify by numerical experiments that these results hold for more general initializations.
Bridging the Information Gap Between Domain-Specific Model and General LLM for Personalized Recommendation
Authors: Wenxuan Zhang, Hongzhi Liu, Yingpeng Du, Chen Zhu, Yang song, HengShu Zhu, Zhonghai Wu
Date: 2023-11-07 00:00:00

Generative large language models(LLMs) are proficient in solving general problems but often struggle to handle domain-specific tasks. This is because most of domain-specific tasks, such as personalized recommendation, rely on task-related information for optimal performance. Current methods attempt to supplement task-related information to LLMs by designing appropriate prompts or employing supervised fine-tuning techniques. Nevertheless, these methods encounter the certain issue that information such as community behavior pattern in RS domain is challenging to express in natural language, which limits the capability of LLMs to surpass state-of-the-art domain-specific models. On the other hand, domain-specific models for personalized recommendation which mainly rely on user interactions are susceptible to data sparsity due to their limited common knowledge capabilities. To address these issues, we proposes a method to bridge the information gap between the domain-specific models and the general large language models. Specifically, we propose an information sharing module which serves as an information storage mechanism and also acts as a bridge for collaborative training between the LLMs and domain-specific models. By doing so, we can improve the performance of LLM-based recommendation with the help of user behavior pattern information mined by domain-specific models. On the other hand, the recommendation performance of domain-specific models can also be improved with the help of common knowledge learned by LLMs. Experimental results on three real-world datasets have demonstrated the effectiveness of the proposed method.
Large Language Model based Long-tail Query Rewriting in Taobao Search
Authors: Wenjun Peng, Guiyang Li, Yue Jiang, Zilong Wang, Dan Ou, Xiaoyi Zeng, Derong Xu, Tong Xu, Enhong Chen
Date: 2023-11-07 00:00:00
Tasks: Contrastive Learning, Language Modeling, Language Modelling, Large Language Model, Retrieval

In the realm of e-commerce search, the significance of semantic matching cannot be overstated, as it directly impacts both user experience and company revenue. Along this line, query rewriting, serving as an important technique to bridge the semantic gaps inherent in the semantic matching process, has attached wide attention from the industry and academia. However, existing query rewriting methods often struggle to effectively optimize long-tail queries and alleviate the phenomenon of "few-recall" caused by semantic gap. In this paper, we present BEQUE, a comprehensive framework that Bridges the sEmantic gap for long-tail QUEries. In detail, BEQUE comprises three stages: multi-instruction supervised fine tuning (SFT), offline feedback, and objective alignment. We first construct a rewriting dataset based on rejection sampling and auxiliary tasks mixing to fine-tune our large language model (LLM) in a supervised fashion. Subsequently, with the well-trained LLM, we employ beam search to generate multiple candidate rewrites, and feed them into Taobao offline system to obtain the partial order. Leveraging the partial order of rewrites, we introduce a contrastive learning method to highlight the distinctions between rewrites, and align the model with the Taobao online objectives. Offline experiments prove the effectiveness of our method in bridging semantic gap. Online A/B tests reveal that our method can significantly boost gross merchandise volume (GMV), number of transaction (#Trans) and unique visitor (UV) for long-tail queries. BEQUE has been deployed on Taobao, one of most popular online shopping platforms in China, since October 2023.
Towards Automated Negative Sampling in Implicit Recommendation
Authors: Fuyuan Lyu, Yaochen Hu, Xing Tang, Yingxue Zhang, Ruiming Tang, Xue Liu
Date: 2023-11-06 00:00:00
Tasks: AutoML

Negative sampling methods are vital in implicit recommendation models as they allow us to obtain negative instances from massive unlabeled data. Most existing approaches focus on sampling hard negative samples in various ways. These studies are orthogonal to the recommendation model and implicit datasets. However, such an idea contradicts the common belief in AutoML that the model and dataset should be matched. Empirical experiments suggest that the best-performing negative sampler depends on the implicit dataset and the specific recommendation model. Hence, we propose a hypothesis that the negative sampler should align with the capacity of the recommendation models as well as the statistics of the datasets to achieve optimal performance. A mismatch between these three would result in sub-optimal outcomes. An intuitive idea to address the mismatch problem is to exhaustively select the best-performing negative sampler given the model and dataset. However, such an approach is computationally expensive and time-consuming, leaving the problem unsolved. In this work, we propose the AutoSample framework that adaptively selects the best-performing negative sampler among candidates. Specifically, we propose a loss-to-instance approximation to transform the negative sampler search task into the learning task over a weighted sum, enabling end-to-end training of the model. We also designed an adaptive search algorithm to extensively and efficiently explore the search space. A specific initialization approach is also obtained to better utilize the obtained model parameters during the search stage, which is similar to curriculum learning and leads to better performance and less computation resource consumption. We evaluate the proposed framework on four benchmarks over three models. Extensive experiments demonstrate the effectiveness and efficiency of our proposed framework.
Enhancing AI Research Paper Analysis: Methodology Component Extraction using Factored Transformer-based Sequence Modeling Approach
Authors: Madhusudan Ghosh, Debasis Ganguly, Partha Basuchowdhuri, Sudip Kumar Naskar
Date: 2023-11-05 00:00:00

Research in scientific disciplines evolves, often rapidly, over time with the emergence of novel methodologies and their associated terminologies. While methodologies themselves being conceptual in nature and rather difficult to automatically extract and characterise, in this paper, we seek to develop supervised models for automatic extraction of the names of the various constituents of a methodology, e.g., `R-CNN', `ELMo' etc. The main research challenge for this task is effectively modeling the contexts around these methodology component names in a few-shot or even a zero-shot setting. The main contributions of this paper towards effectively identifying new evolving scientific methodology names are as follows: i) we propose a factored approach to sequence modeling, which leverages a broad-level category information of methodology domains, e.g., `NLP', `RL' etc.; ii) to demonstrate the feasibility of our proposed approach of identifying methodology component names under a practical setting of fast evolving AI literature, we conduct experiments following a simulated chronological setup (newer methodologies not seen during the training process); iii) our experiments demonstrate that the factored approach outperforms state-of-the-art baselines by margins of up to 9.257\% for the methodology extraction task with the few-shot setup.
Causal Structure Representation Learning of Confounders in Latent Space for Recommendation
Authors: Hangtong Xu, Yuanbo Xu, Chaozhuo Li, Fuzhen Zhuang
Date: 2023-11-02 00:00:00
Tasks: Recommendation Systems, Representation Learning, Specificity

Inferring user preferences from the historical feedback of users is a valuable problem in recommender systems. Conventional approaches often rely on the assumption that user preferences in the feedback data are equivalent to the real user preferences without additional noise, which simplifies the problem modeling. However, there are various confounders during user-item interactions, such as weather and even the recommendation system itself. Therefore, neglecting the influence of confounders will result in inaccurate user preferences and suboptimal performance of the model. Furthermore, the unobservability of confounders poses a challenge in further addressing the problem. To address these issues, we refine the problem and propose a more rational solution. Specifically, we consider the influence of confounders, disentangle them from user preferences in the latent space, and employ causal graphs to model their interdependencies without specific labels. By cleverly combining local and global causal graphs, we capture the user-specificity of confounders on user preferences. We theoretically demonstrate the identifiability of the obtained causal graph. Finally, we propose our model based on Variational Autoencoders, named Causal Structure representation learning of Confounders in latent space (CSC). We conducted extensive experiments on one synthetic dataset and five real-world datasets, demonstrating the superiority of our model. Furthermore, we demonstrate that the learned causal representations of confounders are controllable, potentially offering users fine-grained control over the objectives of their recommendation lists with the learned causal graphs.
Separating and Learning Latent Confounders to Enhancing User Preferences Modeling
Authors: Hangtong Xu, Yuanbo Xu, Yongjian Yang
Date: 2023-11-02 00:00:00
Tasks: counterfactual, Recommendation Systems

Recommender models aim to capture user preferences from historical feedback and then predict user-specific feedback on candidate items. However, the presence of various unmeasured confounders causes deviations between the user preferences in the historical feedback and the true preferences, resulting in models not meeting their expected performance. Existing debias models either (1) specific to solving one particular bias or (2) directly obtain auxiliary information from user historical feedback, which cannot identify whether the learned preferences are true user preferences or mixed with unmeasured confounders. Moreover, we find that the former recommender system is not only a successor to unmeasured confounders but also acts as an unmeasured confounder affecting user preference modeling, which has always been neglected in previous studies. To this end, we incorporate the effect of the former recommender system and treat it as a proxy for all unmeasured confounders. We propose a novel framework, Separating and Learning Latent Confounders For Recommendation (SLFR), which obtains the representation of unmeasured confounders to identify the counterfactual feedback by disentangling user preferences and unmeasured confounders, then guides the target model to capture the true preferences of users. Extensive experiments in five real-world datasets validate the advantages of our method.
Blocked Collaborative Bandits: Online Collaborative Filtering with Per-Item Budget Constraints
Date: 2023-10-31 00:00:00
Proceeding: NeurIPS 2023 11
Tasks: Collaborative Filtering, Matrix Completion

We consider the problem of \emph{blocked} collaborative bandits where there are multiple users, each with an associated multi-armed bandit problem. These users are grouped into \emph{latent} clusters such that the mean reward vectors of users within the same cluster are identical. Our goal is to design algorithms that maximize the cumulative reward accrued by all the users over time, under the \emph{constraint} that no arm of a user is pulled more than $\mathsf{B}$ times. This problem has been originally considered by \cite{Bresler:2014}, and designing regret-optimal algorithms for it has since remained an open problem. In this work, we propose an algorithm called \texttt{B-LATTICE} (Blocked Latent bAndiTs via maTrIx ComplEtion) that collaborates across users, while simultaneously satisfying the budget constraints, to maximize their cumulative rewards. Theoretically, under certain reasonable assumptions on the latent structure, with $\mathsf{M}$ users, $\mathsf{N}$ arms, $\mathsf{T}$ rounds per user, and $\mathsf{C}=O(1)$ latent clusters, \texttt{B-LATTICE} achieves a per-user regret of $\widetilde{O}(\sqrt{\mathsf{T}(1 + \mathsf{N}\mathsf{M}^{-1})}$ under a budget constraint of $\mathsf{B}=\Theta(\log \mathsf{T})$. These are the first sub-linear regret bounds for this problem, and match the minimax regret bounds when $\mathsf{B}=\mathsf{T}$. Empirically, we demonstrate that our algorithm has superior performance over baselines even when $\mathsf{B}=1$. \texttt{B-LATTICE} runs in phases where in each phase it clusters users into groups and collaborates across users within a group to quickly learn their reward models.
Generative AI for Software Metadata: Overview of the Information Retrieval in Software Engineering Track at FIRE 2023
Authors: Srijoni Majumdar, Soumen Paul, Debjyoti Paul, Ayan Bandyopadhyay, Samiran Chattopadhyay, Partha Pratim Das, Paul D Clough, Prasenjit Majumder
Date: 2023-10-27 00:00:00
Tasks: Binary Classification, Information Retrieval, Language Modeling, Language Modelling, Large Language Model, Retrieval

The Information Retrieval in Software Engineering (IRSE) track aims to develop solutions for automated evaluation of code comments in a machine learning framework based on human and large language model generated labels. In this track, there is a binary classification task to classify comments as useful and not useful. The dataset consists of 9048 code comments and surrounding code snippet pairs extracted from open source github C based projects and an additional dataset generated individually by teams using large language models. Overall 56 experiments have been submitted by 17 teams from various universities and software companies. The submissions have been evaluated quantitatively using the F1-Score and qualitatively based on the type of features developed, the supervised learning model used and their corresponding hyper-parameters. The labels generated from large language models increase the bias in the prediction model but lead to less over-fitted results.
Relative Arbitrage Opportunities in an Extended Mean Field System
Authors: Nicole Tianjiao Yang, Tomoyuki Ichiba
Date: 2023-11-05 00:00:00

This paper studies relative arbitrage opportunities in a market with infinitely many interacting investors. We establish a conditional McKean-Vlasov system to study the market dynamics coupled with investors. We then provide a theoretical framework to study a mean-field system, where the mean-field terms consist of a joint distribution of wealth and strategies. The optimal relative arbitrage is characterized by the equilibrium of extended mean-field games. We show the conditions on the existence and the uniqueness of the mean field equilibrium, then prove the propagation of chaos results for the finite-player game, and demonstrate that the Nash equilibrium converges to the mean field equilibrium when the population grows to infinity.
Raman Spectroscopy Reveals Photobiomodulation-Induced α-Helix to β-Sheet Transition in Tubulins: Potential Implications for Alzheimer's and Other Neurodegenerative Diseases
Authors: Elisabetta Di Gregorio, Michael Staelens, Nazanin Hosseinkhah, Mahroo Karimpoor, Janine Liburd, Lew Lim, Karthik Shankar, Jack A. Tuszynski
Date: 2023-11-07 00:00:00

In this study, we employed a Raman spectroscopic analysis of the amide I band of polymerized samples of tubulin exposed to pulsed low-intensity NIR radiation (810 nm, 10 Hz, 22.5 J/cm$^{2}$ dose). Peaks in the Raman fingerprint region (300$\unicode{x2013}$1900 cm$^{-1}$), in particular, in the amide I band (1600$\unicode{x2013}$1700 cm$^{-1}$), can be used to quantify the percentage of protein secondary structures. Under this band, hidden signals of $\mathrm{C}$=$\mathrm{O}$ stretching, belonging to different structures, are superimposed$\unicode{x2014}$producing a complex signal as a result. An accurate decomposition of the amide I band is therefore required for the reliable analysis of the conformation of proteins, which we achieved through a straightforward method employing a Voigt profile. This approach was validated through secondary structure analyses of unexposed control samples, for which comparisons with other values available in the literature could be conducted. Subsequently, using this validated method, we present novel findings of statistically significant alterations in the secondary structures of NIR-exposed tubulin, characterized by a notable decrease in alpha-helix content and a concurrent increase in beta-sheets compared to the control samples. The alpha-helix to beta-sheet transition suggests that PBM reduces microtubule stability and introduces dynamism to allow for the remodeling and, consequently, refreshing of microtubule structures. This newly discovered mechanism could have implications for reducing the risks associated with brain aging, including neurodegenerative diseases like Alzheimer's disease.
A seven-step guide to spatial, agent-based modelling of tumour evolution
Authors: Blair Colyer, Maciej Bak, David Basanta, Robert Noble
Date: 2023-11-06 00:00:00

Spatial agent-based models are increasingly used to investigate the evolution of solid tumours subject to localised cell-cell interactions and microenvironmental heterogeneity. Here we present a non-technical step by step guide to developing such a model from first principles, aimed at both aspiring modellers and other biologists and oncologists who wish to understand the assumptions and limitations of this approach. Stressing the importance of tailoring the model structure to that of the biological system, we describe methods of increasing complexity, from the basic Eden growth model up to off-lattice simulations with diffusible factors. We examine choices that unavoidably arise in model design, such as implementation, parameterisation, visualisation, and reproducibility. Each topic is illustrated with examples drawn from recent research studies and state of the art modelling platforms. We emphasise the benefits of simpler models that aim to match the complexity of the phenomena of interest, rather than that of the entire biological system.
Reaction-diffusion equations in mathematical models arising in epidemiology
Authors: Vasyl' Davydovych, Vasyl' Dutka, Roman Cherniha
Date: 2023-11-05 00:00:00
Tasks: Epidemiology

The review is devoted to analysis of mathematical models used for describing epidemic processes. A main focus is done on the models that are based on partial differential equations (PDEs), especially those that were developed and used for the COVID-19 pandemic modelling. Our attention is paid preferable to the studies in which not only results of numerical simulations are presented but analytical results as well. In particular, travelling fronts (waves), exact solutions, estimation of key epidemic parameters of the epidemic models with governing PDEs (typically reaction-diffusion equations) are discussed. The review may serve as a valuable source for researchers and practitioners in the field of mathematical modelling in epidemiology.