pwc-archive/papers-with-abstracts huggingface dataset:
Longer Fixations, More Computation: Gaze-Guided Recurrent Neural Networks
Authors: Xinting Huang, Jiajing Wan, Ioannis Kritikos, Nora Hollenstein
Date: 2023-10-31 00:00:00
Tasks: Language Modeling, Language Modelling, Sentiment Analysis

Humans read texts at a varying pace, while machine learning models treat each token in the same way in terms of a computational process. Therefore, we ask, does it help to make models act more like humans? In this paper, we convert this intuition into a set of novel models with fixation-guided parallel RNNs or layers and conduct various experiments on language modeling and sentiment analysis tasks to test their effectiveness, thus providing empirical validation for this intuition. Our proposed models achieve good performance on the language modeling task, considerably surpassing the baseline model. In addition, we find that, interestingly, the fixation duration predicted by neural networks bears some resemblance to humans' fixation. Without any explicit guidance, the model makes similar choices to humans. We also investigate the reasons for the differences between them, which explain why "model fixations" are often more suitable than human fixations, when used to guide language models.
On the effect of curriculum learning with developmental data for grammar acquisition
Authors: Mattia Opper, J. Morrison, N. Siddharth
Date: 2023-10-31 00:00:00

This work explores the degree to which grammar acquisition is driven by language `simplicity' and the source modality (speech vs. text) of data. Using BabyBERTa as a probe, we find that grammar acquisition is largely driven by exposure to speech data, and in particular through exposure to two of the BabyLM training corpora: AO-Childes and Open Subtitles. We arrive at this finding by examining various ways of presenting input data to our model. First, we assess the impact of various sequence-level complexity based curricula. We then examine the impact of learning over `blocks' -- covering spans of text that are balanced for the number of tokens in each of the source corpora (rather than number of lines). Finally, we explore curricula that vary the degree to which the model is exposed to different corpora. In all cases, we find that over-exposure to AO-Childes and Open Subtitles significantly drives performance. We verify these findings through a comparable control dataset in which exposure to these corpora, and speech more generally, is limited by design. Our findings indicate that it is not the proportion of tokens occupied by high-utility data that aids acquisition, but rather the proportion of training steps assigned to such data. We hope this encourages future research into the use of more developmentally plausible linguistic data (which tends to be more scarce) to augment general purpose pre-training regimes.
BadLlama: cheaply removing safety fine-tuning from Llama 2-Chat 13B
Authors: Pranav Gade, Simon Lermen, Charlie Rogers-Smith, Jeffrey Ladish
Date: 2023-10-31 00:00:00

Llama 2-Chat is a collection of large language models that Meta developed and released to the public. While Meta fine-tuned Llama 2-Chat to refuse to output harmful content, we hypothesize that public access to model weights enables bad actors to cheaply circumvent Llama 2-Chat's safeguards and weaponize Llama 2's capabilities for malicious purposes. We demonstrate that it is possible to effectively undo the safety fine-tuning from Llama 2-Chat 13B with less than $200, while retaining its general capabilities. Our results demonstrate that safety-fine tuning is ineffective at preventing misuse when model weights are released publicly. Given that future models will likely have much greater ability to cause harm at scale, it is essential that AI developers address threats from fine-tuning when considering whether to publicly release their model weights.
BERTwich: Extending BERT's Capabilities to Model Dialectal and Noisy Text
Authors: Aarohi Srivastava, David Chiang
Date: 2023-10-31 00:00:00
Tasks: Language Modeling, Language Modelling, Masked Language Modeling

Real-world NLP applications often deal with nonstandard text (e.g., dialectal, informal, or misspelled text). However, language models like BERT deteriorate in the face of dialect variation or noise. How do we push BERT's modeling capabilities to encompass nonstandard text? Fine-tuning helps, but it is designed for specializing a model to a task and does not seem to bring about the deeper, more pervasive changes needed to adapt a model to nonstandard language. In this paper, we introduce the novel idea of sandwiching BERT's encoder stack between additional encoder layers trained to perform masked language modeling on noisy text. We find that our approach, paired with recent work on including character-level noise in fine-tuning data, can promote zero-shot transfer to dialectal text, as well as reduce the distance in the embedding space between words and their noisy counterparts.
Tracking capelin spawning migration -- Integrating environmental data and Individual-based modeling
Authors: Salah Alrabeei, Sam Subbey, Talal Rahman
Date: 2023-11-01 00:00:00

This paper presents a modeling framework for tracking the spawning migration of the capelin, which is a fish species in the Barents Sea. The framework combines an individual-based model (IBM) with artificial neural networks (ANNs). The ANNs determine the direction of the fish's movement based on local environmental information, while a genetic algorithm and fitness function assess the suitability of the proposed directions. The framework's efficacy is demonstrated by comparing the spatial distributions of modeled and empirical potential spawners. The proposed model successfully replicates the southeastward movement of capelin during their spawning migration, accurately capturing the distribution of spawning fish over historical spawning sites along the eastern coast of northern Norway. Furthermore, the paper compares three migration models: passive swimmers, taxis movement based on temperature gradients, and restricted-area search, along with our proposed approach. The results reveal that our approach outperforms the other models in mimicking the migration pattern. Most spawning stocks managed to reach the spawning sites, unlike the other models where water currents played a significant role in pushing the fish away from the coast. The temperature gradient detection model and restricted-area search model are found to be inadequate for accurately simulating capelin spawning migration in the Barents Sea due to complex oceanographic conditions.
Variational Gaussian Processes For Linear Inverse Problems
Date: 2023-11-01 00:00:00
Proceeding: NeurIPS 2023 11
Tasks: Gaussian Processes

By now Bayesian methods are routinely used in practice for solving inverse problems. In inverse problems the parameter or signal of interest is observed only indirectly, as an image of a given map, and the observations are typically further corrupted with noise. Bayes offers a natural way to regularize these problems via the prior distribution and provides a probabilistic solution, quantifying the remaining uncertainty in the problem. However, the computational costs of standard, sampling based Bayesian approaches can be overly large in such complex models. Therefore, in practice variational Bayes is becoming increasingly popular. Nevertheless, the theoretical understanding of these methods is still relatively limited, especially in context of inverse problems. In our analysis we investigate variational Bayesian methods for Gaussian process priors to solve linear inverse problems. We consider both mildly and severely ill-posed inverse problems and work with the popular inducing variables variational Bayes approach proposed by Titsias in 2009. We derive posterior contraction rates for the variational posterior in general settings and show that the minimax estimation rate can be attained by correctly tunned procedures. As specific examples we consider a collection of inverse problems including the heat equation, Volterra operator and Radon transform and inducing variable methods based on population and empirical spectral features.
Scalable kernel balancing weights in a nationwide observational study of hospital profit status and heart attack outcomes
Authors: Kwangho Kim, Bijan A. Niknam, José R. Zubizarreta
Date: 2023-11-01 00:00:00

Weighting is a general and often-used method for statistical adjustment. Weighting has two objectives: first, to balance covariate distributions, and second, to ensure that the weights have minimal dispersion and thus produce a more stable estimator. A recent, increasingly common approach directly optimizes the weights toward these two objectives. However, this approach has not yet been feasible in large-scale datasets when investigators wish to flexibly balance general basis functions in an extended feature space. For example, many balancing approaches cannot scale to national-level health services research studies. To address this practical problem, we describe a scalable and flexible approach to weighting that integrates a basis expansion in a reproducing kernel Hilbert space with state-of-the-art convex optimization techniques. Specifically, we use the rank-restricted Nystr\"{o}m method to efficiently compute a kernel basis for balancing in {nearly} linear time and space, and then use the specialized first-order alternating direction method of multipliers to rapidly find the optimal weights. In an extensive simulation study, we provide new insights into the performance of weighting estimators in large datasets, showing that the proposed approach substantially outperforms others in terms of accuracy and speed. Finally, we use this weighting approach to conduct a national study of the relationship between hospital profit status and heart attack outcomes in a comprehensive dataset of 1.27 million patients. We find that for-profit hospitals use interventional cardiology to treat heart attacks at similar rates as other hospitals, but have higher mortality and readmission rates.
Polynomial Chaos Surrogate Construction for Random Fields with Parametric Uncertainty
Authors: Joy N. Mueller, Khachik Sargsyan, Craig J. Daniels, Habib N. Najm
Date: 2023-11-01 00:00:00
Tasks: Uncertainty Quantification

Engineering and applied science rely on computational experiments to rigorously study physical systems. The mathematical models used to probe these systems are highly complex, and sampling-intensive studies often require prohibitively many simulations for acceptable accuracy. Surrogate models provide a means of circumventing the high computational expense of sampling such complex models. In particular, polynomial chaos expansions (PCEs) have been successfully used for uncertainty quantification studies of deterministic models where the dominant source of uncertainty is parametric. We discuss an extension to conventional PCE surrogate modeling to enable surrogate construction for stochastic computational models that have intrinsic noise in addition to parametric uncertainty. We develop a PCE surrogate on a joint space of intrinsic and parametric uncertainty, enabled by Rosenblatt transformations, and then extend the construction to random field data via the Karhunen-Loeve expansion. We then take advantage of closed-form solutions for computing PCE Sobol indices to perform a global sensitivity analysis of the model which quantifies the intrinsic noise contribution to the overall model output variance. Additionally, the resulting joint PCE is generative in the sense that it allows generating random realizations at any input parameter setting that are statistically approximately equivalent to realizations from the underlying stochastic model. The method is demonstrated on a chemical catalysis example model.
Precise Error Rates for Computationally Efficient Testing
Authors: Ankur Moitra, Alexander S. Wein
Date: 2023-11-01 00:00:00

We revisit the fundamental question of simple-versus-simple hypothesis testing with an eye towards computational complexity, as the statistically optimal likelihood ratio test is often computationally intractable in high-dimensional settings. In the classical spiked Wigner model with a general i.i.d. spike prior we show (conditional on a conjecture) that an existing test based on linear spectral statistics achieves the best possible tradeoff curve between type I and type II error rates among all computationally efficient tests, even though there are exponential-time tests that do better. This result is conditional on an appropriate complexity-theoretic conjecture, namely a natural strengthening of the well-established low-degree conjecture. Our result shows that the spectrum is a sufficient statistic for computationally bounded tests (but not for all tests). To our knowledge, our approach gives the first tool for reasoning about the precise asymptotic testing error achievable with efficient computation. The main ingredients required for our hardness result are a sharp bound on the norm of the low-degree likelihood ratio along with (counterintuitively) a positive result on achievability of testing. This strategy appears to be new even in the setting of unbounded computation, in which case it gives an alternate way to analyze the fundamental statistical limits of testing.
Adaptive and non-adaptive minimax rates for weighted Laplacian-eigenmap based nonparametric regression
Authors: Zhaoyang Shi, Krishnakumar Balasubramanian, Wolfgang Polonik
Date: 2023-10-31 00:00:00
Tasks: regression

We show both adaptive and non-adaptive minimax rates of convergence for a family of weighted Laplacian-Eigenmap based nonparametric regression methods, when the true regression function belongs to a Sobolev space and the sampling density is bounded from above and below. The adaptation methodology is based on extensions of Lepski's method and is over both the smoothness parameter ($s\in\mathbb{N}_{+}$) and the norm parameter ($M>0$) determining the constraints on the Sobolev space. Our results extend the non-adaptive result in \cite{green2021minimax}, established for a specific normalized graph Laplacian, to a wide class of weighted Laplacian matrices used in practice, including the unnormalized Laplacian and random walk Laplacian.
NoMoPy: Noise Modeling in Python
Authors: Dylan Albrecht, N. Tobias Jacobson
Date: 2023-10-31 00:00:00
Tasks: Model Selection, parameter estimation

NoMoPy is a code for fitting, analyzing, and generating noise modeled as a hidden Markov model (HMM) or, more generally, factorial hidden Markov model (FHMM). This code, written in Python, implements approximate and exact expectation maximization (EM) algorithms for performing the parameter estimation process, model selection procedures via cross-validation, and parameter confidence region estimation. Here, we describe in detail the functionality implemented in NoMoPy and provide examples of its use and performance on example problems.
GATSY: Graph Attention Network for Music Artist Similarity
Authors: Andrea Giuseppe Di Francesco, Giuliano Giampietro, Indro Spinelli, Danilo Comminiello
Date: 2023-11-01 00:00:00
Tasks: Graph Attention

The artist similarity quest has become a crucial subject in social and scientific contexts, driven by the desire to enhance music discovery according to user preferences. Modern research solutions facilitate music discovery according to user tastes. However, defining similarity among artists remains challenging due to its inherently subjective nature, which can impact recommendation accuracy. This paper introduces GATSY, a novel recommendation system built upon graph attention networks and driven by a clusterized embedding of artists. The proposed framework leverages the graph topology of the input data to achieve outstanding performance results without relying heavily on hand-crafted features. This flexibility allows us the inclusion of fictitious artists within a music dataset, facilitating connections between previously unlinked artists and enabling diverse recommendations from various and heterogeneous sources. Experimental results prove the effectiveness of the proposed method with respect to state-of-the-art solutions while maintaining flexibility. The code to reproduce these experiments is available at https://anonymous.4open.science/r/GATSY-Music_Artist_Similarity-4807/README.md.
LLMRec: Large Language Models with Graph Augmentation for Recommendation
Authors: Wei Wei, Xubin Ren, Jiabin Tang, Qinyong Wang, Lixin Su, Suqi Cheng, Junfeng Wang, Dawei Yin, Chao Huang
Date: 2023-11-01 00:00:00
Tasks: Model Optimization, Recommendation Systems

The problem of data sparsity has long been a challenge in recommendation systems, and previous studies have attempted to address this issue by incorporating side information. However, this approach often introduces side effects such as noise, availability issues, and low data quality, which in turn hinder the accurate modeling of user preferences and adversely impact recommendation performance. In light of the recent advancements in large language models (LLMs), which possess extensive knowledge bases and strong reasoning capabilities, we propose a novel framework called LLMRec that enhances recommender systems by employing three simple yet effective LLM-based graph augmentation strategies. Our approach leverages the rich content available within online platforms (e.g., Netflix, MovieLens) to augment the interaction graph in three ways: (i) reinforcing user-item interaction egde, (ii) enhancing the understanding of item node attributes, and (iii) conducting user node profiling, intuitively from the natural language perspective. By employing these strategies, we address the challenges posed by sparse implicit feedback and low-quality side information in recommenders. Besides, to ensure the quality of the augmentation, we develop a denoised data robustification mechanism that includes techniques of noisy implicit feedback pruning and MAE-based feature enhancement that help refine the augmented data and improve its reliability. Furthermore, we provide theoretical analysis to support the effectiveness of LLMRec and clarify the benefits of our method in facilitating model optimization. Experimental results on benchmark datasets demonstrate the superiority of our LLM-based augmentation approach over state-of-the-art techniques. To ensure reproducibility, we have made our code and augmented data publicly available at: https://github.com/HKUDS/LLMRec.git
AutoSAM: Towards Automatic Sampling of User Behaviors for Sequential Recommender Systems
Authors: Hao Zhang, Mingyue Cheng, Qi Liu, Zhiding Liu, Junzhe Jiang, Enhong Chen
Date: 2023-11-01 00:00:00
Tasks: Future prediction, Recommendation Systems, Sequential Recommendation

Sequential recommender systems (SRS) have gained widespread popularity in recommendation due to their ability to effectively capture dynamic user preferences. One default setting in the current SRS is to uniformly consider each historical behavior as a positive interaction. Actually, this setting has the potential to yield sub-optimal performance, as each item makes a distinct contribution to the user's interest. For example, purchased items should be given more importance than clicked ones. Hence, we propose a general automatic sampling framework, named AutoSAM, to non-uniformly treat historical behaviors. Specifically, AutoSAM augments the standard sequential recommendation architecture with an additional sampler layer to adaptively learn the skew distribution of the raw input, and then sample informative sub-sets to build more generalizable SRS. To overcome the challenges of non-differentiable sampling actions and also introduce multiple decision factors for sampling, we further introduce a novel reinforcement learning based method to guide the training of the sampler. We theoretically design multi-objective sampling rewards including Future Prediction and Sequence Perplexity, and then optimize the whole framework in an end-to-end manner by combining the policy gradient. We conduct extensive experiments on benchmark recommender models and four real-world datasets. The experimental results demonstrate the effectiveness of the proposed approach. We will make our code publicly available after the acceptance.
Caseformer: Pre-training for Legal Case Retrieval Based on Inter-Case Distinctions
Authors: Weihang Su, Qingyao Ai, Yueyue Wu, Yixiao Ma, Haitao Li, Yiqun Liu, Zhijing Wu, Min Zhang
Date: 2023-11-01 00:00:00
Tasks: Fairness, Retrieval

Legal case retrieval aims to help legal workers find relevant cases related to their cases at hand, which is important for the guarantee of fairness and justice in legal judgments. While recent advances in neural retrieval methods have significantly improved the performance of open-domain retrieval tasks (e.g., Web search), their advantages have not been observed in legal case retrieval due to their thirst for annotated data. As annotating large-scale training data in legal domains is prohibitive due to the need for domain expertise, traditional search techniques based on lexical matching such as TF-IDF, BM25, and Query Likelihood are still prevalent in legal case retrieval systems. While previous studies have designed several pre-training methods for IR models in open-domain tasks, these methods are usually suboptimal in legal case retrieval because they cannot understand and capture the key knowledge and data structures in the legal corpus. To this end, we propose a novel pre-training framework named Caseformer that enables the pre-trained models to learn legal knowledge and domain-specific relevance information in legal case retrieval without any human-labeled data. Through three unsupervised learning tasks, Caseformer is able to capture the special language, document structure, and relevance patterns of legal case documents, making it a strong backbone for downstream legal case retrieval tasks. Experimental results show that our model has achieved state-of-the-art performance in both zero-shot and full-data fine-tuning settings. Also, experiments on both Chinese and English legal datasets demonstrate that the effectiveness of Caseformer is language-independent in legal case retrieval.
BFV-Based Homomorphic Encryption for Privacy-Preserving CNN Models
Authors: FebriantiWibawa, Ferhat Ozgur Catak, Salih Sarp, Murat Kuzlu
Date: 2022-07-01 00:00:00
Proceeding: journal 2022 7
Tasks: Federated Learning, Privacy Preserving

Medical data is frequently quite sensitive in terms of data privacy and security. Federated
learning has been used to increase the privacy and security of medical data, which is a sort of machine
learning technique. The training data is disseminated across numerous machines in federated learning,
and the learning process is collaborative. There are numerous privacy attacks on deep learning
(DL) models that attackers can use to obtain sensitive information. As a result, the DL model should
be safeguarded from adversarial attacks, particularly in medical data applications. Homomorphic
encryption-based model security from the adversarial collaborator is one of the answers to this
challenge. Using homomorphic encryption, this research presents a privacy-preserving federated
learning system for medical data. The proposed technique employs a secure multi-party computation
protocol to safeguard the deep learning model from adversaries. The proposed approach is tested in
terms of model performance using a real-world medical dataset in this paper