pwc-archive/papers-with-abstracts huggingface dataset:
The Problem of Alignment
Authors: Tsvetelina Hristova, Liam Magee, Karen Soldatic
Date: 2023-12-30 00:00:00
Tasks: Prompt Engineering

Large Language Models produce sequences learned as statistical patterns from large corpora. In order not to reproduce corpus biases, after initial training models must be aligned with human values, preferencing certain continuations over others. Alignment, which can be viewed as the superimposition of normative structure onto a statistical model, reveals a conflicted and complex interrelationship between language and technology. This relationship shapes theories of language, linguistic practice and subjectivity, which are especially relevant to the current sophistication in artificially produced text. We examine this practice of structuration as a two-way interaction between users and models by analysing how ChatGPT4 redacts perceived `anomalous' language in fragments of Joyce's Ulysses and the new linguistic practice of prompt engineering. We then situate this alignment problem historically, revisiting earlier postwar linguistic debates which counterposed two views of meaning: as discrete structures, and as continuous probability distributions. We discuss the largely occluded work of the Moscow Linguistic School, which sought to reconcile this opposition. Our attention to the Moscow School and later related arguments by Searle and Kristeva casts the problem of alignment in a new light: as one involving attention to the social structuration of linguistic practice, including structuration of anomalies that, like the Joycean text, exist in defiance of expressive conventions. These debates around the communicative orientation toward language can help explain some of the contemporary behaviours and interdependencies that take place between users and LLMs.
Principle Interference in Technical and Scientific Translation
Authors: Mohammad Ibrahim Qani
Date: 2023-12-30 00:00:00
Tasks: Translation

In this article, I will explore the nature of interference in translation, especially in technical and scientific texts, using a descriptivist approach. I will have a brief overview of the historical excursion of interference in technical and scientific translation. My aim is to explain this phenomenon and its causes with all its paradoxes, instead of simply condemning it as an example of supposedly bad translation. Thus, I will focus on its status in the bibliography of translation, on the motives for and consequences of interference in specialized translation, as well as on the nature of the arguments given for and against this phenomenon. Therefore the relationship between different societies has always been possible with the act of translation. When civilizations are examined throughout history, it is seen that the dissemination of knowledge among different societies has been achieved by translation. These societies have often become aware of the advancements in technology and science by means of translation. Therefore; translation becomes very significant in technical contact between societies and humans. Since the translation of technical texts is the preliminary scope of this thesis, it will be beneficial to have a brief look at the history of technical translation in the world.
Mitigating the Impact of False Negatives in Dense Retrieval with Contrastive Confidence Regularization
Authors: Shiqi Wang, Yeqin Zhang, Cam-Tu Nguyen
Date: 2023-12-30 00:00:00
Tasks: Answer Generation, Contrastive Learning, Open-Domain Question Answering, Question Answering, Retrieval, Text Retrieval

In open-domain Question Answering (QA), dense retrieval is crucial for finding relevant passages for answer generation. Typically, contrastive learning is used to train a retrieval model that maps passages and queries to the same semantic space. The objective is to make similar ones closer and dissimilar ones further apart. However, training such a system is challenging due to the false negative issue, where relevant passages may be missed during data annotation. Hard negative sampling, which is commonly used to improve contrastive learning, can introduce more noise in training. This is because hard negatives are those closer to a given query, and thus more likely to be false negatives. To address this issue, we propose a novel contrastive confidence regularizer for Noise Contrastive Estimation (NCE) loss, a commonly used loss for dense retrieval. Our analysis shows that the regularizer helps dense retrieval models be more robust against false negatives with a theoretical guarantee. Additionally, we propose a model-agnostic method to filter out noisy negative passages in the dataset, improving any downstream dense retrieval models. Through experiments on three datasets, we demonstrate that our method achieves better retrieval performance in comparison to existing state-of-the-art dense retrieval systems.
ChatEd: A Chatbot Leveraging ChatGPT for an Enhanced Learning Experience in Higher Education
Authors: Kevin Wang, Jason Ramos, Ramon Lawrence
Date: 2023-12-29 00:00:00
Tasks: Chatbot, Information Retrieval, Retrieval

With the rapid evolution of Natural Language Processing (NLP), Large Language Models (LLMs) like ChatGPT have emerged as powerful tools capable of transforming various sectors. Their vast knowledge base and dynamic interaction capabilities represent significant potential in improving education by operating as a personalized assistant. However, the possibility of generating incorrect, biased, or unhelpful answers are a key challenge to resolve when deploying LLMs in an education context. This work introduces an innovative architecture that combines the strengths of ChatGPT with a traditional information retrieval based chatbot framework to offer enhanced student support in higher education. Our empirical evaluations underscore the high promise of this approach.
A Novel Dual-Stage Evolutionary Algorithm for Finding Robust Solutions
Authors: Wei Du, Wenxuan Fang, Chen Liang, Yang Tang, Yaochu Jin
Date: 2024-01-02 00:00:00

In robust optimization problems, the magnitude of perturbations is relatively small. Consequently, solutions within certain regions are less likely to represent the robust optima when perturbations are introduced. Hence, a more efficient search process would benefit from increased opportunities to explore promising regions where global optima or good local optima are situated. In this paper, we introduce a novel robust evolutionary algorithm named the dual-stage robust evolutionary algorithm (DREA) aimed at discovering robust solutions. DREA operates in two stages: the peak-detection stage and the robust solution-searching stage. The primary objective of the peak-detection stage is to identify peaks in the fitness landscape of the original optimization problem. Conversely, the robust solution-searching stage focuses on swiftly identifying the robust optimal solution using information obtained from the peaks discovered in the initial stage. These two stages collectively enable the proposed DREA to efficiently obtain the robust optimal solution for the optimization problem. This approach achieves a balance between solution optimality and robustness by separating the search processes for optimal and robust optimal solutions. Experimental results demonstrate that DREA significantly outperforms five state-of-the-art algorithms across 18 test problems characterized by diverse complexities. Moreover, when evaluated on higher-dimensional robust optimization problems (100-$D$ and 200-$D$), DREA also demonstrates superior performance compared to all five counterpart algorithms.
Evolutionary Alternating Direction Method of Multipliers for Constrained Multi-Objective Optimization with Unknown Constraints
Authors: Shuang Li, Ke Li, Wei Li, Ming Yang
Date: 2024-01-02 00:00:00

Constrained multi-objective optimization problems (CMOPs) pervade real-world applications in science, engineering, and design. Constraint violation has been a building block in designing evolutionary multi-objective optimization algorithms for solving constrained multi-objective optimization problems. However, in certain scenarios, constraint functions might be unknown or inadequately defined, making constraint violation unattainable and potentially misleading for conventional constrained evolutionary multi-objective optimization algorithms. To address this issue, we present the first of its kind evolutionary optimization framework, inspired by the principles of the alternating direction method of multipliers that decouples objective and constraint functions. This framework tackles CMOPs with unknown constraints by reformulating the original problem into an additive form of two subproblems, each of which is allotted a dedicated evolutionary population. Notably, these two populations operate towards complementary evolutionary directions during their optimization processes. In order to minimize discrepancy, their evolutionary directions alternate, aiding the discovery of feasible solutions. Comparative experiments conducted against five state-of-the-art constrained evolutionary multi-objective optimization algorithms, on 120 benchmark test problem instances with varying properties, as well as two real-world engineering optimization problems, demonstrate the effectiveness and superiority of our proposed framework. Its salient features include faster convergence and enhanced resilience to various Pareto front shapes.
Learn to integrate parts for whole through correlated neural variability
Authors: Zhichao Zhu, Yang Qi, Wenlian Lu, Jianfeng Feng
Date: 2024-01-01 00:00:00

Sensory perception originates from the responses of sensory neurons, which react to a collection of sensory signals linked to various physical attributes of a singular perceptual object. Unraveling how the brain extracts perceptual information from these neuronal responses is a pivotal challenge in both computational neuroscience and machine learning. Here we introduce a statistical mechanical theory, where perceptual information is first encoded in the correlated variability of sensory neurons and then reformatted into the firing rates of downstream neurons. Applying this theory, we illustrate the encoding of motion direction using neural covariance and demonstrate high-fidelity direction recovery by spiking neural networks. Networks trained under this theory also show enhanced performance in classifying natural images, achieving higher accuracy and faster inference speed. Our results challenge the traditional view of neural covariance as a secondary factor in neural coding, highlighting its potential influence on brain function.
Emulating insect brains for neuromorphic navigation
Authors: Korbinian Schreiber, Timo Wunderlich, Philipp Spilger, Sebastian Billaudelle, Benjamin Cramer, Yannik Stradmann, Christian Pehle, Eric Müller, Mihai A. Petrovici, Johannes Schemmel, Karlheinz Meier
Date: 2023-12-31 00:00:00

Bees display the remarkable ability to return home in a straight line after meandering excursions to their environment. Neurobiological imaging studies have revealed that this capability emerges from a path integration mechanism implemented within the insect's brain. In the present work, we emulate this neural network on the neuromorphic mixed-signal processor BrainScaleS-2 to guide bees, virtually embodied on a digital co-processor, back to their home location after randomly exploring their environment. To realize the underlying neural integrators, we introduce single-neuron spike-based short-term memory cells with axo-axonic synapses. All entities, including environment, sensory organs, brain, actuators, and the virtual body, run autonomously on a single BrainScaleS-2 microchip. The functioning network is fine-tuned for better precision and reliability through an evolution strategy. As BrainScaleS-2 emulates neural processes 1000 times faster than biology, 4800 consecutive bee journeys distributed over 320 generations occur within only half an hour on a single neuromorphic core.
Multiplayer Battle Game-Inspired Optimizer for Complex Optimization Problems
Authors: Yuefeng Xu, Rui Zhong, Chao Zhang, Jun Yu
Date: 2023-12-31 00:00:00
Tasks: Diversity

Various popular multiplayer battle royale games share a lot of common elements. Drawing from our observations, we summarized these shared characteristics and subsequently proposed a novel heuristic algorithm named multiplayer battle game-inspired optimizer (MBGO). The proposed MBGO streamlines mainstream multiplayer battle royale games into two discrete phases: movement and battle. Specifically, the movement phase incorporates the principles of commonly encountered ``safe zones'' to incentivize participants to relocate to areas with a higher survival potential. The battle phase simulates a range of strategies adopted by players in various situations to enhance the diversity of the population. To evaluate and analyze the performance of the proposed MBGO, we executed it alongside eight other algorithms, including three classics and five latest ones, across multiple diverse dimensions within the CEC2017 and CEC2020 benchmark functions. In addition, we employed several industrial design problems to evaluate the scalability and practicality of the proposed MBGO. The results of the statistical analysis reveal that the novel MBGO demonstrates significant competitiveness, excelling not only in convergence speed, but also in achieving high levels of convergence accuracy across both benchmark functions and real-world problems.
Benchmarking Hebbian learning rules for associative memory
Authors: Anders Lansner, Naresh B Ravichandran, Pawel Herman
Date: 2023-12-30 00:00:00
Tasks: Benchmarking

Associative memory or content addressable memory is an important component function in computer science and information processing and is a key concept in cognitive and computational brain science. Many different neural network architectures and learning rules have been proposed to model associative memory of the brain while investigating key functions like pattern completion and rivalry, noise reduction, and storage capacity. A less investigated but important function is prototype extraction where the training set comprises pattern instances generated by distorting prototype patterns and the task of the trained network is to recall the correct prototype pattern given a new instance. In this paper we characterize these different aspects of associative memory performance and benchmark six different learning rules on storage capacity and prototype extraction. We consider only models with Hebbian plasticity that operate on sparse distributed representations with unit activities in the interval [0,1]. We evaluate both non-modular and modular network architectures and compare performance when trained and tested on different kinds of sparse random binary pattern sets, including correlated ones. We show that covariance learning has a robust but low storage capacity under these conditions and that the Bayesian Confidence Propagation learning rule (BCPNN) is superior with a good margin in all cases except one, reaching a three times higher composite score than the second best learning rule tested.
Multiform Evolution for High-Dimensional Problems with Low Effective Dimensionality
Authors: Yaqing Hou, Mingyang Sun, Abhishek Gupta, Yaochu Jin, Haiyin Piao, Hongwei Ge, Qiang Zhang
Date: 2023-12-30 00:00:00
Tasks: Evolutionary Algorithms

In this paper, we scale evolutionary algorithms to high-dimensional optimization problems that deceptively possess a low effective dimensionality (certain dimensions do not significantly affect the objective function). To this end, an instantiation of the multiform optimization paradigm is presented, where multiple low-dimensional counterparts of a target high-dimensional task are generated via random embeddings. Since the exact relationship between the auxiliary (low-dimensional) tasks and the target is a priori unknown, a multiform evolutionary algorithm is developed for unifying all formulations into a single multi-task setting. The resultant joint optimization enables the target task to efficiently reuse solutions evolved across various low-dimensional searches via cross-form genetic transfers, hence speeding up overall convergence characteristics. To validate the overall efficacy of our proposed algorithmic framework, comprehensive experimental studies are carried out on well-known continuous benchmark functions as well as a set of practical problems in the hyper-parameter tuning of machine learning models and deep learning models in classification tasks and Predator-Prey games, respectively.
Learning Long Sequences in Spiking Neural Networks
Authors: Matei Ioan Stan, Oliver Rhodes
Date: 2023-12-14 00:00:00
Tasks: image-classification, Image Classification, Sequential Image Classification, State Space Models

Spiking neural networks (SNNs) take inspiration from the brain to enable energy-efficient computations. Since the advent of Transformers, SNNs have struggled to compete with artificial networks on modern sequential tasks, as they inherit limitations from recurrent neural networks (RNNs), with the added challenge of training with non-differentiable binary spiking activations. However, a recent renewed interest in efficient alternatives to Transformers has given rise to state-of-the-art recurrent architectures named state space models (SSMs). This work systematically investigates, for the first time, the intersection of state-of-the-art SSMs with SNNs for long-range sequence modelling. Results suggest that SSM-based SNNs can outperform the Transformer on all tasks of a well-established long-range sequence modelling benchmark. It is also shown that SSM-based SNNs can outperform current state-of-the-art SNNs with fewer parameters on sequential image classification. Finally, a novel feature mixing layer is introduced, improving SNN accuracy while challenging assumptions about the role of binary activations in SNNs. This work paves the way for deploying powerful SSM-based architectures, such as large language models, to neuromorphic hardware for energy-efficient long-range sequence modelling.
Inverting estimating equations for causal inference on quantiles
Authors: Chao Cheng, Fan Li
Date: 2024-01-02 00:00:00
Tasks: Causal Inference

The causal inference literature frequently focuses on estimating the mean of the potential outcome, whereas quantiles of the potential outcome may carry important additional information. We propose a unified approach, based on the inverse estimating equations, to generalize a class of causal inference solutions from estimating the mean of the potential outcome to its quantiles. We assume that a moment function is available to identify the mean of the threshold-transformed potential outcome, based on which a convenient construction of the estimating equation of quantiles of potential outcome is proposed. In addition, we give a general construction of the efficient influence functions of the mean and quantiles of potential outcomes, and explicate their connection. We motivate estimators for the quantile estimands with the efficient influence function, and develop their asymptotic properties when either parametric models or data-adaptive machine learners are used to estimate the nuisance functions. A broad implication of our results is that one can rework the existing result for mean causal estimands to facilitate causal inference on quantiles. Our general results are illustrated by several analytical and numerical examples.
Factor Importance Ranking and Selection using Total Indices
Authors: Chaofan Huang, V. Roshan Joseph
Date: 2024-01-01 00:00:00
Tasks: Binary Classification

Factor importance measures the impact of each feature on output prediction accuracy. Many existing works focus on the model-based importance, but an important feature in one learning algorithm may hold little significance in another model. Hence, a factor importance measure ought to characterize the feature's predictive potential without relying on a specific prediction algorithm. Such algorithm-agnostic importance is termed as intrinsic importance in Williamson et al. (2023), but their estimator again requires model fitting. To bypass the modeling step, we present the equivalence between predictiveness potential and total Sobol' indices from global sensitivity analysis, and introduce a novel consistent estimator that can be directly estimated from noisy data. Integrating with forward selection and backward elimination gives rise to FIRST, Factor Importance Ranking and Selection using Total (Sobol') indices. Extensive simulations are provided to demonstrate the effectiveness of FIRST on regression and binary classification problems, and a clear advantage over the state-of-the-art methods.
Simplicity bias, algorithmic probability, and the random logistic map
Authors: Boumediene Hamzi, Kamaludin Dingle
Date: 2023-12-31 00:00:00

Simplicity bias is an intriguing phenomenon prevalent in various input-output maps, characterized by a preference for simpler, more regular, or symmetric outputs. Notably, these maps typically feature high-probability outputs with simple patterns, whereas complex patterns are exponentially less probable. This bias has been extensively examined and attributed to principles derived from algorithmic information theory and algorithmic probability. In a significant advancement, it has been demonstrated that the renowned logistic map and other one-dimensional maps exhibit simplicity bias when conceptualized as input-output systems. Building upon this work, our research delves into the manifestations of simplicity bias within the random logistic map, specifically focusing on scenarios involving additive noise. We discover that simplicity bias is observable in the random logistic map for specific ranges of $\mu$ and noise magnitudes. Additionally, we find that this bias persists even with the introduction of small measurement noise, though it diminishes as noise levels increase. Our studies also revisit the phenomenon of noise-induced chaos, particularly when $\mu=3.83$, revealing its characteristics through complexity-probability plots. Intriguingly, we employ the logistic map to illustrate a paradoxical aspect of data analysis: more data adhering to a consistent trend can occasionally lead to \emph{reduced} confidence in extrapolation predictions, challenging conventional wisdom. We propose that adopting a probability-complexity perspective in analyzing dynamical systems could significantly enrich statistical learning theories related to series prediction and analysis. This approach not only facilitates a deeper understanding of simplicity bias and its implications but also paves the way for novel methodologies in forecasting complex systems behavior.