pwc-archive/papers-with-abstracts huggingface dataset:
Journey to the Center of the Knowledge Neurons: Discoveries of Language-Independent Knowledge Neurons and Degenerate Knowledge Neurons
Authors: YuHeng Chen, Pengfei Cao, Yubo Chen, Kang Liu, Jun Zhao
Date: 2023-08-25 00:00:00
Tasks: Fact Checking, knowledge editing

Pre-trained language models (PLMs) contain vast amounts of factual knowledge, but how the knowledge is stored in the parameters remains unclear. This paper delves into the complex task of understanding how factual knowledge is stored in multilingual PLMs, and introduces the Architecture-adapted Multilingual Integrated Gradients method, which successfully localizes knowledge neurons more precisely compared to current methods, and is more universal across various architectures and languages. Moreover, we conduct an in-depth exploration of knowledge neurons, leading to the following two important discoveries: (1) The discovery of Language-Independent Knowledge Neurons, which store factual knowledge in a form that transcends language. We design cross-lingual knowledge editing experiments, demonstrating that the PLMs can accomplish this task based on language-independent neurons; (2) The discovery of Degenerate Knowledge Neurons, a novel type of neuron showing that different knowledge neurons can store the same fact. Its property of functional overlap endows the PLMs with a robust mastery of factual knowledge. We design fact-checking experiments, proving that the degenerate knowledge neurons can help the PLMs to detect wrong facts. Experiments corroborate these findings, shedding light on the mechanisms of factual knowledge storage in multilingual PLMs, and contribute valuable insights to the field. The code is available at https://github.com/heng840/AMIG.
Measuring Spurious Correlation in Classification: 'Clever Hans' in Translationese
Authors: Angana Borah, Daria Pylypenko, Cristina Espana-Bonet, Josef van Genabith
Date: 2023-08-25 00:00:00
Tasks: Classification

Recent work has shown evidence of 'Clever Hans' behavior in high-performance neural translationese classifiers, where BERT-based classifiers capitalize on spurious correlations, in particular topic information, between data and target classification labels, rather than genuine translationese signals. Translationese signals are subtle (especially for professional translation) and compete with many other signals in the data such as genre, style, author, and, in particular, topic. This raises the general question of how much of the performance of a classifier is really due to spurious correlations in the data versus the signals actually targeted for by the classifier, especially for subtle target signals and in challenging (low resource) data settings. We focus on topic-based spurious correlation and approach the question from two directions: (i) where we have no knowledge about spurious topic information and its distribution in the data, (ii) where we have some indication about the nature of spurious topic correlations. For (i) we develop a measure from first principles capturing alignment of unsupervised topics with target classification labels as an indication of spurious topic information in the data. We show that our measure is the same as purity in clustering and propose a 'topic floor' (as in a 'noise floor') for classification. For (ii) we investigate masking of known spurious topic carriers in classification. Both (i) and (ii) contribute to quantifying and (ii) to mitigating spurious correlations.
SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Research
Authors: Liangtai Sun, Yang Han, Zihan Zhao, Da Ma, Zhennan Shen, Baocai Chen, Lu Chen, Kai Yu
Date: 2023-08-25 00:00:00
Tasks: Language Model Evaluation, Language Modeling, Language Modelling, Large Language Model

Recently, there has been growing interest in using Large Language Models (LLMs) for scientific research. Numerous benchmarks have been proposed to evaluate the ability of LLMs for scientific research. However, current benchmarks are mostly based on pre-collected objective questions. This design suffers from data leakage problem and lacks the evaluation of subjective Q/A ability. In this paper, we propose SciEval, a comprehensive and multi-disciplinary evaluation benchmark to address these issues. Based on Bloom's taxonomy, SciEval covers four dimensions to systematically evaluate scientific research ability. In particular, we design a "dynamic" subset based on scientific principles to prevent evaluation from potential data leakage. Both objective and subjective questions are included in SciEval. These characteristics make SciEval a more effective benchmark for scientific research ability evaluation of LLMs. Comprehensive experiments on most advanced LLMs show that, although GPT-4 achieves SOTA performance compared to other LLMs, there is still substantial room for improvement, especially for dynamic questions. The codes and data are publicly available on https://github.com/OpenDFM/SciEval.
Sentence Embedding Models for Ancient Greek Using Multilingual Knowledge Distillation
Authors: Kevin Krahn, Derrick Tate, Andrew C. Lamicela
Date: 2023-08-24 00:00:00
Tasks: Authorship Attribution, Knowledge Distillation, Lemmatization, Morphological Tagging, Part-Of-Speech Tagging, Retrieval, Semantic Retrieval, Semantic Similarity, Semantic Textual Similarity, Sentence, Sentence Embedding, Sentence-Embedding, Sentence Embeddings, Translation

Contextual language models have been trained on Classical languages, including Ancient Greek and Latin, for tasks such as lemmatization, morphological tagging, part of speech tagging, authorship attribution, and detection of scribal errors. However, high-quality sentence embedding models for these historical languages are significantly more difficult to achieve due to the lack of training data. In this work, we use a multilingual knowledge distillation approach to train BERT models to produce sentence embeddings for Ancient Greek text. The state-of-the-art sentence embedding approaches for high-resource languages use massive datasets, but our distillation approach allows our Ancient Greek models to inherit the properties of these models while using a relatively small amount of translated sentence data. We build a parallel sentence dataset using a sentence-embedding alignment method to align Ancient Greek documents with English translations, and use this dataset to train our models. We evaluate our models on translation search, semantic similarity, and semantic retrieval tasks and investigate translation bias. We make our training and evaluation datasets freely available at https://github.com/kevinkrahn/ancient-greek-datasets .
Towards a Holistic Approach: Understanding Sociodemographic Biases in NLP Models using an Interdisciplinary Lens
Authors: Pranav Narayanan Venkit
Date: 2023-08-24 00:00:00

The rapid growth in the usage and applications of Natural Language Processing (NLP) in various sociotechnical solutions has highlighted the need for a comprehensive understanding of bias and its impact on society. While research on bias in NLP has expanded, several challenges persist that require attention. These include the limited focus on sociodemographic biases beyond race and gender, the narrow scope of analysis predominantly centered on models, and the technocentric implementation approaches. This paper addresses these challenges and advocates for a more interdisciplinary approach to understanding bias in NLP. The work is structured into three facets, each exploring a specific aspect of bias in NLP.
Formal specification terminology for demographic agent-based models of fixed-step single-clocked simulations
Authors: Atiyah Elsheikh
Date: 2023-08-24 00:00:00

This document presents adequate formal terminology for the mathematical specification of a subset of Agent Based Models (ABMs) in the field of Demography. The simulation of the targeted ABMs follows a fixedstep single-clocked pattern. The proposed terminology further improves the model understanding and can act as a stand-alone protocol for the specification and optionally the documentation of a significant set of (demographic) ABMs. Nevertheless, it is imaginable the this terminology can serve as an inspiring basis for further improvement to the largely-informal widely-used model documentation and communication O.D.D. protocol [Grimm and et al., 2020, Amouroux et al., 2010] to reduce many sources of ambiguity which hinder model replications by other modelers. A published demographic model documentation, largely simplified version of the Lone Parent Model [Gostoli and Silverman, 2020] is separately published in [Elsheikh, 2023c] as illustration for the formal terminology presented here. The model was implemented in the Julia language [Elsheikh, 2023b] based on the Agents.jl julia package [Datseris et al., 2022].
Lexical Diversity in Kinship Across Languages and Dialects
Authors: Hadi Khalilia, Gábor Bella, Abed Alhakim Freihat, Shandy Darma, Fausto Giunchiglia
Date: 2023-08-24 00:00:00
Tasks: Diversity

Languages are known to describe the world in diverse ways. Across lexicons, diversity is pervasive, appearing through phenomena such as lexical gaps and untranslatability. However, in computational resources, such as multilingual lexical databases, diversity is hardly ever represented. In this paper, we introduce a method to enrich computational lexicons with content relating to linguistic diversity. The method is verified through two large-scale case studies on kinship terminology, a domain known to be diverse across languages and cultures: one case study deals with seven Arabic dialects, while the other one with three Indonesian languages. Our results, made available as browseable and downloadable computational resources, extend prior linguistics research on kinship terminology, and provide insight into the extent of diversity even within linguistically and culturally close communities.
Reinforcement Learning-assisted Evolutionary Algorithm: A Survey and Research Opportunities
Authors: Yanjie Song, Yutong Wu, Yangyang Guo, Ran Yan, P. N. Suganthan, Yue Zhang, Witold Pedrycz, Swagatam Das, Rammohan Mallipeddi, Oladayo Solomon Ajani. Qiang Feng
Date: 2023-08-25 00:00:00
Tasks: Attribute, Evolutionary Algorithms, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Evolutionary algorithms (EA), a class of stochastic search methods based on the principles of natural evolution, have received widespread acclaim for their exceptional performance in various real-world optimization problems. While researchers worldwide have proposed a wide variety of EAs, certain limitations remain, such as slow convergence speed and poor generalization capabilities. Consequently, numerous scholars actively explore improvements to algorithmic structures, operators, search patterns, etc., to enhance their optimization performance. Reinforcement learning (RL) integrated as a component in the EA framework has demonstrated superior performance in recent years. This paper presents a comprehensive survey on integrating reinforcement learning into the evolutionary algorithm, referred to as reinforcement learning-assisted evolutionary algorithm (RL-EA). We begin with the conceptual outlines of reinforcement learning and the evolutionary algorithm. We then provide a taxonomy of RL-EA. Subsequently, we discuss the RL-EA integration method, the RL-assisted strategy adopted by RL-EA, and its applications according to the existing literature. The RL-assisted procedure is divided according to the implemented functions including solution generation, learnable objective function, algorithm/operator/sub-population selection, parameter adaptation, and other strategies. Additionally, different attribute settings of RL in RL-EA are discussed. In the applications of RL-EA section, we also demonstrate the excellent performance of RL-EA on several benchmarks and a range of public datasets to facilitate a quick comparative study. Finally, we analyze potential directions for future research.
TC-LIF: A Two-Compartment Spiking Neuron Model for Long-Term Sequential Modelling
Authors: Shimin Zhang, Qu Yang, Chenxiang Ma, Jibin Wu, Haizhou Li, Kay Chen Tan
Date: 2023-08-25 00:00:00

The identification of sensory cues associated with potential opportunities and dangers is frequently complicated by unrelated events that separate useful cues by long delays. As a result, it remains a challenging task for state-of-the-art spiking neural networks (SNNs) to establish long-term temporal dependency between distant cues. To address this challenge, we propose a novel biologically inspired Two-Compartment Leaky Integrate-and-Fire spiking neuron model, dubbed TC-LIF. The proposed model incorporates carefully designed somatic and dendritic compartments that are tailored to facilitate learning long-term temporal dependencies. Furthermore, a theoretical analysis is provided to validate the effectiveness of TC-LIF in propagating error gradients over an extended temporal duration. Our experimental results, on a diverse range of temporal classification tasks, demonstrate superior temporal classification capability, rapid training convergence, and high energy efficiency of the proposed TC-LIF model. Therefore, this work opens up a myriad of opportunities for solving challenging temporal processing tasks on emerging neuromorphic computing systems. Our code is publicly available at https://github.com/ZhangShimin1/TC-LIF.
Eigenvector Dreaming
Authors: Marco Benedetti, Louis Carillo, Enzo Marinari, Marc Mèzard
Date: 2023-08-09 00:00:00

Among the performance-enhancing procedures for Hopfield-type networks that implement associative memory, Hebbian Unlearning (or dreaming) strikes for its simplicity and its clear biological interpretation. Yet, it does not easily lend itself to a clear analytical understanding. Here we show how Hebbian Unlearning can be effectively described in terms of a simple evolution of the spectrum and the eigenvectors of the coupling matrix. We use these ideas to design new dreaming algorithms that are effective from a computational point of view, and are analytically far more transparent than the original scheme.
On the Practicality of Dynamic Updates in Fast Searchable Encryption
Authors: Steven Willoughby
Date: 2023-08-25 00:00:00

Searchable encrypted (SE) indexing systems are a useful tool for utilizing cloud services to store and manage sensitive information. However, much of the work on SE systems to date has remained theoretical. In order to make them of practical use, more work is needed to develop optimal protocols and working models for them. This includes, in particular, the creation of a working update model in order to maintain an encrypted index of a dynamic document set such as an email inbox. I have created a working, real-world end-to-end SE implementation that satisfies these needs, including the first empirical performance evaluation of the dynamic SE update operation. In doing so, I show a viable path to move from the theoretical concepts described by previous researchers to a future production-worthy implementation and identify issues for follow-on investigation.
Learning and Optimization of Implicit Negative Feedback for Industrial Short-video Recommender System
Authors: Yunzhu Pan, Nian Li, Chen Gao, Jianxin Chang, Yanan Niu, Yang song, Depeng Jin, Yong Li
Date: 2023-08-25 00:00:00
Tasks: Recommendation Systems

Short-video recommendation is one of the most important recommendation applications in today's industrial information systems. Compared with other recommendation tasks, the enormous amount of feedback is the most typical characteristic. Specifically, in short-video recommendation, the easiest-to-collect user feedback is the skipping behavior, which leads to two critical challenges for the recommendation model. First, the skipping behavior reflects implicit user preferences, and thus, it is challenging for interest extraction. Second, this kind of special feedback involves multiple objectives, such as total watching time and skipping rate, which is also very challenging. In this paper, we present our industrial solution in Kuaishou, which serves billion-level users every day. Specifically, we deploy a feedback-aware encoding module that extracts user preferences, taking the impact of context into consideration. We further design a multi-objective prediction module which well distinguishes the relation and differences among different model objectives in the short-video recommendation. We conduct extensive online A/B tests, along with detailed and careful analysis, which verify the effectiveness of our solution.
MMBAttn: Max-Mean and Bit-wise Attention for CTR Prediction
Authors: Hasan Saribas, Cagri Yesil, Serdarcan Dilbaz, Halit Orenbas
Date: 2023-08-25 00:00:00
Tasks: Click-Through Rate Prediction, Feature Importance, Recommendation Systems

With the increasing complexity and scale of click-through rate (CTR) prediction tasks in online advertising and recommendation systems, accurately estimating the importance of features has become a critical aspect of developing effective models. In this paper, we propose an attention-based approach that leverages max and mean pooling operations, along with a bit-wise attention mechanism, to enhance feature importance estimation in CTR prediction. Traditionally, pooling operations such as max and mean pooling have been widely used to extract relevant information from features. However, these operations can lead to information loss and hinder the accurate determination of feature importance. To address this challenge, we propose a novel attention architecture that utilizes a bit-based attention structure that emphasizes the relationships between all bits in features, together with maximum and mean pooling. By considering the fine-grained interactions at the bit level, our method aims to capture intricate patterns and dependencies that might be overlooked by traditional pooling operations. To examine the effectiveness of the proposed method, experiments have been conducted on three public datasets. The experiments demonstrated that the proposed method significantly improves the performance of the base models to achieve state-of-the-art results.
A speech corpus of Quechua Collao for automatic dimensional emotion recognition
Authors: Rosa Y. G. Paccotacya-Yanque, Candy A. Huanca-Anquise, Judith Escalante-Calcina, Wilber R. Ramos-Lovón, Álvaro E. Cuno-Parari
Date: 2022-12-24 00:00:00
Proceeding: Scientific Data 2022 12
Tasks: Emotion Recognition, Speech Emotion Recognition

Automatic speech emotion recognition is an important research topic for human-computer interaction and affective computing. Over ten million people speak the Quechua language throughout South America, and one of the most known variants is the Quechua Collao one. However, this language can be considered a low resource for machine emotion recognition, creating a barrier for Quechua speakers who want to use this technology. Therefore, the contribution of this work is a 15 hours speech corpus in Quechua Collao, which is made publicly available to the research community. The corpus was created from a set of words and sentences explicitly collected for this task, divided into nine categorical emotions: happy, sad, bored, fear, sleepy, calm, excited, angry, and neutral. The annotation was performed on a 5-value discrete scale according to 3 dimensions: valence, arousal, and dominance. To demonstrate the usefulness of the corpus, we have performed speech emotion recognition using machine learning methods and neural networks.
Image Segmentation for Radar Signal Deinterleaving Using Deep Learning
Authors: Mustafa Atahan Nuhoglu, Yasar Kemal Alp, Mehmet Ege Can Ulusoy, Hakan Ali Cirpan
Date: 2022-07-11 00:00:00
Proceeding: IEEE Transactions on Aerospace and Electronic Systems 2022 7
Tasks: Deep Learning, Image Segmentation, Semantic Segmentation

Passive systems, such as electronic intelligence and electronic support measures systems, aim to extract necessary information from the received radar signals for situational awareness. To achieve this, the system must first deinterleave the radar signals simultaneously coming from different emitters, so that the pulse repetition interval (PRI) patterns will be revealed for further analysis and identification purposes. PRI transform is a well-known deinterleaving method that utilizes the complex autocorrelation function. There are two main versions of the method. The initial version detects only constant PRI schemes, while the second modified version is capable of detecting varying PRI schemes as well. Miss detection of varying PRI patterns is the drawback for the first version, while producing harmonics, especially at high PRI levels, is the disadvantage of the second one. To alleviate these problems, we propose an image segmentation method based on deep learning. The developed preprocessing step uses both versions of the PRI transform outputs to generate 2-D time–PRI images of the collected radar emissions, so that constant and varying PRI patterns are revealed. The images are concatenated and fed to the proposed network, which uses a practicable U-Net structure. The output of the network directly estimates the PRI levels of the existing radars and the time duration of the transmission jointly. In addition to qualitative and quantitative experiments on the synthetic datasets, qualitative experiments are conducted on real measurements, in which we demonstrate that the proposed method effectively utilizes PRI transform in the preprocessing step and outperforms both versions of the PRI transform in terms of accuracy, Jaccard index, structural similarity, and PRI estimation error metrics.
Understanding Turbo Codes: A Signal Processing Study
Authors: Xiang-Gen Xia
Date: 2023-08-25 00:00:00

In this paper, we study turbo codes from the digital signal processing point of view by defining turbo codes over the complex field. It is known that iterative decoding and interleaving between concatenated parallel codes are two key elements that make turbo codes perform significantly better than the conventional error control codes. This is analytically illustrated in this paper by showing that the decoded noise mean power in the iterative decoding decreases when the number of iterations increases, as long as the interleaving decorrelates the noise after each iterative decoding step. An analytic decreasing rate and the limit of the decoded noise mean power are given. The limit of the decoded noise mean power of the iterative decoding of a turbo code with two parallel codes with their rates less than 1/2 is one third of the noise power before the decoding, which can not be achieved by any non-turbo codes with the same rate. From this study, the role of designing a good interleaver can also be clearly seen.
OCTAL: Graph Representation Learning for LTL Model Checking
Authors: Prasita Mukherjee, Haoteng Yin
Date: 2023-08-19 00:00:00
Tasks: Binary Classification, Graph Representation Learning, Representation Learning

Model Checking is widely applied in verifying the correctness of complex and concurrent systems against a specification. Pure symbolic approaches while popular, suffer from the state space explosion problem due to cross product operations required that make them prohibitively expensive for large-scale systems and/or specifications. In this paper, we propose to use graph representation learning (GRL) for solving linear temporal logic (LTL) model checking, where the system and the specification are expressed by a B{\"u}chi automaton and an LTL formula, respectively. A novel GRL-based framework \model, is designed to learn the representation of the graph-structured system and specification, which reduces the model checking problem to binary classification. Empirical experiments on two model checking scenarios show that \model achieves promising accuracy, with up to $11\times$ overall speedup against canonical SOTA model checkers and $31\times$ for satisfiability checking alone.