pwc-archive/papers-with-abstracts huggingface dataset:
Trillion Dollar Words: A New Financial Dataset, Task & Market Analysis
Authors: Agam Shah, Suvan Paturi, Sudheer Chava
Date: 2023-05-13 00:00:00

Monetary policy pronouncements by Federal Open Market Committee (FOMC) are a major driver of financial market returns. We construct the largest tokenized and annotated dataset of FOMC speeches, meeting minutes, and press conference transcripts in order to understand how monetary policy influences financial markets. In this study, we develop a novel task of hawkish-dovish classification and benchmark various pre-trained language models on the proposed dataset. Using the best-performing model (RoBERTa-large), we construct a measure of monetary policy stance for the FOMC document release days. To evaluate the constructed measure, we study its impact on the treasury market, stock market, and macroeconomic indicators. Our dataset, models, and code are publicly available on Huggingface and GitHub under CC BY-NC 4.0 license.
GPT-Sentinel: Distinguishing Human and ChatGPT Generated Content
Authors: Yutian Chen, Hao Kang, Vivian Zhai, Liangze Li, Rita Singh, Bhiksha Raj
Date: 2023-05-13 00:00:00
Tasks: text-classification, Text Classification

This paper presents a novel approach for detecting ChatGPT-generated vs. human-written text using language models. To this end, we first collected and released a pre-processed dataset named OpenGPTText, which consists of rephrased content generated using ChatGPT. We then designed, implemented, and trained two different models for text classification, using Robustly Optimized BERT Pretraining Approach (RoBERTa) and Text-to-Text Transfer Transformer (T5), respectively. Our models achieved remarkable results, with an accuracy of over 97% on the test dataset, as evaluated through various metrics. Furthermore, we conducted an interpretability study to showcase our model's ability to extract and differentiate key features between human-written and ChatGPT-generated text. Our findings provide important insights into the effective use of language models to detect generated text.
RC3: Regularized Contrastive Cross-lingual Cross-modal Pre-training
Authors: Chulun Zhou, Yunlong Liang, Fandong Meng, Jinan Xu, Jinsong Su, Jie zhou
Date: 2023-05-13 00:00:00
Tasks: Contrastive Learning, Machine Translation

Multilingual vision-language (V&L) pre-training has achieved remarkable progress in learning universal representations across different modalities and languages. In spite of recent success, there still remain challenges limiting further improvements of V&L pre-trained models in multilingual settings. Particularly, current V&L pre-training methods rely heavily on strictly-aligned multilingual image-text pairs generated from English-centric datasets through machine translation. However, the cost of collecting and translating such strictly-aligned datasets is usually unbearable. In this paper, we propose Regularized Contrastive Cross-lingual Cross-modal (RC^3) pre-training, which further exploits more abundant weakly-aligned multilingual image-text pairs. Specifically, we design a regularized cross-lingual visio-textual contrastive learning objective that constrains the representation proximity of weakly-aligned visio-textual inputs according to textual relevance. Besides, existing V&L pre-training approaches mainly deal with visual inputs by either region-of-interest (ROI) features or patch embeddings. We flexibly integrate the two forms of visual features into our model for pre-training and downstream multi-modal tasks. Extensive experiments on 5 downstream multi-modal tasks across 6 languages demonstrate the effectiveness of our proposed method over competitive contrast models with stronger zero-shot capability.
Dual Use Concerns of Generative AI and Large Language Models
Authors: Alexei Grinbaum, Laurynas Adomaitis
Date: 2023-05-13 00:00:00

We suggest the implementation of the Dual Use Research of Concern (DURC) framework, originally designed for life sciences, to the domain of generative AI, with a specific focus on Large Language Models (LLMs). With its demonstrated advantages and drawbacks in biological research, we believe the DURC criteria can be effectively redefined for LLMs, potentially contributing to improved AI governance. Acknowledging the balance that must be struck when employing the DURC framework, we highlight its crucial political role in enhancing societal awareness of the impact of generative AI. As a final point, we offer a series of specific recommendations for applying the DURC approach to LLM research.
The Geometry of Multilingual Language Models: An Equality Lens
Authors: Cheril Shah, Yashashree Chandak, Manan Suri
Date: 2023-05-13 00:00:00

Understanding the representations of different languages in multilingual language models is essential for comprehending their cross-lingual properties, predicting their performance on downstream tasks, and identifying any biases across languages. In our study, we analyze the geometry of three multilingual language models in Euclidean space and find that all languages are represented by unique geometries. Using a geometric separability index we find that although languages tend to be closer according to their linguistic family, they are almost separable with languages from other families. We also introduce a Cross-Lingual Similarity Index to measure the distance of languages with each other in the semantic space. Our findings indicate that the low-resource languages are not represented as good as high resource languages in any of the models
Frequency-aware Dimension Selection for Static Word Embedding by Mixed Product Distance
Authors: Lingfeng Shen, Haiyun Jiang, Lemao Liu, Ying Chen
Date: 2023-05-13 00:00:00
Tasks: Word Embeddings

Static word embedding is still useful, particularly for context-unavailable tasks, because in the case of no context available, pre-trained language models often perform worse than static word embeddings. Although dimension is a key factor determining the quality of static word embeddings, automatic dimension selection is rarely discussed. In this paper, we investigate the impact of word frequency on the dimension selection, and empirically find that word frequency is so vital that it needs to be taken into account during dimension selection. Based on such an empirical finding, this paper proposes a dimension selection method that uses a metric (Mixed Product Distance, MPD) to select a proper dimension for word embedding algorithms without training any word embedding. Through applying a post-processing function to oracle matrices, the MPD-based method can de-emphasize the impact of word frequency. Experiments on both context-unavailable and context-available tasks demonstrate the better efficiency-performance trade-off of our MPD-based dimension selection method over baselines.
ACCENT: An Automatic Event Commonsense Evaluation Metric for Open-Domain Dialogue Systems
Authors: Sarik Ghazarian, Yijia Shao, Rujun Han, Aram Galstyan, Nanyun Peng
Date: 2023-05-12 00:00:00

Commonsense reasoning is omnipresent in human communications and thus is an important feature for open-domain dialogue systems. However, evaluating commonsense in dialogue systems is still an open challenge. We take the first step by focusing on event commonsense that considers events and their relations, and is crucial in both dialogues and general commonsense reasoning. We propose ACCENT, an event commonsense evaluation metric empowered by commonsense knowledge bases (CSKBs). ACCENT first extracts event-relation tuples from a dialogue, and then evaluates the response by scoring the tuples in terms of their compatibility with the CSKB. To evaluate ACCENT, we construct the first public event commonsense evaluation dataset for open-domain dialogues. Our experiments show that ACCENT is an efficient metric for event commonsense evaluation, which achieves higher correlations with human judgments than existing baselines.
Constructing Holistic Measures for Social Biases in Masked Language Models
Authors: Yang Liu, Yuexian Hou
Date: 2023-05-12 00:00:00

Masked Language Models (MLMs) have been successful in many natural language processing tasks. However, real-world stereotype biases are likely to be reflected in MLMs due to their learning from large text corpora. Most of the evaluation metrics proposed in the past adopt different masking strategies, designed with the log-likelihood of MLMs. They lack holistic considerations such as variance for stereotype bias and anti-stereotype bias samples. In this paper, the log-likelihoods of stereotype bias and anti-stereotype bias samples output by MLMs are considered Gaussian distributions. Two evaluation metrics, Kullback Leibler Divergence Score (KLDivS) and Jensen Shannon Divergence Score (JSDivS) are proposed to evaluate social biases in MLMs The experimental results on the public datasets StereoSet and CrowS-Pairs demonstrate that KLDivS and JSDivS are more stable and interpretable compared to the metrics proposed in the past.
NL2TL: Transforming Natural Languages to Temporal Logics using Large Language Models
Authors: Yongchao Chen, Rujul Gandhi, Yang Zhang, Chuchu Fan
Date: 2023-05-12 00:00:00

Temporal Logic (TL) can be used to rigorously specify complex high-level specification for systems in many engineering applications. The translation between natural language (NL) and TL has been under-explored due to the lack of dataset and generalizable model across different application domains. In this paper, we propose an accurate and generalizable transformation framework of English instructions from NL to TL, exploring the use of Large Language Models (LLMs) at multiple stages. Our contributions are twofold. First, we develop a framework to create a dataset of NL-TL pairs combining LLMs and human annotation. We publish a dataset with 28K NL-TL pairs. Then, we finetune T5 models on the lifted versions (i.e., the specific Atomic Propositions (AP) are hidden) of the NL and TL. The enhanced generalizability originates from two aspects: 1) Usage of lifted NL-TL characterizes common logical structures, without constraints of specific domains. 2) Application of LLMs in dataset creation largely enhances corpus richness. We test the generalization of trained models on five varied domains. To achieve full NL-TL transformation, we either combine the lifted model with AP recognition task or do the further finetuning on each specific domain. During the further finetuning, our model achieves higher accuracy (>95%) using only <10% training data, compared with the baseline sequence to sequence (Seq2Seq) model.
Parallel Tree Kernel Computation
Authors: Souad Taouti, Hadda Cherroun, Djelloul Ziadi
Date: 2023-05-12 00:00:00

Tree kernels are fundamental tools that have been leveraged in many applications, particularly those based on machine learning for Natural Language Processing tasks. In this paper, we devise a parallel implementation of the sequential algorithm for the computation of some tree kernels of two finite sets of trees (Ouali-Sebti, 2015). Our comparison is narrowed on a sequential implementation of SubTree kernel computation. This latter is mainly reduced to an intersection of weighted tree automata. Our approach relies on the nature of the data parallelism source inherent in this computation by deploying the MapReduce paradigm. One of the key benefits of our approach is its versatility in being adaptable to a wide range of substructure tree kernel-based learning methods. To evaluate the efficacy of our parallel approach, we conducted a series of experiments that compared it against the sequential version using a diverse set of synthetic tree language datasets that were manually crafted for our analysis. The reached results clearly demonstrate that the proposed parallel algorithm outperforms the sequential one in terms of latency.
Accelerating genetic optimization of nonlinear model predictive control by learning optimal search space size
Authors: Eslam Mostafa, Hussein A. Aly, Ahmed Elliethy
Date: 2023-05-14 00:00:00
Tasks: Evolutionary Algorithms, Model Predictive Control

Genetic algorithm (GA) is typically used to solve nonlinear model predictive control's optimization problem. However, the size of the search space in which the GA searches for the optimal control inputs is crucial for its applicability to fast-response systems. This paper proposes accelerating the genetic optimization of NMPC by learning optimal search space size. The approach trains a multivariate regression model to adaptively predict the best smallest size of the search space in every control cycle. The proposed approach reduces the GA's computational time, improves the chance of convergence to better control inputs, and provides a stable and feasible solution. The proposed approach was evaluated on three nonlinear systems and compared to four other evolutionary algorithms implemented in a processor-in-the-loop fashion. The results show that the proposed approach provides a 17-45\% reduction in computational time and increases the convergence rate by 35-47\%. The source code is available on GitHub.
Quantization in Spiking Neural Networks
Authors: Bernhard A. Moser, Michael Lunglmayr
Date: 2023-05-13 00:00:00
Tasks: Quantization

In spiking neural networks (SNN), at each node, an incoming sequence of weighted Dirac pulses is converted into an output sequence of weighted Dirac pulses by a leaky-integrate-and-fire (LIF) neuron model based on spike aggregation and thresholding. We show that this mapping can be understood as a quantization operator and state a corresponding formula for the quantization error by means of the Alexiewicz norm. This analysis has implications for rethinking re-initialization in the LIF model, leading to the proposal of 'reset-to-mod' as a modulo-based reset variant.
Can the Problem-Solving Benefits of Quality Diversity Be Obtained Without Explicit Diversity Maintenance?
Authors: Ryan Boldi, Lee Spector
Date: 2023-05-12 00:00:00
Tasks: Dimensionality Reduction, Diversity

When using Quality Diversity (QD) optimization to solve hard exploration or deceptive search problems, we assume that diversity is extrinsically valuable. This means that diversity is important to help us reach an objective, but is not an objective in itself. Often, in these domains, practitioners benchmark their QD algorithms against single objective optimization frameworks. In this paper, we argue that the correct comparison should be made to \emph{multi-objective} optimization frameworks. This is because single objective optimization frameworks rely on the aggregation of sub-objectives, which could result in decreased information that is crucial for maintaining diverse populations automatically. In order to facilitate a fair comparison between quality diversity and multi-objective optimization, we present a method that utilizes dimensionality reduction to automatically determine a set of behavioral descriptors for an individual, as well as a set of objectives for an individual to solve. Using the former, one can generate solutions using standard quality diversity optimization techniques, and using the latter, one can generate solutions using standard multi-objective optimization techniques. This allows for a level comparison between these two classes of algorithms, without requiring domain and algorithm specific modifications to facilitate a comparison.
Seeing is Believing: Brain-Inspired Modular Training for Mechanistic Interpretability
Authors: Ziming Liu, Eric Gan, Max Tegmark
Date: 2023-05-04 00:00:00

We introduce Brain-Inspired Modular Training (BIMT), a method for making neural networks more modular and interpretable. Inspired by brains, BIMT embeds neurons in a geometric space and augments the loss function with a cost proportional to the length of each neuron connection. We demonstrate that BIMT discovers useful modular neural networks for many simple tasks, revealing compositional structures in symbolic formulas, interpretable decision boundaries and features for classification, and mathematical structure in algorithmic datasets. The ability to directly see modules with the naked eye can complement current mechanistic interpretability strategies such as probes, interventions or staring at all weights.
On the connections between optimization algorithms, Lyapunov functions, and differential equations: theory and insights
Authors: Paul Dobson, Jesus Maria Sanz-Serna, Konstantinos Zygalakis
Date: 2023-05-15 00:00:00

We revisit the general framework introduced by Fazylab et al. (SIAM J. Optim. 28, 2018) to construct Lyapunov functions for optimization algorithms in discrete and continuous time. For smooth, strongly convex objective functions, we relax the requirements necessary for such a construction. As a result we are able to prove for Polyak's ordinary differential equations and for a two-parameter family of Nesterov algorithms rates of convergence that improve on those available in the literature. We analyse the interpretation of Nesterov algorithms as discretizations of the Polyak equation. We show that the algorithms are instances of Additive Runge-Kutta integrators and discuss the reasons why most discretizations of the differential equation do not result in optimization algorithms with acceleration. We also introduce a modification of Polyak's equation and study its convergence properties. Finally we extend the general framework to the stochastic scenario and consider an application to random algorithms with acceleration for overparameterized models; again we are able to prove convergence rates that improve on those in the literature.
Kernel-based Joint Independence Tests for Multivariate Stationary and Non-stationary Time Series
Authors: Zhaolu Liu, Robert L. Peach, Felix Laumann, Sara Vallejo Mengod, Mauricio Barahona
Date: 2023-05-15 00:00:00
Tasks: Time Series

Multivariate time series data that capture the temporal evolution of interconnected systems are ubiquitous in diverse areas. Understanding the complex relationships and potential dependencies among co-observed variables is crucial for the accurate statistical modelling and analysis of such systems. Here, we introduce kernel-based statistical tests of joint independence in multivariate time series by extending the $d$-variable Hilbert-Schmidt independence criterion (dHSIC) to encompass both stationary and non-stationary processes, thus allowing broader real-world applications. By leveraging resampling techniques tailored for both single- and multiple-realisation time series, we show how the method robustly uncovers significant higher-order dependencies in synthetic examples, including frequency mixing data and logic gates, as well as real-world climate, neuroscience, and socioeconomic data. Our method adds to the mathematical toolbox for the analysis of multivariate time series and can aid in uncovering high-order interactions in data.
Orthogonal polynomial approximation and Extended Dynamic Mode Decomposition in chaos
Authors: Caroline L. Wormell
Date: 2023-05-14 00:00:00

Extended Dynamic Mode Decomposition (EDMD) is a data-driven tool for forecasting and model reduction of dynamics, which has been extensively taken up in the physical sciences. While the method is conceptually simple, in deterministic chaos it is unclear what its properties are or even what it converges to. In particular, it is not clear how EDMD's least-squares approximation treats the classes of differentiable functions on which chaotic systems act. We develop for the first time a general, rigorous theory of EDMD on the simplest examples of chaotic maps: analytic expanding maps of the circle. To do this, we prove a new, basic approximation result in the theory of orthogonal polynomials on the unit circle (OPUC) and apply methods from transfer operator theory. We show that in the infinite-data limit, the least-squares projection error is exponentially small for trigonometric polynomial observable dictionaries. As a result, we show that forecasts and Koopman spectral data produced using EDMD in this setting converge to the physically meaningful limits, exponentially fast with respect to the size of the dictionary. This demonstrates that with only a relatively small polynomial dictionary, EDMD can be very effective, even when the sampling measure is not uniform. Furthermore, our OPUC result suggests that data-based least-squares projection may be a very effective approximation strategy more generally.
Efficient and Effective Tree-based and Neural Learning to Rank
Authors: Sebastian Bruch, Claudio Lucchese, Franco Maria Nardini
Date: 2023-05-15 00:00:00
Tasks: Information Retrieval, Learning-To-Rank, Retrieval

This monograph takes a step towards promoting the study of efficiency in the era of neural information retrieval by offering a comprehensive survey of the literature on efficiency and effectiveness in ranking, and to a limited extent, retrieval. This monograph was inspired by the parallels that exist between the challenges in neural network-based ranking solutions and their predecessors, decision forest-based learning to rank models, as well as the connections between the solutions the literature to date has to offer. We believe that by understanding the fundamentals underpinning these algorithmic and data structure solutions for containing the contentious relationship between efficiency and effectiveness, one can better identify future directions and more efficiently determine the merits of ideas. We also present what we believe to be important research directions in the forefront of efficiency and effectiveness in retrieval and ranking.
GCNSLIM: Graph Convolutional Network with Sparse Linear Methods for E-government Service Recommendation
Authors: Lingyuan Kong, Hao Ding, Guangwei Hu
Date: 2023-05-15 00:00:00
Tasks: Music Recommendation

Graph Convolutional Networks have made significant strides in Collabora-tive Filtering recommendations. However, existing GCN-based CF methods are mainly based on matrix factorization and incorporate some optimization tech-niques to enhance performance, which are not enough to handle the complexities of diverse real-world recommendation scenarios. E-government service recommendation is a crucial area for recommendation re-search as it involves rigid aspects of people's lives. However, it has not received ad-equate attention in comparison to other recommendation scenarios like news and music recommendation. We empirically find that when existing GCN-based CF methods are directly applied to e-government service recommendation, they are limited by the MF framework and showing poor performance. This is because MF's equal treatment of users and items is not appropriate for scenarios where the number of users and items is unbalanced. In this work, we propose a new model, GCNSLIM, which combines GCN and sparse linear methods instead of combining GCN and MF to accommodate e-government service recommendation. In particular, GCNSLIM explicitly injects high-order collaborative signals obtained from multi-layer light graph convolutions into the item similarity matrix in the SLIM frame-work, effectively improving the recommendation accuracy. In addition, we propose two optimization measures, removing layer 0 embedding and adding nonlinear acti-vation, to further adapt to the characteristics of e-government service recommenda-tion scenarios. Furthermore, we propose a joint optimization mode to adapt to more diverse recommendation scenarios. We conduct extensive experiments on a real e-government service dataset and a common public dataset and demonstrate the ef-fectiveness of GCNSLIM in recommendation accuracy and operational performance.