pwc-archive/papers-with-abstracts huggingface dataset:
The geometry of hidden representations of large transformer models
Authors: Lucrezia Valeriani, Diego Doimo, Francesca Cuturello, Alessandro Laio, Alessio Ansuini, Alberto Cazzaniga
Date: 2023-02-01 00:00:00
Proceeding: NeurIPS 2023 11
Tasks: Image Reconstruction

Large transformers are powerful architectures used for self-supervised data analysis across various data types, including protein sequences, images, and text. In these models, the semantic structure of the dataset emerges from a sequence of transformations between one representation and the next. We characterize the geometric and statistical properties of these representations and how they change as we move through the layers. By analyzing the intrinsic dimension (ID) and neighbor composition, we find that the representations evolve similarly in transformers trained on protein language tasks and image reconstruction tasks. In the first layers, the data manifold expands, becoming high-dimensional, and then contracts significantly in the intermediate layers. In the last part of the model, the ID remains approximately constant or forms a second shallow peak. We show that the semantic information of the dataset is better expressed at the end of the first peak, and this phenomenon can be observed across many models trained on diverse datasets. Based on our findings, we point out an explicit strategy to identify, without supervision, the layers that maximize semantic content: representations at intermediate layers corresponding to a relative minimum of the ID profile are more suitable for downstream learning tasks.
Climate modeling with neural advection–diffusion equation
Authors: Hwangyong Choi, Jeongwhan Choi, Jeehyun Hwang, Kookjin Lee, Dongeun Lee, Noseong Park
Date: 2023-01-31 00:00:00
Proceeding: Knowledge and Information Systems 2023 1
Tasks: Weather Forecasting

Owing to the remarkable development of deep learning technology, there have been a series of efforts to build deep learning-based climate models. Whereas most of them utilize recurrent neural networks and/or graph neural networks, we design a novel climate model based on two concepts, the neural ordinary differential equation (NODE) and the advection–diffusion equation. The advection–diffusion equation is widely used for climate modeling because it describes many physical processes involving Brownian and bulk motions in climate systems. On the other hand, NODEs are to learn a latent governing equation of ODE from data. In our presented method, we combine them into a single framework and propose a concept, called neural advection–diffusion equation (NADE). Our NADE, equipped with the advection–diffusion equation and one more additional neural network to model inherent uncertainty, can learn an appropriate latent governing equation that best describes a given climate dataset. In our experiments with three real-world and two synthetic datasets and fourteen baselines, our method consistently outperforms existing baselines by non-trivial margins.
On Contrasting YAGO with GPT-J: An Experiment for Person-Related Attributes
Authors: David Martin-Moncunill, Miguel-Angel Sicilia, Lino González & Diego Rodríguez
Date: 2022-11-01 00:00:00
Proceeding: Conference 2022 11
Tasks: Knowledge Graphs

Language models (LMs) trained or large text corpora have demonstrated their superior performance in different language related tasks in the last years. These models automatically implicitly incorporate factual knowledge that can be used to complement existing Knowledge Graphs (KGs) that in most cases are structured from human curated databases. Here we report an experiment that attempts to gain insights about the extent to which LMs can generate factual information as that present in KGs. Concretely, we have tested such process using the English Wikipedia subset of YAGO and the GPT-J model for attributes related to individuals. Results show that the generation of correct factual information depends on the generation parameters of the model and are unevenly balanced across diverse individuals. Further, the LM can be used to populate further factual information, but it requires intermediate parsing to correctly map to KG attributes.
The IDEFICS community-oriented intervention programme: a new model for childhood obesity prevention in Europe?
Authors: S De Henauw, V Verbestel, S Mårild, G Barba, K Bammann, G Eiben, A Hebestreit, L Iacoviello, K Gallois, K Konstabel, É Kovács, L Lissner, L Maes, D Molnár, L A Moreno, L Reisch, A Siani, M Tornaritis, G Williams
Date: 2011-04-12 00:00:00
Proceeding: journal 2011 4

The European Union—as well as other parts of the world—faces a major challenge of increasing incidence of overweight/obesity. In particular, the increase in childhood obesity gives rise to a strong imperative for immediate action. Yet, little is known about the effectiveness of community interventions, and further research in this field is needed. There is, however, a growing consensus that such research should start from the paradigm that the current living environments tend to counteract healthy lifestyles. Questioning these environments thoroughly can help to develop new pathways for sustainable health-promoting communities. Against this background, the IDEFICS (Identification and prevention of dietary- and lifestyle-induced health effects in children and infants) study developed and implemented innovative community-oriented intervention programmes for obesity prevention and healthy lifestyle primarily in children aged 2–10 years in eight European countries: Sweden, Estonia, Germany, Belgium, Hungary, Italy, Spain and Cyprus.
Dual convolutional neural network with attention for image blind denoising
Authors: Wencong Wu, Guannan Lv, Yingying Duan, Peng Liang, Yungang Zhang, Yuelong Xia
Date: 2024-09-08 00:00:00
Proceeding: Multimedia Systems 2024 9
Tasks: Denoising, Image Denoising, Noise Estimation

Noise removal of images is an essential preprocessing procedure for many computer vision tasks. Currently, many denoising models based on deep neural networks can perform well in removing the noise with known distributions (i.e. the additive Gaussian white noise). However eliminating real noise is still a very challenging task, since real-world noise often does not simply follow one single type of distribution, and the noise may spatially vary. In this paper, we present a novel dual convolutional neural network (CNN) with attention for image blind denoising, named as the DCANet. To the best of our knowledge, the proposed DCANet is the first work that integrates both the dual CNN and attention mechanism for image denoising. The DCANet is composed of a noise estimation network, a spatial and channel attention module (SCAM), and a dual CNN. The noise estimation network is utilized to estimate the spatial distribution and the noise level in an image. The noisy image and its estimated noise are combined as the input of the SCAM, and a dual CNN contains two different branches is designed to learn the complementary features to obtain the denoised image. The experimental results have verified that the proposed DCANet can suppress both synthetic and real noise effectively. The code of DCANet is available at https://github.com/WenCongWu/DCANet.
Simple and precise static analysis of untrusted linux kernel extensions
Authors: Elazar Gershuni, Nadav Amit, Arie Gurfinkel, Nina Narodytska, Jorge A. Navas, Noam Rinetzky, Leonid Ryzhyk, Mooly Sagiv
Date: 2019-06-08 00:00:00
Proceeding: PLDI 2019 6

Extended Berkeley Packet Filter (eBPF) is a Linux subsystem that allows safely executing untrusted user-defined extensions inside the kernel. It relies on static analysis to protect the kernel against buggy and malicious extensions. As the eBPF ecosystem evolves to support more complex and diverse extensions, the limitations of its current verifier, including high rate of false positives, poor scalability, and lack of support for loops, have become a major barrier for developers.
We design a static analyzer for eBPF within the framework of abstract interpretation. Our choice of abstraction is based on common patterns found in many eBPF programs. We observed that eBPF programs manipulate memory in a rather disciplined way which permits analyzing them successfully with a scalable mixture of very-precise abstraction of certain bounded regions with coarser abstractions of other parts of the memory. We use the Zone domain, a simple domain that tracks differences between pairs of registers and offsets, to achieve precise and scalable analysis. We demonstrate that this abstraction is as precise in practice as more costly abstract domains like Octagon and Polyhedra.
Furthermore, our evaluation, based on hundreds of real-world eBPF programs, shows that the new tool generates no more false alarms than the existing Linux verifier, while it supports a wider class of programs (including programs with loops) and has better asymptotic complexity.
GREEN: a lightweight architecture using learnable wavelets and Riemannian geometry for biomarker exploration
Authors: Joseph Paillard, Jörg F. Hipp, Denis A. Engemann
Date: 2024-05-14 00:00:00
Proceeding: bioRxiv 2024 5
Tasks: EEG

Spectral analysis using wavelets has proven useful for analyzing electroencephalographic (EEG) signals and identifying biomarkers in a clinical context. Over the past decade, Riemannian geometry has crystalized as a theoretical framework providing robust methods for modeling biomedical outcomes and brain function from multi-channel EEG recordings. Combining both approaches yields applications with higher interpretability and efficiency. Yet these approaches rely on handcrafted rules and sequential optimization. On the contrary, the growing deep learning (DL) literature provides end-to-end trainable models for processing raw EEG. Despite state-of-the-art performance on various prediction tasks, clinical impact remains a challenge, which is at least in part due to their lack of interpretability and interoperability with established neuroscience concepts. In this work we introduce a new lightweight neural network for processing raw EEG data building on wavelet transforms and Riemannian geometry. The proposed architecture termed GREEN (Gabor Riemann EEGNet) is benchmarked on five different prediction tasks (age, sex, eyes-closed detection, dementia diagnosis, EEG pathology) from three different datasets (TUAB, CAUEEG, TDBRAIN) encompassing data from more than 5000 human participants. This new architecture significantly outperformed state-of-the-art non-deep models on several of the benchmarks and performed favorably compared to large DL models on the CAU bench-mark using orders of magnitude fewer parameters. Computational experiments revealed how GREEN facilitates learning sparse representations without compromising performance. We further demonstrate that the modularity of the GREEN architecture allows computing classical measures of phase synchrony, such as pairwise phase-locking values which are found to convey information relevant for predicting dementia diagnosis. Furthermore, the learned wavelets can easily be interpreted as bandpass filters, therefore making this architecture explainable by design. We illustrate this with a classical example of the Berger effect i.e. modulation of 8-10Hz power when closing the eyes. By integrating domain-knowledge in architecture choices, the proposed model benefits from desirable complexity-performance trade-off and learns interpretable representations of EEG. The source code is made publicly available.
COMEX Copper Futures Volatility Forecasting: Econometric Models and Deep Learning
Authors: Zian Wang, Xinyi Lu
Date: 2024-09-12 00:00:00
Tasks: Deep Learning

This paper investigates the forecasting performance of COMEX copper futures realized volatility across various high-frequency intervals using both econometric volatility models and deep learning recurrent neural network models. The econometric models considered are GARCH and HAR, while the deep learning models include RNN (Recurrent Neural Network), LSTM (Long Short-Term Memory), and GRU (Gated Recurrent Unit). In forecasting daily realized volatility for COMEX copper futures with a rolling window approach, the econometric models, particularly HAR, outperform recurrent neural networks overall, with HAR achieving the lowest QLIKE loss function value. However, when the data is replaced with hourly high-frequency realized volatility, the deep learning models outperform the GARCH model, and HAR attains a comparable QLIKE loss function value. Despite the black-box nature of machine learning models, the deep learning models demonstrate superior forecasting performance, surpassing the fixed QLIKE value of HAR in the experiment. Moreover, as the forecast horizon extends for daily realized volatility, deep learning models gradually close the performance gap with the GARCH model in certain loss function metrics. Nonetheless, HAR remains the most effective model overall for daily realized volatility forecasting in copper futures.
Data Efficient Child-Adult Speaker Diarization with Simulated Conversations
Authors: Anfeng Xu, Tiantian Feng, Helen Tager-Flusberg, Catherine Lord, Shrikanth Narayanan
Date: 2024-09-13 00:00:00
Tasks: speaker-diarization, Speaker Diarization, Transfer Learning

Automating child speech analysis is crucial for applications such as neurocognitive assessments. Speaker diarization, which identifies ``who spoke when'', is an essential component of the automated analysis. However, publicly available child-adult speaker diarization solutions are scarce due to privacy concerns and a lack of annotated datasets, while manually annotating data for each scenario is both time-consuming and costly. To overcome these challenges, we propose a data-efficient solution by creating simulated child-adult conversations using AudioSet. We then train a Whisper Encoder-based model, achieving strong zero-shot performance on child-adult speaker diarization using real datasets. The model performance improves substantially when fine-tuned with only 30 minutes of real train data, with LoRA further improving the transfer learning performance. The source code and the child-adult speaker diarization model trained on simulated conversations are publicly available.
The unknotting number, hard unknot diagrams, and reinforcement learning
Authors: Taylor Applebaum, Sam Blackwell, Alex Davies, Thomas Edlich, András Juhász, Marc Lackenby, Nenad Tomašev, Daniel Zheng
Date: 2024-09-13 00:00:00
Tasks: reinforcement-learning, Reinforcement Learning

We have developed a reinforcement learning agent that often finds a minimal sequence of unknotting crossing changes for a knot diagram with up to 200 crossings, hence giving an upper bound on the unknotting number. We have used this to determine the unknotting number of 57k knots. We took diagrams of connected sums of such knots with oppositely signed signatures, where the summands were overlaid. The agent has found examples where several of the crossing changes in an unknotting collection of crossings result in hyperbolic knots. Based on this, we have shown that, given knots $K$ and $K'$ that satisfy some mild assumptions, there is a diagram of their connected sum and $u(K) + u(K')$ unknotting crossings such that changing any one of them results in a prime knot. As a by-product, we have obtained a dataset of 2.6 million distinct hard unknot diagrams; most of them under 35 crossings. Assuming the additivity of the unknotting number, we have determined the unknotting number of 43 at most 12-crossing knots for which the unknotting number is unknown.
Agents in Software Engineering: Survey, Landscape, and Vision
Authors: Yanlin Wang, Wanjun Zhong, Yanxian Huang, Ensheng Shi, Min Yang, Jiachi Chen, Hui Li, Yuchi Ma, Qianxiang Wang, Zibin Zheng
Date: 2024-09-13 00:00:00
Tasks: Survey

In recent years, Large Language Models (LLMs) have achieved remarkable success and have been widely used in various downstream tasks, especially in the tasks of the software engineering (SE) field. We find that many studies combining LLMs with SE have employed the concept of agents either explicitly or implicitly. However, there is a lack of an in-depth survey to sort out the development context of existing works, analyze how existing works combine the LLM-based agent technologies to optimize various tasks, and clarify the framework of LLM-based agents in SE. In this paper, we conduct the first survey of the studies on combining LLM-based agents with SE and present a framework of LLM-based agents in SE which includes three key modules: perception, memory, and action. We also summarize the current challenges in combining the two fields and propose future opportunities in response to existing challenges. We maintain a GitHub repository of the related papers at: https://github.com/DeepSoftwareAnalytics/Awesome-Agent4SE.
Towards Leveraging Contrastively Pretrained Neural Audio Embeddings for Recommender Tasks
Authors: Florian Grötschla, Luca Strässle, Luca A. Lanzendörfer, Roger Wattenhofer
Date: 2024-09-13 00:00:00
Tasks: Collaborative Filtering, Music Recommendation, Recommendation Systems

Music recommender systems frequently utilize network-based models to capture relationships between music pieces, artists, and users. Although these relationships provide valuable insights for predictions, new music pieces or artists often face the cold-start problem due to insufficient initial information. To address this, one can extract content-based information directly from the music to enhance collaborative-filtering-based methods. While previous approaches have relied on hand-crafted audio features for this purpose, we explore the use of contrastively pretrained neural audio embedding models, which offer a richer and more nuanced representation of music. Our experiments demonstrate that neural embeddings, particularly those generated with the Contrastive Language-Audio Pretraining (CLAP) model, present a promising approach to enhancing music recommendation tasks within graph-based frameworks.
AI-LieDar: Examine the Trade-off Between Utility and Truthfulness in LLM Agents
Authors: Zhe Su, Xuhui Zhou, Sanketh Rangreji, Anubha Kabra, Julia Mendelsohn, Faeze Brahman, Maarten Sap
Date: 2024-09-13 00:00:00
Tasks: AI Agent, Navigate

To be safely and successfully deployed, LLMs must simultaneously satisfy truthfulness and utility goals. Yet, often these two goals compete (e.g., an AI agent assisting a used car salesman selling a car with flaws), partly due to ambiguous or misleading user instructions. We propose AI-LieDar, a framework to study how LLM-based agents navigate scenarios with utility-truthfulness conflicts in a multi-turn interactive setting. We design a set of realistic scenarios where language agents are instructed to achieve goals that are in conflict with being truthful during a multi-turn conversation with simulated human agents. To evaluate the truthfulness at large scale, we develop a truthfulness detector inspired by psychological literature to assess the agents' responses. Our experiment demonstrates that all models are truthful less than 50% of the time, although truthfulness and goal achievement (utility) rates vary across models. We further test the steerability of LLMs towards truthfulness, finding that models follow malicious instructions to deceive, and even truth-steered models can still lie. These findings reveal the complex nature of truthfulness in LLMs and underscore the importance of further research to ensure the safe and reliable deployment of LLMs and AI agents.
VAE Explainer: Supplement Learning Variational Autoencoders with Interactive Visualization
Authors: Donald Bertucci, Alex Endert
Date: 2024-09-13 00:00:00
Tasks: Math

Variational Autoencoders are widespread in Machine Learning, but are typically explained with dense math notation or static code examples. This paper presents VAE Explainer, an interactive Variational Autoencoder running in the browser to supplement existing static documentation (e.g., Keras Code Examples). VAE Explainer adds interactions to the VAE summary with interactive model inputs, latent space, and output. VAE Explainer connects the high-level understanding with the implementation: annotated code and a live computational graph. The VAE Explainer interactive visualization is live at https://xnought.github.io/vae-explainer and the code is open source at https://github.com/xnought/vae-explainer.
Contri(e)ve: Context + Retrieve for Scholarly Question Answering
Authors: Kanchan Shivashankar, Nadine Steinmetz
Date: 2024-09-13 00:00:00
Tasks: Information Retrieval, Knowledge Graphs, Language Modeling, Language Modelling, Large Language Model, Prompt Engineering, Question Answering, Retrieval

Scholarly communication is a rapid growing field containing a wealth of knowledge. However, due to its unstructured and document format, it is challenging to extract useful information from them through conventional document retrieval methods. Scholarly knowledge graphs solve this problem, by representing the documents in a semantic network, providing, hidden insights, summaries and ease of accessibility through queries. Naturally, question answering for scholarly graphs expands the accessibility to a wider audience. But some of the knowledge in this domain is still presented as unstructured text, thus requiring a hybrid solution for question answering systems. In this paper, we present a two step solution using open source Large Language Model(LLM): Llama3.1 for Scholarly-QALD dataset. Firstly, we extract the context pertaining to the question from different structured and unstructured data sources: DBLP, SemOpenAlex knowledge graphs and Wikipedia text. Secondly, we implement prompt engineering to improve the information retrieval performance of the LLM. Our approach achieved an F1 score of 40% and also observed some anomalous responses from the LLM, that are discussed in the final part of the paper.
SGFormer: Single-Layer Graph Transformers with Approximation-Free Linear Complexity
Authors: Qitian Wu, Kai Yang, Hengrui Zhang, David Wipf, Junchi Yan
Date: 2024-09-13 00:00:00
Tasks: Deep Attention, Representation Learning

Learning representations on large graphs is a long-standing challenge due to the inter-dependence nature. Transformers recently have shown promising performance on small graphs thanks to its global attention for capturing all-pair interactions beyond observed structures. Existing approaches tend to inherit the spirit of Transformers in language and vision tasks, and embrace complicated architectures by stacking deep attention-based propagation layers. In this paper, we attempt to evaluate the necessity of adopting multi-layer attentions in Transformers on graphs, which considerably restricts the efficiency. Specifically, we analyze a generic hybrid propagation layer, comprised of all-pair attention and graph-based propagation, and show that multi-layer propagation can be reduced to one-layer propagation, with the same capability for representation learning. It suggests a new technical path for building powerful and efficient Transformers on graphs, particularly through simplifying model architectures without sacrificing expressiveness. As exemplified by this work, we propose a Simplified Single-layer Graph Transformers (SGFormer), whose main component is a single-layer global attention that scales linearly w.r.t. graph sizes and requires none of any approximation for accommodating all-pair interactions. Empirically, SGFormer successfully scales to the web-scale graph ogbn-papers100M, yielding orders-of-magnitude inference acceleration over peer Transformers on medium-sized graphs, and demonstrates competitiveness with limited labeled data.