pwc-archive/papers-with-abstracts huggingface dataset:
Speech ReaLLM -- Real-time Streaming Speech Recognition with Multimodal LLMs by Teaching the Flow of Time
Authors: Frank Seide, Morrie Doulaty, Yangyang Shi, Yashesh Gaur, Junteng Jia, Chunyang Wu
Date: 2024-06-13 00:00:00
Tasks: Decoder, speech-recognition, Speech Recognition

We introduce Speech ReaLLM, a new ASR architecture that marries "decoder-only" ASR with the RNN-T to make multimodal LLM architectures capable of real-time streaming. This is the first "decoder-only" ASR architecture designed to handle continuous audio without explicit end-pointing. Speech ReaLLM is a special case of the more general ReaLLM ("real-time LLM") approach, also introduced here for the first time. The idea is inspired by RNN-T: Instead of generating a response only at the end of a user prompt, generate after every input token received in real time (it is often empty). On Librispeech "test", an 80M Speech ReaLLM achieves WERs of 3.0% and 7.4% in real time (without an external LM or auxiliary loss). This is only slightly above a 3x larger Attention-Encoder-Decoder baseline. We also show that this way, an LLM architecture can learn to represent and reproduce the flow of time; and that a pre-trained 7B LLM can be fine-tuned to do reasonably well on this task.
Talking Heads: Understanding Inter-layer Communication in Transformer Language Models
Authors: Jack Merullo, Carsten Eickhoff, Ellie Pavlick
Date: 2024-06-13 00:00:00
Tasks: Language Modelling

Although it is known that transformer language models (LMs) pass features from early layers to later layers, it is not well understood how this information is represented and routed by the model. We analyze a mechanism used in two LMs to selectively inhibit items in a context in one task, and find that it underlies a commonly used abstraction across many context-retrieval behaviors. Specifically, we find that models write into low-rank subspaces of the residual stream to represent features which are then read out by later layers, forming low-rank communication channels (Elhage et al., 2021) between layers. A particular 3D subspace in model activations in GPT-2 can be traversed to positionally index items in lists, and we show that this mechanism can explain an otherwise arbitrary-seeming sensitivity of the model to the order of items in the prompt. That is, the model has trouble copying the correct information from context when many items ``crowd" this limited space. By decomposing attention heads with the Singular Value Decomposition (SVD), we find that previously described interactions between heads separated by one or more layers can be predicted via analysis of their weight matrices alone. We show that it is possible to manipulate the internal model representations as well as edit model weights based on the mechanism we discover in order to significantly improve performance on our synthetic Laundry List task, which requires recall from a list, often improving task accuracy by over 20%. Our analysis reveals a surprisingly intricate interpretable structure learned from language model pretraining, and helps us understand why sophisticated LMs sometimes fail in simple domains, facilitating future analysis of more complex behaviors.
Newswire: A Large-Scale Structured Database of a Century of Historical News
Authors: Emily Silcock, Abhishek Arora, Luca D'Amico-Wong, Melissa Dell
Date: 2024-06-13 00:00:00
Tasks: Articles, Entity Disambiguation, Language Modeling, Language Modelling, Topic Classification

In the U.S. historically, local newspapers drew their content largely from newswires like the Associated Press. Historians argue that newswires played a pivotal role in creating a national identity and shared understanding of the world, but there is no comprehensive archive of the content sent over newswires. We reconstruct such an archive by applying a customized deep learning pipeline to hundreds of terabytes of raw image scans from thousands of local newspapers. The resulting dataset contains 2.7 million unique public domain U.S. newswire articles, written between 1878 and 1977. Locations in these articles are georeferenced, topics are tagged using customized neural topic classification, named entities are recognized, and individuals are disambiguated to Wikipedia using a novel entity disambiguation model. To construct the Newswire dataset, we first recognize newspaper layouts and transcribe around 138 millions structured article texts from raw image scans. We then use a customized neural bi-encoder model to de-duplicate reproduced articles, in the presence of considerable abridgement and noise, quantifying how widely each article was reproduced. A text classifier is used to ensure that we only include newswire articles, which historically are in the public domain. The structured data that accompany the texts provide rich information about the who (disambiguated individuals), what (topics), and where (georeferencing) of the news that millions of Americans read over the course of a century. We also include Library of Congress metadata information about the newspapers that ran the articles on their front pages. The Newswire dataset is useful both for large language modeling - expanding training data beyond what is available from modern web texts - and for studying a diversity of questions in computational linguistics, social science, and the digital humanities.
Evolving Self-Assembling Neural Networks: From Spontaneous Activity to Experience-Dependent Learning
Authors: Erwan Plantec, Joachin W. Pedersen, Milton L. Montero, Eleni Nisioti, Sebastian Risi
Date: 2024-06-14 00:00:00

Biological neural networks are characterized by their high degree of plasticity, a core property that enables the remarkable adaptability of natural organisms. Importantly, this ability affects both the synaptic strength and the topology of the nervous systems. Artificial neural networks, on the other hand, have been mainly designed as static, fully connected structures that can be notoriously brittle in the face of changing environments and novel inputs. Building on previous works on Neural Developmental Programs (NDPs), we propose a class of self-organizing neural networks capable of synaptic and structural plasticity in an activity and reward-dependent manner which we call Lifelong Neural Developmental Program (LNDP). We present an instance of such a network built on the graph transformer architecture and propose a mechanism for pre-experience plasticity based on the spontaneous activity of sensory neurons. Our results demonstrate the ability of the model to learn from experiences in different control tasks starting from randomly connected or empty networks. We further show that structural plasticity is advantageous in environments necessitating fast adaptation or with non-stationary rewards.
SAGA: Synthesis Augmentation with Genetic Algorithms for In-Memory Sequence Optimization
Authors: Andey Robins, Mike Borowczak
Date: 2024-06-14 00:00:00

The von-Neumann architecture has a bottleneck which limits the speed at which data can be made available for computation. To combat this problem, novel paradigms for computing are being developed. One such paradigm, known as in-memory computing, interleaves computation with the storage of data within the same circuits. MAGIC, or Memristor Aided Logic, is an approach which uses memory circuits which physically perform computation through write operations to memory. Sequencing these operations is a computationally difficult problem which is directly correlated with the cost of solutions using MAGIC based in-memory computation. SAGA models the execution sequences as a topological sorting problem which makes the optimization well-suited for genetic algorithms. We then detail the formation and implementation of these genetic algorithms and evaluate them over a number of open circuit implementations. The memory-footprint needed for evaluating each of these circuits is decreased by up to 52% from existing, greedy-algorithm-based optimization solutions. Over the 10 benchmark circuits evaluated, these modifications lead to an overall improvement in the efficiency of in-memory circuit evaluation of 128% in the best case and 27.5% on average.
Distributed genetic algorithm for application placement in the compute continuum leveraging infrastructure nodes for optimization
Authors: Carlos Guerrero, Isaac Lera, Carlos Juiz
Date: 2024-06-13 00:00:00

The increasing complexity of fog computing environments calls for efficient resource optimization techniques. In this paper, we propose and evaluate three distributed designs of a genetic algorithm (GA) for resource optimization in fog computing, within an increasing degree of distribution. The designs leverage the execution of the GA in the fog devices themselves by dealing with the specific features of this domain: constrained resources and widely geographical distribution of the devices. For their evaluation, we implemented a benchmark case using the NSGA-II for the specific problem of optimizing the fog service placement, according to the guidelines of our three distributed designs. These three experimental scenarios were compared with a control case, a traditional centralized version of this GA algorithm, considering solution quality and network overhead. The results show that the design with the lowest distribution degree, which keeps centralized storage of the objective space, achieves comparable solution quality to the traditional approach but incurs a higher network load. The second design, which completely distributes the population between the workers, reduces network overhead but exhibits lower solution diversity while keeping enough good results in terms of optimization objective minimization. Finally, the proposal with a distributed population and that only interchanges solution between the workers' neighbors achieves the lowest network load but with compromised solution quality.
Differentiable Programming for Differential Equations: A Review
Authors: Facundo Sapienza, Jordi Bolibar, Frank Schäfer, Brian Groenke, Avik Pal, Victor Boussange, Patrick Heimbach, Giles Hooker, Fernando Pérez, Per-Olof Persson, Christopher Rackauckas
Date: 2024-06-14 00:00:00

The differentiable programming paradigm is a cornerstone of modern scientific computing. It refers to numerical methods for computing the gradient of a numerical model's output. Many scientific models are based on differential equations, where differentiable programming plays a crucial role in calculating model sensitivities, inverting model parameters, and training hybrid models that combine differential equations with data-driven approaches. Furthermore, recognizing the strong synergies between inverse methods and machine learning offers the opportunity to establish a coherent framework applicable to both fields. Differentiating functions based on the numerical solution of differential equations is non-trivial. Numerous methods based on a wide variety of paradigms have been proposed in the literature, each with pros and cons specific to the type of problem investigated. Here, we provide a comprehensive review of existing techniques to compute derivatives of numerical solutions of differential equations. We first discuss the importance of gradients of solutions of differential equations in a variety of scientific domains. Second, we lay out the mathematical foundations of the various approaches and compare them with each other. Third, we cover the computational considerations and explore the solutions available in modern scientific software. Last but not least, we provide best-practices and recommendations for practitioners. We hope that this work accelerates the fusion of scientific models and data, and fosters a modern approach to scientific modelling.
New algorithms for sampling and diffusion models
Authors: Xicheng Zhang
Date: 2024-06-14 00:00:00

Drawing from the theory of stochastic differential equations, we introduce a novel sampling method for known distributions and a new algorithm for diffusion generative models with unknown distributions. Our approach is inspired by the concept of the reverse diffusion process, widely adopted in diffusion generative models. Additionally, we derive the explicit convergence rate based on the smooth ODE flow. For diffusion generative models and sampling, we establish a dimension-free particle approximation convergence result. Numerical experiments demonstrate the effectiveness of our method. Notably, unlike the traditional Langevin method, our sampling method does not require any regularity assumptions about the density function of the target distribution. Furthermore, we also apply our method to optimization problems.
ClimRetrieve: A Benchmarking Dataset for Information Retrieval from Corporate Climate Disclosures
Authors: Tobias Schimanski, Jingwei Ni, Roberto Spacey, Nicola Ranger, Markus Leippold
Date: 2024-06-14 00:00:00
Tasks: Answer Generation, Benchmarking, Information Retrieval, RAG, Retrieval, Retrieval-augmented Generation

To handle the vast amounts of qualitative data produced in corporate climate communication, stakeholders increasingly rely on Retrieval Augmented Generation (RAG) systems. However, a significant gap remains in evaluating domain-specific information retrieval - the basis for answer generation. To address this challenge, this work simulates the typical tasks of a sustainability analyst by examining 30 sustainability reports with 16 detailed climate-related questions. As a result, we obtain a dataset with over 8.5K unique question-source-answer pairs labeled by different levels of relevance. Furthermore, we develop a use case with the dataset to investigate the integration of expert knowledge into information retrieval with embeddings. Although we show that incorporating expert knowledge works, we also outline the critical limitations of embeddings in knowledge-intensive downstream domains like climate change communication.
Soil nitrogen forecasting from environmental variables provided by multisensor remote sensing images
Authors: Weiying Zhao, Ganzorig Chuluunbat, Aleksei Unagaev, Natalia Efremova
Date: 2024-06-14 00:00:00
Tasks: Management

This study introduces a framework for forecasting soil nitrogen content, leveraging multi-modal data, including multi-sensor remote sensing images and advanced machine learning methods. We integrate the Land Use/Land Cover Area Frame Survey (LUCAS) database, which covers European and UK territory, with environmental variables from satellite sensors to create a dataset of novel features. We further test a broad range of machine learning algorithms, focusing on tree-based models such as CatBoost, LightGBM, and XGBoost. We test the proposed methods with a variety of land cover classes, including croplands and grasslands to ensure the robustness of this approach. Our results demonstrate that the CatBoost model surpasses other methods in accuracy. This research advances the field of agricultural management and environmental monitoring and demonstrates the significant potential of integrating multisensor remote sensing data with machine learning for environmental analysis.
IFA: Interaction Fidelity Attention for Entire Lifelong Behaviour Sequence Modeling
Authors: Wenhui Yu, Chao Feng, Yanze Zhang, Lantao Hu, Peng Jiang, Han Li
Date: 2024-06-14 00:00:00
Tasks: Recommendation Systems

The lifelong user behavior sequence provides abundant information of user preference and gains impressive improvement in the recommendation task, however increases computational consumption significantly. To meet the severe latency requirement in online service, a short sub-sequence is sampled based on similarity to the target item. Unfortunately, items not in the sub-sequence are abandoned, leading to serious information loss. In this paper, we propose a new efficient paradigm to model the full lifelong sequence, which is named as \textbf{I}nteraction \textbf{F}idelity \textbf{A}ttention (\textbf{IFA}). In IFA, we input all target items in the candidate set into the model at once, and leverage linear transformer to reduce the time complexity of the cross attention between the candidate set and the sequence without any interaction information loss. We also additionally model the relationship of all target items for optimal set generation, and design loss function for better consistency of training and inference. We demonstrate the effectiveness and efficiency of our model by off-line and online experiments in the recommender system of Kuaishou.
Enhancing Text Corpus Exploration with Post Hoc Explanations and Comparative Design
Authors: Michael Gleicher, Keaton Leppenan, Yunyu Bai
Date: 2024-06-14 00:00:00

Text corpus exploration (TCE) spans the range of exploratory search tasks: it goes beyond simple retrieval to include item discovery and learning about the corpus and topic. Systems support TCE with tools such as similarity-based recommendations and embedding-based spatial maps. However, these tools address specific tasks; current systems lack the flexibility to support the range of tasks encountered in practice and the iterative, multiscale, workflows users employ. In this paper, we provide methods that enhance TCE tools with post hoc explanations and multiscale, comparative designs to provide flexible support for user needs. We introduce salience functions as a mechanism to provide post hoc explanations of similarity, recommendations, and spatial placement. This post hoc strategy allows our approach to complement a variety of underlying algorithms; the salience functions provide both exemplar- and feature-based explanations at scales ranging from individual documents through to the entire corpus. These explanations are incorporated into a set of views that operate at multiple scales. The views use design elements that explicitly support comparison to enable flexible integration. Together, these form an approach that provides a flexible toolset that can address a range of tasks. We demonstrate our approach in a prototype system that enables the exploration of corpora of paper abstracts and newspaper archives. Examples illustrate how our approach enables the system to flexibly support a wide range of tasks and workflows that emerge in user scenarios. A user study confirms that researchers are able to use our system to achieve a variety of tasks.
Enhancing Knowledge Retrieval with In-Context Learning and Semantic Search through Generative AI
Authors: Mohammed-Khalil Ghali, Abdelrahman Farrag, Daehan Won, Yu Jin
Date: 2024-06-13 00:00:00
Tasks: In-Context Learning, Information Retrieval, Retrieval, Text Retrieval

Retrieving and extracting knowledge from extensive research documents and large databases presents significant challenges for researchers, students, and professionals in today's information-rich era. Existing retrieval systems, which rely on general-purpose Large Language Models (LLMs), often fail to provide accurate responses to domain-specific inquiries. Additionally, the high cost of pretraining or fine-tuning LLMs for specific domains limits their widespread adoption. To address these limitations, we propose a novel methodology that combines the generative capabilities of LLMs with the fast and accurate retrieval capabilities of vector databases. This advanced retrieval system can efficiently handle both tabular and non-tabular data, understand natural language user queries, and retrieve relevant information without fine-tuning. The developed model, Generative Text Retrieval (GTR), is adaptable to both unstructured and structured data with minor refinement. GTR was evaluated on both manually annotated and public datasets, achieving over 90% accuracy and delivering truthful outputs in 87% of cases. Our model achieved state-of-the-art performance with a Rouge-L F1 score of 0.98 on the MSMARCO dataset. The refined model, Generative Tabular Text Retrieval (GTR-T), demonstrated its efficiency in large database querying, achieving an Execution Accuracy (EX) of 0.82 and an Exact-Set-Match (EM) accuracy of 0.60 on the Spider dataset, using an open-source LLM. These efforts leverage Generative AI and In-Context Learning to enhance human-text interaction and make advanced AI capabilities more accessible. By integrating robust retrieval systems with powerful LLMs, our approach aims to democratize access to sophisticated AI tools, improving the efficiency, accuracy, and scalability of AI-driven information retrieval and database querying.
Exploring Traffic Crash Narratives in Jordan Using Text Mining Analytics
Authors: Shadi Jaradat, Taqwa I. Alhadidi, Huthaifa I. Ashqar, Ahmed Hossain, Mohammed Elhenawy
Date: 2024-06-11 00:00:00
Tasks: Keyword Extraction

This study explores traffic crash narratives in an attempt to inform and enhance effective traffic safety policies using text-mining analytics. Text mining techniques are employed to unravel key themes and trends within the narratives, aiming to provide a deeper understanding of the factors contributing to traffic crashes. This study collected crash data from five major freeways in Jordan that cover narratives of 7,587 records from 2018-2022. An unsupervised learning method was adopted to learn the pattern from crash data. Various text mining techniques, such as topic modeling, keyword extraction, and Word Co-Occurrence Network, were also used to reveal the co-occurrence of crash patterns. Results show that text mining analytics is a promising method and underscore the multifactorial nature of traffic crashes, including intertwining human decisions and vehicular conditions. The recurrent themes across all analyses highlight the need for a balanced approach to road safety, merging both proactive and reactive measures. Emphasis on driver education and awareness around animal-related incidents is paramount.
Rethinking the Elementary Function Fusion for Single-Image Dehazing
Authors: Yesian Rohn
Date: 2024-05-23 00:00:00
Tasks: Image Dehazing, Single Image Dehazing

This paper addresses the limitations of physical models in the current field of image dehazing by proposing an innovative dehazing network (CL2S). Building on the DM2F model, it identifies issues in its ablation experiments and replaces the original logarithmic function model with a trigonometric (sine) model. This substitution aims to better fit the complex and variable distribution of haze. The approach also integrates the atmospheric scattering model and other elementary functions to enhance dehazing performance. Experimental results demonstrate that CL2S achieves outstanding performance on multiple dehazing datasets, particularly in maintaining image details and color authenticity. Additionally, systematic ablation experiments supplementing DM2F validate the concerns raised about DM2F and confirm the necessity and effectiveness of the functional components in the proposed CL2S model. Our code is available at \url{https://github.com/YesianRohn/CL2S}, where the corresponding pre-trained models can also be accessed.
Some contributions to Lagrangian modelling of Power Converters
Authors: Shakir Showkat Sofi, Mosaib Ul Munieeb, Fazil Bashir, Munieeb Ul Hassan, Shahkar Ahmad Nahvi
Date: 2024-06-14 00:00:00

Lagrangian modelling can be used to derive mathematical models for complex power electronic converters. This approach uses scalar quantities (kinetic and potential energy) to derive models, which is simpler than using (vector-based) force balance equations. It employs generalized coordinates, making it easier to deal with complex systems with constraints. This systematic approach results in equations that can be expressed in state-space form, which allows for the simplification of the simulation and design process and the use of many standard software packages for system analysis and simulation. In this work, contributions are made regarding the procedure to be followed for the Lagrangian modelling of power converters and the incorporation of constraints within the Lagrangian framework. Furthermore, for the first time, Lagrangian modelling is extended to non-ideal, high-fidelity descriptions of standard power electronic circuits.