pwc-archive/papers-with-abstracts huggingface dataset:
Dynamic Context Adaptation and Information Flow Control in Transformers: Introducing the Evaluator Adjuster Unit and Gated Residual Connections
Authors: Sahil Rajesh Dhayalkar
Date: 2024-05-22 00:00:00

Transformers have revolutionized various domains of artificial intelligence due to their unique ability to model long-range dependencies in data. However, they lack in nuanced, context-dependent modulation of features and information flow. This paper introduces two significant enhancements to the transformer architecture - the Evaluator Adjuster Unit (EAU) and Gated Residual Connections (GRC) - designed to address these limitations. The EAU dynamically modulates attention outputs based on the relevance of the input context, allowing for more adaptive response patterns. Concurrently, the GRC modifies the transformer's residual connections through a gating mechanism that selectively controls the information flow, thereby enhancing the network's ability to focus on contextually important features. We evaluate the performance of these enhancements across several benchmarks in natural language processing. Our results demonstrate improved adaptability and efficiency, suggesting that these modifications could set new standards for designing flexible and context-aware transformer models.
A theory of neural emulators
Authors: Catalin C. Mitelut
Date: 2024-05-22 00:00:00
Tasks: Prediction

A central goal in neuroscience is to provide explanations for how animal nervous systems can generate actions and cognitive states such as consciousness while artificial intelligence (AI) and machine learning (ML) seek to provide models that are increasingly better at prediction. Despite many decades of research we have made limited progress on providing neuroscience explanations yet there is an increased use of AI and ML methods in neuroscience for prediction of behavior and even cognitive states. Here we propose emulator theory (ET) and neural emulators as circuit- and scale-independent predictive models of biological brain activity and emulator theory (ET) as an alternative research paradigm in neuroscience. ET proposes that predictive models trained solely on neural dynamics and behaviors can generate functionally indistinguishable systems from their sources. That is, compared to the biological organisms which they model, emulators may achieve indistinguishable behavior and cognitive states - including consciousness - without any mechanistic explanations. We posit ET via several conjectures, discuss the nature of endogenous and exogenous activation of neural circuits, and discuss neural causality of phenomenal states. ET provides the conceptual and empirical framework for prediction-based models of neural dynamics and behavior without explicit representations of idiosyncratically evolved nervous systems.
NFCL: Simply interpretable neural networks for a short-term multivariate forecasting
Authors: Wonkeun Jo, Dongil Kim
Date: 2024-05-22 00:00:00
Tasks: Multivariate Time Series Forecasting, Time Series, Time Series Forecasting

Multivariate time-series forecasting (MTSF) stands as a compelling field within the machine learning community. Diverse neural network based methodologies deployed in MTSF applications have demonstrated commendable efficacy. Despite the advancements in model performance, comprehending the rationale behind the model's behavior remains an enigma. Our proposed model, the Neural ForeCasting Layer (NFCL), employs a straightforward amalgamation of neural networks. This uncomplicated integration ensures that each neural network contributes inputs and predictions independently, devoid of interference from other inputs. Consequently, our model facilitates a transparent explication of forecast results. This paper introduces NFCL along with its diverse extensions. Empirical findings underscore NFCL's superior performance compared to nine benchmark models across 15 available open datasets. Notably, NFCL not only surpasses competitors but also provides elucidation for its predictions. In addition, Rigorous experimentation involving diverse model structures bolsters the justification of NFCL's unique configuration.
360Zhinao Technical Report
Authors: 360Zhinao Team
Date: 2024-05-22 00:00:00
Tasks: 4k

We present 360Zhinao models with 7B parameter size and context lengths spanning 4K, 32K and 360K, all available at https://github.com/Qihoo360/360zhinao. For rapid development in pretraining, we establish a stable and sensitive ablation environment to evaluate and compare experiment runs with minimal model size. Under such guidance, we perfect our data cleaning and composition strategies to pretrain $\texttt{360Zhinao-7B-Base}$ on 3.4T tokens. We also mainly emphasize data during alignment, where we strive to balance quantity and quality with filtering and reformatting. With tailored data, 360Zhinao-7B's context window is easily extended to 32K and 360K. RMs and RLHF are trained following SFT and credibly applied to specific tasks. All together these contributions lead to 360Zhinao-7B's competitive performance among models of similar size.
Collaboration of Teachers for Semi-supervised Object Detection
Authors: Liyu Chen, Huaao Tang, Yi Wen, Hanting Chen, Wei Li, Junchao Liu, Jie Hu
Date: 2024-05-22 00:00:00
Tasks: Object, object-detection, Object Detection, Semi-Supervised Object Detection

Recent semi-supervised object detection (SSOD) has achieved remarkable progress by leveraging unlabeled data for training. Mainstream SSOD methods rely on Consistency Regularization methods and Exponential Moving Average (EMA), which form a cyclic data flow. However, the EMA updating training approach leads to weight coupling between the teacher and student models. This coupling in a cyclic data flow results in a decrease in the utilization of unlabeled data information and the confirmation bias on low-quality or erroneous pseudo-labels. To address these issues, we propose the Collaboration of Teachers Framework (CTF), which consists of multiple pairs of teacher and student models for training. In the learning process of CTF, the Data Performance Consistency Optimization module (DPCO) informs the best pair of teacher models possessing the optimal pseudo-labels during the past training process, and these most reliable pseudo-labels generated by the best performing teacher would guide the other student models. As a consequence, this framework greatly improves the utilization of unlabeled data and prevents the positive feedback cycle of unreliable pseudo-labels. The CTF achieves outstanding results on numerous SSOD datasets, including a 0.71% mAP improvement on the 10% annotated COCO dataset and a 0.89% mAP improvement on the VOC dataset compared to LabelMatch and converges significantly faster. Moreover, the CTF is plug-and-play and can be integrated with other mainstream SSOD methods.
"Turing Tests" For An AI Scientist
Authors: Xiaoxin Yin
Date: 2024-05-22 00:00:00
Tasks: AI Agent, Data Compression, Math, scientific discovery

While LLMs have shown impressive capabilities in solving math or coding problems, the ability to make scientific discoveries remains a distinct challenge. This paper proposes a "Turing test for an AI scientist" to assess whether an AI agent can conduct scientific research independently, without relying on human-generated knowledge. Drawing inspiration from the historical development of science, we propose seven benchmark tests that evaluate an AI agent's ability to make groundbreaking discoveries in various scientific domains. These tests include inferring the heliocentric model from celestial observations, discovering the laws of motion in a simulated environment, deriving the differential equation governing vibrating strings, inferring Maxwell's equations from electrodynamics simulations, inventing numerical methods for initial value problems, discovering Huffman coding for data compression, and developing efficient sorting algorithms. To ensure the validity of these tests, the AI agent is provided with interactive libraries or datasets specific to each problem, without access to human knowledge that could potentially contain information about the target discoveries. The ultimate goal is to create an AI scientist capable of making novel and impactful scientific discoveries, surpassing the best human experts in their respective fields. These "Turing tests" serve as intermediate milestones, assessing the AI agent's ability to make discoveries that were groundbreaking in their time. If an AI agent can pass the majority of these seven tests, it would indicate significant progress towards building an AI scientist, paving the way for future advancements in autonomous scientific discovery. This paper aims to establish a benchmark for the capabilities of AI in scientific research and to stimulate further research in this exciting field.
Time-Series Forecasting and Sequence Learning Using Memristor-based Reservoir System
Authors: Abdullah M. Zyarah, Dhireesha Kudithipudi
Date: 2024-05-22 00:00:00
Tasks: Time Series, Time Series Forecasting

Pushing the frontiers of time-series information processing in the ever-growing domain of edge devices with stringent resources has been impeded by the systems' ability to process information and learn locally on the device. Local processing and learning of time-series information typically demand intensive computations and massive storage as the process involves retrieving information and tuning hundreds of parameters back in time. In this work, we developed a memristor-based echo state network accelerator that features efficient temporal data processing and in-situ online learning. The proposed design is benchmarked using various datasets involving real-world tasks, such as forecasting the load energy consumption and weather conditions. The experimental results illustrate that the hardware model experiences a marginal degradation in performance as compared to the software counterpart. This is mainly attributed to the limited precision and dynamic range of network parameters when emulated using memristor devices. The proposed system is evaluated for lifespan, robustness, and energy-delay product. It is observed that the system demonstrates reasonable robustness for device failure below 10%, which may occur due to stuck-at faults. Furthermore, 247X reduction in energy consumption is achieved when compared to a custom CMOS digital design implemented at the same technology node.
DEGAP: Dual Event-Guided Adaptive Prefixes for Templated-Based Event Argument Extraction with Slot Querying
Authors: Guanghui Wang, Dexi Liu, Jian-Yun Nie, Qizhi Wan, Rong Hu, Xiping Liu, Wanlong Liu, Jiaming Liu
Date: 2024-05-22 00:00:00
Tasks: Event Argument Extraction, Retrieval

Recent advancements in event argument extraction (EAE) involve incorporating useful auxiliary information into models during training and inference, such as retrieved instances and event templates. These methods face two challenges: (1) the retrieval results may be irrelevant and (2) templates are developed independently for each event without considering their possible relationship. In this work, we propose DEGAP to address these challenges through a simple yet effective components: dual prefixes, i.e. learnable prompt vectors, where the instance-oriented prefix and template-oriented prefix are trained to learn information from different event instances and templates. Additionally, we propose an event-guided adaptive gating mechanism, which can adaptively leverage possible connections between different events and thus capture relevant information from the prefix. Finally, these event-guided prefixes provide relevant information as cues to EAE model without retrieval. Extensive experiments demonstrate that our method achieves new state-of-the-art performance on four datasets (ACE05, RAMS, WIKIEVENTS, and MLEE). Further analysis shows the impact of different components.
Adversarial Training via Adaptive Knowledge Amalgamation of an Ensemble of Teachers
Authors: Shayan Mohajer Hamidi, Linfeng Ye
Date: 2024-05-22 00:00:00
Tasks: Adversarial Robustness

Adversarial training (AT) is a popular method for training robust deep neural networks (DNNs) against adversarial attacks. Yet, AT suffers from two shortcomings: (i) the robustness of DNNs trained by AT is highly intertwined with the size of the DNNs, posing challenges in achieving robustness in smaller models; and (ii) the adversarial samples employed during the AT process exhibit poor generalization, leaving DNNs vulnerable to unforeseen attack types. To address these dual challenges, this paper introduces adversarial training via adaptive knowledge amalgamation of an ensemble of teachers (AT-AKA). In particular, we generate a diverse set of adversarial samples as the inputs to an ensemble of teachers; and then, we adaptively amalgamate the logtis of these teachers to train a generalized-robust student. Through comprehensive experiments, we illustrate the superior efficacy of AT-AKA over existing AT methods and adversarial robustness distillation techniques against cutting-edge attacks, including AutoAttack.
Theoretical Analysis of Meta Reinforcement Learning: Generalization Bounds and Convergence Guarantees
Authors: Cangqing Wang, Mingxiu Sui, Dan Sun, Zecheng Zhang, Yan Zhou
Date: 2024-05-22 00:00:00
Tasks: Generalization Bounds, Meta Reinforcement Learning

This research delves deeply into Meta Reinforcement Learning (Meta RL) through a exploration focusing on defining generalization limits and ensuring convergence. By employing a approach this article introduces an innovative theoretical framework to meticulously assess the effectiveness and performance of Meta RL algorithms. We present an explanation of generalization limits measuring how well these algorithms can adapt to learning tasks while maintaining consistent results. Our analysis delves into the factors that impact the adaptability of Meta RL revealing the relationship, between algorithm design and task complexity. Additionally we establish convergence assurances by proving conditions under which Meta RL strategies are guaranteed to converge towards solutions. We examine the convergence behaviors of Meta RL algorithms across scenarios providing a comprehensive understanding of the driving forces behind their long term performance. This exploration covers both convergence and real time efficiency offering a perspective, on the capabilities of these algorithms.
Part-based Quantitative Analysis for Heatmaps
Authors: Osman Tursun, Sinan Kalkan, Simon Denman, Sridha Sridharan, Clinton Fookes
Date: 2024-05-22 00:00:00
Tasks: Informativeness

Heatmaps have been instrumental in helping understand deep network decisions, and are a common approach for Explainable AI (XAI). While significant progress has been made in enhancing the informativeness and accessibility of heatmaps, heatmap analysis is typically very subjective and limited to domain experts. As such, developing automatic, scalable, and numerical analysis methods to make heatmap-based XAI more objective, end-user friendly, and cost-effective is vital. In addition, there is a need for comprehensive evaluation metrics to assess heatmap quality at a granular level.
Traffic control using intelligent timing of traffic lights with reinforcement learning technique and real-time processing of surveillance camera images
Authors: Mahdi JameBozorg, Mohsen Hami, Sajjad Deh Deh Jani
Date: 2024-05-22 00:00:00
Tasks: Management, OpenAI Gym, reinforcement-learning, Reinforcement Learning, Transfer Learning, vehicle detection

Optimal management of traffic light timing is one of the most effective factors in reducing urban traffic. In most old systems, fixed timing was used along with human factors to control traffic, which is not very efficient in terms of time and cost. Nowadays, methods in the field of traffic management are based on the use of artificial intelligence. In this method, by using real-time processing of video surveillance camera images along with reinforcement learning, the optimal timing of traffic lights is determined and applied according to several parameters. In the research, deep learning methods were used in vehicle detection using the YOLOv9-C model to estimate the number and other characteristics of vehicles such as speed. Finally, by modeling vehicles in an urban environment simulator at OpenAI Gym using multi-factor reinforcement learning and the DQN Rainbow algorithm, timing is applied to traffic lights at intersections. Additionally, the use of transfer learning along with retraining the model on images of Iranian cars has increased the accuracy of the model. The results of the proposed method show a model that is reasonably accurate in both parts of analyzing surveillance cameras and finding the optimal timing, and it has been observed that it has better accuracy than previous research.
A Survey of Robotic Language Grounding: Tradeoffs between Symbols and Embeddings
Authors: Vanya Cohen, Jason Xinyu Liu, Raymond Mooney, Stefanie Tellex, David Watkins
Date: 2024-05-21 00:00:00

With large language models, robots can understand language more flexibly and more capable than ever before. This survey reviews and situates recent literature into a spectrum with two poles: 1) mapping between language and some manually defined formal representation of meaning, and 2) mapping between language and high-dimensional vector spaces that translate directly to low-level robot policy. Using a formal representation allows the meaning of the language to be precisely represented, limits the size of the learning problem, and leads to a framework for interpretability and formal safety guarantees. Methods that embed language and perceptual data into high-dimensional spaces avoid this manually specified symbolic structure and thus have the potential to be more general when fed enough data but require more data and computing to train. We discuss the benefits and tradeoffs of each approach and finish by providing directions for future work that achieves the best of both worlds.
Goals as Reward-Producing Programs
Authors: Guy Davidson, Graham Todd, Julian Togelius, Todd M. Gureckis, Brenden M. Lake
Date: 2024-05-21 00:00:00
Tasks: Diversity, Program Synthesis

People are remarkably capable of generating their own goals, beginning with child's play and continuing into adulthood. Despite considerable empirical and computational work on goals and goal-oriented behavior, models are still far from capturing the richness of everyday human goals. Here, we bridge this gap by collecting a dataset of human-generated playful goals (in the form of scorable, single-player games), modeling them as reward-producing programs, and generating novel human-like goals through program synthesis. Reward-producing programs capture the rich semantics of goals through symbolic operations that compose, add temporal constraints, and allow for program execution on behavioral traces to evaluate progress. To build a generative model of goals, we learn a fitness function over the infinite set of possible goal programs and sample novel goals with a quality-diversity algorithm. Human evaluators found that model-generated goals, when sampled from partitions of program space occupied by human examples, were indistinguishable from human-created games. We also discovered that our model's internal fitness scores predict games that are evaluated as more fun to play and more human-like.
Multiple Realizability and the Rise of Deep Learning
Authors: Sam Whitman McGrath, Jacob Russin
Date: 2024-05-21 00:00:00
Tasks: Deep Learning, Diversity

The multiple realizability thesis holds that psychological states may be implemented in a diversity of physical systems. The deep learning revolution seems to be bringing this possibility to life, offering the most plausible examples of man-made realizations of sophisticated cognitive functions to date. This paper explores the implications of deep learning models for the multiple realizability thesis. Among other things, it challenges the widely held view that multiple realizability entails that the study of the mind can and must be pursued independently of the study of its implementation in the brain or in artificial analogues. Although its central contribution is philosophical, the paper has substantial methodological upshots for contemporary cognitive science, suggesting that deep neural networks may play a crucial role in formulating and evaluating hypotheses about cognition, even if they are interpreted as implementation-level models. In the age of deep learning, multiple realizability possesses a renewed significance.
Transfer Learning Approach for Railway Technical Map (RTM) Component Identification
Authors: Obadage Rochana Rumalshan, Pramuka Weerasinghe, Mohamed Shaheer, Prabhath Gunathilake, Erunika Dayaratna
Date: 2024-05-21 00:00:00
Tasks: Management, object-detection, Object Detection, Optical Character Recognition, Optical Character Recognition (OCR), Transfer Learning

The extreme popularity over the years for railway transportation urges the necessity to maintain efficient railway management systems around the globe. Even though, at present, there exist a large collection of Computer Aided Designed Railway Technical Maps (RTMs) but available only in the portable document format (PDF). Using Deep Learning and Optical Character Recognition techniques, this research work proposes a generic system to digitize the relevant map component data from a given input image and create a formatted text file per image. Out of YOLOv3, SSD and Faster-RCNN object detection models used, Faster-RCNN yields the highest mean Average Precision (mAP) and the highest F1 score values 0.68 and 0.76 respectively. Further it is proven from the results obtained that, one can improve the results with OCR when the text containing image is being sent through a sophisticated pre-processing pipeline to remove distortions.
Paired Autoencoders for Likelihood-free Estimation in Inverse Problems
Authors: Matthias Chung, Emma Hart, Julianne Chung, Bas Peters, Eldad Haber
Date: 2024-05-21 00:00:00

We consider the solution of nonlinear inverse problems where the forward problem is a discretization of a partial differential equation. Such problems are notoriously difficult to solve in practice and require minimizing a combination of a data-fit term and a regularization term. The main computational bottleneck of typical algorithms is the direct estimation of the data misfit. Therefore, likelihood-free approaches have become appealing alternatives. Nonetheless, difficulties in generalization and limitations in accuracy have hindered their broader utility and applicability. In this work, we use a paired autoencoder framework as a likelihood-free estimator for inverse problems. We show that the use of such an architecture allows us to construct a solution efficiently and to overcome some known open problems when using likelihood-free estimators. In particular, our framework can assess the quality of the solution and improve on it if needed. We demonstrate the viability of our approach using examples from full waveform inversion and inverse electromagnetic imaging.