pwc-archive/papers-with-abstracts huggingface dataset:
On the parametrization of solutions of the Yang--Baxter equations
Authors: J. Hietarinta, C. Viallet
Date: 1995-04-28 00:00:00

We study all five-, six-, and one eight-vertex type two-state solutions of the Yang-Baxter equations in the form $A_{12} B_{13} C_{23} = C_{23} B_{13} A_{12}$, and analyze the interplay of the `gauge' and `inversion' symmetries of these solution. Starting with algebraic solutions, whose parameters have no specific interpretation, and then using these symmetries we can construct a parametrization where we can identify global, color and spectral parameters. We show in particular how the distribution of these parameters may be changed by a change of gauge.
Fuse and Adapt: Investigating the Use of Pre-Trained Self-Supervising Learning Models in Limited Data NLU problems
Authors: S Siriwardhana
Date: 2022-12-02 00:00:00
Proceeding: UOA Thesis 2022 12
Tasks: Domain Adaptation, Emotion Recognition, Natural Language Understanding, Question Answering, Reinforcement Learning (RL), Self-Supervised Learning, Sentiment Analysis, speech-recognition, Speech Recognition

Deep Learning (DL) has become a key element in the field of Artificial Intelligence (AI) over the last decade. DL has improved many applications related to different fields of Natural Language Processing (NLP), Computer Vision (CV), Speech Recognition (SR), and Reinforcement Learning (RL). A significant challenge in applying DL in most domains is the scarcity of labelled data. Usually, DL models need large amounts of annotated data to train models. The novel paradigm of Self Supervised Learning (SSL) has become a gamechanger in the field of Deep Learning due to its ability to answer the problem of scarcity of labeled data. SSL can utilize commonly available unlabelled data to large DL architectures. SSL usually consists of two stages, the pre-training phase, and the downstream phase. The pre-training phase usually needs a large amount of unlabelled data, and it is a computationally expensive process that could cost hundreds of thousands of dollars to millions. But it has become a common practice to open-source the model checkpoints of pre-trained SSL models that can represent different modalities of data such as text, vision, and speech. Usually, expensive and valuable pre-trained SSL models get open-sourced from tech giants like Google, Meta, Amazon, Nvidia, and Microsoft. These Pre-trained models have become a vital part of research communities and the industry due to their effectiveness in solving many downstream tasks. In this thesis, my focus is on exploring the utilization of pre-trained SSL models in the field of Natural Language Understanding (NLU). NLU is the ability of machines to understand human language. It has enabled many practical applications such as Emotion Recognition, Sentiment Analysis, Summarization, and Question Answering. This thesis mainly explores the utilization of pre-trained SSL models in the two main areas of multimodal fusion and domain adaptation. Under these two main topics, I explored four research question that introduces novel fusion and adaptation techniques.
Improved robust weighted averaging for event-related potentials in EEG
Authors: Krzysztof Kotowski, Katarzyna Stapor, Jacek Leski
Date: 2019-09-18 00:00:00
Proceeding: Biocybernetics and Biomedical Engineering 2019 9
Tasks: EEG, EEG Denoising, Electroencephalogram (EEG), ERP

The aim of this study was to improve the robust weighted averaging based on criterion function minimization and assess its effectiveness for extracting event-related brain potentials (ERP) from electroencephalographic (EEG) recordings. The areas of improvement include significantly lower averaging error (45% lower RMSE and 37% lower maximum difference than for original implementation) and increased robustness to local minima, strong outliers and corrupted epochs common to real-life EEG signals, especially from low-cost devices. Our proposed procedure was tested on two datasets, one artificially generated for purposes of this study (including different noise sources) and one real-life dataset collected with Emotiv EPOC+. The lower error results mainly from more effective rejection (lowering the weights) of corrupted epochs by integrating the correlation-based weighting. The advantages of our method over pure correlation-based weighting are lower RMSE (up to two times) and robustness to the algorithm initialization and strong outliers. The performance of the methods was measured using bootstrap testing to avoid dependency of results on data. It shows that our improvements lead to significantly lower error, especially when the EEG signal is not filtered. The values of the parameters were adjusted for EEG signals but they can easily be incorporated in other repetitive electrophysiological measurement techniques.
Using Large Pre-Trained Language Model to Assist FDA in Premarket Medical Device
Authors: Zongzhe Xu
Date: 2022-11-03 00:00:00
Tasks: Language Modeling, Language Modelling, Marketing, Sentence, Sentence Embedding, Sentence-Embedding, Word Embeddings

This paper proposes a possible method using natural language processing that might assist in the FDA medical device marketing process. Actual device descriptions are taken and matched with the device description in FDA Title 21 of CFR to determine their corresponding device type. Both pre-trained word embeddings such as FastText and large pre-trained sentence embedding models such as sentence transformers are evaluated on their accuracy in characterizing a piece of device description. An experiment is also done to test whether these models can identify the devices wrongly classified in the FDA database. The result shows that sentence transformer with T5 and MPNet and GPT-3 semantic search embedding show high accuracy in identifying the correct classification by narrowing down the correct label to be contained in the first 15 most likely results, as compared to 2585 types of device descriptions that must be manually searched through. On the other hand, all methods demonstrate high accuracy in identifying completely incorrectly labeled devices, but all fail to identify false device classifications that are wrong but closely related to the true label.
Model and Data Agreement for Learning with Noisy Labels
Authors: Yuhang Zhang, Weihong Deng, Xingchen Cui, Yunfeng Yin, Hongzhi Shi, Dongchao Wen
Date: 2022-12-02 00:00:00
Tasks: Learning with noisy labels

Learning with noisy labels is a vital topic for practical deep learning as models should be robust to noisy open-world datasets in the wild. The state-of-the-art noisy label learning approach JoCoR fails when faced with a large ratio of noisy labels. Moreover, selecting small-loss samples can also cause error accumulation as once the noisy samples are mistakenly selected as small-loss samples, they are more likely to be selected again. In this paper, we try to deal with error accumulation in noisy label learning from both model and data perspectives. We introduce mean point ensemble to utilize a more robust loss function and more information from unselected samples to reduce error accumulation from the model perspective. Furthermore, as the flip images have the same semantic meaning as the original images, we select small-loss samples according to the loss values of flip images instead of the original ones to reduce error accumulation from the data perspective. Extensive experiments on CIFAR-10, CIFAR-100, and large-scale Clothing1M show that our method outperforms state-of-the-art noisy label learning methods with different levels of label noise. Our method can also be seamlessly combined with other noisy label learning methods to further improve their performance and generalize well to other tasks. The code is available in https://github.com/zyh-uaiaaaa/MDA-noisy-label-learning.
FedALA: Adaptive Local Aggregation for Personalized Federated Learning
Authors: Jianqing Zhang, Yang Hua, Hao Wang, Tao Song, Zhengui Xue, Ruhui Ma, Haibing Guan
Date: 2022-12-02 00:00:00
Tasks: Federated Learning, Personalized Federated Learning

A key challenge in federated learning (FL) is the statistical heterogeneity that impairs the generalization of the global model on each client. To address this, we propose a method Federated learning with Adaptive Local Aggregation (FedALA) by capturing the desired information in the global model for client models in personalized FL. The key component of FedALA is an Adaptive Local Aggregation (ALA) module, which can adaptively aggregate the downloaded global model and local model towards the local objective on each client to initialize the local model before training in each iteration. To evaluate the effectiveness of FedALA, we conduct extensive experiments with five benchmark datasets in computer vision and natural language processing domains. FedALA outperforms eleven state-of-the-art baselines by up to 3.27% in test accuracy. Furthermore, we also apply ALA module to other federated learning methods and achieve up to 24.19% improvement in test accuracy.
Embedding Synthetic Off-Policy Experience for Autonomous Driving via Zero-Shot Curricula
Authors: Eli Bronstein, Sirish Srinivasan, Supratik Paul, Aman Sinha, Matthew O'Kelly, Payam Nikdel, Shimon Whiteson
Date: 2022-12-02 00:00:00
Tasks: Autonomous Driving, Autonomous Vehicles, Imitation Learning, Motion Planning

ML-based motion planning is a promising approach to produce agents that exhibit complex behaviors, and automatically adapt to novel environments. In the context of autonomous driving, it is common to treat all available training data equally. However, this approach produces agents that do not perform robustly in safety-critical settings, an issue that cannot be addressed by simply adding more data to the training set - we show that an agent trained using only a 10% subset of the data performs just as well as an agent trained on the entire dataset. We present a method to predict the inherent difficulty of a driving situation given data collected from a fleet of autonomous vehicles deployed on public roads. We then demonstrate that this difficulty score can be used in a zero-shot transfer to generate curricula for an imitation-learning based planning agent. Compared to training on the entire unbiased training dataset, we show that prioritizing difficult driving scenarios both reduces collisions by 15% and increases route adherence by 14% in closed-loop evaluation, all while using only 10% of the training data.
Designing Ecosystems of Intelligence from First Principles
Authors: Karl J Friston, Maxwell J D Ramstead, Alex B Kiefer, Alexander Tschantz, Christopher L Buckley, Mahault Albarracin, Riddhi J Pitliya, Conor Heins, Brennan Klein, Beren Millidge, Dalton A R Sakthivadivel, Toby St Clere Smithe, Magnus Koudahl, Safae Essafi Tremblay, Capm Petersen, Kaiser Fung, Jason G Fox, Steven Swanson, Dan Mapes, Gabriel René
Date: 2022-12-02 00:00:00
Tasks: Model Selection

This white paper lays out a vision of research and development in the field of artificial intelligence for the next decade (and beyond). Its denouement is a cyber-physical ecosystem of natural and synthetic sense-making, in which humans are integral participants -- what we call ''shared intelligence''. This vision is premised on active inference, a formulation of adaptive behavior that can be read as a physics of intelligence, and which inherits from the physics of self-organization. In this context, we understand intelligence as the capacity to accumulate evidence for a generative model of one's sensed world -- also known as self-evidencing. Formally, this corresponds to maximizing (Bayesian) model evidence, via belief updating over several scales: i.e., inference, learning, and model selection. Operationally, this self-evidencing can be realized via (variational) message passing or belief propagation on a factor graph. Crucially, active inference foregrounds an existential imperative of intelligent systems; namely, curiosity or the resolution of uncertainty. This same imperative underwrites belief sharing in ensembles of agents, in which certain aspects (i.e., factors) of each agent's generative world model provide a common ground or frame of reference. Active inference plays a foundational role in this ecology of belief sharing -- leading to a formal account of collective intelligence that rests on shared narratives and goals. We also consider the kinds of communication protocols that must be developed to enable such an ecosystem of intelligences and motivate the development of a shared hyper-spatial modeling language and transaction protocol, as a first -- and key -- step towards such an ecology.
Nonparametric Masked Language Modeling
Authors: Sewon Min, Weijia Shi, Mike Lewis, Xilun Chen, Wen-tau Yih, Hannaneh Hajishirzi, Luke Zettlemoyer
Date: 2022-12-02 00:00:00
Tasks: Language Modeling, Language Modelling, Masked Language Modeling, Question Answering, Retrieval

Existing language models (LMs) predict tokens with a softmax over a finite vocabulary, which can make it difficult to predict rare tokens or phrases. We introduce NPM, the first nonparametric masked language model that replaces this softmax with a nonparametric distribution over every phrase in a reference corpus. NPM fills in the [MASK] solely from retrieving a token from a text corpus. We show that NPM can be efficiently trained with a contrastive objective and an in-batch approximation to full corpus retrieval. Zero-shot evaluation on 16 tasks including classification, fact probing and question answering demonstrates that NPM outperforms significantly larger parametric models, either with or without a retrieve-and-generate approach. It is particularly better at dealing with rare patterns (word senses or facts) and predicting rare or nearly unseen words (e.g., non-Latin script). We release the model and code at github.com/facebookresearch/NPM.
Guaranteed Conformance of Neurosymbolic Models to Natural Constraints
Authors: Kaustubh Sridhar, Souradeep Dutta, James Weimer, Insup Lee
Date: 2022-12-02 00:00:00

Deep neural networks have emerged as the workhorse for a large section of robotics and control applications, especially as models for dynamical systems. Such data-driven models are in turn used for designing and verifying autonomous systems. They are particularly useful in modeling medical systems where data can be leveraged to individualize treatment. In safety-critical applications, it is important that the data-driven model is conformant to established knowledge from the natural sciences. Such knowledge is often available or can often be distilled into a (possibly black-box) model. For instance, an F1 racing car should conform to Newton's laws (which are encoded within a unicycle model). In this light, we consider the following problem - given a model $M$ and a state transition dataset, we wish to best approximate the system model while being a bounded distance away from $M$. We propose a method to guarantee this conformance. Our first step is to distill the dataset into a few representative samples called memories, using the idea of a growing neural gas. Next, using these memories we partition the state space into disjoint subsets and compute bounds that should be respected by the neural network in each subset. This serves as a symbolic wrapper for guaranteed conformance. We argue theoretically that this only leads to a bounded increase in approximation error; which can be controlled by increasing the number of memories. We experimentally show that on three case studies (Car Model, Drones, and Artificial Pancreas), our constrained neurosymbolic models conform to specified models (each encoding various constraints) with order-of-magnitude improvements compared to the augmented Lagrangian and vanilla training methods. Our code can be found at: https://github.com/kaustubhsridhar/Constrained_Models
Legal Prompting: Teaching a Language Model to Think Like a Lawyer
Authors: FangYi Yu, Lee Quartey, Frank Schilder
Date: 2022-12-02 00:00:00
Tasks: Common Sense Reasoning, Language Modeling, Language Modelling, Legal Reasoning, Prompt Engineering

Large language models that are capable of zero or few-shot prompting approaches have given rise to the new research area of prompt engineering. Recent advances showed that for example Chain-of-Thought (CoT) prompts can improve arithmetic or common sense tasks significantly. We explore how such approaches fare with legal reasoning tasks and take the COLIEE entailment task based on the Japanese Bar exam for testing zero-shot/few-shot and fine-tuning approaches. Our findings show that while CoT prompting and fine-tuning with explanations approaches show improvements, the best results are produced by prompts that are derived from specific legal reasoning techniques such as IRAC (Issue, Rule, Application, Conclusion). Based on our experiments we improve the 2021 best result from 0.7037 accuracy to 0.8148 accuracy and beat the 2022 best system of 0.6789 accuracy with an accuracy of 0.7431.
Selecting Mechanical Parameters of a Monopode Jumping System with Reinforcement Learning
Authors: Andrew Albright, Joshua Vaughan
Date: 2022-12-02 00:00:00
Tasks: Navigate, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Legged systems have many advantages when compared to their wheeled counterparts. For example, they can more easily navigate extreme, uneven terrain. However, there are disadvantages as well, particularly the difficulty seen in modeling the nonlinearities of the system. Research has shown that using flexible components within legged locomotive systems improves performance measures such as efficiency and running velocity. Because of the difficulties encountered in modeling flexible systems, control methods such as reinforcement learning can be used to define control strategies. Furthermore, reinforcement learning can be tasked with learning mechanical parameters of a system to match a control input. It is shown in this work that when deploying reinforcement learning to find design parameters for a pogo-stick jumping system, the designs the agents learn are optimal within the design space provided to the agents.
DeepFT: Fault-Tolerant Edge Computing using a Self-Supervised Deep Surrogate Model
Authors: Shreshth Tuli, Giuliano Casale, Ludmila Cherkasova, Nicholas R. Jennings
Date: 2022-12-02 00:00:00
Tasks: Edge-computing, Fault Detection, Scheduling, Self-Supervised Learning

The emergence of latency-critical AI applications has been supported by the evolution of the edge computing paradigm. However, edge solutions are typically resource-constrained, posing reliability challenges due to heightened contention for compute and communication capacities and faulty application behavior in the presence of overload conditions. Although a large amount of generated log data can be mined for fault prediction, labeling this data for training is a manual process and thus a limiting factor for automation. Due to this, many companies resort to unsupervised fault-tolerance models. Yet, failure models of this kind can incur a loss of accuracy when they need to adapt to non-stationary workloads and diverse host characteristics. To cope with this, we propose a novel modeling approach, called DeepFT, to proactively avoid system overloads and their adverse effects by optimizing the task scheduling and migration decisions. DeepFT uses a deep surrogate model to accurately predict and diagnose faults in the system and co-simulation based self-supervised learning to dynamically adapt the model in volatile settings. It offers a highly scalable solution as the model size scales by only 3 and 1 percent per unit increase in the number of active tasks and hosts. Extensive experimentation on a Raspberry-Pi based edge cluster with DeFog benchmarks shows that DeepFT can outperform state-of-the-art baseline methods in fault-detection and QoS metrics. Specifically, DeepFT gives the highest F1 scores for fault-detection, reducing service deadline violations by up to 37\% while also improving response time by up to 9%.
SARAS-Net: Scale and Relation Aware Siamese Network for Change Detection
Authors: Chao-Peng Chen, Jun-Wei Hsieh, Ping-Yang Chen, Yi-Kuan Hsieh, Bor-Shiun Wang
Date: 2022-12-02 00:00:00
Tasks: Building change detection for remote sensing images, Change Detection, Change detection for remote sensing images, Relation, Scene Change Detection

Change detection (CD) aims to find the difference between two images at different times and outputs a change map to represent whether the region has changed or not. To achieve a better result in generating the change map, many State-of-The-Art (SoTA) methods design a deep learning model that has a powerful discriminative ability. However, these methods still get lower performance because they ignore spatial information and scaling changes between objects, giving rise to blurry or wrong boundaries. In addition to these, they also neglect the interactive information of two different images. To alleviate these problems, we propose our network, the Scale and Relation-Aware Siamese Network (SARAS-Net) to deal with this issue. In this paper, three modules are proposed that include relation-aware, scale-aware, and cross-transformer to tackle the problem of scene change detection more effectively. To verify our model, we tested three public datasets, including LEVIR-CD, WHU-CD, and DSFIN, and obtained SoTA accuracy. Our code is available at https://github.com/f64051041/SARAS-Net.
Loss shaping enhances exact gradient learning with Eventprop in spiking neural networks
Authors: Thomas Nowotny, James P. Turner, James C. Knight
Date: 2022-12-02 00:00:00
Tasks: GPU

Event-based machine learning promises more energy-efficient AI on future neuromorphic hardware. Here, we investigate how the recently discovered Eventprop algorithm for gradient descent on exact gradients in spiking neural networks can be scaled up to challenging keyword recognition benchmarks. We implemented Eventprop in the GPU-enhanced Neural Networks framework and used it for training recurrent spiking neural networks on the Spiking Heidelberg Digits and Spiking Speech Commands datasets. We found that learning depended strongly on the loss function and extended Eventprop to a wider class of loss functions to enable effective training. We then tested a large number of data augmentations and regularisations as well as exploring different network structures; and heterogeneous and trainable timescales. We found that when combined with two specific augmentations, the right regularisation and a delay line input, Eventprop networks with one recurrent layer achieved state-of-the-art performance on Spiking Heidelberg Digits and good accuracy on Spiking Speech Commands. In comparison to a leading surrogate-gradient-based SNN training method, our GeNN Eventprop implementation is 3X faster and uses 4X less memory. This work is a significant step towards a low-power neuromorphic alternative to current machine learning paradigms.