pwc-archive/papers-with-abstracts huggingface dataset:
Machine learning model to predict solar radiation, based on the integration of meteorological data and data obtained from satellite images
Authors: Luis Eduardo Ordoñez Palacios, Víctor Bucheli Guerrero, Hugo Ordoñez
Date: 2022-04-08 00:00:00

Knowing the behavior of solar radiation at a geographic location is essential for the use of energy from the sun using photovoltaic systems; however, the number of stations for measuring meteorological parameters and for determining the size of solar fields in remote areas is limited. In this work, images obtained from the GOES-13 satellite were used, from which variables were extracted that could be integrated into datasets from meteorological stations. From this, 3 different models were built, on which the performance of 5 machine learning algorithms in predicting solar radiation was evaluated. The neural networks had the highest performance in the model that integrated the meteorological variables and the variables obtained from the images, according to an analysis carried out using four evaluation metrics; although if the rRMSE is considered, all results obtained were higher than 20%, which classified the performance of the algorithms as fair. In the 2012 dataset, the estimation results according to the metrics MBE, R2, RMSE, and rRMSE corresponded to -0.051, 0.880, 90.99 and 26.7%, respectively. In the 2017 dataset, the results of MBE, R2, RMSE, and rRMSE were -0.146, 0.917, 40.97 and 22.3%, respectively. Although it is possible to calculate solar radiation from satellite images, it is also true that some statistical methods depend on radiation data and sunshine captured by ground-based instruments, which is not always possible given that the number of measurement stations on the surface is limited.
Evolving Pareto-Optimal Actor-Critic Algorithms for Generalizability and Stability
Authors: Juan Jose Garau-Luis, Yingjie Miao, John D. Co-Reyes, Aaron Parisi, Jie Tan, Esteban Real, Aleksandra Faust
Date: 2022-04-08 00:00:00
Tasks: continuous-control, Continuous Control, Friction, Reinforcement Learning (RL)

Generalizability and stability are two key objectives for operating reinforcement learning (RL) agents in the real world. Designing RL algorithms that optimize these objectives can be a costly and painstaking process. This paper presents MetaPG, an evolutionary method for automated design of actor-critic loss functions. MetaPG explicitly optimizes for generalizability and performance, and implicitly optimizes the stability of both metrics. We initialize our loss function population with Soft Actor-Critic (SAC) and perform multi-objective optimization using fitness metrics encoding single-task performance, zero-shot generalizability to unseen environment configurations, and stability across independent runs with different random seeds. On a set of continuous control tasks from the Real-World RL Benchmark Suite, we find that our method, using a single environment during evolution, evolves algorithms that improve upon SAC's performance and generalizability by 4% and 20%, respectively, and reduce instability up to 67%. Then, we scale up to more complex environments from the Brax physics simulator and replicate generalizability tests encountered in practical settings, such as different friction coefficients. MetaPG evolves algorithms that can obtain 10% better generalizability without loss of performance within the same meta-training environment and obtain similar results to SAC when doing cross-domain evaluations in other Brax environments. The evolution results are interpretable; by analyzing the structure of the best algorithms we identify elements that help optimizing certain objectives, such as regularization terms for the critic loss.
Weight Matrix Dimensionality Reduction in Deep Learning via Kronecker Multi-layer Architectures
Authors: Jarom D. Hogue, Robert M. Kirby, Akil Narayan
Date: 2022-04-08 00:00:00
Tasks: Deep Learning, Dimensionality Reduction

Deep learning using neural networks is an effective technique for generating models of complex data. However, training such models can be expensive when networks have large model capacity resulting from a large number of layers and nodes. For training in such a computationally prohibitive regime, dimensionality reduction techniques ease the computational burden, and allow implementations of more robust networks. We propose a novel type of such dimensionality reduction via a new deep learning architecture based on fast matrix multiplication of a Kronecker product decomposition; in particular our network construction can be viewed as a Kronecker product-induced sparsification of an "extended" fully connected network. Analysis and practical examples show that this architecture allows a neural network to be trained and implemented with a significant reduction in computational time and resources, while achieving a similar error level compared to a traditional feedforward neural network.
HBFL: A Hierarchical Blockchain-based Federated Learning Framework for a Collaborative IoT Intrusion Detection
Authors: Mohanad Sarhan, Wai Weng Lo, Siamak Layeghy, Marius Portmann
Date: 2022-04-08 00:00:00
Tasks: Federated Learning, Intrusion Detection

The continuous strengthening of the security posture of IoT ecosystems is vital due to the increasing number of interconnected devices and the volume of sensitive data shared. The utilisation of Machine Learning (ML) capabilities in the defence against IoT cyber attacks has many potential benefits. However, the currently proposed frameworks do not consider data privacy, secure architectures, and/or scalable deployments of IoT ecosystems. In this paper, we propose a hierarchical blockchain-based federated learning framework to enable secure and privacy-preserved collaborative IoT intrusion detection. We highlight and demonstrate the importance of sharing cyber threat intelligence among inter-organisational IoT networks to improve the model's detection capabilities. The proposed ML-based intrusion detection framework follows a hierarchical federated learning architecture to ensure the privacy of the learning process and organisational data. The transactions (model updates) and processes will run on a secure immutable ledger, and the conformance of executed tasks will be verified by the smart contract. We have tested our solution and demonstrated its feasibility by implementing it and evaluating the intrusion detection performance using a key IoT data set. The outcome is a securely designed ML-based intrusion detection system capable of detecting a wide range of malicious activities while preserving data privacy.
An approach to improving sound-based vehicle speed estimation
Authors: Nikola Bulatovic, Slobodan Djukanovic
Date: 2022-04-08 00:00:00
Tasks: vehicle detection, Vehicle Speed Estimation

We consider improving the performance of a recently proposed sound-based vehicle speed estimation method. In the original method, an intermediate feature, referred to as the modified attenuation (MA), has been proposed for both vehicle detection and speed estimation. The MA feature maximizes at the instant of the vehicle's closest point of approach, which represents a training label extracted from video recording of the vehicle's pass by. In this paper, we show that the original labeling approach is suboptimal and propose a method for label correction. The method is tested on the VS10 dataset, which contains 304 audio-video recordings of ten different vehicles. The results show that the proposed label correction method reduces average speed estimation error from 7.39 km/h to 6.92 km/h. If the speed is discretized into 10 km/h classes, the accuracy of correct class prediction is improved from 53.2% to 53.8%, whereas when tolerance of one class offset is allowed, accuracy is improved from 93.4% to 94.3%.
Convolutional autoencoders for spatially-informed ensemble post-processing
Authors: Sebastian Lerch, Kai L. Polsterer
Date: 2022-04-08 00:00:00

Ensemble weather predictions typically show systematic errors that have to be corrected via post-processing. Even state-of-the-art post-processing methods based on neural networks often solely rely on location-specific predictors that require an interpolation of the physical weather model's spatial forecast fields to the target locations. However, potentially useful predictability information contained in large-scale spatial structures within the input fields is potentially lost in this interpolation step. Therefore, we propose the use of convolutional autoencoders to learn compact representations of spatial input fields which can then be used to augment location-specific information as additional inputs to post-processing models. The benefits of including this spatial information is demonstrated in a case study of 2-m temperature forecasts at surface stations in Germany.
Transformer-Based Self-Supervised Learning for Emotion Recognition
Authors: Juan Vazquez-Rodriguez, Grégoire Lefebvre, Julien Cumin, James L. Crowley
Date: 2022-04-08 00:00:00
Tasks: Emotion Recognition, Self-Supervised Learning, Time Series, Time Series Analysis

In order to exploit representations of time-series signals, such as physiological signals, it is essential that these representations capture relevant information from the whole signal. In this work, we propose to use a Transformer-based model to process electrocardiograms (ECG) for emotion recognition. Attention mechanisms of the Transformer can be used to build contextualized representations for a signal, giving more importance to relevant parts. These representations may then be processed with a fully-connected network to predict emotions. To overcome the relatively small size of datasets with emotional labels, we employ self-supervised learning. We gathered several ECG datasets with no labels of emotion to pre-train our model, which we then fine-tuned for emotion recognition on the AMIGOS dataset. We show that our approach reaches state-of-the-art performances for emotion recognition using ECG signals on AMIGOS. More generally, our experiments show that transformers and pre-training are promising strategies for emotion recognition with physiological signals.
Self-Supervised Graph Neural Network for Multi-Source Domain Adaptation
Authors: Jin Yuan, Feng Hou, Yangzhou Du, Zhongchao shi, Xin Geng, Jianping Fan, Yong Rui
Date: 2022-04-08 00:00:00
Tasks: Domain Adaptation, Graph Neural Network, Self-Supervised Learning, Task 2

Domain adaptation (DA) tries to tackle the scenarios when the test data does not fully follow the same distribution of the training data, and multi-source domain adaptation (MSDA) is very attractive for real world applications. By learning from large-scale unlabeled samples, self-supervised learning has now become a new trend in deep learning. It is worth noting that both self-supervised learning and multi-source domain adaptation share a similar goal: they both aim to leverage unlabeled data to learn more expressive representations. Unfortunately, traditional multi-task self-supervised learning faces two challenges: (1) the pretext task may not strongly relate to the downstream task, thus it could be difficult to learn useful knowledge being shared from the pretext task to the target task; (2) when the same feature extractor is shared between the pretext task and the downstream one and only different prediction heads are used, it is ineffective to enable inter-task information exchange and knowledge sharing. To address these issues, we propose a novel \textbf{S}elf-\textbf{S}upervised \textbf{G}raph Neural Network (SSG), where a graph neural network is used as the bridge to enable more effective inter-task information exchange and knowledge sharing. More expressive representation is learned by adopting a mask token strategy to mask some domain information. Our extensive experiments have demonstrated that our proposed SSG method has achieved state-of-the-art results over four multi-source domain adaptation datasets, which have shown the effectiveness of our proposed SSG method from different aspects.
Forecasting new diseases in low-data settings using transfer learning
Authors: Kirstin Roster, Colm Connaughton, Francisco A. Rodrigues
Date: 2022-04-07 00:00:00
Tasks: Time Series Analysis, Transfer Learning

Recent infectious disease outbreaks, such as the COVID-19 pandemic and the Zika epidemic in Brazil, have demonstrated both the importance and difficulty of accurately forecasting novel infectious diseases. When new diseases first emerge, we have little knowledge of the transmission process, the level and duration of immunity to reinfection, or other parameters required to build realistic epidemiological models. Time series forecasts and machine learning, while less reliant on assumptions about the disease, require large amounts of data that are also not available in early stages of an outbreak. In this study, we examine how knowledge of related diseases can help make predictions of new diseases in data-scarce environments using transfer learning. We implement both an empirical and a theoretical approach. Using empirical data from Brazil, we compare how well different machine learning models transfer knowledge between two different disease pairs: (i) dengue and Zika, and (ii) influenza and COVID-19. In the theoretical analysis, we generate data using different transmission and recovery rates with an SIR compartmental model, and then compare the effectiveness of different transfer learning methods. We find that transfer learning offers the potential to improve predictions, even beyond a model based on data from the target disease, though the appropriate source disease must be chosen carefully. While imperfect, these models offer an additional input for decision makers during pandemic response.
ShiftNAS: Towards Automatic Generation of Advanced Mulitplication-Less Neural Networks
Authors: Xiaoxuan Lou, Guowen Xu, Kangjie Chen, Guanlin Li, Jiwei Li, Tianwei Zhang
Date: 2022-04-07 00:00:00
Tasks: Neural Architecture Search

Multiplication-less neural networks significantly reduce the time and energy cost on the hardware platform, as the compute-intensive multiplications are replaced with lightweight bit-shift operations. However, existing bit-shift networks are all directly transferred from state-of-the-art convolutional neural networks (CNNs), which lead to non-negligible accuracy drop or even failure of model convergence. To combat this, we propose ShiftNAS, the first framework tailoring Neural Architecture Search (NAS) to substantially reduce the accuracy gap between bit-shift neural networks and their real-valued counterparts. Specifically, we pioneer dragging NAS into a shift-oriented search space and endow it with the robust topology-related search strategy and custom regularization and stabilization. As a result, our ShiftNAS breaks through the incompatibility of traditional NAS methods for bit-shift neural networks and achieves more desirable performance in terms of accuracy and convergence. Extensive experiments demonstrate that ShiftNAS sets a new state-of-the-art for bit-shift neural networks, where the accuracy increases (1.69-8.07)% on CIFAR10, (5.71-18.09)% on CIFAR100 and (4.36-67.07)% on ImageNet, especially when many conventional CNNs fail to converge on ImageNet with bit-shift weights.
Exploring the Universal Vulnerability of Prompt-based Learning Paradigm
Authors: Lei Xu, Yangyi Chen, Ganqu Cui, Hongcheng Gao, Zhiyuan Liu
Date: 2022-04-11 00:00:00
Proceeding: Findings (NAACL) 2022 7

Prompt-based learning paradigm bridges the gap between pre-training and fine-tuning, and works effectively under the few-shot setting. However, we find that this learning paradigm inherits the vulnerability from the pre-training stage, where model predictions can be misled by inserting certain triggers into the text. In this paper, we explore this universal vulnerability by either injecting backdoor triggers or searching for adversarial triggers on pre-trained language models using only plain text. In both scenarios, we demonstrate that our triggers can totally control or severely decrease the performance of prompt-based models fine-tuned on arbitrary downstream tasks, reflecting the universal vulnerability of the prompt-based learning paradigm. Further experiments show that adversarial triggers have good transferability among language models. We also find conventional fine-tuning models are not vulnerable to adversarial triggers constructed from pre-trained language models. We conclude by proposing a potential solution to mitigate our attack methods. Code and data are publicly available at https://github.com/leix28/prompt-universal-vulnerability
Single-Turn Debate Does Not Help Humans Answer Hard Reading-Comprehension Questions
Authors: Alicia Parrish, Harsh Trivedi, Ethan Perez, Angelica Chen, Nikita Nangia, Jason Phang, Samuel R. Bowman
Date: 2022-04-11 00:00:00
Proceeding: LNLS (ACL) 2022 5
Tasks: Multiple-choice, Reading Comprehension

Current QA systems can generate reasonable-sounding yet false answers without explanation or evidence for the generated answer, which is especially problematic when humans cannot readily check the model's answers. This presents a challenge for building trust in machine learning systems. We take inspiration from real-world situations where difficult questions are answered by considering opposing sides (see Irving et al., 2018). For multiple-choice QA examples, we build a dataset of single arguments for both a correct and incorrect answer option in a debate-style set-up as an initial step in training models to produce explanations for two candidate answers. We use long contexts -- humans familiar with the context write convincing explanations for pre-selected correct and incorrect answers, and we test if those explanations allow humans who have not read the full context to more accurately determine the correct answer. We do not find that explanations in our set-up improve human accuracy, but a baseline condition shows that providing human-selected text snippets does improve accuracy. We use these findings to suggest ways of improving the debate set up for future data collection efforts.
Entities, Dates, and Languages: Zero-Shot on Historical Texts with T0
Authors: Francesco De Toni, Christopher Akiki, Javier de la Rosa, Clémentine Fourrier, Enrique Manjavacas, Stefan Schweter, Daniel van Strien
Date: 2022-04-11 00:00:00
Proceeding: BigScience (ACL) 2022 5
Tasks: Multilingual Named Entity Recognition, named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER)

In this work, we explore whether the recently demonstrated zero-shot abilities of the T0 model extend to Named Entity Recognition for out-of-distribution languages and time periods. Using a historical newspaper corpus in 3 languages as test-bed, we use prompts to extract possible named entities. Our results show that a naive approach for prompt-based zero-shot multilingual Named Entity Recognition is error-prone, but highlights the potential of such an approach for historical languages lacking labeled datasets. Moreover, we also find that T0-like models can be probed to predict the publication date and language of a document, which could be very relevant for the study of historical texts.
Tokenwise Contrastive Pretraining for Finer Speech-to-BERT Alignment in End-to-End Speech-to-Intent Systems
Authors: Vishal Sunder, Eric Fosler-Lussier, Samuel Thomas, Hong-Kwang J. Kuo, Brian Kingsbury
Date: 2022-04-11 00:00:00
Tasks: Intent Recognition, Spoken Language Understanding

Recent advances in End-to-End (E2E) Spoken Language Understanding (SLU) have been primarily due to effective pretraining of speech representations. One such pretraining paradigm is the distillation of semantic knowledge from state-of-the-art text-based models like BERT to speech encoder neural networks. This work is a step towards doing the same in a much more efficient and fine-grained manner where we align speech embeddings and BERT embeddings on a token-by-token basis. We introduce a simple yet novel technique that uses a cross-modal attention mechanism to extract token-level contextual embeddings from a speech encoder such that these can be directly compared and aligned with BERT based contextual embeddings. This alignment is performed using a novel tokenwise contrastive loss. Fine-tuning such a pretrained model to perform intent recognition using speech directly yields state-of-the-art performance on two widely used SLU datasets. Our model improves further when fine-tuned with additional regularization using SpecAugment especially when speech is noisy, giving an absolute improvement as high as 8% over previous results.
Building an ASR Error Robust Spoken Virtual Patient System in a Highly Class-Imbalanced Scenario Without Speech Data
Authors: Vishal Sunder, Prashant Serai, Eric Fosler-Lussier
Date: 2022-04-11 00:00:00
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), intent-classification, Intent Classification, speech-recognition, Speech Recognition, Spoken Language Understanding

A Virtual Patient (VP) is a powerful tool for training medical students to take patient histories, where responding to a diverse set of spoken questions is essential to simulate natural conversations with a student. The performance of such a Spoken Language Understanding system (SLU) can be adversely affected by both the presence of Automatic Speech Recognition (ASR) errors in the test data and a high degree of class imbalance in the SLU training data. While these two issues have been addressed separately in prior work, we develop a novel two-step training methodology that tackles both these issues effectively in a single dialog agent. As it is difficult to collect spoken data from users without a functioning SLU system, our method does not rely on spoken data for training, rather we use an ASR error predictor to "speechify" the text data. Our method shows significant improvements over strong baselines on the VP intent classification task at various word error rate settings.
Gaining Insights into Unrecognized User Utterances in Task-Oriented Dialog Systems
Authors: Ella Rabinovich, Matan Vetzler, David Boaz, Vineet Kumar, Gaurav Pandey, Ateret Anaby-Tavor
Date: 2022-04-11 00:00:00
Tasks: Attribute, Goal-Oriented Dialog

The rapidly growing market demand for automatic dialogue agents capable of goal-oriented behavior has caused many tech-industry leaders to invest considerable efforts into task-oriented dialog systems. The success of these systems is highly dependent on the accuracy of their intent identification -- the process of deducing the goal or meaning of the user's request and mapping it to one of the known intents for further processing. Gaining insights into unrecognized utterances -- user requests the systems fail to attribute to a known intent -- is therefore a key process in continuous improvement of goal-oriented dialog systems. We present an end-to-end pipeline for processing unrecognized user utterances, deployed in a real-world, commercial task-oriented dialog system, including a specifically-tailored clustering algorithm, a novel approach to cluster representative extraction, and cluster naming. We evaluated the proposed components, demonstrating their benefits in the analysis of unrecognized user requests.