pwc-archive/papers-with-abstracts huggingface dataset:
FedTrees: A Novel Computation-Communication Efficient Federated Learning Framework Investigated in Smart Grids
Authors: Mohammad Al-Quraan, Ahsan Khan, Anthony Centeno, Ahmed Zoha, Muhammad Ali Imran, Lina Mohjazi
Date: 2022-09-30 00:00:00
Tasks: energy management, Ensemble Learning, Federated Learning, Management

Smart energy performance monitoring and optimisation at the supplier and consumer levels is essential to realising smart cities. In order to implement a more sustainable energy management plan, it is crucial to conduct a better energy forecast. The next-generation smart meters can also be used to measure, record, and report energy consumption data, which can be used to train machine learning (ML) models for predicting energy needs. However, sharing fine-grained energy data and performing centralised learning may compromise users' privacy and leave them vulnerable to several attacks. This study addresses this issue by utilising federated learning (FL), an emerging technique that performs ML model training at the user level, where data resides. We introduce FedTrees, a new, lightweight FL framework that benefits from the outstanding features of ensemble learning. Furthermore, we developed a delta-based early stopping algorithm to monitor FL training and stop it when it does not need to continue. The simulation results demonstrate that FedTrees outperforms the most popular federated averaging (FedAvg) framework and the baseline Persistence model for providing accurate energy forecasting patterns while taking only 2% of the computation time and 13% of the communication rounds compared to FedAvg, saving considerable amounts of computation and communication resources.
Kernel Normalized Convolutional Networks for Privacy-Preserving Machine Learning
Authors: Reza Nasirigerdeh, Javad Torkzadehmahani, Daniel Rueckert, Georgios Kaissis
Date: 2022-09-30 00:00:00
Tasks: Federated Learning, Privacy Preserving

Normalization is an important but understudied challenge in privacy-related application domains such as federated learning (FL), differential privacy (DP), and differentially private federated learning (DP-FL). While the unsuitability of batch normalization for these domains has already been shown, the impact of other normalization methods on the performance of federated or differentially private models is not well-known. To address this, we draw a performance comparison among layer normalization (LayerNorm), group normalization (GroupNorm), and the recently proposed kernel normalization (KernelNorm) in FL, DP, and DP-FL settings. Our results indicate LayerNorm and GroupNorm provide no performance gain compared to the baseline (i.e. no normalization) for shallow models in FL and DP. They, on the other hand, considerably enhance the performance of shallow models in DP-FL and deeper models in FL and DP. KernelNorm, moreover, significantly outperforms its competitors in terms of accuracy and convergence rate (or communication efficiency) for both shallow and deeper models in all considered learning environments. Given these key observations, we propose a kernel normalized ResNet architecture called KNResNet-13 for differentially private learning. Using the proposed architecture, we provide new state-of-the-art accuracy values on the CIFAR-10 and Imagenette datasets, when trained from scratch.
Task Formulation Matters When Learning Continually: A Case Study in Visual Question Answering
Authors: Mavina Nikandrou, Lu Yu, Alessandro Suglia, Ioannis Konstas, Verena Rieser
Date: 2022-09-30 00:00:00
Tasks: Continual Learning, Question Answering, Visual Question Answering, Visual Question Answering (VQA)

Continual learning aims to train a model incrementally on a sequence of tasks without forgetting previous knowledge. Although continual learning has been widely studied in computer vision, its application to Vision+Language tasks is not that straightforward, as settings can be parameterized in multiple ways according to their input modalities. In this paper, we present a detailed study of how different settings affect performance for Visual Question Answering. We first propose three plausible task formulations and demonstrate their impact on the performance of continual learning algorithms. We break down several factors of task similarity, showing that performance and sensitivity to task order highly depend on the shift of the output distribution. We also investigate the potential of pretrained models and compare the robustness of transformer models with different visual embeddings. Finally, we provide an analysis interpreting model representations and their impact on forgetting. Our results highlight the importance of stabilizing visual representations in deeper layers.
Direct Estimation of Porosity from Seismic Data using Rock and Wave Physics Informed Neural Networks (RW-PINN)
Authors: Divakar Vashisth, Tapan Mukerji
Date: 2022-09-30 00:00:00
Tasks: Sand

Petrophysical inversion is an important aspect of reservoir modeling. However due to the lack of a unique and straightforward relationship between seismic traces and rock properties, predicting petrophysical properties directly from seismic data is a complex task. Many studies have attempted to identify the direct end-to-end link using supervised machine learning techniques, but face different challenges such as a lack of large petrophysical training dataset or estimates that may not conform with physics or depositional history of the rocks. We present a rock and wave physics informed neural network (RW-PINN) model that can estimate porosity directly from seismic image traces with no or limited number of wells, with predictions that are consistent with rock physics and geologic knowledge of deposition. As an example, we use the uncemented sand rock physics model and normal-incidence wave physics to guide the learning of RW-PINN to eventually get good estimates of porosities from normal-incidence seismic traces and limited well data. Training RW-PINN with few wells (weakly supervised) helps in tackling the problem of non-uniqueness as different porosity logs can give similar seismic traces. We use weighted normalized root mean square error loss function to train the weakly supervised network and demonstrate the impact of different weights on porosity predictions. The RW-PINN estimated porosities and seismic traces are compared to predictions from a completely supervised model, which gives slightly better porosity estimates but poorly matches the seismic traces, in addition to requiring a large amount of labeled training data. In this paper, we demonstrate the complete workflow for executing petrophysical inversion of seismic data using self-supervised or weakly supervised rock physics informed neural networks.
Neural Causal Models for Counterfactual Identification and Estimation
Authors: Kevin Xia, Yushu Pan, Elias Bareinboim
Date: 2022-09-30 00:00:00
Tasks: counterfactual, Counterfactual Reasoning, Fairness

Evaluating hypothetical statements about how the world would be had a different course of action been taken is arguably one key capability expected from modern AI systems. Counterfactual reasoning underpins discussions in fairness, the determination of blame and responsibility, credit assignment, and regret. In this paper, we study the evaluation of counterfactual statements through neural models. Specifically, we tackle two causal problems required to make such evaluations, i.e., counterfactual identification and estimation from an arbitrary combination of observational and experimental data. First, we show that neural causal models (NCMs) are expressive enough and encode the structural constraints necessary for performing counterfactual reasoning. Second, we develop an algorithm for simultaneously identifying and estimating counterfactual distributions. We show that this algorithm is sound and complete for deciding counterfactual identification in general settings. Third, considering the practical implications of these results, we introduce a new strategy for modeling NCMs using generative adversarial networks. Simulations corroborate with the proposed methodology.
Direct Embedding of Temporal Network Edges via Time-Decayed Line Graphs
Authors: Sudhanshu Chanpuriya, Ryan A. Rossi, Sungchul Kim, Tong Yu, Jane Hoffswell, Nedim Lipka, Shunan Guo, Cameron Musco
Date: 2022-09-30 00:00:00
Tasks: Edge Classification, Link Prediction

Temporal networks model a variety of important phenomena involving timed interactions between entities. Existing methods for machine learning on temporal networks generally exhibit at least one of two limitations. First, time is assumed to be discretized, so if the time data is continuous, the user must determine the discretization and discard precise time information. Second, edge representations can only be calculated indirectly from the nodes, which may be suboptimal for tasks like edge classification. We present a simple method that avoids both shortcomings: construct the line graph of the network, which includes a node for each interaction, and weigh the edges of this graph based on the difference in time between interactions. From this derived graph, edge representations for the original network can be computed with efficient classical methods. The simplicity of this approach facilitates explicit theoretical analysis: we can constructively show the effectiveness of our method's representations for a natural synthetic model of temporal networks. Empirical results on real-world networks demonstrate our method's efficacy and efficiency on both edge classification and temporal link prediction.
Artificial Replay: A Meta-Algorithm for Harnessing Historical Data in Bandits
Authors: Siddhartha Banerjee, Sean R. Sinclair, Milind Tambe, Lily Xu, Christina Lee Yu
Date: 2022-09-30 00:00:00
Tasks: Open-Ended Question Answering

Most real-world deployments of bandit algorithms exist somewhere in between the offline and online set-up, where some historical data is available upfront and additional data is collected dynamically online. How best to incorporate historical data to "warm start" bandit algorithms is an open question: naively initializing reward estimates using all historical samples can suffer from spurious data and imbalanced data coverage, leading to data inefficiency (amount of historical data used) - particularly for continuous action spaces. To address these challenges, we propose ArtificialReplay, a meta-algorithm for incorporating historical data into any arbitrary base bandit algorithm. We show that ArtificialReplay uses only a fraction of the historical data compared to a full warm-start approach, while still achieving identical regret for base algorithms that satisfy independence of irrelevant data (IIData), a novel and broadly applicable property that we introduce. We complement these theoretical results with experiments on K-armed bandits and continuous combinatorial bandits, on which we model green security domains using real poaching data. Our results show the practical benefits of ArtificialReplay for improving data efficiency, including for base algorithms that do not satisfy IIData.
On Best-Arm Identification with a Fixed Budget in Non-Parametric Multi-Armed Bandits
Authors: Antoine Barrier, Aurélien Garivier, Gilles Stoltz
Date: 2022-09-30 00:00:00
Tasks: Multi-Armed Bandits

We lay the foundations of a non-parametric theory of best-arm identification in multi-armed bandits with a fixed budget T. We consider general, possibly non-parametric, models D for distributions over the arms; an overarching example is the model D = P(0,1) of all probability distributions over [0,1]. We propose upper bounds on the average log-probability of misidentifying the optimal arm based on information-theoretic quantities that correspond to infima over Kullback-Leibler divergences between some distributions in D and a given distribution. This is made possible by a refined analysis of the successive-rejects strategy of Audibert, Bubeck, and Munos (2010). We finally provide lower bounds on the same average log-probability, also in terms of the same new information-theoretic quantities; these lower bounds are larger when the (natural) assumptions on the considered strategies are stronger. All these new upper and lower bounds generalize existing bounds based, e.g., on gaps between distributions.
The (In)Effectiveness of Intermediate Task Training For Domain Adaptation and Cross-Lingual Transfer Learning
Authors: Sovesh Mohapatra, Somesh Mohapatra
Date: 2022-10-03 00:00:00
Tasks: Cross-Lingual Transfer, Domain Adaptation, Open-Ended Question Answering, Sentence, Sentence Similarity, text-classification, Text Classification, Transfer Learning

Transfer learning from large language models (LLMs) has emerged as a powerful technique to enable knowledge-based fine-tuning for a number of tasks, adaptation of models for different domains and even languages. However, it remains an open question, if and when transfer learning will work, i.e. leading to positive or negative transfer. In this paper, we analyze the knowledge transfer across three natural language processing (NLP) tasks - text classification, sentimental analysis, and sentence similarity, using three LLMs - BERT, RoBERTa, and XLNet - and analyzing their performance, by fine-tuning on target datasets for domain and cross-lingual adaptation tasks, with and without an intermediate task training on a larger dataset. Our experiments showed that fine-tuning without an intermediate task training can lead to a better performance for most tasks, while more generalized tasks might necessitate a preceding intermediate task training step. We hope that this work will act as a guide on transfer learning to NLP practitioners.
Hypothesis Engineering for Zero-Shot Hate Speech Detection
Authors: Janis Goldzycher, Gerold Schneider
Date: 2022-10-03 00:00:00
Proceeding: TRAC (COLING) 2022 10
Tasks: Hate Speech Detection, Natural Language Inference, Text Classification, Zero-Shot Text Classification

Standard approaches to hate speech detection rely on sufficient available hate speech annotations. Extending previous work that repurposes natural language inference (NLI) models for zero-shot text classification, we propose a simple approach that combines multiple hypotheses to improve English NLI-based zero-shot hate speech detection. We first conduct an error analysis for vanilla NLI-based zero-shot hate speech detection and then develop four strategies based on this analysis. The strategies use multiple hypotheses to predict various aspects of an input text and combine these predictions into a final verdict. We find that the zero-shot baseline used for the initial error analysis already outperforms commercial systems and fine-tuned BERT-based hate speech detection models on HateCheck. The combination of the proposed strategies further increases the zero-shot accuracy of 79.4% on HateCheck by 7.9 percentage points (pp), and the accuracy of 69.6% on ETHOS by 10.0pp.
The Effectiveness of Masked Language Modeling and Adapters for Factual Knowledge Injection
Authors: Sondre Wold
Date: 2022-10-03 00:00:00
Proceeding: COLING (TextGraphs) 2022 10
Tasks: Language Modeling, Language Modelling, Masked Language Modeling

This paper studies the problem of injecting factual knowledge into large pre-trained language models. We train adapter modules on parts of the ConceptNet knowledge graph using the masked language modeling objective and evaluate the success of the method by a series of probing experiments on the LAMA probe. Mean P@K curves for different configurations indicate that the technique is effective, increasing the performance on subsets of the LAMA probe for large values of k by adding as little as 2.1% additional parameters to the original models.
Unsupervised Search Algorithm Configuration using Query Performance Prediction
Authors: Haggai Roitman
Date: 2022-10-03 00:00:00
Tasks: Prediction

Search engine configuration can be quite difficult for inexpert developers. Instead, an auto-configuration approach can be used to speed up development time. Yet, such an automatic process usually requires relevance labels to train a supervised model. In this work, we suggest a simple solution based on query performance prediction that requires no relevance labels but only a sample of queries in a given domain. Using two example usecases we demonstrate the merits of our solution.
Lexical semantics enhanced neural word embeddings
Authors: Dongqiang Yang, Ning li, Li Zou, Hongwei Ma
Date: 2022-10-03 00:00:00
Tasks: Metric Learning, Semantic Similarity, Semantic Textual Similarity, Word Embeddings

Current breakthroughs in natural language processing have benefited dramatically from neural language models, through which distributional semantics can leverage neural data representations to facilitate downstream applications. Since neural embeddings use context prediction on word co-occurrences to yield dense vectors, they are inevitably prone to capture more semantic association than semantic similarity. To improve vector space models in deriving semantic similarity, we post-process neural word embeddings through deep metric learning, through which we can inject lexical-semantic relations, including syn/antonymy and hypo/hypernymy, into a distributional space. We introduce hierarchy-fitting, a novel semantic specialization approach to modelling semantic similarity nuances inherently stored in the IS-A hierarchies. Hierarchy-fitting attains state-of-the-art results on the common- and rare-word benchmark datasets for deriving semantic similarity from neural word embeddings. It also incorporates an asymmetric distance function to specialize hypernymy's directionality explicitly, through which it significantly improves vanilla embeddings in multiple evaluation tasks of detecting hypernymy and directionality without negative impacts on semantic similarity judgement. The results demonstrate the efficacy of hierarchy-fitting in specializing neural embeddings with semantic relations in late fusion, potentially expanding its applicability to aggregating heterogeneous data and various knowledge resources for learning multimodal semantic spaces.
An Embarrassingly Simple Approach for Intellectual Property Rights Protection on Recurrent Neural Networks
Authors: Zhi Qin Tan, Hao Shan Wong, Chee Seng Chan
Date: 2022-10-03 00:00:00

Capitalise on deep learning models, offering Natural Language Processing (NLP) solutions as a part of the Machine Learning as a Service (MLaaS) has generated handsome revenues. At the same time, it is known that the creation of these lucrative deep models is non-trivial. Therefore, protecting these inventions intellectual property rights (IPR) from being abused, stolen and plagiarized is vital. This paper proposes a practical approach for the IPR protection on recurrent neural networks (RNN) without all the bells and whistles of existing IPR solutions. Particularly, we introduce the Gatekeeper concept that resembles the recurrent nature in RNN architecture to embed keys. Also, we design the model training scheme in a way such that the protected RNN model will retain its original performance iff a genuine key is presented. Extensive experiments showed that our protection scheme is robust and effective against ambiguity and removal attacks in both white-box and black-box protection schemes on different RNN variants. Code is available at https://github.com/zhiqin1998/RecurrentIPR
SpeechCLIP: Integrating Speech with Pre-Trained Vision and Language Model
Authors: Yi-Jen Shih, Hsuan-Fu Wang, Heng-Jui Chang, Layne Berry, Hung-Yi Lee, David Harwath
Date: 2022-10-03 00:00:00
Tasks: Language Modeling, Language Modelling, Retrieval, Text Retrieval

Data-driven speech processing models usually perform well with a large amount of text supervision, but collecting transcribed speech data is costly. Therefore, we propose SpeechCLIP, a novel framework bridging speech and text through images to enhance speech models without transcriptions. We leverage state-of-the-art pre-trained HuBERT and CLIP, aligning them via paired images and spoken captions with minimal fine-tuning. SpeechCLIP outperforms prior state-of-the-art on image-speech retrieval and performs zero-shot speech-text retrieval without direct supervision from transcriptions. Moreover, SpeechCLIP can directly retrieve semantically related keywords from speech.
Some pointwise and decidable properties of non-uniform cellular automata
Authors: Xuan Kien Phung
Date: 2022-10-03 00:00:00

For non-uniform cellular automata (NUCA) with finite memory over an arbitrary universe with multiple local transition rules, we show that pointwise nilpotency, pointwise periodicity, and pointwise eventual periodicity properties are respectively equivalent to nilpotency, periodicity, and eventual periodicity. Moreover, we prove that every linear NUCA which satisfies pointwise a polynomial equation (which may depend on the configuration) must be an eventually periodic linear NUCA. Generalizing results for higher dimensional group and linear CA, we also establish the decidability results of the above dynamical properties as well as the injectivity for arbitrary NUCA with finite memory which are local perturbations of higher dimensional linear and group CA. Some generalizations to the case of sparse global perturbations of higher dimensional linear and group CA are also obtained.
Probing of Quantitative Values in Abstractive Summarization Models
Authors: Nathan M. White
Date: 2022-10-03 00:00:00
Tasks: Abstractive Text Summarization, Hallucination, Text Summarization

Abstractive text summarization has recently become a popular approach, but data hallucination remains a serious problem, including with quantitative data. We propose a set of probing tests to evaluate the efficacy of abstract summarization models' modeling of quantitative values found in the input text. Our results show that in most cases, the encoders of recent SOTA-performing models struggle to provide embeddings that adequately represent quantitative values in the input compared to baselines, and in particular, they outperform random representations in some, but surprisingly not all, cases. Under our assumptions, this suggests that the encoder's performance contributes to the quantity hallucination problem. One model type in particular, DistilBART-CDM, was observed to underperform randomly initialized representations for several experiments, and performance versus BERT suggests that standard pretraining and fine-tuning approaches for the summarization task may play a role in underperformance for some encoders.
Community Learning: Understanding A Community Through NLP for Positive Impact
Authors: Md Towhidul Absar Chowdhury, Naveen Sharma
Date: 2022-10-02 00:00:00

A post-pandemic world resulted in economic upheaval, particularly for the cities' communities. While significant work in NLP4PI focuses on national and international events, there is a gap in bringing such state-of-the-art methods into the community development field. In order to help with community development, we must learn about the communities we develop. To that end, we propose the task of community learning as a computational task of extracting natural language data about the community, transforming and loading it into a suitable knowledge graph structure for further downstream applications. We study two particular cases of homelessness and education in showing the visualization capabilities of a knowledge graph, and also discuss other usefulness such a model can provide.