pwc-archive/papers-with-abstracts huggingface dataset:
Adversarially Constructed Evaluation Sets Are More Challenging, but May Not Be Fair
Authors: Jason Phang, Angelica Chen, William Huang, Samuel R. Bowman
Date: 2021-11-16 00:00:00
Proceeding: NAACL (DADC) 2022 7

More capable language models increasingly saturate existing task benchmarks, in some cases outperforming humans. This has left little headroom with which to measure further progress. Adversarial dataset creation has been proposed as a strategy to construct more challenging datasets, and two common approaches are: (1) filtering out easy examples and (2) model-in-the-loop data collection. In this work, we study the impact of applying each approach to create more challenging evaluation datasets. We adapt the AFLite algorithm to filter evaluation data, and run experiments against 18 different adversary models. We find that AFLite indeed selects more challenging examples, lowering the performance of evaluated models more as stronger adversary models are used. However, the resulting ranking of models can also be unstable and highly sensitive to the choice of adversary model used. Moreover, AFLite oversamples examples with low annotator agreement, meaning that model comparisons hinge on the most contentiously labeled examples. Smaller-scale experiments on the adversarially collected datasets ANLI and AdversarialQA show similar findings, broadly lowering performance with stronger adversaries while disproportionately affecting the adversary model.
Assessing gender bias in medical and scientific masked language models with StereoSet
Authors: Robert Robinson
Date: 2021-11-15 00:00:00

NLP systems use language models such as Masked Language Models (MLMs) that are pre-trained on large quantities of text such as Wikipedia create representations of language. BERT is a powerful and flexible general-purpose MLM system developed using unlabeled text. Pre-training on large quantities of text also has the potential to transparently embed the cultural and social biases found in the source text into the MLM system. This study aims to compare biases in general purpose and medical MLMs with the StereoSet bias assessment tool. The general purpose MLMs showed significant bias overall, with BERT scoring 57 and RoBERTa scoring 61. The category of gender bias is where the best performances were found, with 63 for BERT and 73 for RoBERTa. Performances for profession, race, and religion were similar to the overall bias scores for the general-purpose MLMs.Medical MLMs showed more bias in all categories than the general-purpose MLMs except for SciBERT, which showed a race bias score of 55, which was superior to the race bias score of 53 for BERT. More gender (Medical 54-58 vs. General 63-73) and religious (46-54 vs. 58) biases were found with medical MLMs. This evaluation of four medical MLMs for stereotyped assessments about race, gender, religion, and profession showed inferior performance to general-purpose MLMs. These medically focused MLMs differ considerably in training source data, which is likely the root cause of the differences in ratings for stereotyped biases from the StereoSet tool.
A Multi-criteria Approach to Evolve Sparse Neural Architectures for Stock Market Forecasting
Authors: Faizal Hafiz, Jan Broekaert, Davide La Torre, Akshya Swain
Date: 2021-11-15 00:00:00
Tasks: feature selection, Neural Architecture Search

This study proposes a new framework to evolve efficacious yet parsimonious neural architectures for the movement prediction of stock market indices using technical indicators as inputs. In the light of a sparse signal-to-noise ratio under the Efficient Market hypothesis, developing machine learning methods to predict the movement of a financial market using technical indicators has shown to be a challenging problem. To this end, the neural architecture search is posed as a multi-criteria optimization problem to balance the efficacy with the complexity of architectures. In addition, the implications of different dominant trading tendencies which may be present in the pre-COVID and within-COVID time periods are investigated. An $\epsilon-$ constraint framework is proposed as a remedy to extract any concordant information underlying the possibly conflicting pre-COVID data. Further, a new search paradigm, Two-Dimensional Swarms (2DS) is proposed for the multi-criteria neural architecture search, which explicitly integrates sparsity as an additional search dimension in particle swarms. A detailed comparative evaluation of the proposed approach is carried out by considering genetic algorithm and several combinations of empirical neural design rules with a filter-based feature selection method (mRMR) as baseline approaches. The results of this study convincingly demonstrate that the proposed approach can evolve parsimonious networks with better generalization capabilities.
Normalizing flows for atomic solids
Authors: Peter Wirnsberger, George Papamakarios, Borja Ibarz, Sébastien Racanière, Andrew J. Ballard, Alexander Pritzel, Charles Blundell
Date: 2021-11-16 00:00:00

We present a machine-learning approach, based on normalizing flows, for modelling atomic solids. Our model transforms an analytically tractable base distribution into the target solid without requiring ground-truth samples for training. We report Helmholtz free energy estimates for cubic and hexagonal ice modelled as monatomic water as well as for a truncated and shifted Lennard-Jones system, and find them to be in excellent agreement with literature values and with estimates from established baseline methods. We further investigate structural properties and show that the model samples are nearly indistinguishable from the ones obtained with molecular dynamics. Our results thus demonstrate that normalizing flows can provide high-quality samples and free energy estimates without the need for multi-staging.
Hierarchical transfer learning with applications for electricity load forecasting
Authors: Anestis Antoniadis, Solenne Gaucher, Yannig Goude
Date: 2021-11-16 00:00:00
Tasks: Additive models, Load Forecasting, Transfer Learning

The recent abundance of data on electricity consumption at different scales opens new challenges and highlights the need for new techniques to leverage information present at finer scales in order to improve forecasts at wider scales. In this work, we take advantage of the similarity between this hierarchical prediction problem and multi-scale transfer learning. We develop two methods for hierarchical transfer learning, based respectively on the stacking of generalized additive models and random forests, and on the use of aggregation of experts. We apply these methods to two problems of electricity load forecasting at national scale, using smart meter data in the first case, and regional data in the second case. For these two usecases, we compare the performances of our methods to that of benchmark algorithms, and we investigate their behaviour using variable importance analysis. Our results demonstrate the interest of both methods, which lead to a significant improvement of the predictions.
QA4PRF: A Question Answering based Framework for Pseudo Relevance Feedback
Authors: Handong Ma, Jiawei Hou, Chenxu Zhu, Weinan Zhang, Ruiming Tang, Jincai Lai, Jieming Zhu, Xiuqiang He, Yong Yu
Date: 2021-11-16 00:00:00
Tasks: Question Answering, Semantic Similarity, Semantic Textual Similarity

Pseudo relevance feedback (PRF) automatically performs query expansion based on top-retrieved documents to better represent the user's information need so as to improve the search results. Previous PRF methods mainly select expansion terms with high occurrence frequency in top-retrieved documents or with high semantic similarity with the original query. However, existing PRF methods hardly try to understand the content of documents, which is very important in performing effective query expansion to reveal the user's information need. In this paper, we propose a QA-based framework for PRF called QA4PRF to utilize contextual information in documents. In such a framework, we formulate PRF as a QA task, where the query and each top-retrieved document play the roles of question and context in the corresponding QA system, while the objective is to find some proper terms to expand the original query by utilizing contextual information, which are similar answers in QA task. Besides, an attention-based pointer network is built on understanding the content of top-retrieved documents and selecting the terms to represent the original query better. We also show that incorporating the traditional supervised learning methods, such as LambdaRank, to integrate PRF information will further improve the performance of QA4PRF. Extensive experiments on three real-world datasets demonstrate that QA4PRF significantly outperforms the state-of-the-art methods.
SPANN: Highly-efficient Billion-scale Approximate Nearest Neighbor Search
Authors: Qi Chen, Bing Zhao, Haidong Wang, Mingqin Li, Chuanjie Liu, Zengzhong Li, Mao Yang, Jingdong Wang
Date: 2021-11-05 00:00:00
Proceeding: NeurIPS 2021 12

The in-memory algorithms for approximate nearest neighbor search (ANNS) have achieved great success for fast high-recall search, but are extremely expensive when handling very large scale database. Thus, there is an increasing request for the hybrid ANNS solutions with small memory and inexpensive solid-state drive (SSD). In this paper, we present a simple but efficient memory-disk hybrid indexing and search system, named SPANN, that follows the inverted index methodology. It stores the centroid points of the posting lists in the memory and the large posting lists in the disk. We guarantee both disk-access efficiency (low latency) and high recall by effectively reducing the disk-access number and retrieving high-quality posting lists. In the index-building stage, we adopt a hierarchical balanced clustering algorithm to balance the length of posting lists and augment the posting list by adding the points in the closure of the corresponding clusters. In the search stage, we use a query-aware scheme to dynamically prune the access of unnecessary posting lists. Experiment results demonstrate that SPANN is 2$\times$ faster than the state-of-the-art ANNS solution DiskANN to reach the same recall quality $90\%$ with same memory cost in three billion-scale datasets. It can reach $90\%$ recall@1 and recall@10 in just around one millisecond with only 32GB memory cost. Code is available at: {\footnotesize\color{blue}{\url{https://github.com/microsoft/SPTAG}}}.
Xspect, estimation of the angular power spectrum by computing cross-power spectra with analytical error bars
Authors: M. Tristram, J. F. Macias-Perez, C. Renault, D. Santos
Date: 2004-05-28 00:00:00

We present Xspect, a method to obtain estimates of the angular power spectrum of the Cosmic Microwave Background (CMB) temperature anisotropies including analytical error bars developed for the Archeops experiment. Cross-power spectra are computed from a set of maps and each of them is in itself an unbiased estimate of the power spectrum as long as the detector noises are uncorrelated. Then, the cross-power spectra are combined into a final temperature power spectrum with error bars analytically derived from the cross-correlation matrix. This method presents three main useful properties : (1) no estimation of the noise power spectrum is needed, (2) complex weighting schemes including sky covering and map noise properties can be easily taken into account, and corrected for, for each input map, (3) error bars are quickly computed analytically from the data themselves with no Monte-Carlo simulations involved. Xspect also permits the study of common fluctuations between maps from different sky surveys such as CMB, Sunyaev-Zel'dovich effect or mass fluctuations from weak lensing observations.
Binary classification of spoken words with passive phononic metamaterials
Authors: Tena Dubček, Daniel Moreno-Garcia, Thomas Haag, Parisa Omidvar, Henrik R. Thomsen, Theodor S. Becker, Lars Gebraad, Christoph Bärlocher, Fredrik Andersson, Sebastian D. Huber, Dirk-Jan van Manen, Luis Guillermo Villanueva, Johan O. A. Robertsson, Marc Serra-Garcia
Date: 2021-11-14 00:00:00
Tasks: Binary Classification, Classification, Event Detection, speech-recognition, Speech Recognition

Mitigating the energy requirements of artificial intelligence requires novel physical substrates for computation. Phononic metamaterials have a vanishingly low power dissipation and hence are a prime candidate for green, always-on computers. However, their use in machine learning applications has not been explored due to the complexity of their design process: Current phononic metamaterials are restricted to simple geometries (e.g. periodic, tapered), and hence do not possess sufficient expressivity to encode machine learning tasks. We design and fabricate a non-periodic phononic metamaterial, directly from data samples, that can distinguish between pairs of spoken words in the presence of a simple readout nonlinearity; hence demonstrating that phononic metamaterials are a viable avenue towards zero-power smart devices.
Blockage Prediction Using Wireless Signatures: Deep Learning Enables Real-World Demonstration
Authors: Shunyao Wu, Muhammad Alrabeiah, Chaitali Chakrabarti, Ahmed Alkhateeb
Date: 2021-11-16 00:00:00

Overcoming the link blockage challenges is essential for enhancing the reliability and latency of millimeter wave (mmWave) and sub-terahertz (sub-THz) communication networks. Previous approaches relied mainly on either (i) multiple-connectivity, which under-utilizes the network resources, or on (ii) the use of out-of-band and non-RF sensors to predict link blockages, which is associated with increased cost and system complexity. In this paper, we propose a novel solution that relies only on in-band mmWave wireless measurements to proactively predict future dynamic line-of-sight (LOS) link blockages. The proposed solution utilizes deep neural networks and special patterns of received signal power, that we call pre-blockage wireless signatures to infer future blockages. Specifically, the developed machine learning models attempt to predict: (i) If a future blockage will occur? (ii) When will this blockage happen? (iii) What is the type of the blockage? And (iv) what is the direction of the moving blockage? To evaluate our proposed approach, we build a large-scale real-world dataset comprising nearly $0.5$ million data points (mmWave measurements) for both indoor and outdoor blockage scenarios. The results, using this dataset, show that the proposed approach can successfully predict the occurrence of future dynamic blockages with more than 85\% accuracy. Further, for the outdoor scenario with highly-mobile vehicular blockages, the proposed model can predict the exact time of the future blockage with less than $80$ms error for blockages happening within the future $500$ms. These results, among others, highlight the promising gains of the proposed proactive blockage prediction solution which could potentially enhance the reliability and latency of future wireless networks.
Entropy optimized semi-supervised decomposed vector-quantized variational autoencoder model based on transfer learning for multiclass text classification and generation
Authors: Shivani Malhotra, Vinay Kumar, Alpana Agarwal
Date: 2021-11-10 00:00:00
Tasks: Decoder, Multi Class Text Classification, Quantization, text-classification, Text Classification, Text Generation, Transfer Learning

Semisupervised text classification has become a major focus of research over the past few years. Hitherto, most of the research has been based on supervised learning, but its main drawback is the unavailability of labeled data samples in practical applications. It is still a key challenge to train the deep generative models and learn comprehensive representations without supervision. Even though continuous latent variables are employed primarily in deep latent variable models, discrete latent variables, with their enhanced understandability and better compressed representations, are effectively used by researchers. In this paper, we propose a semisupervised discrete latent variable model for multi-class text classification and text generation. The proposed model employs the concept of transfer learning for training a quantized transformer model, which is able to learn competently using fewer labeled instances. The model applies decomposed vector quantization technique to overcome problems like posterior collapse and index collapse. Shannon entropy is used for the decomposed sub-encoders, on which a variable DropConnect is applied, to retain maximum information. Moreover, gradients of the Loss function are adaptively modified during backpropagation from decoder to encoder to enhance the performance of the model. Three conventional datasets of diversified range have been used for validating the proposed model on a variable number of labeled instances. Experimental results indicate that the proposed model has surpassed the state-of-the-art models remarkably.
Some Doxastic Łukasiewicz Logic
Authors: Doratossadat Dastgheib, Hadi Farahani
Date: 2021-11-04 00:00:00

We propose a doxastic \L ukasiewicz logic \textbf{B\L} that is sound and complete with respect to the class of Kripke-based models in which atomic propositions and accessibility relations are both infinitely valued in the standard MV-algebra [0,1]. We also introduce some extensions of \textbf{B\L} corresponding to axioms \textbf{D}, \textbf{4}, and \textbf{T} of classical epistemic logic. Furthermore, completeness of these extensions are established corresponding to the appropriate classes of models.
Modelling airborne transmission of SARS-CoV-2 at a local scale
Authors: Simon Rahn, Marion Gödel, Gerta Köster, Gesine Hofinger
Date: 2021-11-16 00:00:00

The coronavirus disease (COVID-19) pandemic has changed our lives and still poses a challenge to science. Numerous studies have contributed to a better understanding of the pandemic. In particular, inhalation of aerosolised pathogens has been identified as essential for transmission. This information is crucial to slow the spread, but the individual likelihood of becoming infected in everyday situations remains uncertain. Mathematical models help estimate such risks. In this study, we propose how to model airborne transmission of SARS-CoV-2 at a local scale. In this regard, we combine microscopic crowd simulation with a new model for disease transmission. Inspired by compartmental models, we describe agents' health status as susceptible, exposed, infectious or recovered. Infectious agents exhale pathogens bound to persistent aerosols, whereas susceptible agents absorb pathogens when moving through an aerosol cloud left by the infectious agent. The transmission depends on the pathogen load of the aerosol cloud, which changes over time. We propose a 'high risk' benchmark scenario to distinguish critical from non-critical situations. Simulating indoor situations show that the new model is suitable to evaluate the risk of exposure qualitatively and, thus, enables scientists or even decision-makers to better assess the spread of COVID-19 and similar diseases.
Computational tools for assessing gene therapy under branching process models of mutation
Authors: Timothy C Stutz, Janet S. Sinsheimer, Mary Sehl, Jason Xu
Date: 2021-11-16 00:00:00

Multitype branching processes are ideal for studying the population dynamics of stem cell populations undergoing mutation accumulation over the years following transplant. In such stochastic models, several quantities are of clinical interest as insertional mutagenesis carries the potential threat of leukemogenesis following gene therapy with autologous stem cell transplantation. In this paper, we develop a three-type branching process model describing accumulations of mutations in a population of stem cells distinguished by their ability for long-term self-renewal. Our outcome of interest is the appearance of a double-mutant cell, which carries a high potential for leukemic transformation. In our model, a single-hit mutation carries a slight proliferative advantage over a wild-type stem cells. We compute marginalized transition probabilities that allow us to capture important quantitative aspects of our model, including the probability of observing a double-hit mutant and relevant moments of a single-hit mutation population over time. We thoroughly explore the model behavior numerically, varying birth rates across the initial sizes and populations of wild type stem cells and single-hit mutants, and compare the probability of observing a double-hit mutant under these conditions. We find that increasing the number of single-mutants over wild-type particles initially present has a large effect on the occurrence of a double-mutant, and that it is relatively safe for single-mutants to be quite proliferative, provided the lentiviral gene addition avoids creating single mutants in the original insertion process. Our approach is broadly applicable to an important set of questions in cancer modeling and other population processes involving multiple stages, compartments, or types.
Unsupervised Speech Enhancement with speech recognition embedding and disentanglement losses
Authors: Viet Anh Trinh, Sebastian Braun
Date: 2021-11-16 00:00:00
Tasks: Disentanglement, Speech Enhancement, speech-recognition, Speech Recognition

Speech enhancement has recently achieved great success with various deep learning methods. However, most conventional speech enhancement systems are trained with supervised methods that impose two significant challenges. First, a majority of training datasets for speech enhancement systems are synthetic. When mixing clean speech and noisy corpora to create the synthetic datasets, domain mismatches occur between synthetic and real-world recordings of noisy speech or audio. Second, there is a trade-off between increasing speech enhancement performance and degrading speech recognition (ASR) performance. Thus, we propose an unsupervised loss function to tackle those two problems. Our function is developed by extending the MixIT loss function with speech recognition embedding and disentanglement loss. Our results show that the proposed function effectively improves the speech enhancement performance compared to a baseline trained in a supervised way on the noisy VoxCeleb dataset. While fully unsupervised training is unable to exceed the corresponding baseline, with joint super- and unsupervised training, the system is able to achieve similar speech quality and better ASR performance than the best supervised baseline.