pwc-archive/papers-with-abstracts huggingface dataset:
Hybrid Micro/Macro Level Convolution for Heterogeneous Graph Learning
Authors: Le Yu, Leilei Sun, Bowen Du, Chuanren Liu, Weifeng Lv, Hui Xiong
Date: 2020-12-29 00:00:00
Tasks: Graph Learning, Node Property Prediction, Representation Learning

Heterogeneous graphs are pervasive in practical scenarios, where each graph consists of multiple types of nodes and edges. Representation learning on heterogeneous graphs aims to obtain low-dimensional node representations that could preserve both node attributes and relation information. However, most of the existing graph convolution approaches were designed for homogeneous graphs, and therefore cannot handle heterogeneous graphs. Some recent methods designed for heterogeneous graphs are also faced with several issues, including the insufficient utilization of heterogeneous properties, structural information loss, and lack of interpretability. In this paper, we propose HGConv, a novel Heterogeneous Graph Convolution approach, to learn comprehensive node representations on heterogeneous graphs with a hybrid micro/macro level convolutional operation. Different from existing methods, HGConv could perform convolutions on the intrinsic structure of heterogeneous graphs directly at both micro and macro levels: A micro-level convolution to learn the importance of nodes within the same relation, and a macro-level convolution to distinguish the subtle difference across different relations. The hybrid strategy enables HGConv to fully leverage heterogeneous information with proper interpretability. Moreover, a weighted residual connection is designed to aggregate both inherent attributes and neighbor information of the focal node adaptively. Extensive experiments on various tasks demonstrate not only the superiority of HGConv over existing methods, but also the intuitive interpretability of our approach for graph analysis.
Super-k: A Piecewise Linear Classifier Based on Voronoi Tessellations
Authors: Rahman Salim Zengin, Volkan Sezer
Date: 2020-12-31 00:00:00

Voronoi tessellations are used to partition the Euclidean space into polyhedral regions, which are called Voronoi cells. Labeling the Voronoi cells with the class information, we can map any classification problem into a Voronoi tessellation. In this way, the classification problem changes into a query of just finding the enclosing Voronoi cell. In order to accomplish this task, we have developed a new algorithm which generates a labeled Voronoi tessellation that partitions the training data into polyhedral regions and obtains interclass boundaries as an indirect result. It is called Supervised k-Voxels or in short Super-k. We are introducing Super-k as a foundational new algorithm and opening the possibility of a new family of algorithms. In this paper, it is shown via comparisons on certain datasets that the Super-k algorithm has the potential of providing comparable performance of the well-known SVM family of algorithms with less complexity.
Intrinsic Bias Metrics Do Not Correlate with Application Bias
Authors: Seraphina Goldfarb-Tarrant, Rebecca Marchant, Ricardo Muñoz Sanchez, Mugdha Pandya, Adam Lopez
Date: 2020-12-31 00:00:00
Proceeding: ACL 2021 5
Tasks: Word Embeddings

Natural Language Processing (NLP) systems learn harmful societal biases that cause them to amplify inequality as they are deployed in more and more situations. To guide efforts at debiasing these systems, the NLP community relies on a variety of metrics that quantify bias in models. Some of these metrics are intrinsic, measuring bias in word embedding spaces, and some are extrinsic, measuring bias in downstream tasks that the word embeddings enable. Do these intrinsic and extrinsic metrics correlate with each other? We compare intrinsic and extrinsic metrics across hundreds of trained models covering different tasks and experimental conditions. Our results show no reliable correlation between these metrics that holds in all scenarios across tasks and languages. We urge researchers working on debiasing to focus on extrinsic measures of bias, and to make using these measures more feasible via creation of new challenge sets and annotated test data. To aid this effort, we release code, a new intrinsic metric, and an annotated test set focused on gender bias in hate speech.
Studying Strategically: Learning to Mask for Closed-book QA
Authors: Qinyuan Ye, Belinda Z. Li, Sinong Wang, Benjamin Bolte, Hao Ma, Wen-tau Yih, Xiang Ren, Madian Khabsa
Date: 2020-12-31 00:00:00
Tasks: Language Modeling, Language Modelling, Question Answering, TriviaQA

Closed-book question-answering (QA) is a challenging task that requires a model to directly answer questions without access to external knowledge. It has been shown that directly fine-tuning pre-trained language models with (question, answer) examples yields surprisingly competitive performance, which is further improved upon through adding an intermediate pre-training stage between general pre-training and fine-tuning. Prior work used a heuristic during this intermediate stage, whereby named entities and dates are masked, and the model is trained to recover these tokens. In this paper, we aim to learn the optimal masking strategy for the intermediate pre-training stage. We first train our masking policy to extract spans that are likely to be tested, using supervision from the downstream task itself, then deploy the learned policy during intermediate pre-training. Thus, our policy packs task-relevant knowledge into the parameters of a language model. Our approach is particularly effective on TriviaQA, outperforming strong heuristics when used to pre-train BART.
Uncovering the invariant structural organization of the human connectome
Authors: Anand Pathak, Shakti N. Menon, Sitabhra Sinha
Date: 2020-12-31 00:00:00

In order to understand the complex cognitive functions of the human brain, it is essential to study the structural connectome, i.e., the wiring of different brain regions to each other through axonal pathways. However, the high degree of plasticity and cross-population variability in human brains makes it difficult to relate structure to function, motivating a search for invariant patterns in the connectivity. At the same time, variability within a population can provide information about generative mechanisms. In this paper we analyze the connection topology and link-weight distribution of human structural connectomes obtained from a database comprising 196 subjects. By demonstrating a correspondence between the occurrence frequency of individual links and their average weight across the population, we show that the process by which the brain is wired is not independent of the process by which the link weights of the connectome are determined. Furthermore, using the specific distribution of the weights associated with each link over the entire population, we show that a single parameter that is specific to a link can account for its frequency of occurrence, as well as, the variation in its weight across different subjects. This parameter provides a basis for ``rescaling'' the link weights in each connectome, allowing us to obtain a generic network representative of the human brain, distinct from a simple average over the connectomes. We obtain functional connectomes by implementing a neural mass model on each of the vertices of the corresponding structural connectomes. By comparing with the empirical functional brain networks, we demonstrate that the rescaling procedure yields a closer structure-function correspondence. Finally, we show that the representative network can be decomposed into a basal component that is stable across the population and a highly variable superstructure.
Real-time Webcam Heart-Rate and Variability Estimation with Clean Ground Truth for Evaluation
Authors: Amogh Gudi, Marian Bittner, Jan van Gemert
Date: 2020-12-31 00:00:00
Tasks: Benchmarking, Heart Rate Variability

Remote photo-plethysmography (rPPG) uses a camera to estimate a person's heart rate (HR). Similar to how heart rate can provide useful information about a person's vital signs, insights about the underlying physio/psychological conditions can be obtained from heart rate variability (HRV). HRV is a measure of the fine fluctuations in the intervals between heart beats. However, this measure requires temporally locating heart beats with a high degree of precision. We introduce a refined and efficient real-time rPPG pipeline with novel filtering and motion suppression that not only estimates heart rates, but also extracts the pulse waveform to time heart beats and measure heart rate variability. This unsupervised method requires no rPPG specific training and is able to operate in real-time. We also introduce a new multi-modal video dataset, VicarPPG 2, specifically designed to evaluate rPPG algorithms on HR and HRV estimation. We validate and study our method under various conditions on a comprehensive range of public and self-recorded datasets, showing state-of-the-art results and providing useful insights into some unique aspects. Lastly, we make available CleanerPPG, a collection of human-verified ground truth peak/heart-beat annotations for existing rPPG datasets. These verified annotations should make future evaluations and benchmarking of rPPG algorithms more accurate, standardized and fair.
A Tale of Two Efficient and Informative Negative Sampling Distributions
Authors: Shabnam Daghaghi, Tharun Medini, Nicholas Meisburger, Beidi Chen, Mengnan Zhao, Anshumali Shrivastava
Date: 2020-12-31 00:00:00
Tasks: CPU, GPU, Information Retrieval, Retrieval, Vocal Bursts Valence Prediction

Softmax classifiers with a very large number of classes naturally occur in many applications such as natural language processing and information retrieval. The calculation of full softmax is costly from the computational and energy perspective. There have been various sampling approaches to overcome this challenge, popularly known as negative sampling (NS). Ideally, NS should sample negative classes from a distribution that is dependent on the input data, the current parameters, and the correct positive class. Unfortunately, due to the dynamically updated parameters and data samples, there is no sampling scheme that is provably adaptive and samples the negative classes efficiently. Therefore, alternative heuristics like random sampling, static frequency-based sampling, or learning-based biased sampling, which primarily trade either the sampling cost or the adaptivity of samples per iteration are adopted. In this paper, we show two classes of distributions where the sampling scheme is truly adaptive and provably generates negative samples in near-constant time. Our implementation in C++ on CPU is significantly superior, both in terms of wall-clock time and accuracy, compared to the most optimized TensorFlow implementations of other popular negative sampling approaches on powerful NVIDIA V100 GPU.
Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers
Authors: Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip H. S. Torr, Li Zhang
Date: 2020-12-31 00:00:00
Proceeding: CVPR 2021 1
Tasks: Decoder, Medical Image Segmentation, Segmentation, Semantic Segmentation

Most recent semantic segmentation methods adopt a fully-convolutional network (FCN) with an encoder-decoder architecture. The encoder progressively reduces the spatial resolution and learns more abstract/semantic visual concepts with larger receptive fields. Since context modeling is critical for segmentation, the latest efforts have been focused on increasing the receptive field, through either dilated/atrous convolutions or inserting attention modules. However, the encoder-decoder based FCN architecture remains unchanged. In this paper, we aim to provide an alternative perspective by treating semantic segmentation as a sequence-to-sequence prediction task. Specifically, we deploy a pure transformer (ie, without convolution and resolution reduction) to encode an image as a sequence of patches. With the global context modeled in every layer of the transformer, this encoder can be combined with a simple decoder to provide a powerful segmentation model, termed SEgmentation TRansformer (SETR). Extensive experiments show that SETR achieves new state of the art on ADE20K (50.28% mIoU), Pascal Context (55.83% mIoU) and competitive results on Cityscapes. Particularly, we achieve the first position in the highly competitive ADE20K test server leaderboard on the day of submission.
Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans
Authors: Sida Peng, Yuanqing Zhang, Yinghao Xu, Qianqian Wang, Qing Shuai, Hujun Bao, Xiaowei Zhou
Date: 2020-12-31 00:00:00
Proceeding: CVPR 2021 1
Tasks: Novel View Synthesis, Representation Learning

This paper addresses the challenge of novel view synthesis for a human performer from a very sparse set of camera views. Some recent works have shown that learning implicit neural representations of 3D scenes achieves remarkable view synthesis quality given dense input views. However, the representation learning will be ill-posed if the views are highly sparse. To solve this ill-posed problem, our key idea is to integrate observations over video frames. To this end, we propose Neural Body, a new human body representation which assumes that the learned neural representations at different frames share the same set of latent codes anchored to a deformable mesh, so that the observations across frames can be naturally integrated. The deformable mesh also provides geometric guidance for the network to learn 3D representations more efficiently. To evaluate our approach, we create a multi-view dataset named ZJU-MoCap that captures performers with complex motions. Experiments on ZJU-MoCap show that our approach outperforms prior works by a large margin in terms of novel view synthesis quality. We also demonstrate the capability of our approach to reconstruct a moving person from a monocular video on the People-Snapshot dataset. The code and dataset are available at https://zju3dv.github.io/neuralbody/.
Using Natural Language Relations between Answer Choices for Machine Comprehension
Authors: Rajkumar Pujari, Dan Goldwasser
Date: 2020-12-31 00:00:00
Proceeding: NAACL 2019 6
Tasks: Natural Language Inference, Question Answering, Reading Comprehension

When evaluating an answer choice for Reading Comprehension task, other answer choices available for the question and the answers of related questions about the same paragraph often provide valuable information. In this paper, we propose a method to leverage the natural language relations between the answer choices, such as entailment and contradiction, to improve the performance of machine comprehension. We use a stand-alone question answering (QA) system to perform QA task and a Natural Language Inference (NLI) system to identify the relations between the choice pairs. Then we perform inference using an Integer Linear Programming (ILP)-based relational framework to re-evaluate the decisions made by the standalone QA system in light of the relations identified by the NLI system. We also propose a multitask learning model that learns both the tasks jointly.
Semantic Modeling with SUMO
Authors: Robert B. Allen
Date: 2020-12-31 00:00:00

We explore using the Suggested Upper Merged Ontology (SUMO) to develop a semantic simulation. We provide two proof-of-concept demonstrations modeling transitions in a simulated gasoline engine using a general-purpose programming language. Rather than focusing on computationally highly intensive techniques, we explore a less computationally intensive approach related to familiar software engineering testing procedures. In addition, we propose structured representations of terms based on linguistic approaches to lexicography.
Topological obstructions in neural networks learning
Authors: Serguei Barannikov, Daria Voronkova, Ilya Trofimov, Alexander Korotin, Grigorii Sotnikov, Evgeny Burnaev
Date: 2020-12-31 00:00:00
Tasks: Topological Data Analysis

We apply topological data analysis methods to loss functions to gain insights into learning of deep neural networks and deep neural networks generalization properties. We use the Morse complex of the loss function to relate the local behavior of gradient descent trajectories with global properties of the loss surface. We define the neural network Topological Obstructions score, "TO-score", with the help of robust topological invariants, barcodes of the loss function, that quantify the "badness" of local minima for gradient-based optimization. We have made experiments for computing these invariants for fully-connected, convolutional and ResNet-like neural networks on different datasets: MNIST, Fashion MNIST, CIFAR10, CIFAR100 and SVHN. Our two principal observations are as follows. Firstly, the neural network barcode and TO score decrease with the increase of the neural network depth and width, thus the topological obstructions to learning diminish. Secondly, in certain situations there is an intriguing connection between the lengths of minima segments in the barcode and the minima generalization errors.
Shortformer: Better Language Modeling using Shorter Inputs
Authors: Ofir Press, Noah A. Smith, Mike Lewis
Date: 2020-12-31 00:00:00
Proceeding: ACL 2021 5
Tasks: Language Modeling, Language Modelling, Position, Word Embeddings

Increasing the input length has been a driver of progress in language modeling with transformers. We identify conditions where shorter inputs are not harmful, and achieve perplexity and efficiency improvements through two new methods that decrease input length. First, we show that initially training a model on short subsequences before moving on to longer ones both reduces overall training time and, surprisingly, substantially improves perplexity. Second, we show how to improve the efficiency of recurrence methods in transformers, which let models condition on previously processed tokens when generating sequences that exceed the maximal length the transformer can handle at once. Existing methods require computationally expensive relative position embeddings; we introduce a simple alternative of adding absolute position embeddings to queries and keys instead of to word embeddings, which efficiently produces superior results. We show that these recurrent models also benefit from short input lengths. Combining these techniques speeds up training by a factor of 1.65, reduces memory usage, and substantially improves perplexity on WikiText-103, without adding any parameters.
Fully Non-autoregressive Neural Machine Translation: Tricks of the Trade
Authors: Jiatao Gu, Xiang Kong
Date: 2020-12-31 00:00:00
Proceeding: Findings (ACL) 2021 8
Tasks: Machine Translation, Translation

Fully non-autoregressive neural machine translation (NAT) is proposed to simultaneously predict tokens with single forward of neural networks, which significantly reduces the inference latency at the expense of quality drop compared to the Transformer baseline. In this work, we target on closing the performance gap while maintaining the latency advantage. We first inspect the fundamental issues of fully NAT models, and adopt dependency reduction in the learning space of output tokens as the basic guidance. Then, we revisit methods in four different aspects that have been proven effective for improving NAT models, and carefully combine these techniques with necessary modifications. Our extensive experiments on three translation benchmarks show that the proposed system achieves the new state-of-the-art results for fully NAT models, and obtains comparable performance with the autoregressive and iterative NAT systems. For instance, one of the proposed models achieves 27.49 BLEU points on WMT14 En-De with approximately 16.5X speed up at inference time.
Timely Communication in Federated Learning
Authors: Baturalp Buyukates, Sennur Ulukus
Date: 2020-12-31 00:00:00
Tasks: Federated Learning

We consider a federated learning framework in which a parameter server (PS) trains a global model by using $n$ clients without actually storing the client data centrally at a cloud server. Focusing on a setting where the client datasets are fast changing and highly temporal in nature, we investigate the timeliness of model updates and propose a novel timely communication scheme. Under the proposed scheme, at each iteration, the PS waits for $m$ available clients and sends them the current model. Then, the PS uses the local updates of the earliest $k$ out of $m$ clients to update the global model at each iteration. We find the average age of information experienced by each client and numerically characterize the age-optimal $m$ and $k$ values for a given $n$. Our results indicate that, in addition to ensuring timeliness, the proposed communication scheme results in significantly smaller average iteration times compared to random client selection without hurting the convergence of the global learning task.
Continuity of Generalized Entropy and Statistical Learning
Authors: Aolin Xu
Date: 2020-12-31 00:00:00
Tasks: Decision Making, Learning Theory

We study the continuity property of the generalized entropy as a function of the underlying probability distribution, defined with an action space and a loss function, and use this property to answer the basic questions in statistical learning theory: the excess risk analyses for various learning methods. We first derive upper and lower bounds for the entropy difference of two distributions in terms of several commonly used f-divergences, the Wasserstein distance, a distance that depends on the action space and the loss function, and the Bregman divergence generated by the entropy, which also induces bounds in terms of the Euclidean distance between the two distributions. Examples are given along with the discussion of each general result, comparisons are made with the existing entropy difference bounds, and new mutual information upper bounds are derived based on the new results. We then apply the entropy difference bounds to the theory of statistical learning. It is shown that the excess risks in the two popular learning paradigms, the frequentist learning and the Bayesian learning, both can be studied with the continuity property of different forms of the generalized entropy. The analysis is then extended to the continuity of generalized conditional entropy. The extension provides performance bounds for Bayes decision making with mismatched distributions. It also leads to excess risk bounds for a third paradigm of learning, where the decision rule is optimally designed under the projection of the empirical distribution to a predefined family of distributions. We thus establish a unified method of excess risk analysis for the three major paradigms of statistical learning, through the continuity of generalized entropy.