pwc-archive/papers-with-abstracts huggingface dataset:
Multi-scale Transformer Language Models
Authors: Sandeep Subramanian, Ronan Collobert, Marc'Aurelio Ranzato, Y-Lan Boureau
Date: 2020-05-01 00:00:00
Tasks: Inductive Bias, Language Modeling, Language Modelling

We investigate multi-scale transformer language models that learn representations of text at multiple scales, and present three different architectures that have an inductive bias to handle the hierarchical nature of language. Experiments on large-scale language modeling benchmarks empirically demonstrate favorable likelihood vs memory footprint trade-offs, e.g. we show that it is possible to train a hierarchical variant with 30 layers that has 23% smaller memory footprint and better perplexity, compared to a vanilla transformer with less than half the number of layers, on the Toronto BookCorpus. We analyze the advantages of learned representations at multiple scales in terms of memory footprint, compute time, and perplexity, which are particularly appealing given the quadratic scaling of transformers' run time and memory usage with respect to sequence length.
Evaluating Robustness to Input Perturbations for Neural Machine Translation
Authors: Xing Niu, Prashant Mathur, Georgiana Dinu, Yaser Al-Onaizan
Date: 2020-05-01 00:00:00
Proceeding: ACL 2020 6
Tasks: Machine Translation, NMT, Translation

Neural Machine Translation (NMT) models are sensitive to small perturbations in the input. Robustness to such perturbations is typically measured using translation quality metrics such as BLEU on the noisy input. This paper proposes additional metrics which measure the relative degradation and changes in translation when small perturbations are added to the input. We focus on a class of models employing subword regularization to address robustness and perform extensive evaluations of these models using the robustness measures proposed. Results show that our proposed metrics reveal a clear trend of improved robustness to perturbations when subword regularization methods are used.
Clinical Reading Comprehension: A Thorough Analysis of the emrQA Dataset
Authors: Xiang Yue, Bernal Jimenez Gutierrez, Huan Sun
Date: 2020-05-01 00:00:00
Proceeding: ACL 2020 6
Tasks: Machine Reading Comprehension, Question Answering, Reading Comprehension

Machine reading comprehension has made great progress in recent years owing to large-scale annotated datasets. In the clinical domain, however, creating such datasets is quite difficult due to the domain expertise required for annotation. Recently, Pampari et al. (EMNLP'18) tackled this issue by using expert-annotated question templates and existing i2b2 annotations to create emrQA, the first large-scale dataset for question answering (QA) based on clinical notes. In this paper, we provide an in-depth analysis of this dataset and the clinical reading comprehension (CliniRC) task. From our qualitative analysis, we find that (i) emrQA answers are often incomplete, and (ii) emrQA questions are often answerable without using domain knowledge. From our quantitative experiments, surprising results include that (iii) using a small sampled subset (5%-20%), we can obtain roughly equal performance compared to the model trained on the entire dataset, (iv) this performance is close to human expert's performance, and (v) BERT models do not beat the best performing base model. Following our analysis of the emrQA, we further explore two desired aspects of CliniRC systems: the ability to utilize clinical domain knowledge and to generalize to unseen questions and contexts. We argue that both should be considered when creating future datasets.
Exploring Pre-training with Alignments for RNN Transducer based End-to-End Speech Recognition
Authors: Hu Hu, Rui Zhao, Jinyu Li, Liang Lu, Yifan Gong
Date: 2020-05-01 00:00:00
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), Language Modeling, Language Modelling, speech-recognition, Speech Recognition

Recently, the recurrent neural network transducer (RNN-T) architecture has become an emerging trend in end-to-end automatic speech recognition research due to its advantages of being capable for online streaming speech recognition. However, RNN-T training is made difficult by the huge memory requirements, and complicated neural structure. A common solution to ease the RNN-T training is to employ connectionist temporal classification (CTC) model along with RNN language model (RNNLM) to initialize the RNN-T parameters. In this work, we conversely leverage external alignments to seed the RNN-T model. Two different pre-training solutions are explored, referred to as encoder pre-training, and whole-network pre-training respectively. Evaluated on Microsoft 65,000 hours anonymized production data with personally identifiable information removed, our proposed methods can obtain significant improvement. In particular, the encoder pre-training solution achieved a 10% and a 8% relative word error rate reduction when compared with random initialization and the widely used CTC+RNNLM initialization strategy, respectively. Our solutions also significantly reduce the RNN-T model latency from the baseline.
Learning Collaborative Agents with Rule Guidance for Knowledge Graph Reasoning
Authors: Deren Lei, Gangrong Jiang, Xiaotao Gu, Kexuan Sun, Yuning Mao, Xiang Ren
Date: 2020-05-01 00:00:00
Proceeding: EMNLP 2020 11
Tasks: reinforcement-learning, Reinforcement Learning (RL)

Walk-based models have shown their advantages in knowledge graph (KG) reasoning by achieving decent performance while providing interpretable decisions. However, the sparse reward signals offered by the KG during traversal are often insufficient to guide a sophisticated walk-based reinforcement learning (RL) model. An alternate approach is to use traditional symbolic methods (e.g., rule induction), which achieve good performance but can be hard to generalize due to the limitation of symbolic representation. In this paper, we propose RuleGuider, which leverages high-quality rules generated by symbolic-based methods to provide reward supervision for walk-based agents. Experiments on benchmark datasets show that RuleGuider improves the performance of walk-based models without losing interpretability.
When Ensembling Smaller Models is More Efficient than Single Large Models
Authors: Dan Kondratyuk, Mingxing Tan, Matthew Brown, Boqing Gong
Date: 2020-05-01 00:00:00
Tasks: Diversity

Ensembling is a simple and popular technique for boosting evaluation performance by training multiple models (e.g., with different initializations) and aggregating their predictions. This approach is commonly reserved for the largest models, as it is commonly held that increasing the model size provides a more substantial reduction in error than ensembling smaller models. However, we show results from experiments on CIFAR-10 and ImageNet that ensembles can outperform single models with both higher accuracy and requiring fewer total FLOPs to compute, even when those individual models' weights and hyperparameters are highly optimized. Furthermore, this gap in improvement widens as models become large. This presents an interesting observation that output diversity in ensembling can often be more efficient than training larger models, especially when the models approach the size of what their dataset can foster. Instead of using the common practice of tuning a single large model, one can use ensembles as a more flexible trade-off between a model's inference speed and accuracy. This also potentially eases hardware design, e.g., an easier way to parallelize the model across multiple workers for real-time or distributed inference.
Adversarial domain adaptation to reduce sample bias of a high energy physics classifier
Authors: Jose M. Clavijo, Paul Glaysher, Judith M. Katzy, Jenia Jitsev
Date: 2020-05-01 00:00:00
Tasks: BIG-bench Machine Learning, Classification, Domain Adaptation, domain classification, General Classification

We apply adversarial domain adaptation in unsupervised setting to reduce sample bias in a supervised high energy physics events classifier training. We make use of a neural network containing event and domain classifier with a gradient reversal layer to simultaneously enable signal versus background events classification on the one hand, while on the other hand minimising the difference in response of the network to background samples originating from different MC models via adversarial domain classification loss. We show the successful bias removal on the example of simulated events at the LHC with $t\bar{t}H$ signal versus $t\bar{t}b\bar{b}$ background classification and discuss implications and limitations of the method
Smart Containers With Bidding Capacity: A Policy Gradient Algorithm for Semi-Cooperative Learning
Authors: Wouter van Heeswijk
Date: 2020-05-01 00:00:00

Smart modular freight containers -- as propagated in the Physical Internet paradigm -- are equipped with sensors, data storage capability and intelligence that enable them to route themselves from origin to destination without manual intervention or central governance. In this self-organizing setting, containers can autonomously place bids on transport services in a spot market setting. However, for individual containers it may be difficult to learn good bidding policies due to limited observations. By sharing information and costs between one another, smart containers can jointly learn bidding policies, even though simultaneously competing for the same transport capacity. We replicate this behavior by learning stochastic bidding policies in a semi-cooperative multi agent setting. To this end, we develop a reinforcement learning algorithm based on the policy gradient framework. Numerical experiments show that sharing solely bids and acceptance decisions leads to stable bidding policies. Additional system information only marginally improves performance; individual job properties suffice to place appropriate bids. Furthermore, we find that carriers may have incentives not to share information with the smart containers. The experiments give rise to several directions for follow-up research, in particular the interaction between smart containers and transport services in self-organizing logistics.
When BERT Plays the Lottery, All Tickets Are Winning
Authors: Sai Prasanna, Anna Rogers, Anna Rumshisky
Date: 2020-05-01 00:00:00
Proceeding: EMNLP 2020 11
Tasks: All

Large Transformer-based models were shown to be reducible to a smaller number of self-attention heads and layers. We consider this phenomenon from the perspective of the lottery ticket hypothesis, using both structured and magnitude pruning. For fine-tuned BERT, we show that (a) it is possible to find subnetworks achieving performance that is comparable with that of the full model, and (b) similarly-sized subnetworks sampled from the rest of the model perform worse. Strikingly, with structured pruning even the worst possible subnetworks remain highly trainable, indicating that most pre-trained BERT weights are potentially useful. We also study the "good" subnetworks to see if their success can be attributed to superior linguistic knowledge, but find them unstable, and not explained by meaningful self-attention patterns.
RigNet: Neural Rigging for Articulated Characters
Authors: Zhan Xu, Yang Zhou, Evangelos Kalogerakis, Chris Landreth, Karan Singh
Date: 2020-05-01 00:00:00
Tasks: Skeleton Rig Prediction

We present RigNet, an end-to-end automated method for producing animation rigs from input character models. Given an input 3D model representing an articulated character, RigNet predicts a skeleton that matches the animator expectations in joint placement and topology. It also estimates surface skin weights based on the predicted skeleton. Our method is based on a deep architecture that directly operates on the mesh representation without making assumptions on shape class and structure. The architecture is trained on a large and diverse collection of rigged models, including their mesh, skeletons and corresponding skin weights. Our evaluation is three-fold: we show better results than prior art when quantitatively compared to animator rigs; qualitatively we show that our rigs can be expressively posed and animated at multiple levels of detail; and finally, we evaluate the impact of various algorithm choices on our output rigs.
POINTER: Constrained Progressive Text Generation via Insertion-based Generative Pre-training
Authors: Yizhe Zhang, Guoyin Wang, Chunyuan Li, Zhe Gan, Chris Brockett, Bill Dolan
Date: 2020-05-01 00:00:00
Proceeding: EMNLP 2020 11
Tasks: Language Modelling, Representation Learning, Text Generation

Large-scale pre-trained language models, such as BERT and GPT-2, have achieved excellent performance in language representation learning and free-form text generation. However, these models cannot be directly employed to generate text under specified lexical constraints. To address this challenge, we present POINTER (PrOgressive INsertion-based TransformER), a simple yet novel insertion-based approach for hard-constrained text generation. The proposed method operates by progressively inserting new tokens between existing tokens in a parallel manner. This procedure is recursively applied until a sequence is completed. The resulting coarse-to-fine hierarchy makes the generation process intuitive and interpretable. We pre-train our model with the proposed progressive insertion-based objective on a 12GB Wikipedia dataset, and fine-tune it on downstream hard-constrained generation tasks. Non-autoregressive decoding yields an empirically logarithmic time complexity during inference time. Experimental results on both News and Yelp datasets demonstrate that POINTER achieves state-of-the-art performance on constrained text generation. We released the pre-trained models and the source code to facilitate future research (https://github.com/dreasysnail/POINTER).
Low-Dimensional Hyperbolic Knowledge Graph Embeddings
Authors: Ines Chami, Adva Wolf, Da-Cheng Juan, Frederic Sala, Sujith Ravi, Christopher Ré
Date: 2020-05-01 00:00:00
Proceeding: ACL 2020 6
Tasks: Knowledge Graph Embeddings

Knowledge graph (KG) embeddings learn low-dimensional representations of entities and relations to predict missing facts. KGs often exhibit hierarchical and logical patterns which must be preserved in the embedding space. For hierarchical data, hyperbolic embedding methods have shown promise for high-fidelity and parsimonious representations. However, existing hyperbolic embedding methods do not account for the rich logical patterns in KGs. In this work, we introduce a class of hyperbolic KG embedding models that simultaneously capture hierarchical and logical patterns. Our approach combines hyperbolic reflections and rotations with attention to model complex relational patterns. Experimental results on standard KG benchmarks show that our method improves over previous Euclidean- and hyperbolic-based efforts by up to 6.1% in mean reciprocal rank (MRR) in low dimensions. Furthermore, we observe that different geometric transformations capture different types of relations while attention-based transformations generalize to multiple relations. In high dimensions, our approach yields new state-of-the-art MRRs of 49.6% on WN18RR and 57.7% on YAGO3-10.
GoEmotions: A Dataset of Fine-Grained Emotions
Authors: Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, Sujith Ravi
Date: 2020-05-01 00:00:00
Proceeding: ACL 2020 6
Tasks: Emotion Classification, Transfer Learning

Understanding emotion expressed in language has a wide range of applications, from building empathetic chatbots to detecting harmful online behavior. Advancement in this area can be improved using large-scale datasets with a fine-grained typology, adaptable to multiple downstream tasks. We introduce GoEmotions, the largest manually annotated dataset of 58k English Reddit comments, labeled for 27 emotion categories or Neutral. We demonstrate the high quality of the annotations via Principal Preserved Component Analysis. We conduct transfer learning experiments with existing emotion benchmarks to show that our dataset generalizes well to other domains and different emotion taxonomies. Our BERT-based model achieves an average F1-score of .46 across our proposed taxonomy, leaving much room for improvement.
Variational Quantum Eigensolver for Frustrated Quantum Systems
Authors: Alexey Uvarov, Jacob Biamonte, Dmitry Yudin
Date: 2020-05-01 00:00:00

Hybrid quantum-classical algorithms have been proposed as a potentially viable application of quantum computers. A particular example - the variational quantum eigensolver, or VQE - is designed to determine a global minimum in an energy landscape specified by a quantum Hamiltonian, which makes it appealing for the needs of quantum chemistry. Experimental realizations have been reported in recent years and theoretical estimates of its efficiency are a subject of intense effort. Here we consider the performance of the VQE technique for a Hubbard-like model describing a one-dimensional chain of fermions with competing nearest- and next-nearest-neighbor interactions. We find that recovering the VQE solution allows one to obtain the correlation function of the ground state consistent with the exact result. We also study the barren plateau phenomenon for the Hamiltonian in question and find that the severity of this effect depends on the encoding of fermions to qubits. Our results are consistent with the current knowledge about the barren plateaus in quantum optimization.
An Adaptive Enhancement Based Hybrid CNN Model for Digital Dental X-ray Positions Classification
Authors: Yaqi Wang, Lingling Sun, Yifang Zhang, Dailin Lv, Zhixing Li, Wuteng Qi
Date: 2020-05-01 00:00:00
Tasks: Diagnostic, General Classification, Specificity

Analysis of dental radiographs is an important part of the diagnostic process in daily clinical practice. Interpretation by an expert includes teeth detection and numbering. In this project, a novel solution based on adaptive histogram equalization and convolution neural network (CNN) is proposed, which automatically performs the task for dental x-rays. In order to improve the detection accuracy, we propose three pre-processing techniques to supplement the baseline CNN based on some prior domain knowledge. Firstly, image sharpening and median filtering are used to remove impulse noise, and the edge is enhanced to some extent. Next, adaptive histogram equalization is used to overcome the problem of excessive amplification noise of HE. Finally, a multi-CNN hybrid model is proposed to classify six different locations of dental slices. The results showed that the accuracy and specificity of the test set exceeded 90\%, and the AUC reached 0.97. In addition, four dentists were invited to manually annotate the test data set (independently) and then compare it with the labels obtained by our proposed algorithm. The results show that our method can effectively identify the X-ray location of teeth.
A cascade network for Detecting COVID-19 using chest x-rays
Authors: Dailin Lv, Wuteng Qi, Yunxiang Li, Lingling Sun, Yaqi Wang
Date: 2020-05-01 00:00:00
Tasks: Diagnostic

The worldwide spread of pneumonia caused by a novel coronavirus poses an unprecedented challenge to the world's medical resources and prevention and control measures. Covid-19 attacks not only the lungs, making it difficult to breathe and life-threatening, but also the heart, kidneys, brain and other vital organs of the body, with possible sequela. At present, the detection of COVID-19 needs to be realized by the reverse transcription-polymerase Chain Reaction (RT-PCR). However, many countries are in the outbreak period of the epidemic, and the medical resources are very limited. They cannot provide sufficient numbers of gene sequence detection, and many patients may not be isolated and treated in time. Given this situation, we researched the analytical and diagnostic capabilities of deep learning on chest radiographs and proposed Cascade-SEMEnet which is cascaded with SEME-ResNet50 and SEME-DenseNet169. The two cascade networks of Cascade - SEMEnet both adopt large input sizes and SE-Structure and use MoEx and histogram equalization to enhance the data. We first used SEME-ResNet50 to screen chest X-ray and diagnosed three classes: normal, bacterial, and viral pneumonia. Then we used SEME-DenseNet169 for fine-grained classification of viral pneumonia and determined if it is caused by COVID-19. To exclude the influence of non-pathological features on the network, we preprocessed the data with U-Net during the training of SEME-DenseNet169. The results showed that our network achieved an accuracy of 85.6\% in determining the type of pneumonia infection and 97.1\% in the fine-grained classification of COVID-19. We used Grad-CAM to visualize the judgment based on the model and help doctors understand the chest radiograph while verifying the effectivene.
TRIPDECODER: Study Travel Time Attributes and Route Preferences of Metro Systems from Smart Card Data
Authors: Xiancai Tian, Baihua Zheng, Yazhe Wang, Hsiao-Ting Huang, Chih-Chieh Hung
Date: 2020-05-01 00:00:00

In this paper, we target at recovering the exact routes taken by commuters inside a metro system that arenot captured by an Automated Fare Collection (AFC) system and hence remain unknown. We strategicallypropose two inference tasks to handle the recovering, one to infer the travel time of each travel link thatcontributes to the total duration of any trip inside a metro network and the other to infer the route preferencesbased on historical trip records and the travel time of each travel link inferred in the previous inferencetask. As these two inference tasks have interrelationship, most of existing works perform these two taskssimultaneously. However, our solutionTripDecoderadopts a totally different approach. To the best of ourknowledge,TripDecoderis the first model that points out and fully utilizes the fact that there are some tripsinside a metro system with only one practical route available. It strategically decouples these two inferencetasks by only taking those trip records with only one practical route as the input for the first inference taskof travel time and feeding the inferred travel time to the second inference task as an additional input whichnot only improves the accuracy but also effectively reduces the complexity of both inference tasks. Twocase studies have been performed based on the city-scale real trip records captured by the AFC systems inSingapore and Taipei to compare the accuracy and efficiency ofTripDecoderand its competitors. As expected,TripDecoderhas achieved the best accuracy in both datasets, and it also demonstrates its superior efficiencyand scalability.
MeshfreeFlowNet: A Physics-Constrained Deep Continuous Space-Time Super-Resolution Framework
Authors: Chiyu Max Jiang, Soheil Esmaeilzadeh, Kamyar Azizzadenesheli, Karthik Kashinath, Mustafa Mustafa, Hamdi A. Tchelepi, Philip Marcus, Prabhat, Anima Anandkumar
Date: 2020-05-01 00:00:00
Tasks: Super-Resolution

We propose MeshfreeFlowNet, a novel deep learning-based super-resolution framework to generate continuous (grid-free) spatio-temporal solutions from the low-resolution inputs. While being computationally efficient, MeshfreeFlowNet accurately recovers the fine-scale quantities of interest. MeshfreeFlowNet allows for: (i) the output to be sampled at all spatio-temporal resolutions, (ii) a set of Partial Differential Equation (PDE) constraints to be imposed, and (iii) training on fixed-size inputs on arbitrarily sized spatio-temporal domains owing to its fully convolutional encoder. We empirically study the performance of MeshfreeFlowNet on the task of super-resolution of turbulent flows in the Rayleigh-Benard convection problem. Across a diverse set of evaluation metrics, we show that MeshfreeFlowNet significantly outperforms existing baselines. Furthermore, we provide a large scale implementation of MeshfreeFlowNet and show that it efficiently scales across large clusters, achieving 96.80% scaling efficiency on up to 128 GPUs and a training time of less than 4 minutes.