pwc-archive/papers-with-abstracts huggingface dataset:
Automated Concatenation of Embeddings for Structured Prediction
Authors: Xinyu Wang, Yong Jiang, Nguyen Bach, Tao Wang, Zhongqiang Huang, Fei Huang, Kewei Tu
Date: 2020-10-10 00:00:00
Proceeding: ACL 2021 5
Tasks: Aspect Extraction, Chunking, Dependency Parsing, Named Entity Recognition (NER), Neural Architecture Search, Part-Of-Speech Tagging, Prediction, Semantic Dependency Parsing, Structured Prediction

Pretrained contextualized embeddings are powerful word representations for structured prediction tasks. Recent work found that better word representations can be obtained by concatenating different types of embeddings. However, the selection of embeddings to form the best concatenated representation usually varies depending on the task and the collection of candidate embeddings, and the ever-increasing number of embedding types makes it a more difficult problem. In this paper, we propose Automated Concatenation of Embeddings (ACE) to automate the process of finding better concatenations of embeddings for structured prediction tasks, based on a formulation inspired by recent progress on neural architecture search. Specifically, a controller alternately samples a concatenation of embeddings, according to its current belief of the effectiveness of individual embedding types in consideration for a task, and updates the belief based on a reward. We follow strategies in reinforcement learning to optimize the parameters of the controller and compute the reward based on the accuracy of a task model, which is fed with the sampled concatenation as input and trained on a task dataset. Empirical results on 6 tasks and 21 datasets show that our approach outperforms strong baselines and achieves state-of-the-art performance with fine-tuned embeddings in all the evaluations.
Second-Order Neural Dependency Parsing with Message Passing and End-to-End Training
Authors: Xinyu Wang, Kewei Tu
Date: 2020-10-10 00:00:00
Proceeding: Asian Chapter of the Association for Computational Linguistics 2020
Tasks: Dependency Parsing

In this paper, we propose second-order graph-based neural dependency parsing using message passing and end-to-end neural networks. We empirically show that our approaches match the accuracy of very recent state-of-the-art second-order graph-based neural dependency parsers and have significantly faster speed in both training and testing. We also empirically show the advantage of second-order parsing over first-order parsing and observe that the usefulness of the head-selection structured constraint vanishes when using BERT embedding.
Weakly Supervised Medication Regimen Extraction from Medical Conversations
Authors: Dhruvesh Patel, Sandeep Konam, Sai P. Selvaraj
Date: 2020-10-11 00:00:00
Proceeding: EMNLP (ClinicalNLP) 2020 11
Tasks: General Classification

Automated Medication Regimen (MR) extraction from medical conversations can not only improve recall and help patients follow through with their care plan, but also reduce the documentation burden for doctors. In this paper, we focus on extracting spans for frequency, route and change, corresponding to medications discussed in the conversation. We first describe a unique dataset of annotated doctor-patient conversations and then present a weakly supervised model architecture that can perform span extraction using noisy classification data. The model utilizes an attention bottleneck inside a classification model to perform the extraction. We experiment with several variants of attention scoring and projection functions and propose a novel transformer-based attention scoring function (TAScore). The proposed combination of TAScore and Fusedmax projection achieves a 10 point increase in Longest Common Substring F1 compared to the baseline of additive scoring plus softmax projection.
Google Landmark Recognition 2020 Competition Third Place Solution
Authors: Qishen Ha, Bo Liu, Fuxu Liu, Peiyuan Liao
Date: 2020-10-11 00:00:00
Tasks: Landmark Recognition

We present our third place solution to the Google Landmark Recognition 2020 competition. It is an ensemble of global features only Sub-center ArcFace models. We introduce dynamic margins for ArcFace loss, a family of tune-able margin functions of class size, designed to deal with the extreme imbalance in GLDv2 dataset. Progressive finetuning and careful postprocessing are also key to the solution. Our two submissions scored 0.6344 and 0.6289 on private leaderboard, both ranking third place out of 736 teams.
A BERT-based Distractor Generation Scheme with Multi-tasking and Negative Answer Training Strategies
Authors: Ho-Lam Chung, Ying-Hong Chan, Yao-Chung Fan
Date: 2020-10-12 00:00:00
Tasks: Distractor Generation, Multiple-choice, Text Generation

In this paper, we investigate the following two limitations for the existing distractor generation (DG) methods. First, the quality of the existing DG methods are still far from practical use. There is still room for DG quality improvement. Second, the existing DG designs are mainly for single distractor generation. However, for practical MCQ preparation, multiple distractors are desired. Aiming at these goals, in this paper, we present a new distractor generation scheme with multi-tasking and negative answer training strategies for effectively generating \textit{multiple} distractors. The experimental results show that (1) our model advances the state-of-the-art result from 28.65 to 39.81 (BLEU 1 score) and (2) the generated multiple distractors are diverse and show strong distracting power for multiple choice question.
Efficient Long-Range Convolutions for Point Clouds
Authors: Yifan Peng, Lin Lin, Lexing Ying, Leonardo Zepeda-Núñez
Date: 2020-10-11 00:00:00

The efficient treatment of long-range interactions for point clouds is a challenging problem in many scientific machine learning applications. To extract global information, one usually needs a large window size, a large number of layers, and/or a large number of channels. This can often significantly increase the computational cost. In this work, we present a novel neural network layer that directly incorporates long-range information for a point cloud. This layer, dubbed the long-range convolutional (LRC)-layer, leverages the convolutional theorem coupled with the non-uniform Fourier transform. In a nutshell, the LRC-layer mollifies the point cloud to an adequately sized regular grid, computes its Fourier transform, multiplies the result by a set of trainable Fourier multipliers, computes the inverse Fourier transform, and finally interpolates the result back to the point cloud. The resulting global all-to-all convolution operation can be performed in nearly-linear time asymptotically with respect to the number of input points. The LRC-layer is a particularly powerful tool when combined with local convolution as together they offer efficient and seamless treatment of both short and long range interactions. We showcase this framework by introducing a neural network architecture that combines LRC-layers with short-range convolutional layers to accurately learn the energy and force associated with a $N$-body potential. We also exploit the induced two-level decomposition and propose an efficient strategy to train the combined architecture with a reduced number of samples.
IF-Defense: 3D Adversarial Point Cloud Defense via Implicit Function based Restoration
Authors: Ziyi Wu, Yueqi Duan, He Wang, Qingnan Fan, Leonidas J. Guibas
Date: 2020-10-11 00:00:00

Point cloud is an important 3D data representation widely used in many essential applications. Leveraging deep neural networks, recent works have shown great success in processing 3D point clouds. However, those deep neural networks are vulnerable to various 3D adversarial attacks, which can be summarized as two primary types: point perturbation that affects local point distribution, and surface distortion that causes dramatic changes in geometry. In this paper, we simultaneously address both the aforementioned attacks by learning to restore the clean point clouds from the attacked ones. More specifically, we propose an IF-Defense framework to directly optimize the coordinates of input points with geometry-aware and distribution-aware constraints. The former aims to recover the surface of point cloud through implicit function, while the latter encourages evenly-distributed points. Our experimental results show that IF-Defense achieves the state-of-the-art defense performance against existing 3D adversarial attacks on PointNet, PointNet++, DGCNN, PointConv and RS-CNN. For example, compared with previous methods, IF-Defense presents 20.02% improvement in classification accuracy against salient point dropping attack and 16.29% against LG-GAN attack on PointNet. Our code is available at https://github.com/Wuziyi616/IF-Defense.
Locality Preserving Dense Graph Convolutional Networks with Graph Context-Aware Node Representations
Authors: Wenfeng Liu, Maoguo Gong, Zedong Tang, A. K. Qin
Date: 2020-10-12 00:00:00
Tasks: Classification, Decoder, General Classification, Graph Classification, Representation Learning

Graph convolutional networks (GCNs) have been widely used for representation learning on graph data, which can capture structural patterns on a graph via specifically designed convolution and readout operations. In many graph classification applications, GCN-based approaches have outperformed traditional methods. However, most of the existing GCNs are inefficient to preserve local information of graphs -- a limitation that is especially problematic for graph classification. In this work, we propose a locality-preserving dense GCN with graph context-aware node representations. Specifically, our proposed model incorporates a local node feature reconstruction module to preserve initial node features into node representations, which is realized via a simple but effective encoder-decoder mechanism. To capture local structural patterns in neighbourhoods representing different ranges of locality, dense connectivity is introduced to connect each convolutional layer and its corresponding readout with all previous convolutional layers. To enhance node representativeness, the output of each convolutional layer is concatenated with the output of the previous layer's readout to form a global context-aware node representation. In addition, a self-attention module is introduced to aggregate layer-wise representations to form the final representation. Experiments on benchmark datasets demonstrate the superiority of the proposed model over state-of-the-art methods in terms of classification accuracy.
Regularizing Neural Networks via Adversarial Model Perturbation
Authors: Yaowei Zheng, Richong Zhang, Yongyi Mao
Date: 2020-10-10 00:00:00
Proceeding: CVPR 2021 1
Tasks: Image Classification, model

Effective regularization techniques are highly desired in deep learning for alleviating overfitting and improving generalization. This work proposes a new regularization scheme, based on the understanding that the flat local minima of the empirical risk cause the model to generalize better. This scheme is referred to as adversarial model perturbation (AMP), where instead of directly minimizing the empirical risk, an alternative "AMP loss" is minimized via SGD. Specifically, the AMP loss is obtained from the empirical risk by applying the "worst" norm-bounded perturbation on each point in the parameter space. Comparing with most existing regularization schemes, AMP has strong theoretical justifications, in that minimizing the AMP loss can be shown theoretically to favour flat local minima of the empirical risk. Extensive experiments on various modern deep architectures establish AMP as a new state of the art among regularization schemes. Our code is available at https://github.com/hiyouga/AMP-Regularizer.
Partial Multi-label Learning with Noisy Label Identification
Authors: Ming-Kun Xie and Sheng-Jun Huang
Date: 2020-04-03 00:00:00
Tasks: Multi-Label Learning

Partial multi-label learning (PML) deals with problems where each instance is assigned with a candidate label set, which contains multiple relevant labels and some noisy labels. Recent studies usually solve PML problems with the disambiguation strategy, which recovers ground-truth labels from the candidate label set by simply assuming that the noisy labels are generated randomly. In real applications, however,noisy labels are usually caused by some ambiguous contents of the example. Based on this observation, we propose a partial multi-label learning approach to simultaneously recover the ground-truth information and identify the noisy labels. The two objectives are formalized in a unified framework with trace norm and ‘1norm regularizers. Under the supervision of the observed noise-corrupted label matrix, the multi-label classifier and noisy label identifier are jointly optimized by incorporating the label correlation exploitation and feature-induced noise model. Extensive experiments on synthetic as well as real-world data sets validate the effectiveness of the proposed approach.
A Simplified 2D-3D CNN Architecture for Hyperspectral Image Classification Based on Spatial–Spectral Fusion
Authors: Chunyan Yu, Rui Han, Meiping Song, Caiyu Liu, and Chein-I Chang, Life Fellow, IEEE
Date: 2020-06-05 00:00:00
Tasks: General Classification, Hyperspectral Image Classification, image-classification, Image Classification

Abstract—Convolutional neural networks (CNN) have led to
a successful breakthrough for hyperspectral image classification
(HSIC). Due to the intrinsic spatial-spectral specificities of a hyperspectral cube, feature extraction with 3-D convolution operation
is a straightforward way for HSIC. However, the overwhelming
features obtained from the original 3-D CNN network suffers from
the overfitting and more training cost problem. To address this
issue, in this article, a novel HSIC framework based on a simplified
2D-3D CNN is implemented by the cooperation between a 2-D CNN
and a 3-D convolution layer. First, the 2-D convolution block aims to
extract the spatial features abundantly involved spectral information as a training channel. Then, the 3-D CNN approach primarily
concentrates on exploiting band co-relation data by using a reduced
kernel. The proposed architecture achieves the spatial and spectral
features simultaneously based on a joint 2D-3D pattern to achieve
superior fused feature for the subsequent classification. Furthermore, a deconvolution layer intends to enhance the robustness of the
deep features is utilized in the proposed CNN network. The results
and analysis of extensive real HSIC experiments demonstrate that
the proposed light-weighted 2D-3D CNN network can effectively
extract refined features and improve the classification accuracy
TSPNet: Hierarchical Feature Learning via Temporal Semantic Pyramid for Sign Language Translation
Authors: Dongxu Li, Chenchen Xu, Xin Yu, Kaihao Zhang, Ben Swift, Hanna Suominen, Hongdong Li
Date: 2020-10-12 00:00:00
Proceeding: NeurIPS 2020 12
Tasks: Sign Language Recognition, Sign Language Translation, Translation, Video Segmentation, Video Semantic Segmentation

Sign language translation (SLT) aims to interpret sign video sequences into text-based natural language sentences. Sign videos consist of continuous sequences of sign gestures with no clear boundaries in between. Existing SLT models usually represent sign visual features in a frame-wise manner so as to avoid needing to explicitly segmenting the videos into isolated signs. However, these methods neglect the temporal information of signs and lead to substantial ambiguity in translation. In this paper, we explore the temporal semantic structures of signvideos to learn more discriminative features. To this end, we first present a novel sign video segment representation which takes into account multiple temporal granularities, thus alleviating the need for accurate video segmentation. Taking advantage of the proposed segment representation, we develop a novel hierarchical sign video feature learning method via a temporal semantic pyramid network, called TSPNet. Specifically, TSPNet introduces an inter-scale attention to evaluate and enhance local semantic consistency of sign segments and an intra-scale attention to resolve semantic ambiguity by using non-local video context. Experiments show that our TSPNet outperforms the state-of-the-art with significant improvements on the BLEU score (from 9.58 to 13.41) and ROUGE score (from 31.80 to 34.96)on the largest commonly-used SLT dataset. Our implementation is available at https://github.com/verashira/TSPNet.
SMYRF: Efficient Attention using Asymmetric Clustering
Authors: Giannis Daras, Nikita Kitaev, Augustus Odena, Alexandros G. Dimakis
Date: 2020-10-11 00:00:00
Tasks: 16k, Clustering

We propose a novel type of balanced clustering algorithm to approximate attention. Attention complexity is reduced from $O(N^2)$ to $O(N \log N)$, where $N$ is the sequence length. Our algorithm, SMYRF, uses Locality Sensitive Hashing (LSH) in a novel way by defining new Asymmetric transformations and an adaptive scheme that produces balanced clusters. The biggest advantage of SMYRF is that it can be used as a drop-in replacement for dense attention layers without any retraining. On the contrary, prior fast attention methods impose constraints (e.g. queries and keys share the same vector representations) and require re-training from scratch. We apply our method to pre-trained state-of-the-art Natural Language Processing and Computer Vision models and we report significant memory and speed benefits. Notably, SMYRF-BERT outperforms (slightly) BERT on GLUE, while using $50\%$ less memory. We also show that SMYRF can be used interchangeably with dense attention before and after training. Finally, we use SMYRF to train GANs with attention in high resolutions. Using a single TPU, we were able to scale attention to 128x128=16k and 256x256=65k tokens on BigGAN on CelebA-HQ.
Data Agnostic RoBERTa-based Natural Language to SQL Query Generation
Authors: Debaditya Pal, Harsh Sharma, Kaustubh Chaudhari
Date: 2020-10-11 00:00:00
Tasks: valid, Zero-Shot Learning

Relational databases are among the most widely used architectures to store massive amounts of data in the modern world. However, there is a barrier between these databases and the average user. The user often lacks the knowledge of a query language such as SQL required to interact with the database. The NL2SQL task aims at finding deep learning approaches to solve this problem by converting natural language questions into valid SQL queries. Given the sensitive nature of some databases and the growing need for data privacy, we have presented an approach with data privacy at its core. We have passed RoBERTa embeddings and data-agnostic knowledge vectors into LSTM based submodels to predict the final query. Although we have not achieved state of the art results, we have eliminated the need for the table data, right from the training of the model, and have achieved a test set execution accuracy of 76.7%. By eliminating the table data dependency while training we have created a model capable of zero shot learning based on the natural language question and table schema alone.
From Hero to Zéroe: A Benchmark of Low-Level Adversarial Attacks
Authors: Steffen Eger, Yannik Benz
Date: 2020-10-12 00:00:00
Tasks: Natural Language Inference, Part-Of-Speech Tagging, Toxic Comment Classification

Adversarial attacks are label-preserving modifications to inputs of machine learning classifiers designed to fool machines but not humans. Natural Language Processing (NLP) has mostly focused on high-level attack scenarios such as paraphrasing input texts. We argue that these are less realistic in typical application scenarios such as in social media, and instead focus on low-level attacks on the character-level. Guided by human cognitive abilities and human robustness, we propose the first large-scale catalogue and benchmark of low-level adversarial attacks, which we dub Z\'eroe, encompassing nine different attack modes including visual and phonetic adversaries. We show that RoBERTa, NLP's current workhorse, fails on our attacks. Our dataset provides a benchmark for testing robustness of future more human-like NLP models.
MH-COVIDNet: Diagnosis of COVID-19 using Deep Neural Networks and Meta-heuristic-based Feature Selection on X-ray Images
Authors: Murat CANAYAZ
Date: 2020-10-06 00:00:00
Tasks: COVID-19 Diagnosis, Deep Learning, Diagnostic, feature selection, Medical Diagnosis

COVID-19 is a disease that causes symptoms in the lungs and causes deaths around the world. Studies are ongoing for the diagnosis and treatment of this disease, which is defined as a pandemic. Early diagnosis of this disease is important for human life. This process is progressing rapidly with diagnostic studies based on deep learning. Therefore, to contribute to this field, a deep learning-based approach that can be used for early diagnosis of the disease is proposed in our study. In this approach, a data set consisting of 3 classes of COVID19, normal and pneumonia lung X-ray images was created, with each class containing 364 images. Pre-processing was performed using the image contrast enhancement algorithm on the prepared data set and a new data set was obtained. Feature extraction was completed from this data set with deep learning models such as AlexNet, VGG19, GoogleNet, and ResNet. For the selection of the best potential features, two metaheuristic algorithms of binary particle swarm optimization and binary gray wolf optimization were used. After combining the features obtained in the feature selection of the enhancement data set, they were classified using SVM. The overall accuracy of the proposed approach was obtained as 99.38%. The results obtained by verification with two different metaheuristic algorithms proved that the approach we propose can help experts during COVID-19 diagnostic studies.
Load What You Need: Smaller Versions of Multilingual BERT
Authors: Amine Abdaoui, Camille Pradel, Grégoire Sigel
Date: 2020-10-12 00:00:00

Pre-trained Transformer-based models are achieving state-of-the-art results on a variety of Natural Language Processing data sets. However, the size of these models is often a drawback for their deployment in real production applications. In the case of multilingual models, most of the parameters are located in the embeddings layer. Therefore, reducing the vocabulary size should have an important impact on the total number of parameters. In this paper, we propose to generate smaller models that handle fewer number of languages according to the targeted corpora. We present an evaluation of smaller versions of multilingual BERT on the XNLI data set, but we believe that this method may be applied to other multilingual transformers. The obtained results confirm that we can generate smaller models that keep comparable results, while reducing up to 45% of the total number of parameters. We compared our models with DistilmBERT (a distilled version of multilingual BERT) and showed that unlike language reduction, distillation induced a 1.7% to 6% drop in the overall accuracy on the XNLI data set. The presented models and code are publicly available.
On Feature Selection Using Anisotropic General Regression Neural Network
Authors: Federico Amato, Fabian Guignard, Philippe Jacquet, Mikhail Kanevski
Date: 2020-10-12 00:00:00
Tasks: BIG-bench Machine Learning, feature selection, regression, Sensitivity

The presence of irrelevant features in the input dataset tends to reduce the interpretability and predictive quality of machine learning models. Therefore, the development of feature selection methods to recognize irrelevant features is a crucial topic in machine learning. Here we show how the General Regression Neural Network used with an anisotropic Gaussian Kernel can be used to perform feature selection. A number of numerical experiments are conducted using simulated data to study the robustness of the proposed methodology and its sensitivity to sample size. Finally, a comparison with four other feature selection methods is performed on several real world datasets.