pwc-archive/papers-with-abstracts huggingface dataset:
Latin script keyboards for South Asian languages with finite-state normalization
Authors: Lawrence Wolf-Sonkin, Vlad Schogol, Brian Roark, Michael Riley
Date: 2019-09-01 00:00:00
Proceeding: WS 2019 9
Tasks: Transliteration

The use of the Latin script for text entry of South Asian languages is common, even though there is no standard orthography for these languages in the script. We explore several compact finite-state architectures that permit variable spellings of words during mobile text entry. We find that approaches making use of transliteration transducers provide large accuracy improvements over baselines, but that simpler approaches involving a compact representation of many attested alternatives yields much of the accuracy gain. This is particularly important when operating under constraints on model size (e.g., on inexpensive mobile devices with limited storage and memory for keyboard models), and on speed of inference, since people typing on mobile keyboards expect no perceptual delay in keyboard responsiveness.
Transition-Based Coding and Formal Language Theory for Ordered Digraphs
Authors: Anssi Yli-Jyr{\"a}
Date: 2019-09-01 00:00:00
Proceeding: WS 2019 9

Transition-based parsing of natural language uses transition systems to build directed annotation graphs (digraphs) for sentences. In this paper, we define, for an arbitrary ordered digraph, a unique decomposition and a corresponding linear encoding that are associated bijectively with each other via a new transition system. These results give us an efficient and succinct representation for digraphs and sets of digraphs. Based on the system and our analysis of its syntactic properties, we give structural bounds under which the set of encoded digraphs is restricted and becomes a context-free or a regular string language. The context-free restriction is essentially a superset of the encodings used previously to characterize properties of noncrossing digraphs and to solve maximal subgraphs problems. The regular restriction with a tight bound is shown to capture the Universal Dependencies v2.4 treebanks in linguistics.
Improving Semantic Dependency Parsing with Syntactic Features
Authors: Robin Kurtz, Daniel Roxbo, Marco Kuhlmann
Date: 2019-09-01 00:00:00
Proceeding: WS 2019 9
Tasks: Dependency Parsing, Semantic Dependency Parsing, Semantic Parsing

We extend a state-of-the-art deep neural architecture for semantic dependency parsing with features defined over syntactic dependency trees. Our empirical results show that only gold-standard syntactic information leads to consistent improvements in semantic parsing accuracy, and that the magnitude of these improvements varies with the specific combination of the syntactic and the semantic representation used. In contrast, automatically predicted syntax does not seem to help semantic parsing. Our error analysis suggests that there is a significant overlap between syntactic and semantic representations.
Multilingual Probing of Deep Pre-Trained Contextual Encoders
Authors: Vinit Ravishankar, Memduh G{\"o}k{\i}rmak, Lilja {\O}vrelid, Erik Velldal
Date: 2019-09-01 00:00:00
Proceeding: WS 2019 9
Tasks: Diagnostic, Sentence

Encoders that generate representations based on context have, in recent years, benefited from adaptations that allow for pre-training on large text corpora. Earlier work on evaluating fixed-length sentence representations has included the use of {`}probing{'} tasks, that use diagnostic classifiers to attempt to quantify the extent to which these encoders capture specific linguistic phenomena. The principle of probing has also resulted in extended evaluations that include relatively newer word-level pre-trained encoders. We build on probing tasks established in the literature and comprehensively evaluate and analyse {--} from a typological perspective amongst others {--} multilingual variants of existing encoders on probing datasets constructed for 6 non-English languages. Specifically, we probe each layer of a multiple monolingual RNN-based ELMo models, the transformer-based BERT{'}s cased and uncased multilingual variants, and a variant of BERT that uses a cross-lingual modelling scheme (XLM).
Cross-Domain Sentiment Classification using Vector Embedded Domain Representations
Authors: Nicolaj Filrup Rasmussen, Kristian N{\o}rgaard Jensen, Marco Placenti, Thai Wang
Date: 2019-09-01 00:00:00
Proceeding: WS 2019 9
Tasks: Classification, General Classification, Sentiment Analysis, Sentiment Classification

Due to the differences between reviews in different product categories, creating a general model for cross-domain sentiment classification can be a difficult task. This paper proposes an architecture that incorporates domain knowledge into a neural sentiment classification model. In addition to providing a cross-domain model, this also provides a quantifiable representation of the domains as numeric vectors. We show that it is possible to cluster the domain vectors and provide qualitative insights into the inter-domain relations. We also a) present a new data set for sentiment classification that includes a domain parameter and preprocessed data points, and b) perform an ablation study in order to determine whether some word groups impact performance.
Multiclass Text Classification on Unbalanced, Sparse and Noisy Data
Authors: Matthias Damaschk, Tillmann D{\"o}nicke, Florian Lux
Date: 2019-09-01 00:00:00
Proceeding: WS 2019 9
Tasks: Classification, General Classification, text-classification, Text Classification

This paper discusses methods to improve the performance of text classification on data that is difficult to classify due to a large number of unbalanced classes with noisy examples. A variety of features are tested, in combination with three different neural-network-based methods with increasing complexity. The classifiers are applied to a songtext{--}artist dataset which is large, unbalanced and noisy. We come to the conclusion that substantial improvement can be obtained by removing unbalancedness and sparsity from the data. This fulfils a classification task unsatisfactorily{---}however, with contemporary methods, it is a practical step towards fairly satisfactory results.
Fooling Detection Alone is Not Enough: Adversarial Attack against Multiple Object Tracking
Authors: Yunhan Jia, Yantao Lu, Junjie Shen, Qi Alfred Chen, Hao Chen, Zhenyu Zhong, Tao Wei
Date: 2020-01-01 00:00:00
Proceeding: ICLR 2020 1
Tasks: Adversarial Attack, Autonomous Driving, BIG-bench Machine Learning, Multiple Object Tracking, Object, object-detection, Object Detection, Object Tracking

Recent work in adversarial machine learning started to focus on the visual perception in autonomous driving and studied Adversarial Examples (AEs) for object detection models. However, in such visual perception pipeline the detected objects must also be tracked, in a process called Multiple Object Tracking (MOT), to build the moving trajectories of surrounding obstacles. Since MOT is designed to be robust against errors in object detection, it poses a general challenge to existing attack techniques that blindly target objection detection: we find that a success rate of over 98% is needed for them to actually affect the tracking results, a requirement that no existing attack technique can satisfy. In this paper, we are the first to study adversarial machine learning attacks against the complete visual perception pipeline in autonomous driving, and discover a novel attack technique, tracker hijacking, that can effectively fool MOT using AEs on object detection. Using our technique, successful AEs on as few as one single frame can move an existing object in to or out of the headway of an autonomous vehicle to cause potential safety hazards. We perform evaluation using the Berkeley Deep Drive dataset and find that on average when 3 frames are attacked, our attack can have a nearly 100% success rate while attacks that blindly target object detection only have up to 25%.
Differentiable Architecture Compression
Authors: Shashank Singh, Ashish Khetan, Zohar Karnin
Date: 2020-01-01 00:00:00
Proceeding: ICLR 2020 1
Tasks: image-classification, Image Classification, Model Compression

In many learning situations, resources at inference time are significantly more constrained than resources at training time. This paper studies a general paradigm, called Differentiable ARchitecture Compression (DARC), that combines model compression and architecture search to learn models that are resource-efficient at inference time. Given a resource-intensive base architecture, DARC utilizes the training data to learn which sub-components can be replaced by cheaper alternatives. The high-level technique can be applied to any neural architecture, and we report experiments on state-of-the-art convolutional neural networks for image classification. For a WideResNet with 97.2% accuracy on CIFAR-10, we improve single-sample inference speed by 2.28X and memory footprint by 5.64X, with no accuracy loss. For a ResNet with 79.15% Top-1 accuracy on ImageNet, we improve batch inference speed by 1.29X and memory footprint by 3.57X with 1% accuracy loss. We also give theoretical Rademacher complexity bounds in simplified cases, showing how DARC avoids over-fitting despite over-parameterization.
FoveaBox: Beyound Anchor-based Object Detection
Authors: Tao Kong, Fuchun Sun, Huaping Liu, Yuning Jiang, Lei LI, Jianbo Shi
Date: 2020-01-01 00:00:00
Proceeding: ICLR 2020 1
Tasks: Object, object-detection, Object Detection

We present FoveaBox, an accurate, flexible, and completely anchor-free framework for object detection. While almost all state-of-the-art object detectors utilize predefined anchors to enumerate possible locations, scales and aspect ratios for the search of the objects, their performance and generalization ability are also limited to the design of anchors. Instead, FoveaBox directly learns the object existing possibility and the bounding box coordinates without anchor reference. This is achieved by: (a) predicting category-sensitive semantic maps for the object existing possibility, and (b) producing category-agnostic bounding box for each position that potentially contains an object. The scales of target boxes are naturally associated with feature pyramid representations. We demonstrate its effectiveness on standard benchmarks and report extensive experimental analysis.  Without bells and whistles, FoveaBox achieves state-of-the-art single model performance on the standard COCO detection benchmark. More importantly, FoveaBox avoids all computation and hyper-parameters related to anchor boxes, which are often sensitive to the final detection performance.  We believe the simple and effective approach will serve as a solid baseline and help ease future research for object detection.
New Loss Functions for Fast Maximum Inner Product Search
Authors: Ruiqi Guo, Quan Geng, David Simcha, Felix Chern, Phil Sun, Sanjiv Kumar
Date: 2020-01-01 00:00:00
Proceeding: ICLR 2020 1
Tasks: Benchmarking, Quantization

Quantization based methods are popular for solving large scale maximum inner product search problems. However, in most traditional quantization works, the objective is to minimize the reconstruction error for datapoints to be searched. In this work, we focus directly on minimizing error in inner product approximation and derive a new class of quantization loss functions. One key aspect of the new loss functions is that we weight the error term based on the value of the inner product, giving more importance to pairs of queries and datapoints whose inner products are high. We provide theoretical grounding to the new quantization loss function, which is simple, intuitive and able to work with a variety of quantization techniques, including binary quantization and product quantization. We conduct experiments on public benchmarking datasets \url{http://ann-benchmarks.com} to demonstrate that our method using the new objective outperforms other state-of-the-art methods. We are committed to release our source code.
Implicit λ-Jeffreys Autoencoders: Taking the Best of Both Worlds
Authors: Aibek Alanov, Max Kochurov, Artem Sobolev, Daniil Yashkov, Dmitry Vetrov
Date: 2020-01-01 00:00:00
Proceeding: ICLR 2020 1

We propose a new form of an autoencoding model which incorporates the best properties of variational autoencoders (VAE) and generative adversarial networks (GAN). It is known that GAN can produce very realistic samples while VAE does not suffer from mode collapsing problem. Our model optimizes λ-Jeffreys divergence between the model distribution and the true data distribution. We show that it takes the best properties of VAE and GAN objectives. It consists of two parts. One of these parts can be optimized by using the standard adversarial training, and the second one is the very objective of the VAE model. However, the straightforward way of substituting the VAE loss does not work well if we use an explicit likelihood such as Gaussian or Laplace which have limited flexibility in high dimensions and are unnatural for modelling images in the space of pixels. To tackle this problem we propose a novel approach to train the VAE model with an implicit likelihood by an adversarially trained discriminator. In an extensive set of experiments on CIFAR-10 and TinyImagent datasets, we show that our model achieves the state-of-the-art generation and reconstruction quality and demonstrate how we can balance between mode-seeking and mode-covering behaviour of our model by adjusting the weight λ in our objective.
LOSSLESS SINGLE IMAGE SUPER RESOLUTION FROM LOW-QUALITY JPG IMAGES
Authors: Yong Shi, Biao Li, Bo wang, Zhiquan Qi, Jiabin Liu, Fan Meng
Date: 2020-01-01 00:00:00
Proceeding: ICLR 2020 1
Tasks: Image Super-Resolution, Super-Resolution

Super Resolution (SR) is a fundamental and important low-level computer vision (CV) task. Different from traditional SR models, this study concentrates on a specific but realistic SR issue: How can we obtain satisfied SR results from compressed JPG (C-JPG) image, which widely exists on the Internet. In general, C-JPG can release storage space while keeping considerable quality in visual. However, further image processing operations, e.g., SR, will suffer from enlarging inner artificial details and result in unacceptable outputs. To address this problem, we propose a novel SR structure with two specifically designed components, as well as a cycle loss. In short, there are mainly three contributions to this paper. First, our research can generate high-qualified SR images for prevalent C-JPG images. Second, we propose a functional sub-model to recover information for C-JPG images, instead of the perspective of noise elimination in traditional SR approaches. Third, we further integrate cycle loss into SR solver to build a hybrid loss function for better SR generation. Experiments show that our approach achieves outstanding performance among state-of-the-art methods.
Improving Batch Normalization with Skewness Reduction for Deep Neural Networks
Authors: Pak Lun Kevin Ding, Sarah Martin, Baoxin Li
Date: 2020-01-01 00:00:00
Proceeding: ICLR 2020 1

Batch Normalization (BN) is a well-known technique used in training deep neural networks.
    The main idea behind batch normalization is to normalize the features of the layers ($i.e.$, transforming them to have a mean equal to zero and a variance equal to one).
    Such a procedure encourages the optimization landscape of the loss function to be smoother, and improve the learning of the networks for both speed and performance.
    In this paper,
    we demonstrate that the performance of the network can be improved,
    if the distributions of the features of the output in the same layer are similar.
    As normalizing based on mean and variance does not necessarily make the features to have the same distribution, we propose a new normalization scheme: Batch Normalization with Skewness Reduction (BNSR).
    Comparing with other normalization approaches,
    BNSR transforms not just only the mean and variance,
    but also the skewness of the data.
    By tackling this property of a distribution, we are able to make the output distributions of the layers to be further similar. The nonlinearity of BNSR may further improve the expressiveness of the underlying network.
    Comparisons with other normalization schemes are tested on the CIFAR-100 and ImageNet datasets. Experimental results show that the proposed approach can outperform other state-of-the-arts that are not equipped with BNSR.
Cascade Style Transfer
Authors: Zhizhong Wang, Lei Zhao, Qihang Mo, Sihuan Lin, Zhiwen Zuo, Wei Xing, Dongming Lu
Date: 2020-01-01 00:00:00
Proceeding: ICLR 2020 1
Tasks: Serial Style Transfer, Style Transfer

Recent studies have made tremendous progress in style transfer for specific domains, e.g., artistic, semantic and photo-realistic. However, existing approaches have limited flexibility in extending to other domains, as different style representations are often specific to particular domains. This also limits the stylistic quality. To address these limitations, we propose Cascade Style Transfer, a simple yet effective framework that can improve the quality and flexibility of style transfer by combining multiple existing approaches directly. Our cascade framework contains two architectures, i.e., Serial Style Transfer (SST) and Parallel Style Transfer (PST). The SST takes the stylized output of one method as the input content of the others. This could help improve the stylistic quality. The PST uses a shared backbone and a loss module to optimize the loss functions of different methods in parallel. This could help improve the quality and flexibility, and guide us to find domain-independent approaches. Our experiments are conducted on three major style transfer domains: artistic, semantic and photo-realistic. In all these domains, our methods have shown superiority over the state-of-the-art methods.
Improved Modeling of Complex Systems Using Hybrid Physics/Machine Learning/Stochastic Models
Authors: Anand Ramakrishnan, Warren B. Jackson, Kent Evans
Date: 2020-01-01 00:00:00
Proceeding: ICLR 2020 1
Tasks: BIG-bench Machine Learning, Time Series, Time Series Analysis

Combining domain knowledge models with neural models has been challenging.  End-to-end trained neural models often perform better (lower Mean Square Error) than domain knowledge models or domain/neural combinations, and the combination is inefficient to train.  In this paper, we demonstrate that by composing domain models with machine learning models, by using extrapolative testing sets, and invoking decorrelation objective functions, we create models which can predict more complex systems. The models are interpretable, extrapolative, data-efficient, and capture predictable but complex non-stochastic behavior such as unmodeled degrees of freedom and systemic measurement noise.  We apply this improved modeling paradigm to several simulated systems and an actual physical system in the context of system identification.   Several ways of composing domain models with neural models are examined for time series, boosting, bagging, and auto-encoding on various systems of varying complexity and non-linearity.  Although this work is preliminary, we show that the ability to combine models is a very promising direction for neural modeling.
Insights on Visual Representations for Embodied Navigation Tasks
Authors: Erik Wijmans, Julian Straub, Irfan Essa, Dhruv Batra, Judy Hoffman, Ari Morcos
Date: 2020-01-01 00:00:00
Proceeding: ICLR 2020 1
Tasks: Deep Reinforcement Learning, Reinforcement Learning

Recent advances in deep reinforcement learning require a large amount of training data and generally result in representations that are often over specialized to the target task. In this work, we study the underlying potential causes for this specialization by measuring the similarity between representations trained on related, but distinct tasks. We use the recently proposed projection weighted Canonical Correlation Analysis (PWCCA) to examine the task dependence of visual representations learned across different embodied navigation tasks.  Surprisingly, we find that slight differences in task have no measurable effect on the visual representation for both SqueezeNet and ResNet architectures.  We then empirically demonstrate that visual representations learned on one task can be effectively transferred to a different task.  Interestingly, we show that if the tasks constrain the agent to spatially disjoint parts of the environment, differences in representation emerge for SqueezeNet models but less-so for ResNets, suggesting that ResNets feature inductive biases which encourage more task-agnostic representations, even in the context of spatially separated tasks.  We generalize our analysis to examine permutations of an environment and find, surprisingly, permutations of an environment also do not influence the visual representation. Our analysis provides insight on the overfitting of representations in RL and provides suggestions of how to design tasks that induce task-agnostic representations.
A Training Scheme for the Uncertain Neuromorphic Computing Chips
Authors: Qingtian Zhang, Bin Gao, Huaqiang Wu
Date: 2020-01-01 00:00:00
Proceeding: ICLR 2020 1

Uncertainty is a very important feature of the intelligence and helps the brain become a flexible, creative and powerful intelligent system. The crossbar-based neuromorphic computing chips, in which the computing is mainly performed by analog circuits, have the uncertainty and can be used to imitate the brain. However, most of the current deep neural networks have not taken the uncertainty of the neuromorphic computing chip into consideration. Therefore, their performances on the neuromorphic computing chips are not as good as on the original platforms (CPUs/GPUs). In this work, we proposed the uncertainty adaptation training scheme (UATS) that tells the uncertainty to the neural network in the training process. The experimental results show that the neural networks can achieve comparable inference performances on the uncertain neuromorphic computing chip compared to the results on the original platforms, and much better than the performances without this training scheme.
CZ-GEM: A FRAMEWORK FOR DISENTANGLED REPRESENTATION LEARNING
Authors: Akash Srivastava, Yamini Bansal, Yukun Ding, Bernhard Egger, Prasanna Sattigeri, Josh Tenenbaum, David D. Cox, Dan Gutfreund
Date: 2020-01-01 00:00:00
Proceeding: ICLR 2020 1
Tasks: Disentanglement, Representation Learning

Learning disentangled representations of  data is one of the central themes in unsupervised learning in general and generative modelling in particular.  In this work,  we tackle a slightly more intricate scenario where the observations are generated from a conditional distribution of some known control variate and some latent noise variate.  To this end, we present a hierarchical model and a training method (CZ-GEM) that leverages some of the recent developments in likelihood-based and likelihood-free generative models.  We show that by formulation, CZ-GEM introduces the right inductive biases that ensure the disentanglement of the control from the noise variables, while also keeping the components of the control variate disentangled. This is achieved without compromising on the quality of the generated samples. Our approach is simple, general, and can be applied both in supervised and unsupervised settings.
Non-Sequential Melody Generation
Authors: Mitchell Billard, Robert Bishop, Moustafa Elsisy, Laura Graves, Antonina Kolokolova, Vineel Nagisetty, Zachary Northcott, Heather Patey
Date: 2020-01-01 00:00:00
Proceeding: ICLR 2020 1
Tasks: Generative Adversarial Network, Music Generation

In this paper we present a method for algorithmic melody generation using a generative adversarial network without recurrent components. Music generation has been successfully done using recurrent neural networks, where the model learns sequence information that can help create authentic sounding melodies.  Here, we use DCGAN architecture with dilated convolutions and towers to capture sequential information as spatial image information, and learn long-range dependencies in fixed-length melody forms such as Irish traditional reel.