pwc-archive/papers-with-abstracts huggingface dataset:
A Survey of Methods to Leverage Monolingual Data in Low-resource Neural Machine Translation
Authors: Ilshat Gibadullin, Aidar Valeev, Albina Khusainova, Adil Khan
Date: 2019-10-01 00:00:00
Tasks: Low Resource Neural Machine Translation, Low-Resource Neural Machine Translation, Machine Translation, Transfer Learning, Translation

Neural machine translation has become the state-of-the-art for language pairs with large parallel corpora. However, the quality of machine translation for low-resource languages leaves much to be desired. There are several approaches to mitigate this problem, such as transfer learning, semi-supervised and unsupervised learning techniques. In this paper, we review the existing methods, where the main idea is to exploit the power of monolingual data, which, compared to parallel, is usually easier to obtain and significantly greater in amount.
Sub-Architecture Ensemble Pruning in Neural Architecture Search
Authors: Yijun Bian, Qingquan Song, Mengnan Du, Jun Yao, Huanhuan Chen, Xia Hu
Date: 2019-10-01 00:00:00
Tasks: Diversity, Ensemble Learning, Ensemble Pruning, Neural Architecture Search

Neural architecture search (NAS) is gaining more and more attention in recent years due to its flexibility and remarkable capability to reduce the burden of neural network design. To achieve better performance, however, the searching process usually costs massive computations that might not be affordable for researchers and practitioners. While recent attempts have employed ensemble learning methods to mitigate the enormous computational cost, however, they neglect a key property of ensemble methods, namely diversity, which leads to collecting more similar sub-architectures with potential redundancy in the final design. To tackle this problem, we propose a pruning method for NAS ensembles called "Sub-Architecture Ensemble Pruning in Neural Architecture Search (SAEP)." It targets to leverage diversity and to achieve sub-ensemble architectures at a smaller size with comparable performance to ensemble architectures that are not pruned. Three possible solutions are proposed to decide which sub-architectures to prune during the searching process. Experimental results exhibit the effectiveness of the proposed method by largely reducing the number of sub-architectures without degrading the performance.
Application of Low-resource Machine Translation Techniques to Russian-Tatar Language Pair
Authors: Aidar Valeev, Ilshat Gibadullin, Albina Khusainova, Adil Khan
Date: 2019-10-01 00:00:00
Tasks: Machine Translation, Transfer Learning, Translation

Neural machine translation is the current state-of-the-art in machine translation. Although it is successful in a resource-rich setting, its applicability for low-resource language pairs is still debatable. In this paper, we explore the effect of different techniques to improve machine translation quality when a parallel corpus is as small as 324 000 sentences, taking as an example previously unexplored Russian-Tatar language pair. We apply such techniques as transfer learning and semi-supervised learning to the base Transformer model, and empirically show that the resulting models improve Russian to Tatar and Tatar to Russian translation quality by +2.57 and +3.66 BLEU, respectively.
Truth or Backpropaganda? An Empirical Investigation of Deep Learning Theory
Authors: Micah Goldblum, Jonas Geiping, Avi Schwarzschild, Michael Moeller, Tom Goldstein
Date: 2019-10-01 00:00:00
Proceeding: ICLR 2020 1
Tasks: Deep Learning, Learning Theory

We empirically evaluate common assumptions about neural networks that are widely held by practitioners and theorists alike. In this work, we: (1) prove the widespread existence of suboptimal local minima in the loss landscape of neural networks, and we use our theory to find examples; (2) show that small-norm parameters are not optimal for generalization; (3) demonstrate that ResNets do not conform to wide-network theories, such as the neural tangent kernel, and that the interaction between skip connections and batch normalization plays a role; (4) find that rank does not correlate with generalization or robustness in a practical setting.
Grammatical Error Correction in Low-Resource Scenarios
Authors: Jakub Náplava, Milan Straka
Date: 2019-10-01 00:00:00
Proceeding: WS 2019 11
Tasks: Grammatical Error Correction, Machine Translation, Translation

Grammatical error correction in English is a long studied problem with many existing systems and datasets. However, there has been only a limited research on error correction of other languages. In this paper, we present a new dataset AKCES-GEC on grammatical error correction for Czech. We then make experiments on Czech, German and Russian and show that when utilizing synthetic parallel corpus, Transformer neural machine translation model can reach new state-of-the-art results on these datasets. AKCES-GEC is published under CC BY-NC-SA 4.0 license at https://hdl.handle.net/11234/1-3057 and the source code of the GEC model is available at https://github.com/ufal/low-resource-gec-wnut2019.
Proximal Policy Optimization for Improved Convergence in IRGAN
Authors: Moksh Jain, Sowmya Kamath S
Date: 2019-10-01 00:00:00
Tasks: Information Retrieval, Retrieval

IRGAN is an information retrieval (IR) modeling approach that uses a theoretical minimax game between a generative and a discriminative model to iteratively optimize both of them, hence unifying the generative and discriminative approaches. Despite significant performance improvements in several information retrieval tasks, IRGAN training is an unstable process, and the solution varies largely with the random parameter initialization. In this work, we present an improved training objective based on proximal policy optimization objective and Gumbel-Softmax based sampling for the generator. We also propose a modified training algorithm which takes a single gradient update on both the generator as well as discriminator for each iteration step. We present empirical evidence of the improved convergence of the proposed model over the original IRGAN and a comparison on three different IR tasks on benchmark datasets is also discussed, emphasizing the proposed model's superior performance.
Additional Shared Decoder on Siamese Multi-view Encoders for Learning Acoustic Word Embeddings
Authors: Myunghun Jung, Hyungjun Lim, Jahyun Goo, Youngmoon Jung, Hoirin Kim
Date: 2019-10-01 00:00:00
Tasks: Decoder, speech-recognition, Speech Recognition, Triplet, Word Embeddings

Acoustic word embeddings --- fixed-dimensional vector representations of arbitrary-length words --- have attracted increasing interest in query-by-example spoken term detection. Recently, on the fact that the orthography of text labels partly reflects the phonetic similarity between the words' pronunciation, a multi-view approach has been introduced that jointly learns acoustic and text embeddings. It showed that it is possible to learn discriminative embeddings by designing the objective which takes text labels as well as word segments. In this paper, we propose a network architecture that expands the multi-view approach by combining the Siamese multi-view encoders with a shared decoder network to maximize the effect of the relationship between acoustic and text embeddings in embedding space. Discriminatively trained with multi-view triplet loss and decoding loss, our proposed approach achieves better performance on acoustic word discrimination task with the WSJ dataset, resulting in 11.1% relative improvement in average precision. We also present experimental results on cross-view word discrimination and word level speech recognition tasks.
VOnDA: A Framework for Ontology-Based Dialogue Management
Authors: Bernd Kiefer, Anna Welker, Christophe Biwer
Date: 2019-10-01 00:00:00
Tasks: Dialogue Management, Management

We present VOnDA, a framework to implement the dialogue management functionality in dialogue systems. Although domain-independent, VOnDA is tailored towards dialogue systems with a focus on social communication, which implies the need of long-term memory and high user adaptivity. For these systems, which are used in health environments or elderly care, margin of error is very low and control over the dialogue process is of topmost importance. The same holds for commercial applications, where customer trust is at risk. VOnDA's specification and memory layer relies upon (extended) RDF/OWL, which provides a universal and uniform representation, and facilitates interoperability with external data sources, e.g., from physical sensors.
TMLab: Generative Enhanced Model (GEM) for adversarial attacks
Authors: Piotr Niewinski, Maria Pszona, Maria Janicka
Date: 2019-10-01 00:00:00
Tasks: Articles, Language Modeling, Language Modelling, Text Generation

We present our Generative Enhanced Model (GEM) that we used to create samples awarded the first prize on the FEVER 2.0 Breakers Task. GEM is the extended language model developed upon GPT-2 architecture. The addition of novel target vocabulary input to the already existing context input enabled controlled text generation. The training procedure resulted in creating a model that inherited the knowledge of pretrained GPT-2, and therefore was ready to generate natural-like English sentences in the task domain with some additional control. As a result, GEM generated malicious claims that mixed facts from various articles, so it became difficult to classify their truthfulness.
Towards French Smart Building Code: Compliance Checking Based on Semantic Rules
Authors: Nicolas Bus, Ana Roxin, Guillaume Picinbono, Muhammad Fahad
Date: 2019-10-01 00:00:00

Manually checking models for compliance against building regulation is a time-consuming task for architects and construction engineers. There is thus a need for algorithms that process information from construction projects and report non-compliant elements. Still automated code-compliance checking raises several obstacles. Building regulations are usually published as human readable texts and their content is often ambiguous or incomplete. Also, the vocabulary used for expressing such regulations is very different from the vocabularies used to express Building Information Models (BIM). Furthermore, the high level of details associated to BIM-contained geometries induces complex calculations. Finally, the level of complexity of the IFC standard also hinders the automation of IFC processing tasks. Model chart, formal rules and pre-processors approach allows translating construction regulations into semantic queries. We further demonstrate the usefulness of this approach through several use cases. We argue our approach is a step forward in bridging the gap between regulation texts and automated checking algorithms. Finally with the recent building ontology BOT recommended by the W3C Linked Building Data Community Group, we identify perspectives for standardizing and extending our approach.
A Multi-Modal Feature Embedding Approach to Diagnose Alzheimer Disease from Spoken Language
Authors: S. Soroush Haj Zargarbashi, Bagher BabaAli
Date: 2019-10-01 00:00:00
Tasks: BIG-bench Machine Learning

Introduction: Alzheimer's disease is a type of dementia in which early diagnosis plays a major rule in the quality of treatment. Among new works in the diagnosis of Alzheimer's disease, there are many of them analyzing the voice stream acoustically, syntactically or both. The mostly used tools to perform these analysis usually include machine learning techniques. Objective: Designing an automatic machine learning based diagnosis system will help in the procedure of early detection. Also, systems, using noninvasive data are preferable. Methods: We used are classification system based on spoken language. We use three (statistical and neural) approaches to classify audio signals from spoken language into two classes of dementia and control. Result: This work designs a multi-modal feature embedding on the spoken language audio signal using three approaches; N-gram, i-vector, and x-vector. The evaluation of the system is done on the cookie picture description task from Pitt Corpus dementia bank with the accuracy of 83:6
Graph convolutional networks for learning with few clean and many noisy labels
Authors: Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, Ondrej Chum, Cordelia Schmid
Date: 2019-10-01 00:00:00
Proceeding: ECCV 2020 8
Tasks: Few-Shot Learning, General Classification

In this work we consider the problem of learning a classifier from noisy labels when a few clean labeled examples are given. The structure of clean and noisy data is modeled by a graph per class and Graph Convolutional Networks (GCN) are used to predict class relevance of noisy examples. For each class, the GCN is treated as a binary classifier, which learns to discriminate clean from noisy examples using a weighted binary cross-entropy loss function. The GCN-inferred "clean" probability is then exploited as a relevance measure. Each noisy example is weighted by its relevance when learning a classifier for the end task. We evaluate our method on an extended version of a few-shot learning problem, where the few clean examples of novel classes are supplemented with additional noisy data. Experimental results show that our GCN-based cleaning process significantly improves the classification accuracy over not cleaning the noisy data, as well as standard few-shot classification where only few clean examples are used.
BioNLP-OST 2019 RDoC Tasks: Multi-grain Neural Relevance Ranking Using Topics and Attention Based Query-Document-Sentence Interactions
Authors: Yatin Chaudhary, Pankaj Gupta, Hinrich Schütze
Date: 2019-10-01 00:00:00
Proceeding: WS 2019 11
Tasks: Re-Ranking, Retrieval, Sentence, Task 2

This paper presents our system details and results of participation in the RDoC Tasks of BioNLP-OST 2019. Research Domain Criteria (RDoC) construct is a multi-dimensional and broad framework to describe mental health disorders by combining knowledge from genomics to behaviour. Non-availability of RDoC labelled dataset and tedious labelling process hinders the use of RDoC framework to reach its full potential in Biomedical research community and Healthcare industry. Therefore, Task-1 aims at retrieval and ranking of PubMed abstracts relevant to a given RDoC construct and Task-2 aims at extraction of the most relevant sentence from a given PubMed abstract. We investigate (1) attention based supervised neural topic model and SVM for retrieval and ranking of PubMed abstracts and, further utilize BM25 and other relevance measures for re-ranking, (2) supervised and unsupervised sentence ranking models utilizing multi-view representations comprising of query-aware attention-based sentence representation (QAR), bag-of-words (BoW) and TF-IDF. Our best systems achieved 1st rank and scored 0.86 mean average precision (mAP) and 0.58 macro average accuracy (MAA) in Task-1 and Task-2 respectively.
A note on the empirical comparison of RBG and Ludii
Authors: Jakub Kowalski, Maksymilian Mika, Jakub Sutowicz, Marek Szykuła
Date: 2019-10-01 00:00:00

We present an experimental comparison of the efficiency of three General Game Playing systems in their current versions: Regular Boardgames (RBG 1.0), Ludii~0.3.0, and a Game Description Language (GDL) propnet. We show that in general, RBG is currently the fastest GGP system. For example, for chess, we demonstrate that RBG is about 37 times faster than Ludii, and Ludii is about 3 times slower than a GDL propnet. Referring to the recent comparison [An Empirical Evaluation of Two General Game Systems: Ludii and RBG, CoG 2019], we show evidences that the benchmark presented there contains a number of significant flaws that lead to wrong conclusions.
Insect pest image detection and recognition based on bio-inspired methods
Authors: Loris Nanni, Gianluca Maguolo, Fabio Pancino
Date: 2019-10-01 00:00:00

Insect pests recognition is necessary for crop protection in many areas of the world. In this paper we propose an automatic classifier based on the fusion between saliency methods and convolutional neural networks. Saliency methods are famous image processing algorithms that highlight the most relevant pixels of an image. In this paper, we use three different saliency methods as image preprocessing and create three different images for every saliency method. Hence, we create 3x3=9 new images for every original image to train different convolutional neural networks. We evaluate the performance of every preprocessing/network couple and we also evaluate the performance of their ensemble. We test our approach on both a small dataset and the large IP102 dataset. Our best ensembles reaches the state of the art accuracy on both the smaller dataset (92.43%) and the IP102 dataset (61.93%), approaching the performance of human experts on the smaller one. Besides, we share our MATLAB code at: https://github.com/LorisNanni/.
When and Why is Document-level Context Useful in Neural Machine Translation?
Authors: Yunsu Kim, Duc Thanh Tran, Hermann Ney
Date: 2019-10-01 00:00:00
Proceeding: WS 2019 11
Tasks: Machine Translation, NMT, Translation

Document-level context has received lots of attention for compensating neural machine translation (NMT) of isolated sentences. However, recent advances in document-level NMT focus on sophisticated integration of the context, explaining its improvement with only a few selected examples or targeted test sets. We extensively quantify the causes of improvements by a document-level model in general test sets, clarifying the limit of the usefulness of document-level context in NMT. We show that most of the improvements are not interpretable as utilizing the context. We also show that a minimal encoding is sufficient for the context modeling and very long context is not helpful for NMT.
Generalization in Generation: A closer look at Exposure Bias
Authors: Florian Schmidt
Date: 2019-10-01 00:00:00
Proceeding: WS 2019 11
Tasks: Language Modeling, Language Modelling, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL), Sentence

Exposure bias refers to the train-test discrepancy that seemingly arises when an autoregressive generative model uses only ground-truth contexts at training time but generated ones at test time. We separate the contributions of the model and the learning framework to clarify the debate on consequences and review proposed counter-measures. In this light, we argue that generalization is the underlying property to address and propose unconditional generation as its fundamental benchmark. Finally, we combine latent variable modeling with a recent formulation of exploration in reinforcement learning to obtain a rigorous handling of true and generated contexts. Results on language modeling and variational sentence auto-encoding confirm the model's generalization capability.
Distance-Based Approaches to Repair Semantics in Ontology-based Data Access
Authors: César Prouté, Bruno Yun, Madalina Croitoru
Date: 2019-10-01 00:00:00

In the presence of inconsistencies, repair techniques thrive to restore consistency by reasoning with several repairs. However, since the number of repairs can be large, standard inconsistent tolerant semantics usually yield few answers. In this paper, we use the notion of syntactic distance between repairs following the intuition that it can allow us to cluster some repairs "close" to each other. In this way, we propose a generic framework to answer queries in a more personalise fashion.
Identifying Supporting Facts for Multi-hop Question Answering with Document Graph Networks
Authors: Mokanarangan Thayaparan, Marco Valentino, Viktor Schlegel, Andre Freitas
Date: 2019-10-01 00:00:00
Proceeding: WS 2019 11
Tasks: Multi-hop Question Answering, Question Answering, Reading Comprehension

Recent advances in reading comprehension have resulted in models that surpass human performance when the answer is contained in a single, continuous passage of text. However, complex Question Answering (QA) typically requires multi-hop reasoning - i.e. the integration of supporting facts from different sources, to infer the correct answer. This paper proposes Document Graph Network (DGN), a message passing architecture for the identification of supporting facts over a graph-structured representation of text. The evaluation on HotpotQA shows that DGN obtains competitive results when compared to a reading comprehension baseline operating on raw text, confirming the relevance of structured representations for supporting multi-hop reasoning.
Unsupervised Generative 3D Shape Learning from Natural Images
Authors: Attila Szabó, Givi Meishvili, Paolo Favaro
Date: 2019-10-01 00:00:00
Tasks: Image Generation

In this paper we present, to the best of our knowledge, the first method to learn a generative model of 3D shapes from natural images in a fully unsupervised way. For example, we do not use any ground truth 3D or 2D annotations, stereo video, and ego-motion during the training. Our approach follows the general strategy of Generative Adversarial Networks, where an image generator network learns to create image samples that are realistic enough to fool a discriminator network into believing that they are natural images. In contrast, in our approach the image generation is split into 2 stages. In the first stage a generator network outputs 3D objects. In the second, a differentiable renderer produces an image of the 3D objects from random viewpoints. The key observation is that a realistic 3D object should yield a realistic rendering from any plausible viewpoint. Thus, by randomizing the choice of the viewpoint our proposed training forces the generator network to learn an interpretable 3D representation disentangled from the viewpoint. In this work, a 3D representation consists of a triangle mesh and a texture map that is used to color the triangle surface by using the UV-mapping technique. We provide analysis of our learning approach, expose its ambiguities and show how to overcome them. Experimentally, we demonstrate that our method can learn realistic 3D shapes of faces by using only the natural images of the FFHQ dataset.
Bad Form: Comparing Context-Based and Form-Based Few-Shot Learning in Distributional Semantic Models
Authors: Jeroen Van Hautte, Guy Emerson, Marek Rei
Date: 2019-10-01 00:00:00
Proceeding: WS 2019 11
Tasks: Few-Shot Learning, Form, Word Embeddings

Word embeddings are an essential component in a wide range of natural language processing applications. However, distributional semantic models are known to struggle when only a small number of context sentences are available. Several methods have been proposed to obtain higher-quality vectors for these words, leveraging both this context information and sometimes the word forms themselves through a hybrid approach. We show that the current tasks do not suffice to evaluate models that use word-form information, as such models can easily leverage word forms in the training data that are related to word forms in the test data. We introduce 3 new tasks, allowing for a more balanced comparison between models. Furthermore, we show that hyperparameters that have largely been ignored in previous work can consistently improve the performance of both baseline and advanced models, achieving a new state of the art on 4 out of 6 tasks.