pwc-archive/papers-with-abstracts huggingface dataset:
Addressing the Representation Bottleneck in Neural Machine Translation with Lexical Shortcuts
Authors: Anonymous
Date: 2019-01-24 00:00:00
Tasks: Decoder, Machine Translation, Translation

The transformer is a state-of-the-art neural translation model that uses attention to iteratively refine lexical representations with information drawn from the surrounding context. Lexical features are fed into the first layer and propagated through a deep network of hidden layers. We argue that the need to represent and propagate lexical features in each layer limits the model’s capacity for learning and representing other information relevant to the task. To alleviate this bottleneck, we introduce gated shortcut connections between the embedding layer and each subsequent layer within the encoder and decoder. This enables the model to access relevant lexical content dynamically, without expending limited resources on storing it within intermediate states. We show that the proposed modification yields consistent improvements on standard WMT translation tasks and reduces the amount of lexical information passed along the hidden layers. We furthermore evaluate different ways to integrate lexical connections into the transformer architecture and present ablation experiments exploring the effect of proposed shortcuts on model behavior.
MORTY Embedding: Improved Embeddings without Supervision
Authors: Anonymous
Date: 2019-01-24 00:00:00
Tasks: Word Embeddings

We demonstrate a low effort method that unsupervisedly constructs task-optimized embeddings from existing word embeddings to gain performance on a supervised end-task. This avoids additional labeling or building more complex model architectures by instead providing specialized embeddings better fit for the end-task(s). Furthermore, the method can be used to roughly estimate whether a specific kind of end-task(s) can be learned form, or is represented in, a given unlabeled dataset, e.g. using publicly available probing tasks. We evaluate our method for diverse word embedding probing tasks and by size of embedding training corpus -- i.e. to explore its use in reduced (pretraining-resource) settings.
Load Balancing in Large-Scale Heterogeneous Systems with Multiple Dispatchers
Authors: Anonymous
Date: 2018-12-27 00:00:00

In recent years, the efficiency and even the feasibility of traditional load-balancing policies are challenged by the rapid growth of cloud infrastructure with increasing levels of server heterogeneity and increasing size of cloud services and applications. In such many software-load-balancers heterogeneous systems, traditional solutions, such as JSQ, incur an increasing communication overhead, whereas low-communication alternatives, such as JSQ(d) and the recently proposed JIQ scheme are either unstable or provide poor performance.

We argue that a better low-communication load balancing scheme can be established by allowing each dispatcher to have a different view of the system and keep using JSQ, rather than greedily trying to avoid starvation on a per-decision basis. 
accordingly, we introduce the Loosely-Shortest -Queue family of load balancing algorithms. Roughly speaking, in Loosely-shortest -Queue, each dispatcher keeps a different approximation of the server queue lengths and routes jobs to the shortest among them. Communication is used only to update the approximations and make sure that they are not too far from the real queue lengths in expectation. We formally establish the strong stability of any Loosely-Shortest -Queue policy and provide an easy-to-verify sufficient condition for verifying that a policy is Loosely-Shortest -Queue. We further demonstrate that the Loosely-Shortest -Queue approach allows constructing throughput optimal policies with an arbitrarily low communication budget.

Finally, using extensive simulations that consider homogeneous, heterogeneous and highly skewed heterogeneous systems in scenarios with a single dispatcher as well as with multiple dispatchers, we show that the examined Loosely-Shortest -Queue example policies are always stable as dictated by theory. Moreover, it exhibits an appealing performance and significantly outperforms well-known low-communication policies, such as JSQ(d) and JIQ, while using a similar communication budget.
A Review of Meta-Reinforcement Learning for Deep Neural Networks Architecture Search
Authors: Anonymous
Date: 2018-12-20 00:00:00
Tasks: Meta Reinforcement Learning, Natural Language Understanding, reinforcement-learning, Reinforcement Learning (RL), speech-recognition, Speech Recognition

Deep Neural networks are efficient and flexible models that perform well for a variety of tasks such as image, speech recognition and natural language understanding. In particular, convolutional neural networks (CNN) generate a keen interest among researchers in computer vision and more specifically in classification tasks. CNN architecture and related hyperparameters are generally correlated to the nature of the processed task as the network extracts complex and relevant characteristics allowing the optimal convergence. Designing such architectures requires significant human expertise, substantial computation time and doesnt always lead to the optimal network. Model configuration topic has been extensively studied in machine learning without leading to a standard automatic method. This survey focuses on reviewing and discussing the current progress in automating CNN architecture search.
Grammatical Analysis of Pretrained Sentence Encoders with Acceptability Judgments
Authors: Anonymous
Date: 2018-12-11 00:00:00
Tasks: CoLA, Linguistic Acceptability, Sentence

Recent pretrained sentence encoders achieve state of the art results on language understanding tasks, but does this mean they have implicit knowledge of syntactic structures? We introduce a grammatically annotated development set for the Corpus of Linguistic Acceptability (CoLA; Warstadt et al., 2018), which we use to investigate the grammatical knowledge of three pretrained encoders, including the popular OpenAI Transformer (Radford et al., 2018) and BERT (Devlin et al., 2018). We fine-tune these encoders to do acceptability classification over CoLA and compare the models’ performance on the annotated analysis set. Some phenomena, e.g. modification by adjuncts, are easy to learn for all models, while others, e.g. long-distance movement, are learned effectively only by models with strong overall performance, and others still, e.g. morphological agreement, are hardly learned by any model.
DeepProteomics: Protein family classification using Shallow and Deep Networks
Authors: Anonymous
Date: 2018-10-12 00:00:00
Tasks: Classification

The knowledge regarding the function of proteins is necessary as it gives a clear picture of biological processes. Nevertheless, there are many protein sequences found and added to the databases but lacks functional annotation. The laboratory experiments take a considerable amount of time for annotation of the sequences. This arises the need to use computational techniques to classify proteins based on their functions. In our work, we have collected the data from Swiss-Prot containing 40433 proteins which is grouped into 30 families. We pass it to recurrent neural network(RNN), long short term memory(LSTM) and gated recurrent unit(GRU) model and compare it by applying trigram with deep neural network and shallow neural network on the same dataset. Through this approach, we could achieve maximum of around 78% accuracy for the classification of protein families.
Protein Family Classification using Deep Learning
Authors: Anonymous
Date: 2018-10-12 00:00:00
Tasks: Classification, Deep Learning

Protein classification is responsible for the biological sequence, we came up with an idea whichdeals with the classification of proteomics using deep learning algorithm. This algorithm focusesmainly to classify sequences of protein-vector which is used for the representation of proteomics.Selection  of  the  type  protein  representation  is  challenging  based  on  which  output  in  terms  ofaccuracy is depended on, The protein representation used here is n-gram i.e.  3-gram and Kerasembedding used for biological sequences like protein. In this paper we are working on the Proteinclassification to show the strength and representation of biological sequence of the proteins
RNNSecureNet: Recurrent neural networks for Cybersecurity use-cases
Authors: Anonymous
Date: 2018-10-12 00:00:00
Tasks: Fraud Detection, Malware Classification

Recurrent neural network(RNN) is an effective neural network in solving very complex supervised and unsupervised tasks.There has been a significant improvement in RNN field such as natural language processing, speech processing, computer vision and other multiple domains. This paper deals with RNN application on different use cases like Incident Detection , Fraud Detection , and Android Malware Classification. The best performing neural network architecture is chosen
by conducting different chain of experiments for different network parameters and structures.The network is run up to 1000 epochs with learning rate set in the range of 0.01 to 0.5.Obviously, RNN performed very well when compared to classical machine learning algorithms. This is mainly possible because RNNs implicitly extracts the underlying features and also identifies the characteristics of the data. This lead to better accuracy.
Deep-Net: Deep Neural Network for Cyber Security Use Cases
Authors: Anonymous
Date: 2018-10-12 00:00:00
Tasks: Fraud Detection, Malware Classification

Deep neural networks (DNNs) have witnessed as a powerful approach in this year by solving long-standing Artificial
intelligence (AI) supervised and unsupervised tasks exists in natural language processing, speech processing, computer vision and others. In this paper, we attempt to apply DNNs on three different cyber security use cases: Android malware classification, incident detection and fraud detection. The data set of each use case contains real known benign and malicious activities samples. These use cases are part of Cybersecurity Data Mining Competition (CDMC) 2017. The efficient network architecture for DNNs is chosen by conducting various trails of experiments for network parameters and network structures. The experiments of such chosen efficient configurations of DNNs are run up to 1000 epochs with learning rate set in the range [0.01-0.5]. Experiments of DNNs performed well in comparison to the classical machine learning algorithm in all cases of experiments of cyber security use cases. This is due to the fact that DNNs implicitly extract and build better features, identifies the characteristics of the data that lead to better accuracy. The best accuracy obtained by DNNs and XGBoost on Android malware classification 0.940 and 0.741, incident detection 1.00 and 0.997, and fraud detection 0.972 and 0.916 respectively. The accuracy obtained by DNNs varies -0.05%, +0.02%, -0.01% from the top scored system in CDMC 2017 tasks.
Evaluation of automatic collocation extraction methods for language learning
Authors: Anonymous
Date: 2018-05-25 00:00:00
Tasks: Dependency Parsing, POS

A number of methods have been proposed to automatically extract collocations, i.e., conventionalized lexical combinations, from text corpora. However, the attempts to evaluate and compare them with a specific application in mind lag behind. This paper compares three end-to-end resources for collocation learning, all of which used the same corpus but different methods. Adopting a gold-standard evaluation method, the results show that the method of dependency parsing outperforms regex-over-pos in collocation identification. The lexical association measures (AMs) used for collocation ranking perform about the same overall but differently for individual collocation types. Further analysis has also revealed that there are considerable differences between other commonly used AMs.
FR-MRInet: A Deep Convolutional Encoder-Decoder for Brain Tumor Segmentation with Relu-RGB and Sliding-window
Authors: Anonymous
Date: 2018-07-26 00:00:00
Tasks: Benchmarking, Brain Tumor Segmentation, Decoder, Semantic Segmentation, Tumor Segmentation

Automatic detection of tumorous tissue in MRI scans plays an important role in computer-aided diagnosis. We present a novel deep fully convolutional encoding architecture for semantic segmentation of brain MRI scans termed, FR-MRInet. This trainable encoder works with a corresponding decoder of a fully connected network. The 32 layer deep encoding architecture is inspired by VGG16 and InceptionV3. The novelty of FR-MRInet is its architectural design that efﬁciently reduces input to a lower resolution feature map(s). The encoder uses strides instead of pooling in various layers to reduce feature maps without loosing spacial information. We used a non-overlapping sliding window with and a novel activation function called, Relu-RGB to train the model so that the model directly produces the ﬁnal output instead of a mask. We compared our model with well known imagenets such as Alexnet and VGGnet, other recent models proposed by researchers testing for pixel wise accuracy,intersection over union(IoU)and mean square loss value. We conducted our experiment on BRATS dataset for benchmarking and one of the latest dataset which was proposed in 2016 consisting of, 3064T1-weighted contrast-enhanced images from 233 patients. We also show that FR-MRINet provides an impressive performance on live images detecting tumors as well. To further investigate the matter, we have consulted with an MD about the usefulness and the future of these kind of projects. Our code is open sourced and freely available at github.com/xxxxxxxxx/FR-MRInet.
Detecting cognitive impairments by agreeing on interpretations of linguistic features
Authors: Anonymous
Date: 2018-07-26 00:00:00

Linguistic features have shown promising applications for detecting various cognitive impairments. To improve detection accuracies, increasing the amount of data or linguistic features have been two applicable approaches. However, acquiring additional clinical data could be expensive, and hand-carving features are hard. In this paper, we take a third approach, putting forward the scheme ``diagnosis after reaching consensus'', where non-overlapping subsets (modalities) of linguistic features are compressed into low-dimension interpretation vectors by neural networks (''ePhysicians''). By encouraging interpretation vectors from multiple modalities to be indistinguishable, the ``ePhysicians'' extract important information for classification. 
We show that with the same subsets of features, our models outperform baseline neural network classifiers on clinical data. Using all of the 413 linguistic features, our best models have accuracies in detecting cognitive impairments comparable to the state-of-the-art models on several balanced datasets (.82 on DementiaBank in detecting Alzheimer's Disease (AD) and .66 in detecting Mild Cognitive Impairment (MCI)).
Sliced-Wasserstein Autoencoder: An Embarrassingly Simple Generative Model
Authors: Anonymous
Date: 2018-04-05 00:00:00
Tasks: model

In this paper we study generative modeling via autoencoders while using the elegant geometric properties of the optimal transport (OT) problem and the Wasserstein distances. We introduce Sliced-Wasserstein Autoencoders (SWAE), which are generative models that enable one to shape the distribution of the latent space into any samplable probability distribution without the need for training an adversarial network or defining a closed-form for the distribution. In short, we regularize the autoencoder loss with the sliced-Wasserstein distance between the distribution of the encoded training samples and a predefined samplable distribution. We show that the proposed formulation has an efficient numerical solution that provides similar capabilities to Wasserstein Autoencoders (WAE) and Variational Autoencoders (VAE), while benefiting from an embarrassingly simple implementation.
Reducing Distant Supervision Noise with Maxpooled Attention and Sentence-Level Supervision
Authors: Anonymous
Date: 2018-06-06 00:00:00
Tasks: Sentence

We propose an effective multitask learning setup for reducing distant supervision noise by leveraging sentence-level supervision. We show how sentence-level supervision can be used to improve the encoding of individual sentences, and to learn which input sentences are more likely to express the relationship between a pair of entities. We also introduce a novel neural architecture for collecting signals from multiple input sentences, which combines the benefits of attention and maxpooling. The proposed method increases AUC by 10% (from 0.261 to 0.284), and outperforms recently published results on the FB-NYT dataset.
Aggregating Crowdsourced Labels in Subjective Domains
Authors: Anonymous
Date: 2018-05-29 00:00:00

Supervised learning problems---particularly those involving social data---are often subjective. That is, human readers, looking at the same data, might come to legitimate but completely different conclusions based on their personal experiences. Yet in machine learning settings feedback from multiple human annotators is often reduced to a single ``ground truth'' label, thus hiding the true, potentially rich and diverse interpretations of the data found across the social spectrum. We explore the rewards and challenges of discovering and learning representative distributions of the labeling opinions of a large human population. A major, critical cost to this approach is the number of humans needed to provide enough labels not only to obtain representative samples but also to train a machine to predict representative distributions on unlabeled data. We propose aggregating label distributions over, not just individuals, but also data items, in order to maximize the costs of humans in the loop. We test different aggregation approaches on state-of-the-art deep learning models. Our results suggest that careful label aggregation methods can greatly reduce the number of samples needed to obtain representative distributions.
Mind Your Language: Learning Visually Grounded Dialog in a Multi-Agent Setting
Authors: Anonymous
Date: 2018-05-25 00:00:00

The task of visually grounded dialog involves learning goal-oriented cooperative dialog between autonomous agents who exchange information about a scene through several rounds of questions and answers. We posit that requiring agents to adhere to rules of human language while also maximizing information exchange is an ill-posed problem, and observe that humans do not stray from a common language, because they are social creatures and have to communicate with many people everyday, and it is far easier to stick to a common language even at the cost of some efficiency loss. Using this as inspiration, we propose and evaluate a multi-agent dialog framework where each agent interacts with, and learns from, multiple agents, and show that this results in more relevant and coherent dialog (as judged by human evaluators) without sacrificing task performance (as judged by quantitative metrics).
Omega: An Architecture for AI Unification
Authors: Anonymous
Date: 2018-05-24 00:00:00
Tasks: Diversity

We introduce the open-ended, modular, self-improving Omega AI unification architecture which is a refinement of Solomonoff's Alpha architecture, as considered from first principles. The architecture embodies several crucial principles of general intelligence including diversity of representations, diversity of data types, integrated memory, modularity, and higher-order cognition. We retain the basic design of a fundamental algorithmic substrate called an ``AI kernel'' for problem solving and basic cognitive functions like memory, and a larger, modular architecture that re-uses the kernel in many ways. Omega includes eight representation languages and six classes of neural networks, which are briefly introduced. The architecture is intended to initially address data science automation, hence it includes many problem solving methods for statistical tasks. We review the broad software architecture, higher-order cognition, self-improvement, modular neural architectures, intelligent agents, the process and memory hierarchy, hardware abstraction, peer-to-peer computing, and data abstraction facility.
Fake Sentence Detection as a Training Task for Sentence Encoding
Authors: Anonymous
Date: 2018-05-23 00:00:00
Tasks: Binary Classification, Language Modeling, Language Modelling, Sentence

 Sentence encoders are typically trained on language modeling tasks which enable them to use large unlabeled datasets. While these models achieve state-of-the-art results on many sentence-level tasks, they are difficult to train with long training cycles. 
    We introduce fake sentence detection as a new training task for learning sentence encodings. We automatically generate fake sentences by corrupting some original sentence and train the encoders to produce representations that are effective at detecting fake sentences. This binary classification task allows for efficient training and forces the encoder to learn the distinctions introduced by a small edit to sentences. We train a basic BiLSTM encoder to produce sentence representations and find that it outperforms a strong sentence encoding model trained on language modeling tasks, while also training much faster on smaller amount of data (20 hours instead of weeks). Further analysis shows the learned representations capture many syntactic and semantic properties expected from good sentence representations.
Lexicosyntactic Inference in Neural Models
Authors: Anonymous
Date: 2018-05-23 00:00:00
Tasks: Prediction

We investigate neural models' ability to capture "lexicosyntactic inferences": inferences triggered by the interaction of lexical and syntactic information. We take the task of event factuality prediction as a case study and build a factuality judgment dataset for all English clause-embedding predicates in various syntactic contexts. We use this dataset, which we make publicly available, to probe the behavior of current state-of-the-art neural systems, showing that these systems make certain systematic errors that are clearly visible through the lens of factuality prediction.
Meta-Learning for Low-Resource Neural Machine Translation
Authors: Anonymous
Date: 2018-05-23 00:00:00
Tasks: Low Resource Neural Machine Translation, Low-Resource Neural Machine Translation, Machine Translation, Meta-Learning, NMT, Transfer Learning, Translation

In this paper, we propose to extend the recently introduced model-agnostic meta-learning algorithm (MAML, Finn et al., 2017) for low resource neural machine translation (NMT). We frame low-resource translation as a meta-learning problem, and we learn to adapt to low-resource languages based on multilingual high-resource language tasks. We use the universal lexical representation (Gu et al., 2018b) to overcome the input-output mismatch across different languages. We evaluate the proposed meta-learning strategy using eighteen European languages (Bg, Cs, Da, De, El, Es, Et, Fr, Hu, It, Lt, Nl, Pl, Pt, Sk, Sl, Sv and Ru) as source tasks and five diverse languages (Ro, Lv, Fi, Tr, and Ko) as target tasks. We show that the proposed approach significantly outperforms the multilingual, transfer learning based approach (Zoph et al., 2016) and enables us to train a competitive NMT system with only a fraction of training examples. For instance, the proposed approach can achieve as high as 22.04 BLEU on Romanian-English WMT’16 by seeing only 16,000 translated words (~600 parallel sentences).