pwc-archive/papers-with-abstracts huggingface dataset:
Improving Language Understanding by Generative Pre-Training
Authors: Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever
Date: 2018-06-11 00:00:00
Proceeding: Preprint 2018 6
Tasks: Cloze Test, Document Classification, Language Modeling, Language Modelling, Natural Language Inference, Natural Language Understanding, Question Answering, Semantic Similarity, Semantic Textual Similarity

Natural language understanding comprises a wide range of diverse tasks such
as textual entailment, question answering, semantic similarity assessment, and
document classification. Although large unlabeled text corpora are abundant,
labeled data for learning these specific tasks is scarce, making it challenging for
discriminatively trained models to perform adequately. We demonstrate that large
gains on these tasks can be realized by generative pre-training of a language model
on a diverse corpus of unlabeled text, followed by discriminative fine-tuning on each
specific task. In contrast to previous approaches, we make use of task-aware input
transformations during fine-tuning to achieve effective transfer while requiring
minimal changes to the model architecture. We demonstrate the effectiveness of
our approach on a wide range of benchmarks for natural language understanding.
Our general task-agnostic model outperforms discriminatively trained models that
use architectures specifically crafted for each task, significantly improving upon the
state of the art in 9 out of the 12 tasks studied. For instance, we achieve absolute
improvements of 8.9% on commonsense reasoning (Stories Cloze Test), 5.7% on
question answering (RACE), and 1.5% on textual entailment (MultiNLI).
Multiway Attention Networks for Modeling Sentence Pairs
Authors: Chuanqi Tan, Furu Wei, Wenhui Wang, Weifeng Lv, Ming Zhou
Date: 2018-07-01 00:00:00
Proceeding: IJCAI 2018 7
Tasks: Natural Language Inference, Paraphrase Identification, Sentence

Modeling sentence pairs plays the vital role for
judging the relationship between two sentences,
such as paraphrase identification, natural language
inference, and answer sentence selection. Previous
work achieves very promising results using neural
networks with attention mechanism. In this paper,
we propose the multiway attention networks which
employ multiple attention functions to match sentence pairs under the matching-aggregation framework. Specifically, we design four attention functions to match words in corresponding sentences.
Then, we aggregate the matching information from
each function, and combine the information from
all functions to obtain the final representation. Experimental results demonstrate that the proposed
multiway attention networks improve the result on
the Quora Question Pairs, SNLI, MultiNLI, and answer sentence selection task on the SQuAD dataset.
Robust Discovery of Positive and Negative Rules in Knowledge-Bases
Authors: Stefano Ortona, Venkata Vamsikrishna Meduri, Paolo Papotti
Date: 2018-04-16 00:00:00
Proceeding: ICDE 2018 4
Tasks: Knowledge Graphs, Knowledge Graphs Data Curation

We present RUDIK, a system for the discovery of declarative rules over knowledge-bases (KBs). RUDIK discovers rules that express positive relationships between entities, such as “if two persons have the same parent, they are siblings”, and negative rules, i.e., patterns that identify contradictions in the data, such as “if two persons are married, one cannot be the child of the other”. While the former class infers new facts in the KB, the latter class is crucial for other tasks, such as detecting erroneous triples in data cleaning, or the creation of negative examples to bootstrap learning algorithms. The system is designed to: (i) enlarge the expressive power of the rule language to obtain complex rules and wide coverage of the facts in the KB, (ii) discover approximate rules (soft constraints) to be robust to errors and incompleteness in the KB, (iii) use disk-based algorithms, effectively enabling rule mining in commodity machines. In contrast with traditional ranking of all rules based on a measure of support, we propose an approach to identify the subset of useful rules to be exposed to the user. We model the mining process as an incremental graph exploration problem and prove that our search strategy has guarantees on the optimality of the results. We have conducted extensive experiments using real-world KBs to show that RUDIK outperforms previous proposals in terms of efficiency and that it discovers more effective rules for the application at hand.
Automatic Seismic Salt Interpretation with Deep Convolutional Neural Networks
Authors: Yu Zeng, Kebei Jiang, Jie Chen
Date: 2018-11-24 00:00:00

One of the most crucial tasks in seismic reflection imaging is to identify
the salt bodies with high precision. Traditionally, this is accomplished by
visually picking the salt/sediment boundaries, which requires a great amount of
manual work and may introduce systematic bias. With recent progress of deep
learning algorithm and growing computational power, a great deal of efforts
have been made to replace human effort with machine power in salt body
interpretation. Currently, the method of Convolutional neural networks (CNN) is
revolutionizing the computer vision field and has been a hot topic in the image
analysis. In this paper, the benefits of CNN-based classification are
demonstrated by using a state-of-art network structure U-Net, along with the
residual learning framework ResNet, to delineate salt body with high precision.
Network adjustments, including the Exponential Linear Units (ELU) activation
function, the Lov\'{a}sz-Softmax loss function, and stratified $K$-fold
cross-validation, have been deployed to further improve the prediction
accuracy. The preliminary result using SEG Advanced Modeling (SEAM) data shows
good agreement between the predicted salt body and manually interpreted salt
body, especially in areas with weak reflections. This indicates the great
potential of applying CNN for salt-related interpretations.
From genome to phenome: Predicting multiple cancer phenotypes based on somatic genomic alterations via the genomic impact transformer
Authors: Yifeng Tao, Chunhui Cai, William Cohen, Xinghua Lu
Date: 2019-01-31 00:00:00

Motivation: Cancers are mainly caused by somatic genomic alterations (SGAs)
that perturb cellular signaling systems and eventually activate oncogenic
processes. Therefore, understanding the functional impact of SGAs is a
fundamental task in cancer biology and precision oncology. Here, we present a
deep neural network model with encoder-decoder architecture, referred to as
genomic impact transformer (GIT), to infer the functional impact of SGAs on
cellular signaling systems through modeling the statistical relationships
between SGA events and differentially expressed genes (DEGs) in tumors. The
model utilizes multi-head self-attention mechanism to identify SGAs that likely
cause DEGs, or in other words differentiating potential driver SGAs from
passenger ones in a tumor. GIT model learns a vector (gene embedding) as an
abstract representation of functional impact for each SGA-affected gene. Given
SGAs of a tumor, the model can instantiate the states of the hidden layer,
providing abstract representation (tumor embedding) reflecting characteristics
of perturbed molecular/cellular processes in the tumor, which in turn can be
used to predict multiple phenotypes. Results: We apply the GIT model to 4,468
tumors profiled by The Cancer Genome Atlas (TCGA) project. The attention
mechanism enables the model to better capture the statistical relationship
between SGAs and DEGs than conventional methods, and distinguishes cancer
drivers from passengers. The learned gene embeddings capture the functional
similarity of SGAs perturbing common pathways. The tumor embeddings are shown
to be useful for tumor status representation, and phenotype prediction
including patient survival time and drug response of cancer cell lines.
Porter 5: fast, state-of-the-art ab initio prediction of protein secondary structure in 3 and 8 classes
Authors: Mirko Torrisi, Manaz Kaleel, Gianluca Pollastri
Date: 2018-10-05 00:00:00
Proceeding: bioRxiv 2018 10
Tasks: Protein Secondary Structure Prediction

Motivation: Although secondary structure predictors have been developed for decades, current ab initio methods have still some way to go to reach their theoretical limits. Moreover, the continuous effort towards harnessing ever-expanding data sets and more sophisticated, deeper Machine Learning techniques, has not come to an end. 
Results: Here we present Porter 5, the latest release of one of the best performing ab initio secondary structure predictors. Version 5 achieves 84% accuracy (84% SOV) when tested on 3 classes, and 73% accuracy (77% SOV) on 8 classes, on a large independent set, significantly outperforming all the most recent ab initio predictors we have tested. 
Availability: The web and standalone versions of Porter5 are available at http://distilldeep.ucd.ie/porter/.
oAdaBoost: An AdaBoost Variant for Ordinal Classification
Authors: Joao Costa, Jaime S. Cardoso
Date: 2015-01-12 00:00:00
Proceeding: ICPRAM 2015 1
Tasks: Classification, General Classification, Information Retrieval, Multi-Label Classification, Ordinal Classification, Retrieval

Ordinal data classification (ODC) has a wide range of applications in areas where human evaluation plays an important role, ranging from psychology and medicine to information retrieval.  In ODC the output variable has a natural order; however, there is not a precise notion of the distance between classes.  The Data Replication Method was proposed as tool for solving the ODC problem using a single binary classifier.  Due to its characteristics, the Data Replication Method is straightforwardly mapped into methods that optimize the decision function globally.  However, the mapping process is not applicable when the methods construct the decision function locally and iteratively, like decision trees and AdaBoost (with decision stumps).  In this paper we adapt the Data Replication Method for AdaBoost, by softening the constraints resulting from the data replication process. Experimental comparison with state-of-the-art AdaBoost variants in synthetic and real data show the advantages of our proposal.
Side Information Fusion for Recommender Systems over Heterogeneous Information Network
Authors: Huan Zhao, Quanming Yao, Yangqiu Song, James Kwok, Dik Lun Lee
Date: 2018-01-08 00:00:00
Tasks: Collaborative Filtering, Recommendation Systems

Collaborative filtering (CF) has been one of the most important and popular recommendation methods, which aims at predicting users' preferences (ratings) based on their past behaviors. Recently, various types of side information beyond the explicit ratings users give to items, such as social connections among users and metadata of items, have been introduced into CF and shown to be useful for improving recommendation performance. However, previous works process different types of information separately, thus failing to capture the correlations that might exist across them. To address this problem, in this work, we study the application of heterogeneous information network (HIN) to enhance CF-based recommendation methods. However, we face challenging issues in HIN-based recommendation, i.e., how to capture similarities of complex semantics between users and items in a HIN, and how to effectively fuse these similarities to improve final recommendation performance. To address these issues, we apply metagraph to similarity computation and solve the information fusion problem with a ``matrix factorization (MF) + factorization machine (FM)'' framework. For the MF part, we obtain the user-item similarity matrix from each metagraph and then apply low-rank matrix approximation to obtain latent features for both users and items. For the FM part, we apply FM with Group lasso (FMG) on the features obtained from the MF part to train the recommending model and, at the same time, identify the useful metagraphs. Besides FMG, a two-stage method, we further propose an end-to-end method, hierarchical attention fusing (HAF), to fuse metagraph based similarities for the final recommendation. Experimental results on four large real-world datasets show that the two proposed frameworks significantly outperform existing state-of-the-art methods in terms of recommendation performance.
From Neural Re-Ranking to Neural Ranking: Learning a Sparse Representation for Inverted Indexing
Authors: Hamed Zamani, Mostafa Dehghani, W. Bruce Croft, Erik Learned-Miller, and Jaap Kamps
Date: 2018-10-22 00:00:00
Proceeding: 27th ACM International Conference on Information and Knowledge Management (CIKM '18) 2018 10
Tasks: Ad-Hoc Information Retrieval, Blocking, Information Retrieval, Re-Ranking, Retrieval

The availability of massive data and computing power allowing for effective data driven neural approaches is having a major impact
on machine learning and information retrieval research, but these models have a basic problem with efficiency. Current neural ranking models are implemented as multistage rankers: for efficiency reasons, the neural model only re-ranks the top ranked documents retrieved by a first-stage efficient ranker in response to a given query. Neural ranking models learn dense representations causing essentially every query term to match every document term, making it highly inefficient or intractable to rank the whole collection. The reliance on a first stage ranker creates a dual problem: First, the interaction and combination effects are not well understood. Second, the first stage ranker serves as a “gate-keeper” or filter, effectively blocking the potential of neural models to uncover new relevant documents.
In this work, we propose a standalone neural ranking model (SNRM) by introducing a sparsity property to learn a latent sparse representation for each query and document. This representation captures the semantic relationship between the query and documents, but is also sparse enough to enable constructing an inverted index for the whole collection. We parameterize the sparsity of the model to yield a retrieval model as efficient as conventional term based models. Our model gains in efficiency without loss of effectiveness: it not only outperforms the existing term matching baselines, but also performs similarly to the recent re-ranking based neural models with dense representations. Our model can also take advantage of pseudo-relevance feedback for further improvements. More generally, our results demonstrate the importance of sparsity in neuralIR models and show that dense representations can be pruned effectively, giving new insights about essential semantic features and their distributions.
A Network-centric Framework for Auditing Recommendation Systems
Authors: Abhisek Dash, Animesh Mukherjee, Saptarshi Ghosh
Date: 2019-02-07 00:00:00
Tasks: Recommendation Systems

To improve the experience of consumers, all social media, commerce and
entertainment sites deploy Recommendation Systems (RSs) that aim to help users
locate interesting content. These RSs are black-boxes - the way a chunk of
information is filtered out and served to a user from a large information base
is mostly opaque. No one except the parent company generally has access to the
entire information required for auditing these systems - neither the details of
the algorithm nor the user-item interactions are ever made publicly available
for third-party auditors. Hence auditing RSs remains an important challenge,
especially with the recent concerns about how RSs are affecting the views of
the society at large with new technical jargons like "echo chambers",
"confirmation biases", "filter bubbles" etc. in place. Many prior works have
evaluated different properties of RSs such as diversity, novelty, etc. However,
most of these have focused on evaluating static snapshots of RSs. Today,
auditors are not only interested in these static evaluations on a snapshot of
the system, but also interested in how these systems are affecting the society
in course of time. In this work, we propose a novel network-centric framework
which is not only able to quantify various static properties of RSs, but also
is able to quantify dynamic properties such as how likely RSs are to lead to
polarization or segregation of information among their users. We apply the
framework to several popular movie RSs to demonstrate its utility.
A Comparison of Information Retrieval Techniques for Detecting Source Code Plagiarism
Authors: Vasishtha Sriram Jayapati, Ajay Venkitaraman
Date: 2019-02-06 00:00:00
Tasks: Information Retrieval, Retrieval

Plagiarism is a commonly encountered problem in the academia. While there are
several tools and techniques to efficiently determine plagiarism in text, the
same cannot be said about source code plagiarism. To make the existing systems
more efficient, we use several information retrieval techniques to find the
similarity between source code files written in Java. We later use JPlag, which
is a string-based plagiarism detection tool used in academia to match the
plagiarized source codes. In this paper, we aim to generalize on the efficiency
and effectiveness of detecting plagiarism using different information retrieval
models rather than using just string manipulation algorithms.
Automated Brain Disorders Diagnosis Through Deep Neural Networks
Authors: Gabriel Maggiotti
Date: 2019-01-11 00:00:00
Proceeding: vixra.org 2019 1
Tasks: EEG, Electroencephalogram (EEG)

In most cases, the diagnosis of brain disorders such as epilepsy is slow and requires endless visits to doctors and EEG technicians. This project aims to automate brain disorder diagnosis by using Artificial Intelligence and deep learning. Brain could have many disorders that can be detected by reading an Electroencephalography. Using an EEG device and collecting the electrical signals directly from the brain with a non-invasive procedure gives significant information about its health. Classifying and detecting anomalies on these signals is what currently doctors do when reading an Electroencephalography. With the right amount of data and the use of Artificial Intelligence, it could be possible to learn and classify these signals into groups like (i.e: anxiety, epilepsy spikes, etc). Then, a trained Neural Network to interpret those signals and identify evidence of a disorder to finally automate the detection and classification of those disorders found.
Hierarchical RL Using an Ensemble of Proprioceptive Periodic Policies
Authors: Kenneth Marino, Abhinav Gupta, Rob Fergus, Arthur Szlam
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5
Tasks: MuJoCo

In this paper we introduce a simple, robust approach to hierarchically training an agent in the setting of sparse reward tasks.
The agent is split into a low-level and a high-level policy. The low-level policy only accesses internal, proprioceptive dimensions of the state observation. The low-level policies are trained with a simple reward that encourages changing the values of the non-proprioceptive dimensions. Furthermore, it is induced to be periodic with the use a ``phase function.'' The high-level policy is trained using a sparse, task-dependent reward, and operates by choosing which of the low-level policies to run at any given time. Using this approach, we solve difficult maze and navigation tasks with sparse rewards using the Mujoco Ant and Humanoid agents and show improvement over recent hierarchical methods.
A 3D Probabilistic Deep Learning System for Detection and Diagnosis of Lung Cancer Using Low-Dose CT Scans
Authors: Onur Ozdemir, Rebecca L. Russell, Andrew A. Berlin
Date: 2019-02-08 00:00:00
Tasks: Decision Making, Diagnostic, General Classification, Lung Nodule Detection, Probabilistic Deep Learning

We introduce a new computer aided detection and diagnosis system for lung cancer screening with low-dose CT scans that produces meaningful probability assessments. Our system is based entirely on 3D convolutional neural networks and achieves state-of-the-art performance for both lung nodule detection and malignancy classification tasks on the publicly available LUNA16 and Kaggle Data Science Bowl challenges. While nodule detection systems are typically designed and optimized on their own, we find that it is important to consider the coupling between detection and diagnosis components. Exploiting this coupling allows us to develop an end-to-end system that has higher and more robust performance and eliminates the need for a nodule detection false positive reduction stage. Furthermore, we characterize model uncertainty in our deep learning systems, a first for lung CT analysis, and show that we can use this to provide well-calibrated classification probabilities for both nodule detection and patient malignancy diagnosis. These calibrated probabilities informed by model uncertainty can be used for subsequent risk-based decision making towards diagnostic interventions or disease treatments, as we demonstrate using a probability-based patient referral strategy to further improve our results.
Adaptive and Safe Bayesian Optimization in High Dimensions via One-Dimensional Subspaces
Authors: Johannes Kirschner, Mojmír Mutný, Nicole Hiller, Rasmus Ischebeck, Andreas Krause
Date: 2019-02-08 00:00:00
Tasks: Bayesian Optimization

Bayesian optimization is known to be difficult to scale to high dimensions, because the acquisition step requires solving a non-convex optimization problem in the same search space. In order to scale the method and keep its benefits, we propose an algorithm (LineBO) that restricts the problem to a sequence of iteratively chosen one-dimensional sub-problems that can be solved efficiently. We show that our algorithm converges globally and obtains a fast local rate when the function is strongly convex. Further, if the objective has an invariant subspace, our method automatically adapts to the effective dimension without changing the algorithm. When combined with the SafeOpt algorithm to solve the sub-problems, we obtain the first safe Bayesian optimization algorithm with theoretical guarantees applicable in high-dimensional settings. We evaluate our method on multiple synthetic benchmarks, where we obtain competitive performance. Further, we deploy our algorithm to optimize the beam intensity of the Swiss Free Electron Laser with up to 40 parameters while satisfying safe operation constraints.
A Smoother Way to Train Structured Prediction Models
Authors: Krishna Pillutla, Vincent Roulet, Sham M. Kakade, Zaid Harchaoui
Date: 2019-02-08 00:00:00
Proceeding: NeurIPS 2018 12
Tasks: named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER), Object Localization, Prediction, Structured Prediction

We present a framework to train a structured prediction model by performing
smoothing on the inference algorithm it builds upon. Smoothing overcomes the
non-smoothness inherent to the maximum margin structured prediction objective,
and paves the way for the use of fast primal gradient-based optimization
algorithms. We illustrate the proposed framework by developing a novel primal
incremental optimization algorithm for the structural support vector machine.
The proposed algorithm blends an extrapolation scheme for acceleration and an
adaptive smoothing scheme and builds upon the stochastic variance-reduced
gradient algorithm. We establish its worst-case global complexity bound and
study several practical variants, including extensions to deep structured
prediction. We present experimental results on two real-world problems, namely
named entity recognition and visual object localization. The experimental
results show that the proposed framework allows us to build upon efficient
inference algorithms to develop large-scale optimization algorithms for
structured prediction which can achieve competitive performance on the two
real-world problems.