pwc-archive/papers-with-abstracts huggingface dataset:
Dual Attention Model for Citation Recommendation
Authors: Yang Zhang, Qiang Ma
Date: 2020-10-01 00:00:00
Proceeding: COLING 2020 8
Tasks: Articles, Citation Recommendation, model

Based on an exponentially increasing number of academic articles, discovering and citing comprehensive and appropriate resources has become a non-trivial task. Conventional citation recommender methods suffer from severe information loss. For example, they do not consider the section of the paper that the user is writing and for which they need to find a citation, the relatedness between the words in the local context (the text span that describes a citation), or the importance on each word from the local context. These shortcomings make such methods insufficient for recommending adequate citations to academic manuscripts. In this study, we propose a novel embedding-based neural network called "dual attention model for citation recommendation (DACR)" to recommend citations during manuscript preparation. Our method adapts embedding of three dimensions of semantic information: words in the local context, structural contexts, and the section on which a user is working. A neural network is designed to maximize the similarity between the embedding of the three input (local context words, section and structural contexts) and the target citation appearing in the context. The core of the neural network is composed of self-attention and additive attention, where the former aims to capture the relatedness between the contextual words and structural context, and the latter aims to learn the importance of them. The experiments on real-world datasets demonstrate the effectiveness of the proposed approach.
WeChat Neural Machine Translation Systems for WMT20
Authors: Fandong Meng, Jianhao Yan, Yijin Liu, Yuan Gao, Xianfeng Zeng, Qinsong Zeng, Peng Li, Ming Chen, Jie zhou, Sifan Liu, Hao Zhou
Date: 2020-10-01 00:00:00
Proceeding: WMT (EMNLP) 2020 11
Tasks: Knowledge Distillation, Machine Translation, Synthetic Data Generation, Transfer Learning, Translation

We participate in the WMT 2020 shared news translation task on Chinese to English. Our system is based on the Transformer (Vaswani et al., 2017a) with effective variants and the DTMT (Meng and Zhang, 2019) architecture. In our experiments, we employ data selection, several synthetic data generation approaches (i.e., back-translation, knowledge distillation, and iterative in-domain knowledge transfer), advanced finetuning approaches and self-bleu based model ensemble. Our constrained Chinese to English system achieves 36.9 case-sensitive BLEU score, which is the highest among all submissions.
CoLAKE: Contextualized Language and Knowledge Embedding
Authors: Tianxiang Sun, Yunfan Shao, Xipeng Qiu, Qipeng Guo, Yaru Hu, Xuanjing Huang, Zheng Zhang
Date: 2020-10-01 00:00:00
Proceeding: COLING 2020 8
Tasks: Entity Embeddings, Knowledge Graph Completion, Knowledge Probing

With the emerging branch of incorporating factual knowledge into pre-trained language models such as BERT, most existing models consider shallow, static, and separately pre-trained entity embeddings, which limits the performance gains of these models. Few works explore the potential of deep contextualized knowledge representation when injecting knowledge. In this paper, we propose the Contextualized Language and Knowledge Embedding (CoLAKE), which jointly learns contextualized representation for both language and knowledge with the extended MLM objective. Instead of injecting only entity embeddings, CoLAKE extracts the knowledge context of an entity from large-scale knowledge bases. To handle the heterogeneity of knowledge context and language context, we integrate them in a unified data structure, word-knowledge graph (WK graph). CoLAKE is pre-trained on large-scale WK graphs with the modified Transformer encoder. We conduct experiments on knowledge-driven tasks, knowledge probing tasks, and language understanding tasks. Experimental results show that CoLAKE outperforms previous counterparts on most of the tasks. Besides, CoLAKE achieves surprisingly high performance on our synthetic task called word-knowledge graph completion, which shows the superiority of simultaneously contextualizing language and knowledge representation.
Workflow Provenance in the Lifecycle of Scientific Machine Learning
Authors: Renan Souza, Leonardo G. Azevedo, Vítor Lourenço, Elton Soares, Raphael Thiago, Rafael Brandão, Daniel Civitarese, Emilio Vital Brazil, Marcio Moreno, Patrick Valduriez, Marta Mattoso, Renato Cerqueira, Marco A. S. Netto
Date: 2020-09-30 00:00:00
Tasks: BIG-bench Machine Learning

Machine Learning (ML) has already fundamentally changed several businesses. More recently, it has also been profoundly impacting the computational science and engineering domains, like geoscience, climate science, and health science. In these domains, users need to perform comprehensive data analyses combining scientific data and ML models to provide for critical requirements, such as reproducibility, model explainability, and experiment data understanding. However, scientific ML is multidisciplinary, heterogeneous, and affected by the physical constraints of the domain, making such analyses even more challenging. In this work, we leverage workflow provenance techniques to build a holistic view to support the lifecycle of scientific ML. We contribute with (i) characterization of the lifecycle and taxonomy for data analyses; (ii) design principles to build this view, with a W3C PROV compliant data representation and a reference system architecture; and (iii) lessons learned after an evaluation in an Oil & Gas case using an HPC cluster with 393 nodes and 946 GPUs. The experiments show that the principles enable queries that integrate domain semantics with ML models while keeping low overhead (<1%), high scalability, and an order of magnitude of query acceleration under certain workloads against without our representation.
Meta-Consolidation for Continual Learning
Authors: K J Joseph, Vineeth N. Balasubramanian
Date: 2020-10-01 00:00:00
Proceeding: NeurIPS 2020 12
Tasks: Continual Learning

The ability to continuously learn and adapt itself to new tasks, without losing grasp of already acquired knowledge is a hallmark of biological learning systems, which current deep learning systems fall short of. In this work, we present a novel methodology for continual learning called MERLIN: Meta-Consolidation for Continual Learning. We assume that weights of a neural network $\boldsymbol \psi$, for solving task $\boldsymbol t$, come from a meta-distribution $p(\boldsymbol{\psi|t})$. This meta-distribution is learned and consolidated incrementally. We operate in the challenging online continual learning setting, where a data point is seen by the model only once. Our experiments with continual learning benchmarks of MNIST, CIFAR-10, CIFAR-100 and Mini-ImageNet datasets show consistent improvement over five baselines, including a recent state-of-the-art, corroborating the promise of MERLIN.
Detecting White Supremacist Hate Speech using Domain Specific Word Embedding with Deep Learning and BERT
Authors: Hind Saleh Alatawi, Areej Maatog Alhothali, Kawthar Mustafa Moria
Date: 2020-10-01 00:00:00
Tasks: Language Modelling

White supremacists embrace a radical ideology that considers white people superior to people of other races. The critical influence of these groups is no longer limited to social media; they also have a significant effect on society in many ways by promoting racial hatred and violence. White supremacist hate speech is one of the most recently observed harmful content on social media.Traditional channels of reporting hate speech have proved inadequate due to the tremendous explosion of information, and therefore, it is necessary to find an automatic way to detect such speech in a timely manner. This research investigates the viability of automatically detecting white supremacist hate speech on Twitter by using deep learning and natural language processing techniques. Through our experiments, we used two approaches, the first approach is by using domain-specific embeddings which are extracted from white supremacist corpus in order to catch the meaning of this white supremacist slang with bidirectional Long Short-Term Memory (LSTM) deep learning model, this approach reached a 0.74890 F1-score. The second approach is by using the one of the most recent language model which is BERT, BERT model provides the state of the art of most NLP tasks. It reached to a 0.79605 F1-score. Both approaches are tested on a balanced dataset given that our experiments were based on textual data only. The dataset was combined from dataset created from Twitter and a Stormfront dataset compiled from that white supremacist forum.
A Survey on Explainability in Machine Reading Comprehension
Authors: Mokanarangan Thayaparan, Marco Valentino, André Freitas
Date: 2020-10-01 00:00:00
Tasks: Machine Reading Comprehension, Reading Comprehension, Survey

This paper presents a systematic review of benchmarks and approaches for explainability in Machine Reading Comprehension (MRC). We present how the representation and inference challenges evolved and the steps which were taken to tackle these challenges. We also present the evaluation methodologies to assess the performance of explainable systems. In addition, we identify persisting open research questions and highlight critical directions for future work.
A Feasibility-Driven Approach to Control-Limited DDP
Authors: Carlos Mastalli, Wolfgang Merkt, Josep Marti-Saumell, Henrique Ferrolho, Joan Sola, Nicolas Mansard, Sethu Vijayakumar
Date: 2020-10-01 00:00:00
Tasks: Model Predictive Control

Differential dynamic programming (DDP) is a direct single shooting method for trajectory optimization. Its efficiency derives from the exploitation of temporal structure (inherent to optimal control problems) and explicit roll-out/integration of the system dynamics. However, it suffers from numerical instability and, when compared to direct multiple shooting methods, it has limited initialization options (allows initialization of controls, but not of states) and lacks proper handling of control constraints. In this work, we tackle these issues with a feasibility-driven approach that regulates the dynamic feasibility during the numerical optimization and ensures control limits. Our feasibility search emulates the numerical resolution of a direct multiple shooting problem with only dynamics constraints. We show that our approach (named BOX-FDDP) has better numerical convergence than BOX-DDP+ (a single shooting method), and that its convergence rate and runtime performance are competitive with state-of-the-art direct transcription formulations solved using the interior point and active set algorithms available in KNITRO. We further show that BOX-FDDP decreases the dynamic feasibility error monotonically--as in state-of-the-art nonlinear programming algorithms. We demonstrate the benefits of our approach by generating complex and athletic motions for quadruped and humanoid robots. Finally, we highlight that BOX-FDDP is suitable for model predictive control in legged robots.
High Quality Remote Sensing Image Super-Resolution Using Deep Memory Connected Network
Authors: Wenjia Xu, Guangluan Xu, Yang Wang, Xian Sun, Daoyu Lin, Yirong Wu
Date: 2020-10-01 00:00:00
Tasks: image-classification, Image Classification, Image Super-Resolution, Super-Resolution

Single image super-resolution is an effective way to enhance the spatial resolution of remote sensing image, which is crucial for many applications such as target detection and image classification. However, existing methods based on the neural network usually have small receptive fields and ignore the image detail. We propose a novel method named deep memory connected network (DMCN) based on a convolutional neural network to reconstruct high-quality super-resolution images. We build local and global memory connections to combine image detail with environmental information. To further reduce parameters and ease time-consuming, we propose downsampling units, shrinking the spatial size of feature maps. We test DMCN on three remote sensing datasets with different spatial resolution. Experimental results indicate that our method yields promising improvements in both accuracy and visual performance over the current state-of-the-art.
A Multi-modal Machine Learning Approach and Toolkit to Automate Recognition of Early Stages of Dementia among British Sign Language Users
Authors: Xing Liang, Anastassia Angelopoulou, Epaminondas Kapetanios, Bencie Woll, Reda Al-Batat, Tyron Woolfe
Date: 2020-10-01 00:00:00
Tasks: BIG-bench Machine Learning, Diagnostic

The ageing population trend is correlated with an increased prevalence of acquired cognitive impairments such as dementia. Although there is no cure for dementia, a timely diagnosis helps in obtaining necessary support and appropriate medication. Researchers are working urgently to develop effective technological tools that can help doctors undertake early identification of cognitive disorder. In particular, screening for dementia in ageing Deaf signers of British Sign Language (BSL) poses additional challenges as the diagnostic process is bound up with conditions such as quality and availability of interpreters, as well as appropriate questionnaires and cognitive tests. On the other hand, deep learning based approaches for image and video analysis and understanding are promising, particularly the adoption of Convolutional Neural Network (CNN), which require large amounts of training data. In this paper, however, we demonstrate novelty in the following way: a) a multi-modal machine learning based automatic recognition toolkit for early stages of dementia among BSL users in that features from several parts of the body contributing to the sign envelope, e.g., hand-arm movements and facial expressions, are combined, b) universality in that it is possible to apply our technique to users of any sign language, since it is language independent, c) given the trade-off between complexity and accuracy of machine learning (ML) prediction models as well as the limited amount of training and testing data being available, we show that our approach is not over-fitted and has the potential to scale up.
ISAAQ -- Mastering Textbook Questions with Pre-trained Transformers and Bottom-Up and Top-Down Attention
Authors: Jose Manuel Gomez-Perez, Raul Ortega
Date: 2020-10-01 00:00:00
Tasks: Multiple-choice, Question Answering, Reading Comprehension, Visual Question Answering, Visual Question Answering (VQA)

Textbook Question Answering is a complex task in the intersection of Machine Comprehension and Visual Question Answering that requires reasoning with multimodal information from text and diagrams. For the first time, this paper taps on the potential of transformer language models and bottom-up and top-down attention to tackle the language and visual understanding challenges this task entails. Rather than training a language-visual transformer from scratch we rely on pre-trained transformers, fine-tuning and ensembling. We add bottom-up and top-down attention to identify regions of interest corresponding to diagram constituents and their relationships, improving the selection of relevant visual information for each question and answer options. Our system ISAAQ reports unprecedented success in all TQA question types, with accuracies of 81.36%, 71.11% and 55.12% on true/false, text-only and diagram multiple choice questions. ISAAQ also demonstrates its broad applicability, obtaining state-of-the-art results in other demanding datasets.
Understanding tables with intermediate pre-training
Authors: Julian Martin Eisenschlos, Syrine Krichene, Thomas Müller
Date: 2020-10-01 00:00:00
Proceeding: Findings of the Association for Computational Linguistics 2020
Tasks: Binary Classification, Data Augmentation, Natural Language Inference, Sentence, Table-based Fact Verification

Table entailment, the binary classification task of finding if a sentence is supported or refuted by the content of a table, requires parsing language and table structure as well as numerical and discrete reasoning. While there is extensive work on textual entailment, table entailment is less well studied. We adapt TAPAS (Herzig et al., 2020), a table-based BERT model, to recognize entailment. Motivated by the benefits of data augmentation, we create a balanced dataset of millions of automatically created training examples which are learned in an intermediate step prior to fine-tuning. This new data is not only useful for table entailment, but also for SQA (Iyyer et al., 2017), a sequential table QA task. To be able to use long examples as input of BERT models, we evaluate table pruning techniques as a pre-processing step to drastically improve the training and prediction efficiency at a moderate drop in accuracy. The different methods set the new state-of-the-art on the TabFact (Chen et al., 2020) and SQA datasets.
Quantifying social organization and political polarization in online platforms
Authors: Isaac Waller, Ashton Anderson
Date: 2020-10-01 00:00:00
Tasks: Cultural Vocal Bursts Intensity Prediction

Optimism about the Internet's potential to bring the world together has been tempered by concerns about its role in inflaming the 'culture wars'. Via mass selection into like-minded groups, online society may be becoming more fragmented and polarized, particularly with respect to partisan differences. However, our ability to measure the social makeup of online communities, and in turn understand the social organization of online platforms, is limited by the pseudonymous, unstructured, and large-scale nature of digital discussion. We develop a neural embedding methodology to quantify the positioning of online communities along social dimensions by leveraging large-scale patterns of aggregate behaviour. Applying our methodology to 5.1B Reddit comments made in 10K communities over 14 years, we measure how the macroscale community structure is organized with respect to age, gender, and U.S. political partisanship. Examining political content, we find Reddit underwent a significant polarization event around the 2016 U.S. presidential election, and remained highly polarized for years afterward. Contrary to conventional wisdom, however, individual-level polarization is rare; the system-level shift in 2016 was disproportionately driven by the arrival of new and newly political users. Political polarization on Reddit is unrelated to previous activity on the platform, and is instead temporally aligned with external events. We also observe a stark ideological asymmetry, with the sharp increase in 2016 being entirely attributable to changes in right-wing activity. Our methodology is broadly applicable to the study of online interaction, and our findings have implications for the design of online platforms, understanding the social contexts of online behaviour, and quantifying the dynamics and mechanisms of online polarization.
A Niching Indicator-Based Multi-modal Many-objective Optimizer
Authors: Ryoji Tanabe, Hisao Ishibuchi
Date: 2020-10-01 00:00:00
Tasks: Diversity, Evolutionary Algorithms

Multi-modal multi-objective optimization is to locate (almost) equivalent Pareto optimal solutions as many as possible. Some evolutionary algorithms for multi-modal multi-objective optimization have been proposed in the literature. However, there is no efficient method for multi-modal many-objective optimization, where the number of objectives is more than three. To address this issue, this paper proposes a niching indicator-based multi-modal multi- and many-objective optimization algorithm. In the proposed method, the fitness calculation is performed among a child and its closest individuals in the solution space to maintain the diversity. The performance of the proposed method is evaluated on multi-modal multi-objective test problems with up to 15 objectives. Results show that the proposed method can handle a large number of objectives and find a good approximation of multiple equivalent Pareto optimal solutions. The results also show that the proposed method performs significantly better than eight multi-objective evolutionary algorithms.
Review and Analysis of Three Components of Differential Evolution Mutation Operator in MOEA/D-DE
Authors: Ryoji Tanabe, Hisao Ishibuchi
Date: 2020-10-01 00:00:00

A decomposition-based multi-objective evolutionary algorithm with a differential evolution variation operator (MOEA/D-DE) shows high performance on challenging multi-objective problems (MOPs). The DE mutation consists of three key components: a mutation strategy, an index selection method for parent individuals, and a bound-handling method. However, the configuration of the DE mutation operator that should be used for MOEA/D-DE has not been thoroughly investigated in the literature. This configuration choice confuses researchers and users of MOEA/D-DE. To address this issue, we present a review of the existing configurations of the DE mutation operator in MOEA/D-DE and systematically examine the influence of each component on the performance of MOEA/D-DE. Our review reveals that the configuration of the DE mutation operator differs depending on the source code of MOEA/D-DE. In our analysis, a total of 30 configurations (three index selection methods, two mutation strategies, and five bound handling methods) are investigated on 16 MOPs with up to five objectives. Results show that each component significantly affects the performance of MOEA/D-DE. We also present the most suitable configuration of the DE mutation operator, which maximizes the effectiveness of MOEA/D-DE.
A Supervised Machine Learning Approach for Accelerating the Design of Particulate Composites: Application to Thermal Conductivity
Authors: Mohammad Saber Hashemi, Masoud Safdari, Azadeh Sheidaei
Date: 2020-09-30 00:00:00
Tasks: BIG-bench Machine Learning

A supervised machine learning (ML) based computational methodology for the design of particulate multifunctional composite materials with desired thermal conductivity (TC) is presented. The design variables are physical descriptors of the material microstructure that directly link microstructure to the material's properties. A sufficiently large and uniformly sampled database was generated based on the Sobol sequence. Microstructures were realized using an efficient dense packing algorithm, and the TCs were obtained using our previously developed Fast Fourier Transform (FFT) homogenization method. Our optimized ML method is trained over the generated database and establishes the complex relationship between the structure and properties. Finally, the application of the trained ML model in the inverse design of a new class of composite materials, liquid metal (LM) elastomer, with desired TC is discussed. The results show that the surrogate model is accurate in predicting the microstructure behavior with respect to high-fidelity FFT simulations, and inverse design is robust in finding microstructure parameters according to case studies.