pwc-archive/papers-with-abstracts huggingface dataset:
SENT: Sentence-level Distant Relation Extraction via Negative Training
Authors: Ruotian Ma, Tao Gui, Linyang Li, Qi Zhang, Yaqian Zhou, Xuanjing Huang
Date: 2021-06-22 00:00:00
Proceeding: ACL 2021 5
Tasks: Relation, Relation Extraction, Sentence

Distant supervision for relation extraction provides uniform bag labels for each sentence inside the bag, while accurate sentence labels are important for downstream applications that need the exact relation type. Directly using bag labels for sentence-level training will introduce much noise, thus severely degrading performance. In this work, we propose the use of negative training (NT), in which a model is trained using complementary labels regarding that ``the instance does not belong to these complementary labels". Since the probability of selecting a true label as a complementary label is low, NT provides less noisy information. Furthermore, the model trained with NT is able to separate the noisy data from the training data. Based on NT, we propose a sentence-level framework, SENT, for distant relation extraction. SENT not only filters the noisy data to construct a cleaner dataset, but also performs a re-labeling process to transform the noisy data into useful training data, thus further benefiting the model's performance. Experimental results show the significant improvement of the proposed method over previous methods on sentence-level evaluation and de-noise effect.
BARTScore: Evaluating Generated Text as Text Generation
Authors: Weizhe Yuan, Graham Neubig, PengFei Liu
Date: 2021-06-22 00:00:00
Proceeding: NeurIPS 2021 12
Tasks: Decoder, Informativeness, Machine Translation, Text Generation, Text Summarization, Translation

A wide variety of NLP applications, such as machine translation, summarization, and dialog, involve text generation. One major challenge for these applications is how to evaluate whether such generated texts are actually fluent, accurate, or effective. In this work, we conceptualize the evaluation of generated text as a text generation problem, modeled using pre-trained sequence-to-sequence models. The general idea is that models trained to convert the generated text to/from a reference output or the source text will achieve higher scores when the generated text is better. We operationalize this idea using BART, an encoder-decoder based pre-trained model, and propose a metric BARTScore with a number of variants that can be flexibly applied in an unsupervised fashion to evaluation of text from different perspectives (e.g. informativeness, fluency, or factuality). BARTScore is conceptually simple and empirically effective. It can outperform existing top-scoring metrics in 16 of 22 test settings, covering evaluation of 16 datasets (e.g., machine translation, text summarization) and 7 different perspectives (e.g., informativeness, factuality). Code to calculate BARTScore is available at https://github.com/neulab/BARTScore, and we have released an interactive leaderboard for meta-evaluation at http://explainaboard.nlpedia.ai/leaderboard/task-meval/ on the ExplainaBoard platform, which allows us to interactively understand the strengths, weaknesses, and complementarity of each metric.
Fine-tune the Entire RAG Architecture (including DPR retriever) for Question-Answering
Authors: Shamane Siriwardhana, Rivindu Weerasekera, Elliott Wen, Suranga Nanayakkara
Date: 2021-06-22 00:00:00
Tasks: Open-Domain Question Answering, Question Answering, RAG, Retrieval

In this paper, we illustrate how to fine-tune the entire Retrieval Augment Generation (RAG) architecture in an end-to-end manner. We highlighted the main engineering challenges that needed to be addressed to achieve this objective. We also compare how end-to-end RAG architecture outperforms the original RAG architecture for the task of question answering. We have open-sourced our implementation in the HuggingFace Transformers library.
A Comprehensive Comparison of Pre-training Language Models
Authors: Tong Guo
Date: 2021-06-22 00:00:00

Recently, the development of pre-trained language models has brought natural language processing (NLP) tasks to the new state-of-the-art. In this paper we explore the efficiency of various pre-trained language models. We pre-train a list of transformer-based models with the same amount of text and the same training steps. The experimental results shows that the most improvement upon the origin BERT is adding the RNN-layer to capture more contextual information for short text understanding. But the conclusion is: There are no remarkable improvement for short text understanding for similar BERT structures. Data-centric method[12] can achieve better performance.
A Survey of Race, Racism, and Anti-Racism in NLP
Authors: Anjalie Field, Su Lin Blodgett, Zeerak Waseem, Yulia Tsvetkov
Date: 2021-06-21 00:00:00
Proceeding: ACL 2021 5
Tasks: Survey

Despite inextricable ties between race and language, little work has considered race in NLP research and development. In this work, we survey 79 papers from the ACL anthology that mention race. These papers reveal various types of race-related bias in all stages of NLP model development, highlighting the need for proactive consideration of how NLP systems can uphold racial hierarchies. However, persistent gaps in research on race and NLP remain: race has been siloed as a niche topic and remains ignored in many NLP tasks; most work operationalizes race as a fixed single-dimensional variable with a ground-truth label, which risks reinforcing differences produced by historical racism; and the voices of historically marginalized people are nearly absent in NLP literature. By identifying where and how NLP literature has and has not considered race, especially in comparison to related fields, our work calls for inclusion and racial justice in NLP research practices.
Deep Learning Models in Detection of Dietary Supplement Adverse Event Signals from Twitter
Authors: Yefeng Wang, Yunpeng Zhao, Jiang Bian, Rui Zhang
Date: 2021-06-21 00:00:00
Tasks: Deep Learning, Relation Extraction, Word Embeddings

Objective: The objective of this study is to develop a deep learning pipeline to detect signals on dietary supplement-related adverse events (DS AEs) from Twitter. Material and Methods: We obtained 247,807 tweets ranging from 2012 to 2018 that mentioned both DS and AE. We annotated biomedical entities and relations on 2,000 randomly selected tweets. For the concept extraction task, we compared the performance of traditional word embeddings with SVM, CRF and LSTM-CRF classifiers to BERT models. For the relation extraction task, we compared GloVe vectors with CNN classifiers to BERT models. We chose the best performing models in each task to assemble an end-to-end deep learning pipeline to detect DS AE signals and compared the results to the known DS AEs from a DS knowledge base (i.e., iDISK). Results: In both tasks, the BERT-based models outperformed traditional word embeddings. The best performing concept extraction model is the BioBERT model that can identify supplement, symptom, and body organ entities with F1-scores of 0.8646, 0.8497, and 0.7104, respectively. The best performing relation extraction model is the BERT model that can identify purpose and AE relations with F1-scores of 0.8335 and 0.7538, respectively. The end-to-end pipeline was able to extract DS indication and DS AEs with an F1-score of 0.7459 and 0,7414, respectively. Comparing to the iDISK, we could find both known and novel DS-AEs. Conclusion: We have demonstrated the feasibility of detecting DS AE signals from Twitter with a BioBERT-based deep learning pipeline.
Online Ordering Platform City Distribution Based on Genetic Algorithm
Authors: Yu Du
Date: 2021-06-22 00:00:00

Since the rising of the takeaway ordering platform, the M platform has taken the lead in the industry with its high-quality service. The increasing order volume leads the competition between platforms to reduce the distribution cost, which increases rapidly because of the unreasonable distribution route. By analyzing platform distribution's current situation, we study the vehicle routing problem of urban distribution on the M platform and minimize the distribution cost. Considering the constraints of the customer's expected delivery time and vehicle condition, we combine the different arrival times of the vehicle routing problem model using three soft time windows and solve the problem using a genetic algorithm (GA). The results show that our model and algorithm can design the vehicle path superior to the original model in terms of distribution cost and delivery time, thus providing decision support for the M platform to save distribution cost in urban distribution in the future.
Deep Phasor Networks: Connecting Conventional and Spiking Neural Networks
Authors: Wilkie Olin-Ammentorp, Maxim Bazhenov
Date: 2021-06-15 00:00:00

In this work, we extend standard neural networks by building upon an assumption that neuronal activations correspond to the angle of a complex number lying on the unit circle, or 'phasor.' Each layer in such a network produces new activations by taking a weighted superposition of the previous layer's phases and calculating the new phase value. This generalized architecture allows models to reach high accuracy and carries the singular advantage that mathematically equivalent versions of the network can be executed with or without regard to a temporal variable. Importantly, the value of a phase angle in the temporal domain can be sparsely represented by a periodically repeating series of delta functions or 'spikes'. We demonstrate the atemporal training of a phasor network on standard deep learning tasks and show that these networks can then be executed in either the traditional atemporal domain or spiking temporal domain with no conversion step needed. This provides a novel basis for constructing deep networkswhich operate via temporal, spike-based calculations suitable for neuromorphic computing hardware.
Any equation is a forest: Symbolic genetic algorithm for discovering open-form partial differential equations (SGA-PDE)
Authors: Yuntian Chen, Yingtao Luo, Qiang Liu, Hao Xu, Dongxiao Zhang
Date: 2021-06-09 00:00:00
Tasks: Form

Partial differential equations (PDEs) are concise and understandable representations of domain knowledge, which are essential for deepening our understanding of physical processes and predicting future responses. However, the PDEs of many real-world problems are uncertain, which calls for PDE discovery. We propose the symbolic genetic algorithm (SGA-PDE) to discover open-form PDEs directly from data without prior knowledge about the equation structure. SGA-PDE focuses on the representation and optimization of PDE. Firstly, SGA-PDE uses symbolic mathematics to realize the flexible representation of any given PDE, transforms a PDE into a forest, and converts each function term into a binary tree. Secondly, SGA-PDE adopts a specially designed genetic algorithm to efficiently optimize the binary trees by iteratively updating the tree topology and node attributes. The SGA-PDE is gradient-free, which is a desirable characteristic in PDE discovery since it is difficult to obtain the gradient between the PDE loss and the PDE structure. In the experiment, SGA-PDE not only successfully discovered nonlinear Burgers' equation, Korteweg-de Vries (KdV) equation, and Chafee-Infante equation, but also handled PDEs with fractional structure and compound functions that cannot be solved by conventional PDE discovery methods.
MONCAE: Multi-Objective Neuroevolution of Convolutional Autoencoders
Authors: Daniel Dimanov, Emili Balaguer-Ballester, Colin Singleton, Shahin Rostami
Date: 2021-06-07 00:00:00
Tasks: AutoML, image-classification, Image Classification, Image Compression, Neural Architecture Search

In this paper, we present a novel neuroevolutionary method to identify the architecture and hyperparameters of convolutional autoencoders. Remarkably, we used a hypervolume indicator in the context of neural architecture search for autoencoders, for the first time to our current knowledge. Results show that images were compressed by a factor of more than 10, while still retaining enough information to achieve image classification for the majority of the tasks. Thus, this new approach can be used to speed up the AutoML pipeline for image compression.
Selecting Miners within Blockchain-based Systems Using Evolutionary Algorithms for Energy Optimisation
Authors: Akram Alofi, Mahmoud A. Bokhari, Robert Hendley, Rami Bahsoon
Date: 2021-06-06 00:00:00
Tasks: Evolutionary Algorithms

In this paper, we represent the problem of selecting miners within a blockchain-based system as a subset selection problem. We formulate the problem of minimising blockchain energy consumption as an optimisation problem with two conflicting objectives: energy consumption and trust. The proposed model is compared across different algorithms to demonstrate its performance.
Speed Benchmarking of Genetic Programming Frameworks
Authors: Francisco Baeta, João Correia, Tiago Martins, Penousal Machado
Date: 2021-05-25 00:00:00
Tasks: Benchmarking

Genetic Programming (GP) is known to suffer from the burden of being computationally expensive by design. While, over the years, many techniques have been developed to mitigate this issue, data vectorization, in particular, is arguably still the most attractive strategy due to the parallel nature of GP. In this work, we employ a series of benchmarks meant to compare both the performance and evolution capabilities of different vectorized and iterative implementation approaches across several existing frameworks. Namely, TensorGP, a novel open-source engine written in Python, is shown to greatly benefit from the TensorFlow library to accelerate the domain evaluation phase in GP. The presented performance benchmarks demonstrate that the TensorGP engine manages to pull ahead, with relative speedups above two orders of magnitude for problems with a higher number of fitness cases. Additionally, as a consequence of being able to compute larger domains, we argue that TensorGP performance gains aid the discovery of more accurate candidate solutions.
Heterogeneous Treatment Effects in Regression Discontinuity Designs
Authors: Ágoston Reguly
Date: 2021-06-22 00:00:00
Tasks: regression

The paper proposes a supervised machine learning algorithm to uncover treatment effect heterogeneity in classical regression discontinuity (RD) designs. Extending Athey and Imbens (2016), I develop a criterion for building an honest "regression discontinuity tree", where each leaf of the tree contains the RD estimate of a treatment (assigned by a common cutoff rule) conditional on the values of some pre-treatment covariates. It is a priori unknown which covariates are relevant for capturing treatment effect heterogeneity, and it is the task of the algorithm to discover them, without invalidating inference. I study the performance of the method through Monte Carlo simulations and apply it to the data set compiled by Pop-Eleches and Urquiola (2013) to uncover various sources of heterogeneity in the impact of attending a better secondary school in Romania.
Solving Stochastic Optimization with Expectation Constraints Efficiently by a Stochastic Augmented Lagrangian-Type Algorithm
Authors: Liwei Zhang, Yule Zhang, Jia Wu, Xiantao Xiao
Date: 2021-06-22 00:00:00
Tasks: Stochastic Optimization

This paper considers the problem of minimizing a convex expectation function with a set of inequality convex expectation constraints. We present a computable stochastic approximation type algorithm, namely the stochastic linearized proximal method of multipliers, to solve this convex stochastic optimization problem. This algorithm can be roughly viewed as a hybrid of stochastic approximation and the traditional proximal method of multipliers. Under mild conditions, we show that this algorithm exhibits $O(K^{-1/2})$ expected convergence rates for both objective reduction and constraint violation if parameters in the algorithm are properly chosen, where $K$ denotes the number of iterations. Moreover, we show that, with high probability, the algorithm has $O(\log(K)K^{-1/2})$ constraint violation bound and $O(\log^{3/2}(K)K^{-1/2})$ objective bound. Some preliminary numerical results demonstrate the performance of the proposed algorithm.
Robust and Heterogenous Odds Ratio: Estimating Price Sensitivity for Unbought Items
Authors: Jean Pauphilet
Date: 2021-06-21 00:00:00
Tasks: Causal Inference, Imputation, Management, Sensitivity

Problem definition: Mining for heterogeneous responses to an intervention is a crucial step for data-driven operations, for instance to personalize treatment or pricing. We investigate how to estimate price sensitivity from transaction-level data. In causal inference terms, we estimate heterogeneous treatment effects when (a) the response to treatment (here, whether a customer buys a product) is binary, and (b) treatment assignments are partially observed (here, full information is only available for purchased items). Methodology/Results: We propose a recursive partitioning procedure to estimate heterogeneous odds ratio, a widely used measure of treatment effect in medicine and social sciences. We integrate an adversarial imputation step to allow for robust estimation even in presence of partially observed treatment assignments. We validate our methodology on synthetic data and apply it to three case studies from political science, medicine, and revenue management. Managerial Implications: Our robust heterogeneous odds ratio estimation method is a simple and intuitive tool to quantify heterogeneity in patients or customers and personalize interventions, while lifting a central limitation in many revenue management data.
Turing Award elites revisited: patterns of productivity, collaboration, authorship and impact
Authors: Yinyu Jin, Sha Yuan, Zhou Shao, Wendy Hall, Jie Tang
Date: 2021-06-22 00:00:00
Tasks: Articles

The Turing Award is recognized as the most influential and prestigious award in the field of computer science(CS). With the rise of the science of science (SciSci), a large amount of bibliographic data has been analyzed in an attempt to understand the hidden mechanism of scientific evolution. These include the analysis of the Nobel Prize, including physics, chemistry, medicine, etc. In this article, we extract and analyze the data of 72 Turing Award laureates from the complete bibliographic data, fill the gap in the lack of Turing Award analysis, and discover the development characteristics of computer science as an independent discipline. First, we show most Turing Award laureates have long-term and high-quality educational backgrounds, and more than 61% of them have a degree in mathematics, which indicates that mathematics has played a significant role in the development of computer science. Secondly, the data shows that not all scholars have high productivity and high h-index; that is, the number of publications and h-index is not the leading indicator for evaluating the Turing Award. Third, the average age of awardees has increased from 40 to around 70 in recent years. This may be because new breakthroughs take longer, and some new technologies need time to prove their influence. Besides, we have also found that in the past ten years, international collaboration has experienced explosive growth, showing a new paradigm in the form of collaboration. It is also worth noting that in recent years, the emergence of female winners has also been eye-catching. Finally, by analyzing the personal publication records, we find that many people are more likely to publish high-impact articles during their high-yield periods.
Quantifying the Impact of Human Capital, Job History, and Language Factors on Job Seniority with a Large-scale Analysis of Resumes
Authors: Austin P Wright, Caleb Ziems, Haekyu Park, Jon Saad-Falcon, Duen Horng Chau, Diyi Yang, Maria Tomprou
Date: 2021-06-15 00:00:00

As job markets worldwide have become more competitive and applicant selection criteria have become more opaque, and different (and sometimes contradictory) information and advice is available for job seekers wishing to progress in their careers, it has never been more difficult to determine which factors in a r\'esum\'e most effectively help career progression. In this work we present a novel, large scale dataset of over half a million r\'esum\'es with preliminary analysis to begin to answer empirically which factors help or hurt people wishing to transition to more senior roles as they progress in their career. We find that previous experience forms the most important factor, outweighing other aspects of human capital, and find which language factors in a r\'esum\'e have significant effects. This lays the groundwork for future inquiry in career trajectories using large scale data analysis and natural language processing techniques.
A systems framework for remedying dysfunction in U.S. democracy
Authors: Samuel S. -H. Wang, Jonathan Cervas, Bernard Grofman, Keena Lipsitz
Date: 2021-06-22 00:00:00

Democracy often fails to meet its ideals, and these failures may be made worse by electoral institutions. Unwanted outcomes include polarized institutions, unresponsive representatives, and the ability of a faction of voters to gain power at the expense of the majority. Various reforms have been proposed to address these problems, but their effectiveness is difficult to predict against a backdrop of complex interactions. Here we outline a path for systems-level modeling to help understand and optimize repairs to U.S. democracy. Following the tradition of engineering and biology, models of systems include mechanisms with dynamical properties that include nonlinearities and amplification (voting rules), positive feedback mechanisms (single-party control, gerrymandering), negative feedback (checks and balances), integration over time (lifetime judicial appointments), and low dimensionality (polarization). To illustrate a systems-level approach we analyze three emergent phenomena: low dimensionality, elite polarization, and anti-majoritarianism in legislatures. In each case, long-standing rules now contribute to undesirable outcomes as a consequence of changes in the political environment. Theoretical understanding at a general level will also help evaluate whether a proposed reform's benefits will materialize and be lasting, especially as conditions change again. In this way, rigorous modeling may not only shape new lines of research, but aid in the design of effective and lasting reform.