pwc-archive/papers-with-abstracts huggingface dataset:
The statistical advantage of automatic NLG metrics at the system level
Authors: Johnny Tian-Zheng Wei, Robin Jia
Date: 2021-05-26 00:00:00
Proceeding: ACL 2021 5

Estimating the expected output quality of generation systems is central to NLG. This paper qualifies the notion that automatic metrics are not as good as humans in estimating system-level quality. Statistically, humans are unbiased, high variance estimators, while metrics are biased, low variance estimators. We compare these estimators by their error in pairwise prediction (which generation system is better?) using the bootstrap. Measuring this error is complicated: predictions are evaluated against noisy, human predicted labels instead of the ground truth, and metric predictions fluctuate based on the test sets they were calculated on. By applying a bias-variance-noise decomposition, we adjust this error to a noise-free, infinite test set setting. Our analysis compares the adjusted error of metrics to humans and a derived, perfect segment-level annotator, both of which are unbiased estimators dependent on the number of judgments collected. In MT, we identify two settings where metrics outperform humans due to a statistical advantage in variance: when the number of human judgments used is small, and when the quality difference between compared systems is small. The data and code to reproduce our analyses are available at https://github.com/johntzwei/metric-statistical-advantage .
Neural Morphology Dataset and Models for Multiple Languages, from the Large to the Endangered
Authors: Mika Hämäläinen, Niko Partanen, Jack Rueter, Khalid Alnajjar
Date: 2021-05-26 00:00:00
Proceeding: NoDaLiDa 2021 5
Tasks: Lemmatization, Morphological Analysis

We train neural models for morphological analysis, generation and lemmatization for morphologically rich languages. We present a method for automatically extracting substantially large amount of training data from FSTs for 22 languages, out of which 17 are endangered. The neural models follow the same tagset as the FSTs in order to make it possible to use them as fallback systems together with the FSTs. The source code, models and datasets have been released on Zenodo.
Joint Optimization of Tokenization and Downstream Model
Authors: Tatsuya Hiraoka, Sho Takase, Kei Uchiumi, Atsushi Keyaki, Naoaki Okazaki
Date: 2021-05-26 00:00:00
Proceeding: Findings (ACL) 2021 8
Tasks: Machine Translation, model, text-classification, Text Classification, Translation

Since traditional tokenizers are isolated from a downstream task and model, they cannot output an appropriate tokenization depending on the task and model, although recent studies imply that the appropriate tokenization improves the performance. In this paper, we propose a novel method to find an appropriate tokenization to a given downstream model by jointly optimizing a tokenizer and the model. The proposed method has no restriction except for using loss values computed by the downstream model to train the tokenizer, and thus, we can apply the proposed method to any NLP task. Moreover, the proposed method can be used to explore the appropriate tokenization for an already trained model as post-processing. Therefore, the proposed method is applicable to various situations. We evaluated whether our method contributes to improving performance on text classification in three languages and machine translation in eight language pairs. Experimental results show that our proposed method improves the performance by determining appropriate tokenizations.
Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Trigger
Authors: Fanchao Qi, Mukai Li, Yangyi Chen, Zhengyan Zhang, Zhiyuan Liu, Yasheng Wang, Maosong Sun
Date: 2021-05-26 00:00:00
Proceeding: ACL 2021 5
Tasks: Backdoor Attack

Backdoor attacks are a kind of insidious security threat against machine learning models. After being injected with a backdoor in training, the victim model will produce adversary-specified outputs on the inputs embedded with predesigned triggers but behave properly on normal inputs during inference. As a sort of emergent attack, backdoor attacks in natural language processing (NLP) are investigated insufficiently. As far as we know, almost all existing textual backdoor attack methods insert additional contents into normal samples as triggers, which causes the trigger-embedded samples to be detected and the backdoor attacks to be blocked without much effort. In this paper, we propose to use the syntactic structure as the trigger in textual backdoor attacks. We conduct extensive experiments to demonstrate that the syntactic trigger-based attack method can achieve comparable attack performance (almost 100% success rate) to the insertion-based methods but possesses much higher invisibility and stronger resistance to defenses. These results also reveal the significant insidiousness and harmfulness of textual backdoor attacks. All the code and data of this paper can be obtained at https://github.com/thunlp/HiddenKiller.
SentEmojiBot: Empathising Conversations Generation with Emojis
Authors: Akhilesh Ravi, Amit Yadav, Jainish Chauhan, Jatin Dholakia, Naman jain, Mayank Singh
Date: 2021-05-26 00:00:00

The increasing use of dialogue agents makes it extremely desirable for them to understand and acknowledge the implied emotions to respond like humans with empathy. Chatbots using traditional techniques analyze emotions based on the context and meaning of the text and lack the understanding of emotions expressed through face. Emojis representing facial expressions present a promising way to express emotions. However, none of the AI systems utilizes emojis for empathetic conversation generation. We propose, SentEmojiBot, based on the SentEmoji dataset, to generate empathetic conversations with a combination of emojis and text. Evaluation metrics show that the BERT-based model outperforms the vanilla transformer model. A user study indicates that the dialogues generated by our model were understandable and adding emojis improved empathetic traits in conversations by 9.8%
Unsupervised Pronoun Resolution via Masked Noun-Phrase Prediction
Authors: Ming Shen, Pratyay Banerjee, Chitta Baral
Date: 2021-05-26 00:00:00
Proceeding: ACL 2021 5
Tasks: Prediction, Winogrande

In this work, we propose Masked Noun-Phrase Prediction (MNPP), a pre-training strategy to tackle pronoun resolution in a fully unsupervised setting. Firstly, We evaluate our pre-trained model on various pronoun resolution datasets without any finetuning. Our method outperforms all previous unsupervised methods on all datasets by large margins. Secondly, we proceed to a few-shot setting where we finetune our pre-trained model on WinoGrande-S and XS separately. Our method outperforms RoBERTa-large baseline with large margins, meanwhile, achieving a higher AUC score after further finetuning on the remaining three official splits of WinoGrande.
Read, Listen, and See: Leveraging Multimodal Information Helps Chinese Spell Checking
Authors: Heng-Da Xu, Zhongli Li, Qingyu Zhou, Chao Li, Zizhen Wang, Yunbo Cao, Heyan Huang, Xian-Ling Mao
Date: 2021-05-26 00:00:00
Proceeding: Findings (ACL) 2021 8
Tasks: Chinese Spell Checking

Chinese Spell Checking (CSC) aims to detect and correct erroneous characters for user-generated text in the Chinese language. Most of the Chinese spelling errors are misused semantically, phonetically or graphically similar characters. Previous attempts noticed this phenomenon and try to use the similarity for this task. However, these methods use either heuristics or handcrafted confusion sets to predict the correct character. In this paper, we propose a Chinese spell checker called ReaLiSe, by directly leveraging the multimodal information of the Chinese characters. The ReaLiSe model tackles the CSC task by (1) capturing the semantic, phonetic and graphic information of the input characters, and (2) selectively mixing the information in these modalities to predict the correct output. Experiments on the SIGHAN benchmarks show that the proposed model outperforms strong baselines by a large margin.
SGPT: Semantic Graphs based Pre-training for Aspect-based Sentiment Analysis
Authors: Yong Qian, Zhongqing Wang, Rong Xiao, Chen Chen, Haihong Tang
Date: 2021-05-26 00:00:00
Tasks: Aspect-Based Sentiment Analysis, Aspect-Based Sentiment Analysis (ABSA), Language Modeling, Language Modelling, Sentiment Analysis

Previous studies show effective of pre-trained language models for sentiment analysis. However, most of these studies ignore the importance of sentimental information for pre-trained models.Therefore, we fully investigate the sentimental information for pre-trained models and enhance pre-trained language models with semantic graphs for sentiment analysis.In particular, we introduce Semantic Graphs based Pre-training(SGPT) using semantic graphs to obtain synonym knowledge for aspect-sentiment pairs and similar aspect/sentiment terms.We then optimize the pre-trained language model with the semantic graphs.Empirical studies on several downstream tasks show that proposed model outperforms strong pre-trained baselines. The results also show the effectiveness of proposed semantic graphs for pre-trained model.
Word Embedding Transformation for Robust Unsupervised Bilingual Lexicon Induction
Authors: Hailong Cao, Tiejun Zhao
Date: 2021-05-26 00:00:00
Tasks: Bilingual Lexicon Induction, Word Embeddings

Great progress has been made in unsupervised bilingual lexicon induction (UBLI) by aligning the source and target word embeddings independently trained on monolingual corpora. The common assumption of most UBLI models is that the embedding spaces of two languages are approximately isomorphic. Therefore the performance is bound by the degree of isomorphism, especially on etymologically and typologically distant languages. To address this problem, we propose a transformation-based method to increase the isomorphism. Embeddings of two languages are made to match with each other by rotating and scaling. The method does not require any form of supervision and can be applied to any language pair. On a benchmark data set of bilingual lexicon induction, our approach can achieve competitive or superior performance compared to state-of-the-art methods, with particularly strong results being found on distant languages.
Impact of detecting clinical trial elements in exploration of COVID-19 literature
Authors: Simon Šuster, Karin Verspoor, Timothy Baldwin, Jey Han Lau, Antonio Jimeno Yepes, David Martinez, Yulia Otmakhova
Date: 2021-05-25 00:00:00
Tasks: Efficient Exploration, PICO, Retrieval

The COVID-19 pandemic has driven ever-greater demand for tools which enable efficient exploration of biomedical literature. Although semi-structured information resulting from concept recognition and detection of the defining elements of clinical trials (e.g. PICO criteria) has been commonly used to support literature search, the contributions of this abstraction remain poorly understood, especially in relation to text-based retrieval. In this study, we compare the results retrieved by a standard search engine with those filtered using clinically-relevant concepts and their relations. With analysis based on the annotations from the TREC-COVID shared task, we obtain quantitative as well as qualitative insights into characteristics of relational and concept-based literature exploration. Most importantly, we find that the relational concept selection filters the original retrieved collection in a way that decreases the proportion of unjudged documents and increases the precision, which means that the user is likely to be exposed to a larger number of relevant documents.
NukeLM: Pre-Trained and Fine-Tuned Language Models for the Nuclear and Energy Domains
Authors: Lee Burke, Karl Pazdernik, Daniel Fortin, Benjamin Wilson, Rustam Goychayev, John Mattingly
Date: 2021-05-25 00:00:00
Tasks: Articles, Language Modelling, named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER), text-classification, Text Classification, Transfer Learning

Natural language processing (NLP) tasks (text classification, named entity recognition, etc.) have seen revolutionary improvements over the last few years. This is due to language models such as BERT that achieve deep knowledge transfer by using a large pre-trained model, then fine-tuning the model on specific tasks. The BERT architecture has shown even better performance on domain-specific tasks when the model is pre-trained using domain-relevant texts. Inspired by these recent advancements, we have developed NukeLM, a nuclear-domain language model pre-trained on 1.5 million abstracts from the U.S. Department of Energy Office of Scientific and Technical Information (OSTI) database. This NukeLM model is then fine-tuned for the classification of research articles into either binary classes (related to the nuclear fuel cycle [NFC] or not) or multiple categories related to the subject of the article. We show that continued pre-training of a BERT-style architecture prior to fine-tuning yields greater performance on both article classification tasks. This information is critical for properly triaging manuscripts, a necessary task for better understanding citation networks that publish in the nuclear space, and for uncovering new areas of research in the nuclear (or nuclear-relevant) domains.
IntelliCAT: Intelligent Machine Translation Post-Editing with Quality Estimation and Translation Suggestion
Authors: Dongjun Lee, Junhyeong Ahn, Heesoo Park, Jaemin Jo
Date: 2021-05-25 00:00:00
Proceeding: ACL 2021 5
Tasks: Machine Translation, Sentence, Translation

We present IntelliCAT, an interactive translation interface with neural models that streamline the post-editing process on machine translation output. We leverage two quality estimation (QE) models at different granularities: sentence-level QE, to predict the quality of each machine-translated sentence, and word-level QE, to locate the parts of the machine-translated sentence that need correction. Additionally, we introduce a novel translation suggestion model conditioned on both the left and right contexts, providing alternatives for specific words or phrases for correction. Finally, with word alignments, IntelliCAT automatically preserves the original document's styles in the translated document. The experimental results show that post-editing based on the proposed QE and translation suggestions can significantly improve translation quality. Furthermore, a user study reveals that three features provided in IntelliCAT significantly accelerate the post-editing task, achieving a 52.9\% speedup in translation time compared to translating from scratch. The interface is publicly available at https://intellicat.beringlab.com/.
Time Complexity Analysis of Randomized Search Heuristics for the Dynamic Graph Coloring Problem
Authors: Jakob Bossek, Frank Neumann, Pan Peng, Dirk Sudholt
Date: 2021-05-26 00:00:00

We contribute to the theoretical understanding of randomized search heuristics for dynamic problems. We consider the classical vertex coloring problem on graphs and investigate the dynamic setting where edges are added to the current graph. We then analyze the expected time for randomized search heuristics to recompute high quality solutions. The (1+1)~Evolutionary Algorithm and RLS operate in a setting where the number of colors is bounded and we are minimizing the number of conflicts. Iterated local search algorithms use an unbounded color palette and aim to use the smallest colors and, consequently, the smallest number of colors. We identify classes of bipartite graphs where reoptimization is as hard as or even harder than optimization from scratch, i.e., starting with a random initialization. Even adding a single edge can lead to hard symmetry problems. However, graph classes that are hard for one algorithm turn out to be easy for others. In most cases our bounds show that reoptimization is faster than optimizing from scratch. We further show that tailoring mutation operators to parts of the graph where changes have occurred can significantly reduce the expected reoptimization time. In most settings the expected reoptimization time for such tailored algorithms is linear in the number of added edges. However, tailored algorithms cannot prevent exponential times in settings where the original algorithm is inefficient.
An algorithm-based multiple detection influence measure for high dimensional regression using expectile
Authors: Amadou Barry, Nikhil Bhagwat, Bratislav Misic, Jean-Baptiste Poline, Celia M. T. Greenwood
Date: 2021-05-26 00:00:00
Tasks: regression

The identification of influential observations is an important part of data analysis that can prevent erroneous conclusions drawn from biased estimators. However, in high dimensional data, this identification is challenging. Classical and recently-developed methods often perform poorly when there are multiple influential observations in the same dataset. In particular, current methods can fail when there is masking several influential observations with similar characteristics, or swamping when the influential observations are near the boundary of the space spanned by well-behaved observations. Therefore, we propose an algorithm-based, multi-step, multiple detection procedure to identify influential observations that addresses current limitations. Our three-step algorithm to identify and capture undesirable variability in the data, $\asymMIP,$ is based on two complementary statistics, inspired by asymmetric correlations, and built on expectiles. Simulations demonstrate higher detection power than competing methods. Use of the resulting asymptotic distribution leads to detection of influential observations without the need for computationally demanding procedures such as the bootstrap. The application of our method to the Autism Brain Imaging Data Exchange neuroimaging dataset resulted in a more balanced and accurate prediction of brain maturity based on cortical thickness. See our GitHub for a free R package that implements our algorithm: \texttt{asymMIP} (\url{github.com/AmBarry/hidetify}).
Private Recommender Systems: How Can Users Build Their Own Fair Recommender Systems without Log Data?
Authors: Ryoma Sato
Date: 2021-05-26 00:00:00
Tasks: Fairness, Recommendation Systems

Fairness is a crucial property in recommender systems. Although some online services have adopted fairness aware systems recently, many other services have not adopted them yet. In this work, we propose methods to enable the users to build their own fair recommender systems. Our methods can generate fair recommendations even when the service does not (or cannot) provide fair recommender systems. The key challenge is that a user does not have access to the log data of other users or the latent representations of items. This restriction prohibits us from adopting existing methods designed for service providers. The main idea is that a user has access to unfair recommendations shown by the service provider. Our methods leverage the outputs of an unfair recommender system to construct a new fair recommender system. We empirically validate that our proposed method improves fairness substantially without harming much performance of the original unfair system.
Climate Action During COVID-19 Recovery and Beyond: A Twitter Text Mining Study
Authors: Mohammad S. Parsa, Lukasz Golab, Srinivasan Keshav
Date: 2021-05-25 00:00:00

The Coronavirus pandemic created a global crisis that prompted immediate large-scale action, including economic shutdowns and mobility restrictions. These actions have had devastating effects on the economy, but some positive effects on the environment. As the world recovers from the pandemic, we ask the following question: What is the public attitude towards climate action during COVID-19 recovery and beyond? We answer this question by analyzing discussions on the Twitter social media platform. We find that most discussions support climate action and point out lessons learned during pandemic response that can shape future climate policy, although skeptics continue to have a presence. Additionally, concerns arise in the context of climate action during the pandemic, such as mitigating the risk of COVID-19 transmission on public transit.
Exploiting Temporal Dependencies for Cross-Modal Music Piece Identification
Authors: Luis Carvalho, Gerhard Widmer
Date: 2021-05-26 00:00:00
Tasks: Retrieval

This paper addresses the problem of cross-modal musical piece identification and retrieval: finding the appropriate recording(s) from a database given a sheet music query, and vice versa, working directly with audio and scanned sheet music images. The fundamental approach to this is to learn a cross-modal embedding space with a suitable similarity structure for audio and sheet image snippets, using a deep neural network, and identifying candidate pieces by cross-modal near neighbour search in this space. However, this method is oblivious of temporal aspects of music. In this paper, we introduce two strategies that address this shortcoming. First, we present a strategy that aligns sequences of embeddings learned from sheet music scans and audio snippets. A series of experiments on whole piece and fragment-level retrieval on 24 hours worth of classical piano recordings demonstrates significant improvement. Second, we show that the retrieval can be further improved by introducing an attention mechanism to the embedding learning model that reduces the effects of tempo variations in music. To conclude, we assess the scalability of our method and discuss potential measures to make it suitable for truly large-scale applications.
A Structural Model of Business Card Exchange Networks
Authors: Juan Nelson Martínez Dahbura, Shota Komatsu, Takanori Nishida, Angelo Mele
Date: 2021-05-26 00:00:00
Tasks: Management, model

Social and professional networks affect labor market dynamics, knowledge diffusion and new business creation. To understand the determinants of how these networks are formed in the first place, we analyze a unique dataset of business cards exchanges among a sample of over 240,000 users of the multi-platform contact management and professional social networking tool for individuals Eight. We develop a structural model of network formation with strategic interactions, and we estimate users' payoffs that depend on the composition of business relationships, as well as indirect business interactions. We allow heterogeneity of users in both observable and unobservable characteristics to affect how relationships form and are maintained. The model's stationary equilibrium delivers a likelihood that is a mixture of exponential random graph models that we can characterize in closed-form. We overcome several econometric and computational challenges in estimation, by exploiting a two-step estimation procedure, variational approximations and minorization-maximization methods. Our algorithm is scalable, highly parallelizable and makes efficient use of computer memory to allow estimation in massive networks. We show that users payoffs display homophily in several dimensions, e.g. location; furthermore, users unobservable characteristics also display homophily.