pwc-archive/papers-with-abstracts huggingface dataset:
Learning to Acquire Knowledge from a Search Engine for Dialogue Response Generation
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Articles, Chatbot, Response Generation

Knowledge-aided dialogue response generation aims at augmenting chatbots with relevant external knowledge in the hope of generating more informative responses.
The majority of previous work assumes that the relevant knowledge is given as input or retrieved from a static pool of knowledge. However, this assumption violates the real-world situation, where knowledge is continually updated and a chatbot has to \emph{dynamically} retrieve useful knowledge.
In this paper, we propose a dialogue model that can access the vast and dynamic information from any search engine for response generation. To this end, we design a query producer that generates queries from a dialogue context to interact with a search engine. The query producer is trained without any human annotation of gold queries, making it easily transferable to other domains and search engines. More specifically, we design a reinforcement learning algorithm to train the query producer, where rewards are obtained by comparing retrieved articles and gold responses. 
Experiments show that our query producer can achieve R@$1$ and R@$5$ rates of 62.4\% and 74.8\% for retrieving gold knowledge, and the overall model generates better responses over a strong BART (Lewis et al., 2020) model and other typical baselines.
Re-Examining System-Level Correlations of Automatic Summarization Evaluation Metrics
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10

How reliably an automatic summarization evaluation metric replicates human judgments of summary quality is quantified by system-level correlations. We identify two ways in which the definition of the system-level correlation is inconsistent with how metrics are used to evaluate systems in practice and propose changes to rectify this disconnect. First, we calculate the system score for an automatic metric using the full test set instead of the subset of summaries annotated by humans, which is currently standard practice. We demonstrate how this small change leads to more precise estimates of system-level correlations. Second, we propose to calculate correlations only on pairs of systems which are separated by differences in automatic scores that are commonly used to argue one system is of higher quality. This allows us to demonstrate that our best estimate of the correlation of ROUGE to human judgments is near 0 in realistic scenarios. Finally, the results from both analyses point to the need for future research to focus on developing more consistent and reliable human evaluations of summaries.
A Federated Approach to Predict Emojis in Hindi Tweets
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Federated Learning, Privacy Preserving

The use of emojis provide for adding a visual modality to textual communication.The task of predicting emojis however provides a challenge for computational approaches as emoji use tends to cluster into the frequently used and the rarely used emojis. Much of the research on emoji use has focused on high resource languages and conceptualised the task of predicting emojis around traditional servers-side machine learning approaches, which can introduce privacy concerns, as user data is transmitted to a central storage. In this paper, we provide a benchmark dataset of $118$k tweets for emoji prediction in Hindi.Specifically, we show that a privacy preserving approach, Federated Learning exhibits comparable performance to traditional servers-side transformer models.
Best K-best: Efficient Item Selection for Rapid Data Annotation
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10

We wish to leverage low-resource parsers in a human-in-the-loop process for rapidly increasing available training data. Historically, constructing rich natural language interfaces has been enabled through large annotated datasets with complex annotations. Research has shifted to developing low resource semantic parsers that make more efficient use of limited training data. While it is significant that initial parsing solutions are now viable based on limited examples, these parsers are not state of the art: the goal remains to enable highly accurate and rich interactions, the sort supported by earlier models when provided sufficient data. Here we investigate how to improve upon the simple idea of selecting new training examples from a list of model predictions. What makes for the "best $K$-best" solution for minimizing the effort of a human annotator, and maximizing the amount of new training data we can collect on a fixed budget?
DialogueScript: Using Dialogue Agents to Produce a Script
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Natural Language Inference

We present a novel approach to generating scripts by using agents with different personality types. To manage character interaction in the script, we employ simulated dramatic networks. Automatic and human evaluation on multiple criteria shows that our approach outperforms a vanilla-GPT2-based baseline. We further introduce a new metric to evaluate dialogue consistency based on natural language inference and demonstrate its validity.
Semantic Tokenizer for Enhanced Natural Language Processing
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Sentence, Sentence Embeddings

Traditionally, NLP performance improvement has been focused on improving models and increasing the number of parameters. Little attention has been paid to vocabulary optimization. We present a novel tokenizer that uses semantics to drive subword formation. 
The tokenizer includes a trainer that uses stemming to enhance subword formation. Further optimizations and adaptations are implemented to minimize the number of words that cannot be encoded. The encoder is updated to integrate with the trainer. The tokenizer is implemented as a drop-in replacement for the SentencePiece tokenizer.
The new tokenizer more than doubles the number of wordforms represented in the vocabulary. The enhanced vocabulary significantly improves model convergence, quality of word and sentence embeddings. Our experimental results show top performance on two Glue tasks using BERT-base, improving on models more than $20\times$ in size.
WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Cross-Lingual Transfer, Word Embeddings

Recently, large pretrained language models (LMs) have gained popularity. Training these models requires ever more computational resources and most of the existing models are trained on English text only. It is exceedingly expensive to train these models in other languages. To alleviate this problem, we introduce a method – called WECHSEL – to transfer English models to new languages. We exchange the tokenizer of the English model to a tokenizer in the target language and initialize token embeddings such that they are close to semantically similar English tokens by utilizing multilingual static word embeddings covering English and the target language. We use WECHSEL to transfer GPT-2 and RoBERTa models to 4 other languages (French, German, Chinese and Swahili). WECHSEL improves over a previously proposed method for cross-lingual parameter transfer and outperforms models of comparable size trained from scratch in the target language with up to 64x less training effort. Our method makes training large language models for new languages more accessible and less damaging to the environment. We make our code and models publicly available.
Hallucinated but Factual! Inspecting the Factuality of Hallucinations in Abstractive Summarization
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Abstractive Text Summarization, Reinforcement Learning (RL), World Knowledge

State-of-the-art abstractive summarization systems often generate hallucinations; i.e., content that is not directly inferable from the source text. Despite being assumed to be incorrect, we find that much hallucinated content is actually consistent with world knowledge, which we call factual hallucinations. Including these factual hallucinations in a summary can be beneficial because they provide useful background information. 
In this work, we propose a novel detection approach that separates factual from non-factual hallucinations of entities. 
Our method is based on an entity's prior and posterior probabilities according to pre-trained and finetuned masked language models, respectively. Empirical results suggest that our method vastly outperforms two baselines in both accuracy and F1 scores and has a strong correlation with human judgments on factuality classification tasks.
Furthermore, we use our method as a reward signal to train a summarization system using an off-line reinforcement learning (RL) algorithm that can significantly improve the factuality of generated summaries while maintaining the level of abstractiveness.
Gendered Language in Resumes
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10

Despite growing concerns around gender bias in NLP models used in algorithmic hiring, there is little empirical work studying the extent and nature of gendered language in resumes.
Using a corpus of 709k resumes from IT firms, we train a series of models to classify the gender of the applicant, thereby measuring the extent of gendered information encoded in resumes.
We also investigate whether it is possible to obfuscate gender from resumes by removing gender identifiers, removing gender sub-space in embedding models, etc.
We find that there is a significant amount of gendered information in resumes even after obfuscation.
A simple Tf-Idf model can learn to classify gender with AUROC=0.75, and more sophisticated transformer-based models achieve AUROC=0.8.
We further find that gender predictive values have little correlation with gender direction of embeddings -- meaning that, what is predictive of gender is not necessarily ``gendered'' in the masculine/feminine sense.
We discuss the implications of these findings in the algorithmic hiring context.
TableFormer: Robust Transformer Modeling for Table-Text Encoding
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Natural Language Understanding

Understanding tables is an important aspect of natural language understanding. Existing models for table understanding require linearization of the table structure, where row or column order is encoded as an unwanted bias. Such spurious biases make the model vulnerable to row and column order perturbations. Additionally, prior work has not explicitly modeled the table structure, hindering the table-text modeling ability. In this work, we propose a robust and structurally aware table-text encoding architecture TableFormer, where tabular structural biases are incorporated completely through learnable attention biases. TableFormer is strictly invariant to row and column orders, and could understand tables better due to its tabular inductive biases. Our evaluations showed that TableFormer outperforms strong baselines in all settings on SQA, WTQ and TabFact table reasoning datasets, and achieves state-of-the-art performance on SQA, especially when facing answer-invariant row and column order perturbations (6% improvement over the best baseline), because previous SOTA models' performance drops by 4% - 6% when facing such perturbations while TableFormer is not affected.
Slangvolution: A Causal Analysis of Semantic Change and Frequency Dynamics in Slang
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Causal Discovery, Causal Inference

Words are not static in their usage and meaning, but evolve over time. An interesting phenomenon in languages is slang, which is an informal language that is considered ephemeral and is often associated with contemporary trends. In this work, we study the semantic change and relative frequency shift of slang words and compare this change with standard, nonslang words. To measure semantic change, we obtain contextualized representations of words, reduce their dimensionality and propose a metric to measure their average pairwise distances between two time periods. We apply causal discovery algorithms and causal inference to uncover the dynamics of language evolution and measure the effect that word type (slang/nonslang) has on both semantic change and frequency shift, as well as its relationship to absolute frequency and polysemy. Our causal analysis shows that slang words undergo less semantic change even though they have larger frequency shifts over time.
SPE: Symmetrical Prompt Enhancement for Factual Knowledge Retrieval
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Retrieval

Pretrained language models (PLMs) have been shown to accumulate factual knowledge from their unsupervised pretraining procedures  (Petroni et al., 2019). Prompting is an effective way to query such knowledge from PLMs. Recently, continuous prompt methods have been shown to have a larger potential than discrete prompt methods in generating effective queries (Liu et al., 2021a). However, these methods do not consider symmetry of the task. In this work, we propose Symmetrical Prompt Enhancement (SPE), a continuous prompt-based method for fact retrieval that leverages the symmetry of the task. Our results on LAMA, a popular fact retrieval dataset, show significant improvement of SPE over previous prompt methods.
Dataset Geography: Mapping Language Data to Language Users
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Diversity

As language technologies become more ubiquitous, there are increasing efforts towards expanding the language diversity and coverage of natural language processing (NLP) systems. Arguably, the most important factor influencing the quality of modern NLP systems is data availability. In this work, we study the geographical representativeness of NLP datasets, aiming to quantify if and by how much do NLP datasets match the expected needs of the language speakers. In doing so, we use entity recognition and linking systems, also making important observations about their cross-lingual consistency and giving suggestions for more robust evaluation. Last, we explore some geographical and economic factors that may explain the observed dataset distributions.
Joint Content-Context Analysis of Scientific Publications: Identifying Opportunities for Collaboration in Cognitive Science
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Community Detection

This work studies publications in cognitive science and utilizes natural language processing and graph theoretical techniques to connect the analysis of the papers' content (abstracts) to the context (citation, journals). We apply hierarchical topic modeling on the abstracts and community detection algorithms on the citation network, and measure content-context discrepancy to find academic fields that study similar topics but do not cite each other or publish in the same venues. These results show a promising, systemic framework to identify opportunities for scientific collaboration in highly interdisciplinary fields such as cognitive science and machine learning.
Re-evaluating Extreme Multi-label Text Classification Methods in Tail Label Prediction
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Multi Label Text Classification, Multi-Label Text Classification, text-classification, Text Classification

Extreme multi-label text classification (XMTC) is the task of tagging each document with the relevant labels in a very large set of predefined category labels. The most challenging part of the problem is due to a highly skewed label distribution where the majority of the categories (namely the tail labels) have very few training instances. Recent benchmark evaluations
have focused on micro-averaging metrics, where the performance on tail labels can be easily overshadowed by that on the
high-frequency labels (namely the head labels). This paper presents a re-evaluation of state-of-the-art (SOTA) methods based on the binned macro-averaging F1 instead, revealing new insights into the strengths and weaknesses of representative methods, especially in tail label prediction.
Ditch the Gold Standard: Re-evaluating Conversational Question Answering
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Conversational Question Answering, Question Answering, Question Rewriting

Conversational question answering (CQA) systems aim to provide natural-language answers to users in information-seeking conversations. Existing benchmarks compare CQA models on pre-collected human-human conversations, with ground-truth answers provided in conversational history. It remains unclear whether we can rely on this static evaluation for model development, or current systems can well generalize to real-world human-machine conversations. In this work, we conduct the first large-scale human evaluation of state-of-the-art CQA systems, where human evaluators converse with models and judge the correctness of their answers. We ﬁnd that the distribution of human-machine conversations drastically differs from that of human-human conversations, and evaluations using gold answers are inconsistent with human evaluations. We further investigate how to improve automatic evaluations and propose a question rewriting mechanism based on predicted history, which better correlates with human judgments. Finally, we analyze the impact of various modeling strategies. We hope that our ﬁndings can shed light on how to develop better CQA systems in the future.
Event Detection for Suicide Understanding
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Event Detection

Suicide is a serious problem in every society. Understanding life events of a potential patient is essential for successful suicide-risk assessment and prevention. In this work, we focus on the Event Detection (ED) task to identify event trigger words of suicide-related events in public posts of discussion forums. In particular, we introduce a new dataset for ED (called SuicideED) that features seven suicidal event types to comprehensively capture suicide actions and ideation, and general risk and protective factors. Our experiments with current state-of-the-art ED systems suggest that there is still room for improvement of ED models in this domain. We will publicly release SuicideED to support future research in this important area.
A Word is Worth A Thousand Dollars: Adversarial Attack on Tweets Fools Meme Stock Prediction
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Adversarial Attack, Combinatorial Optimization, Stock Prediction

More and more investors and machine learning models rely on social media (e.g., Twitter and Reddit) to gather information and predict certain stocks' prices (meme stock). However, text-based models are known to be vulnerable to adversarial attacks, but whether stock prediction models have similar adversarial vulnerability is underexplored.
In this paper, we experiment with a variety of adversarial attack configurations to fool three stock prediction victim models (StockNet, FinGRU, FinLSTM). We address the task of adversarial generation by solving combinatorial optimization problems with semantics and budget constraints. Our results show that the proposed attack method can achieve consistent success rates, with capabilities of causing thousands of dollars loss (with Long-Only Buy-Hold-Sell investing strategy) by simply concatenating a perturbed but semantically similar tweet.
Learning to Prioritize: Precision-Driven Sentence Filtering for Long Text Summarization
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Articles, Sentence, Text Summarization

Neural text summarization has shown great potential in recent years. However, current state-of-the-art summarization models are limited by their maximum input length, posing a challenge to summarize longer texts comprehensively. As part of a layered summarization architecture, we introduce PureText, a simple yet effective precision-driven sentence filtering layer that learns to remove low-quality sentences in texts to improve existing summarization models. When evaluated on popular datasets like WikiHow and Reddit TIFU, we show up to 3 and 8 point Rouge-1 absolute improvement on the full test set and the long article subset, respectively, for state-of-the-art summarization models such as BertSum and Bart. Our approach provides downstream models with higher-quality sentences for summarization, improving overall model performance, especially on long text articles.
Hierarchical Transformer Networks for Long-sequence and Multiple Clinical Documents Classification
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Sentence

We present a Hierarchical Transformer Network for modeling long-term dependencies across clinical notes for the purpose of patient-level prediction. The network is equipped with three levels of Transformer-based encoders to learn progressively from words to sentences, sentences to notes, and finally notes to patients. The first level from word to sentence directly applies a pre-trained BERT model as a fully trainable component. While the second and third levels both implement a stack of transformer-based encoders, before the final patient representation is fed into a classification layer for clinical predictions. Compared to conventional BERT models, our model increases the maximum input length from 512 tokens to much longer sequences that are appropriate for modeling large numbers of clinical notes.
We empirically examine different hyper-parameters to identify an optimal trade-off given computational resource limits. Our experiment results on the MIMIC-III dataset for different prediction tasks demonstrate that the proposed Hierarchical Transformer Network outperforms previous state-of-the-art models, including but not limited to BigBird.
Can Synthetic Translations Improve Bitext Quality?
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Data Augmentation, NMT

Synthetic translations have been used for a wide range of NLP tasks primarily as a means of data augmentation. This work explores instead, how we can use synthetic translations to selectively replace potentially imperfect reference translations in mined bitext. We find that synthetic samples can improve bitext quality without any additional bilingual supervision, when they replace the originals based on a semantic equivalence classifier that helps mitigate NMT noise. The improved quality of the revised bitext is confirmed intrinsically via human evaluation and extrinsically through bilingual induction and MT tasks.
About Time: Do Transformers Learn Temporal Verbal Aspect?
Authors: Anonymous
Date: 2021-10-16 00:00:00
Proceeding: ACL ARR October 2021 10

Aspect is a linguistic concept that describes how an action, event, or state of a verb phrase is situated in time. In this paper, we explore whether different transformer models are capable of identifying aspectual features. We focus on two specific aspectual features: telicity and duration. Telicity marks whether the verb's action or state has an endpoint or not (telic/atelic), and duration denotes whether a verb expresses an action (dynamic) or a state (stative). These features are integral to the interpretation of natural language, but also hard to annotate and identify with NLP methods. Our results show that transformer models adequately capture information on telicity and duration in their vectors, even in their pretrained forms, but are somewhat biased with regard to verb tense and word order.