pwc-archive/papers-with-abstracts huggingface dataset:
HIBRIDS: Attention with Hierarchical Biases for Structure-aware Long Document Summarization
Authors: Shuyang Cao, Lu Wang
Date: 2022-03-21 00:00:00
Proceeding: ACL 2022 5
Tasks: Articles, Document Summarization

Document structure is critical for efficient information consumption. However, it is challenging to encode it efficiently into the modern Transformer architecture. In this work, we present HIBRIDS, which injects Hierarchical Biases foR Incorporating Document Structure into the calculation of attention scores. We further present a new task, hierarchical question-summary generation, for summarizing salient content in the source document into a hierarchy of questions and summaries, where each follow-up question inquires about the content of its parent question-summary pair. We also annotate a new dataset with 6,153 question-summary hierarchies labeled on long government reports. Experiment results show that our model produces better question-summary hierarchies than comparisons on both hierarchy quality and content coverage, a finding also echoed by human judges. Additionally, our model improves the generation of long-form summaries from lengthy government reports and Wikipedia articles, as measured by ROUGE scores.
A Prompting-based Approach for Adversarial Example Generation and Robustness Enhancement
Authors: Yuting Yang, Pei Huang, Juan Cao, Jintao Li, Yun Lin, Jin Song Dong, Feifei Ma, Jian Zhang
Date: 2022-03-21 00:00:00
Tasks: Adversarial Attack

Recent years have seen the wide application of NLP models in crucial areas such as finance, medical treatment, and news media, raising concerns of the model robustness and vulnerabilities. In this paper, we propose a novel prompt-based adversarial attack to compromise NLP models and robustness enhancement technique. We first construct malicious prompts for each instance and generate adversarial examples via mask-and-filling under the effect of a malicious purpose. Our attack technique targets the inherent vulnerabilities of NLP models, allowing us to generate samples even without interacting with the victim NLP model, as long as it is based on pre-trained language models (PLMs). Furthermore, we design a prompt-based adversarial training method to improve the robustness of PLMs. As our training method does not actually generate adversarial samples, it can be applied to large-scale training sets efficiently. The experimental results show that our attack method can achieve a high attack success rate with more diverse, fluent and natural adversarial examples. In addition, our robustness enhancement method can significantly improve the robustness of models to resist adversarial attacks. Our work indicates that prompting paradigm has great potential in probing some fundamental flaws of PLMs and fine-tuning them for downstream tasks.
Better Language Model with Hypernym Class Prediction
Authors: He Bai, Tong Wang, Alessandro Sordoni, Peng Shi
Date: 2022-03-21 00:00:00
Proceeding: ACL 2022 5
Tasks: Language Modeling, Language Modelling, model, Prediction

Class-based language models (LMs) have been long devised to address context sparsity in $n$-gram LMs. In this study, we revisit this approach in the context of neural LMs. We hypothesize that class-based prediction leads to an implicit context aggregation for similar words and thus can improve generalization for rare words. We map words that have a common WordNet hypernym to the same class and train large neural LMs by gradually annealing from predicting the class to token prediction during training. Empirically, this curriculum learning strategy consistently improves perplexity over various large, highly-performant state-of-the-art Transformer-based models on two datasets, WikiText-103 and Arxiv. Our analysis shows that the performance improvement is achieved without sacrificing performance on rare words. Finally, we document other attempts that failed to yield empirical gains, and discuss future directions for the adoption of class-based LMs on a larger scale.
Who Shares Fake News? Uncovering Insights from Social Media Users' Post Histories
Authors: Verena Schoenmueller, Simon J. Blanchard, Gita V. Johar
Date: 2022-03-20 00:00:00
Tasks: Fact Checking, Misinformation

We propose that social-media users' own post histories are an underused yet valuable resource for studying fake-news sharing. By extracting textual cues from their prior posts, and contrasting their prevalence against random social-media users and others (e.g., those with similar socio-demographics, political news-sharers, and fact-check sharers), researchers can identify cues that distinguish fake-news sharers, predict those most likely to share fake news, and identify promising constructs to build interventions. Our research includes studies along these lines. In Study 1, we explore the distinctive language patterns of fake-news sharers, highlighting elements such as their higher use of anger and power-related words. In Study 2, we show that adding textual cues into predictive models enhances their accuracy in predicting fake-news sharers. In Study 3, we explore the contrasting role of trait and situational anger, and show trait anger is associated with a greater propensity to share both true and fake news. In Study 4, we introduce a way to authenticate Twitter accounts in surveys, before using it to explore how crafting an ad copy that resonates with users' sense of power encourages the adoption of fact-checking tools. We hope to encourage the use of novel research methods for marketers and misinformation researchers.
A Neural-Symbolic Approach to Natural Language Understanding
Authors: Zhixuan Liu, ZiHao Wang, Yuan Lin, Hang Li
Date: 2022-03-20 00:00:00
Tasks: Logical Reasoning, Natural Language Inference, Natural Language Understanding, Question Answering

Deep neural networks, empowered by pre-trained language models, have achieved remarkable results in natural language understanding (NLU) tasks. However, their performances can drastically deteriorate when logical reasoning is needed. This is because NLU in principle depends on not only analogical reasoning, which deep neural networks are good at, but also logical reasoning. According to the dual-process theory, analogical reasoning and logical reasoning are respectively carried out by System 1 and System 2 in the human brain. Inspired by the theory, we present a novel framework for NLU called Neural-Symbolic Processor (NSP), which performs analogical reasoning based on neural processing and logical reasoning based on both neural and symbolic processing. As a case study, we conduct experiments on two NLU tasks, question answering (QA) and natural language inference (NLI), when numerical reasoning (a type of logical reasoning) is necessary. The experimental results show that our method significantly outperforms state-of-the-art methods in both tasks.
Parallel Instance Query Network for Named Entity Recognition
Authors: Yongliang Shen, Xiaobin Wang, Zeqi Tan, Guangwei Xu, Pengjun Xie, Fei Huang, Weiming Lu, Yueting Zhuang
Date: 2022-03-20 00:00:00
Proceeding: ACL 2022 5
Tasks: Chinese Named Entity Recognition, named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER), NER, Nested Named Entity Recognition, Reading Comprehension, Sentence

Named entity recognition (NER) is a fundamental task in natural language processing. Recent works treat named entity recognition as a reading comprehension task, constructing type-specific queries manually to extract entities. This paradigm suffers from three issues. First, type-specific queries can only extract one type of entities per inference, which is inefficient. Second, the extraction for different types of entities is isolated, ignoring the dependencies between them. Third, query construction relies on external knowledge and is difficult to apply to realistic scenarios with hundreds of entity types. To deal with them, we propose Parallel Instance Query Network (PIQN), which sets up global and learnable instance queries to extract entities from a sentence in a parallel manner. Each instance query predicts one entity, and by feeding all instance queries simultaneously, we can query all entities in parallel. Instead of being constructed from external knowledge, instance queries can learn their different query semantics during training. For training the model, we treat label assignment as a one-to-many Linear Assignment Problem (LAP) and dynamically assign gold entities to instance queries with minimal assignment cost. Experiments on both nested and flat NER datasets demonstrate that our proposed method outperforms previous state-of-the-art models.
Entailment Relation Aware Paraphrase Generation
Authors: Abhilasha Sancheti, Balaji Vasan Srinivasan, Rachel Rudinger
Date: 2022-03-20 00:00:00
Tasks: Natural Language Inference, Paraphrase Generation, reinforcement-learning, Reinforcement Learning (RL), Relation

We introduce a new task of entailment relation aware paraphrase generation which aims at generating a paraphrase conforming to a given entailment relation (e.g. equivalent, forward entailing, or reverse entailing) with respect to a given input. We propose a reinforcement learning-based weakly-supervised paraphrasing system, ERAP, that can be trained using existing paraphrase and natural language inference (NLI) corpora without an explicit task-specific corpus. A combination of automated and human evaluations show that ERAP generates paraphrases conforming to the specified entailment relation and are of good quality as compared to the baselines and uncontrolled paraphrasing systems. Using ERAP for augmenting training data for downstream textual entailment task improves performance over an uncontrolled paraphrasing system, and introduces fewer training artifacts, indicating the benefit of explicit control during paraphrasing.
DEIM: An effective deep encoding and interaction model for sentence matching
Authors: Kexin Jiang, Yahui Zhao, Rongyi Cui, Zhenguo Zhang
Date: 2022-03-20 00:00:00
Tasks: Answer Selection, Natural Language Inference, Reading Comprehension, Sentence

Natural language sentence matching is the task of comparing two sentences and identifying the relationship between them.It has a wide range of applications in natural language processing tasks such as reading comprehension, question and answer systems. The main approach is to compute the interaction between text representations and sentence pairs through an attention mechanism, which can extract the semantic information between sentence pairs well. However,this kind of method can not gain satisfactory results when dealing with complex semantic features. To solve this problem, we propose a sentence matching method based on deep encoding and interaction to extract deep semantic information. In the encoder layer,we refer to the information of another sentence in the process of encoding a single sentence, and later use a heuristic algorithm to fuse the information. In the interaction layer, we use a bidirectional attention mechanism and a self-attention mechanism to obtain deep semantic information.Finally, we perform a pooling operation and input it to the MLP for classification. we evaluate our model on three tasks: recognizing textual entailment, paraphrase recognition, and answer selection. We conducted experiments on the SNLI and SciTail datasets for the recognizing textual entailment task, the Quora dataset for the paraphrase recognition task, and the WikiQA dataset for the answer selection task. The experimental results show that the proposed algorithm can effectively extract deep semantic features that verify the effectiveness of the algorithm on sentence matching tasks.
MetaOnce: A Metaverse Framework Based on Multi-scene Relations and Entity-relation-event Game
Authors: Hongyin Zhu
Date: 2022-03-20 00:00:00
Tasks: Relation

Existing metaverse systems lack rich relation types between entities and events. The challenge is that there is no portable framework to introduce rich concepts, relations, events into the metaverse. This paper introduces a new metaverse framework, MetaOnce. This framework proposes to build multi-scene graphs. This framework not only describes rich relations in a single scene but also combines multiple scene graphs into a complete graph for more comprehensive analysis and inference. Prior social network systems mainly describe friend relations. They ignore the effect of entity-relation-event games on the metaverse system and existing rule constraints. We propose a rule controller and impose constraints on the relations that allow the framework to behave in a compliant manner. We build a metaverse system to test the features of the framework, and experimental results show that our framework can build a multi-scene metaverse with memory and rule constraints.
How does the pre-training objective affect what large language models learn about linguistic properties?
Authors: Ahmed Alajrami, Nikolaos Aletras
Date: 2022-03-20 00:00:00
Proceeding: ACL 2022 5
Tasks: Language Modeling, Language Modelling, Masked Language Modeling

Several pre-training objectives, such as masked language modeling (MLM), have been proposed to pre-train language models (e.g. BERT) with the aim of learning better language representations. However, to the best of our knowledge, no previous work so far has investigated how different pre-training objectives affect what BERT learns about linguistics properties. We hypothesize that linguistically motivated objectives such as MLM should help BERT to acquire better linguistic knowledge compared to other non-linguistically motivated objectives that are not intuitive or hard for humans to guess the association between the input and the label to be predicted. To this end, we pre-train BERT with two linguistically motivated objectives and three non-linguistically motivated ones. We then probe for linguistic characteristics encoded in the representation of the resulting models. We find strong evidence that there are only small differences in probing performance between the representations learned by the two different types of objectives. These surprising results question the dominant narrative of linguistically informed pre-training.
The Online Behaviour of the Algerian Abusers in Social Media Networks
Authors: Kheireddine Abainia
Date: 2022-03-19 00:00:00
Tasks: Abuse Detection

Connecting to social media networks becomes a daily task for the majority of people around the world, and the amount of shared information is growing exponentially. Thus, controlling the way in which people communicate is necessary, in order to protect them from disorientation, conflicts, aggressions, etc. In this paper, we conduct a statistical study on the cyber-bullying and the abusive content in social media (i.e. Facebook), where we try to spot the online behaviour of the abusers in the Algerian community. More specifically, we have involved 200 Facebook users from different regions among 600 to carry out this study. The aim of this investigation is to aid automatic systems of abuse detection to take decision by incorporating the online activity. Abuse detection systems require a large amount of data to perform better on such kind of texts (i.e. unstructured and informal texts), and this is due to the lack of standard orthography, where there are various Algerian dialects and languages spoken.
Pretraining with Artificial Language: Studying Transferable Knowledge in Language Models
Authors: Ryokan Ri, Yoshimasa Tsuruoka
Date: 2022-03-19 00:00:00
Proceeding: ACL 2022 5
Tasks: Position

We investigate what kind of structural knowledge learned in neural network encoders is transferable to processing natural language. We design artificial languages with structural properties that mimic natural language, pretrain encoders on the data, and see how much performance the encoder exhibits on downstream tasks in natural language. Our experimental results show that pretraining with an artificial language with a nesting dependency structure provides some knowledge transferable to natural language. A follow-up probing analysis indicates that its success in the transfer is related to the amount of encoded contextual information and what is transferred is the knowledge of position-aware context dependence of language. Our results provide insights into how neural network encoders process human languages and the source of cross-lingual transferability of recent multilingual language models.
From meaning to perception -- exploring the space between word and odor perception embeddings
Authors: Janek Amann, Manex Agirrezabal
Date: 2022-03-19 00:00:00
Tasks: Word Embeddings

In this paper we propose the use of the Word2vec algorithm in order to obtain odor perception embeddings (or smell embeddings), only using publicly available perfume descriptions. Besides showing meaningful similarity relationships among each other, these embeddings also demonstrate to possess some shared information with their respective word embeddings. The meaningfulness of these embeddings suggests that aesthetics might provide enough constraints for using algorithms motivated by distributional semantics on non-randomly combined data. Furthermore, they provide possibilities for new ways of classifying odors and analyzing perfumes. We have also employed the embeddings in an attempt to understand the aesthetic nature of perfumes, based on the difference between real and randomly generated perfumes. In an additional tentative experiment we explore the possibility of a mapping between the word embedding space and the odor perception embedding space by fitting a regressor on the shared vocabulary and then predict the odor perception embeddings of words without an a priori associated smell, such as night or sky.
Clickbait Spoiling via Question Answering and Passage Retrieval
Authors: Matthias Hagen, Maik Fröbe, Artur Jurk, Martin Potthast
Date: 2022-03-19 00:00:00
Proceeding: ACL 2022 5
Tasks: Passage Retrieval, Question Answering, Retrieval

We introduce and study the task of clickbait spoiling: generating a short text that satisfies the curiosity induced by a clickbait post. Clickbait links to a web page and advertises its contents by arousing curiosity instead of providing an informative summary. Our contributions are approaches to classify the type of spoiler needed (i.e., a phrase or a passage), and to generate appropriate spoilers. A large-scale evaluation and error analysis on a new corpus of 5,000 manually spoiled clickbait posts -- the Webis Clickbait Spoiling Corpus 2022 -- shows that our spoiler type classifier achieves an accuracy of 80%, while the question answering model DeBERTa-large outperforms all others in generating spoilers for both types.
Dependency-based Mixture Language Models
Authors: Zhixian Yang, Xiaojun Wan
Date: 2022-03-19 00:00:00
Proceeding: ACL 2022 5
Tasks: Language Modeling, Language Modelling, Text Generation

Various models have been proposed to incorporate knowledge of syntactic structures into neural language models. However, previous works have relied heavily on elaborate components for a specific language model, usually recurrent neural network (RNN), which makes themselves unwieldy in practice to fit into other neural language models, such as Transformer and GPT-2. In this paper, we introduce the Dependency-based Mixture Language Models. In detail, we first train neural language models with a novel dependency modeling objective to learn the probability distribution of future dependent tokens given context. We then formulate the next-token probability by mixing the previous dependency modeling probability distributions with self-attention. Extensive experiments and human evaluations show that our method can be easily and effectively applied to different neural language models while improving neural text generation on various tasks.
Read Top News First: A Document Reordering Approach for Multi-Document News Summarization
Authors: Chao Zhao, Tenghao Huang, Somnath Basu Roy Chowdhury, Muthu Kumar Chandrasekaran, Kathleen McKeown, Snigdha Chaturvedi
Date: 2022-03-19 00:00:00
Proceeding: Findings (ACL) 2022 5
Tasks: Document Summarization, News Summarization

A common method for extractive multi-document news summarization is to re-formulate it as a single-document summarization problem by concatenating all documents as a single meta-document. However, this method neglects the relative importance of documents. We propose a simple approach to reorder the documents according to their relative importance before concatenating and summarizing them. The reordering makes the salient content easier to learn by the summarization model. Experiments show that our approach outperforms previous state-of-the-art methods with more complex architectures.
Similarity and Content-based Phonetic Self Attention for Speech Recognition
Authors: Kyuhong Shim, Wonyong Sung
Date: 2022-03-19 00:00:00
Tasks: speech-recognition, Speech Recognition

Transformer-based speech recognition models have achieved great success due to the self-attention (SA) mechanism that utilizes every frame in the feature extraction process. Especially, SA heads in lower layers capture various phonetic characteristics by the query-key dot product, which is designed to compute the pairwise relationship between frames. In this paper, we propose a variant of SA to extract more representative phonetic features. The proposed phonetic self-attention (phSA) is composed of two different types of phonetic attention; one is similarity-based and the other is content-based. In short, similarity-based attention captures the correlation between frames while content-based attention only considers each frame without being affected by other frames. We identify which parts of the original dot product equation are related to two different attention patterns and improve each part with simple modifications. Our experiments on phoneme classification and speech recognition show that replacing SA with phSA for lower layers improves the recognition performance without increasing the latency and the parameter size.
Meta-X$_{NLG}$: A Meta-Learning Approach Based on Language Clustering for Zero-Shot Cross-Lingual Transfer and Generation
Authors: Kaushal Kumar Maurya, Maunendra Sankar Desarkar
Date: 2022-03-19 00:00:00
Tasks: Abstractive Text Summarization, Cross-Lingual Transfer, Meta-Learning, Question Generation, Question-Generation, Text Summarization, Zero-Shot Cross-Lingual Transfer

Recently, the NLP community has witnessed a rapid advancement in multilingual and cross-lingual transfer research where the supervision is transferred from high-resource languages (HRLs) to low-resource languages (LRLs). However, the cross-lingual transfer is not uniform across languages, particularly in the zero-shot setting. Towards this goal, one promising research direction is to learn shareable structures across multiple tasks with limited annotated data. The downstream multilingual applications may benefit from such a learning setup as most of the languages across the globe are low-resource and share some structures with other languages. In this paper, we propose a novel meta-learning framework (called Meta-X$_{NLG}$) to learn shareable structures from typologically diverse languages based on meta-learning and language clustering. This is a step towards uniform cross-lingual transfer for unseen languages. We first cluster the languages based on language representations and identify the centroid language of each cluster. Then, a meta-learning algorithm is trained with all centroid languages and evaluated on the other languages in the zero-shot setting. We demonstrate the effectiveness of this modeling on two NLG tasks (Abstractive Text Summarization and Question Generation), 5 popular datasets and 30 typologically diverse languages. Consistent improvements over strong baselines demonstrate the efficacy of the proposed framework. The careful design of the model makes this end-to-end NLG setup less vulnerable to the accidental translation problem, which is a prominent concern in zero-shot cross-lingual NLG tasks.