pwc-archive/papers-with-abstracts huggingface dataset:
Towards Layered Events and Schema Representations in Long Documents
Authors: Hans Ole Hatzel, Chris Biemann
Date: 2021-06-01 00:00:00
Proceeding: NAACL 2021 4
Tasks: Event Extraction

In this thesis proposal, we explore the application of event extraction to literary texts. Considering the lengths of literary documents modeling events in different granularities may be more adequate to extract meaningful information, as individual elements contribute little to the overall semantics. We adapt the concept of schemas as sequences of events all describing a single process, connected through shared participants extending it to for multiple schemas in a document. Segmentation of event sequences into schemas is approached by modeling event sequences, on such task as the narrative cloze task, the prediction of missing events in sequences. We propose building on sequences of event embeddings to form schema embeddings, thereby summarizing sections of documents using a single representation. This approach will allow for the comparisons of different sections of documents and entire literary works. Literature is a challenging domain based on its variety of genres, yet the representation of literary content has received relatively little attention.
Parallel Text Alignment and Monolingual Parallel Corpus Creation from Philosophical Texts for Text Simplification
Authors: Stefan Paun
Date: 2021-06-01 00:00:00
Proceeding: NAACL 2021 4
Tasks: Text Simplification

Text simplification is a growing field with many potential useful applications. Training text simplification algorithms generally requires a lot of annotated data, however there are not many corpora suitable for this task. We propose a new unsupervised method for aligning text based on Doc2Vec embeddings and a new alignment algorithm, capable of aligning texts at different levels. Initial evaluation shows promising results for the new approach. We used the newly developed approach to create a new monolingual parallel corpus composed of the works of English early modern philosophers and their corresponding simplified versions.
Syntax-Based Attention Masking for Neural Machine Translation
Authors: Colin McDonald, David Chiang
Date: 2021-06-01 00:00:00
Proceeding: NAACL 2021 4
Tasks: Machine Translation, Translation

We present a simple method for extending transformers to source-side trees. We define a number of masks that limit self-attention based on relationships among tree nodes, and we allow each attention head to learn which mask or masks to use. On translation from English to various low-resource languages, and translation in both directions between English and German, our method always improves over simple linearization of the source-side parse tree and almost always improves over a sequence-to-sequence baseline, by up to +2.1 BLEU.
Shuffled-token Detection for Refining Pre-trained RoBERTa
Authors: Subhadarshi Panda, Anjali Agrawal, Jeewon Ha, Benjamin Bloch
Date: 2021-06-01 00:00:00
Proceeding: NAACL 2021 4
Tasks: Sentence

State-of-the-art transformer models have achieved robust performance on a variety of NLP tasks. Many of these approaches have employed domain agnostic pre-training tasks to train models that yield highly generalized sentence representations that can be fine-tuned for specific downstream tasks. We propose refining a pre-trained NLP model using the objective of detecting shuffled tokens. We use a sequential approach by starting with the pre-trained RoBERTa model and training it using our approach. Applying random shuffling strategy on the word-level, we found that our approach enables the RoBERTa model achieve better performance on 4 out of 7 GLUE tasks. Our results indicate that learning to detect shuffled tokens is a promising approach to learn more coherent sentence representations.
Morphology-Aware Meta-Embeddings for Tamil
Authors: Arjun Sai Krishnan, Seyoon Ragavan
Date: 2021-06-01 00:00:00
Proceeding: NAACL 2021 4
Tasks: Word Embeddings

In this work, we explore generating morphologically enhanced word embeddings for Tamil, a highly agglutinative South Indian language with rich morphology that remains low-resource with regards to NLP tasks. We present here the first-ever word analogy dataset for Tamil, consisting of 4499 hand-curated word tetrads across 10 semantic and 13 morphological relation types. Using a rules-based segmenter to capture morphology as well as meta-embedding techniques, we train meta-embeddings that outperform existing baselines by 16{\%} on our analogy task and appear to mitigate a previously observed trade-off between semantic and morphological accuracy.
Parallel sentences mining with transfer learning in an unsupervised setting
Authors: Yu Sun, Shaolin Zhu, Feng Yifan, Chenggang Mi
Date: 2021-06-01 00:00:00
Proceeding: NAACL 2021 4
Tasks: Machine Translation, NMT, Transfer Learning, Translation

The quality and quantity of parallel sentences are known as very important training data for constructing neural machine translation (NMT) systems. However, these resources are not available for many low-resource language pairs. Many existing methods need strong supervision are not suitable. Although several attempts at developing unsupervised models, they ignore the language-invariant between languages. In this paper, we propose an approach based on transfer learning to mine parallel sentences in the unsupervised setting.With the help of bilingual corpora of rich-resource language pairs, we can mine parallel sentences without bilingual supervision of low-resource language pairs. Experiments show that our approach improves the performance of mined parallel sentences compared with previous methods. In particular, we achieve excellent results at two real-world low-resource language pairs.
Hie-BART: Document Summarization with Hierarchical BART
Authors: Kazuki Akiyama, Akihiro Tamura, Takashi Ninomiya
Date: 2021-06-01 00:00:00
Proceeding: NAACL 2021 4
Tasks: Document Summarization, Machine Translation, Sentence, Translation

This paper proposes a new abstractive document summarization model, hierarchical BART (Hie-BART), which captures hierarchical structures of a document (i.e., sentence-word structures) in the BART model. Although the existing BART model has achieved a state-of-the-art performance on document summarization tasks, the model does not have the interactions between sentence-level information and word-level information. In machine translation tasks, the performance of neural machine translation models has been improved by incorporating multi-granularity self-attention (MG-SA), which captures the relationships between words and phrases. Inspired by the previous work, the proposed Hie-BART model incorporates MG-SA into the encoder of the BART model for capturing sentence-word structures. Evaluations on the CNN/Daily Mail dataset show that the proposed Hie-BART model outperforms some strong baselines and improves the performance of a non-hierarchical BART model (+0.23 ROUGE-L).
Knowledge Router: Learning Disentangled Representations for Knowledge Graphs
Authors: Shuai Zhang, Xi Rao, Yi Tay, Ce Zhang
Date: 2021-06-01 00:00:00
Proceeding: NAACL 2021 4
Tasks: Knowledge Graphs, Representation Learning

The design of expressive representations of entities and relations in a knowledge graph is an important endeavor. While many of the existing approaches have primarily focused on learning from relational patterns and structural information, the intrinsic complexity of KG entities has been more or less overlooked. More concretely, we hypothesize KG entities may be more complex than we think, i.e., an entity may wear many hats and relational triplets may form due to more than a single reason. To this end, this paper proposes to learn disentangled representations of KG entities - a new method that disentangles the inner latent properties of KG entities. Our disentangled process operates at the graph level and a neighborhood mechanism is leveraged to disentangle the hidden properties of each entity. This disentangled representation learning approach is model agnostic and compatible with canonical KG embedding approaches. We conduct extensive experiments on several benchmark datasets, equipping a variety of models (DistMult, SimplE, and QuatE) with our proposed disentangling mechanism. Experimental results demonstrate that our proposed approach substantially improves performance on key metrics.
Abstract Meaning Representation Guided Graph Encoding and Decoding for Joint Information Extraction
Authors: Zixuan Zhang, Heng Ji
Date: 2021-06-01 00:00:00
Proceeding: NAACL 2021 4
Tasks: Abstract Meaning Representation, Decoder, Semantic Parsing

The tasks of Rich Semantic Parsing, such as Abstract Meaning Representation (AMR), share similar goals with Information Extraction (IE) to convert natural language texts into structured semantic representations. To take advantage of such similarity, we propose a novel AMR-guided framework for joint information extraction to discover entities, relations, and events with the help of a pre-trained AMR parser. Our framework consists of two novel components: 1) an AMR based semantic graph aggregator to let the candidate entity and event trigger nodes collect neighborhood information from AMR graph for passing message among related knowledge elements; 2) an AMR guided graph decoder to extract knowledge elements based on the order decided by the hierarchical structures in AMR. Experiments on multiple datasets have shown that the AMR graph encoder and decoder have provided significant gains and our approach has achieved new state-of-the-art performance on all IE subtasks.
Event Time Extraction and Propagation via Graph Attention Networks
Authors: Haoyang Wen, Yanru Qu, Heng Ji, Qiang Ning, Jiawei Han, Avi Sil, Hanghang Tong, Dan Roth
Date: 2021-06-01 00:00:00
Proceeding: NAACL 2021 4
Tasks: Graph Attention, Natural Language Understanding, Sentence, slot-filling, Slot Filling

Grounding events into a precise timeline is important for natural language understanding but has received limited attention in recent work. This problem is challenging due to the inherent ambiguity of language and the requirement for information propagation over inter-related events. This paper first formulates this problem based on a 4-tuple temporal representation used in entity slot filling, which allows us to represent fuzzy time spans more conveniently. We then propose a graph attention network-based approach to propagate temporal information over document-level event graphs constructed by shared entity arguments and temporal relations. To better evaluate our approach, we present a challenging new benchmark on the ACE2005 corpus, where more than 78{\%} of events do not have time spans mentioned explicitly in their local contexts. The proposed approach yields an absolute gain of 7.0{\%} in match rate over contextualized embedding approaches, and 16.3{\%} higher match rate compared to sentence-level manual event time argument annotation.
Data Filtering using Cross-Lingual Word Embeddings
Authors: Christian Herold, Jan Rosendahl, Joris Vanvinckenroye, Hermann Ney
Date: 2021-06-01 00:00:00
Proceeding: NAACL 2021 4
Tasks: Cross-Lingual Word Embeddings, Language Identification, Machine Translation, Word Embeddings

Data filtering for machine translation (MT) describes the task of selecting a subset of a given, possibly noisy corpus with the aim to maximize the performance of an MT system trained on this selected data. Over the years, many different filtering approaches have been proposed. However, varying task definitions and data conditions make it difficult to draw a meaningful comparison. In the present work, we aim for a more systematic approach to the task at hand. First, we analyze the performance of language identification, a tool commonly used for data filtering in the MT community and identify specific weaknesses. Based on our findings, we then propose several novel methods for data filtering, based on cross-lingual word embeddings. We compare our approaches to one of the winning methods from the WMT 2018 shared task on parallel corpus filtering on three real-life, high resource MT tasks. We find that said method, which was performing very strong in the WMT shared task, does not perform well within our more realistic task conditions. While we find that our approaches come out at the top on all three tasks, different variants perform best on different tasks. Further experiments on the WMT 2020 shared task for parallel corpus filtering show that our methods achieve comparable results to the strongest submissions of this campaign.
Counterfactual Data Augmentation for Neural Machine Translation
Authors: Qi Liu, Matt Kusner, Phil Blunsom
Date: 2021-06-01 00:00:00
Proceeding: NAACL 2021 4
Tasks: counterfactual, Data Augmentation, Language Modeling, Language Modelling, Machine Translation, Translation

We propose a data augmentation method for neural machine translation. It works by interpreting language models and phrasal alignment causally. Specifically, it creates augmented parallel translation corpora by generating (path-specific) counterfactual aligned phrases. We generate these by sampling new source phrases from a masked language model, then sampling an aligned counterfactual target phrase by noting that a translation language model can be interpreted as a Gumbel-Max Structural Causal Model (Oberst and Sontag, 2019). Compared to previous work, our method takes both context and alignment into account to maintain the symmetry between source and target sequences. Experiments on IWSLT{'}15 English → Vietnamese, WMT{'}17 English → German, WMT{'}18 English → Turkish, and WMT{'}19 robust English → French show that the method can improve the performance of translation, backtranslation and translation robustness.
Cultural and Geographical Influences on Image Translatability of Words across Languages
Authors: Nikzad Khani, Isidora Tourni, Mohammad Sadegh Rasooli, Chris Callison-Burch, Derry Tanti Wijaya
Date: 2021-06-01 00:00:00
Proceeding: NAACL 2021 4
Tasks: Cultural Vocal Bursts Intensity Prediction, Low Resource Neural Machine Translation, Low-Resource Neural Machine Translation, Machine Translation, Multilingual NLP, Multimodal Machine Translation, NMT, Translation

Neural Machine Translation (NMT) models have been observed to produce poor translations when there are few/no parallel sentences to train the models. In the absence of parallel data, several approaches have turned to the use of images to learn translations. Since images of words, e.g., horse may be unchanged across languages, translations can be identified via images associated with words in different languages that have a high degree of visual similarity. However, translating via images has been shown to improve upon text-only models only marginally. To better understand when images are useful for translation, we study image translatability of words, which we define as the translatability of words via images, by measuring intra- and inter-cluster similarities of image representations of words that are translations of each other. We find that images of words are not always invariant across languages, and that language pairs with shared culture, meaning having either a common language family, ethnicity or religion, have improved image translatability (i.e., have more similar images for similar words) compared to its converse, regardless of their geographic proximity. In addition, in line with previous works that show images help more in translating concrete words, we found that concrete words have improved image translatability compared to abstract ones.
EnSidNet: Enhanced Hybrid Siamese-Deep Network for grouping clinical trials into drug-development pathways
Authors: Lucia Pagani
Date: 2021-06-01 00:00:00
Proceeding: NAACL 2021 4

Siamese Neural Networks have been widely used to perform similarity classification in multi-class settings. Their architecture can be used to group the clinical trials belonging to the same drug-development pathway along the several clinical trial phases. Here we present an approach for the unmet need of drug-development pathway reconstruction, based on an Enhanced hybrid Siamese-Deep Neural Network (EnSidNet). The proposed model demonstrates significant improvement above baselines in a 1-shot evaluation setting and in a classical similarity setting. EnSidNet can be an essential tool in a semi-supervised learning environment: by selecting clinical trials highly likely to belong to the same drug-development pathway it is possible to speed up the labelling process of human experts, allowing the check of a consistent volume of data, further used in the model{'}s training dataset.
SGL: Speaking the Graph Languages of Semantic Parsing via Multilingual Translation
Authors: Luigi Procopio, Rocco Tripodi, Roberto Navigli
Date: 2021-06-01 00:00:00
Proceeding: NAACL 2021 4
Tasks: AMR Parsing, Machine Translation, Semantic Parsing, Translation, UCCA Parsing

Graph-based semantic parsing aims to represent textual meaning through directed graphs. As one of the most promising general-purpose meaning representations, these structures and their parsing have gained a significant interest momentum during recent years, with several diverse formalisms being proposed. Yet, owing to this very heterogeneity, most of the research effort has focused mainly on solutions specific to a given formalism. In this work, instead, we reframe semantic parsing towards multiple formalisms as Multilingual Neural Machine Translation (MNMT), and propose SGL, a many-to-many seq2seq architecture trained with an MNMT objective. Backed by several experiments, we show that this framework is indeed effective once the learning procedure is enhanced with large parallel corpora coming from Machine Translation: we report competitive performances on AMR and UCCA parsing, especially once paired with pre-trained architectures. Furthermore, we find that models trained under this configuration scale remarkably well to tasks such as cross-lingual AMR parsing: SGL outperforms all its competitors by a large margin without even explicitly seeing non-English to AMR examples at training time and, once these examples are included as well, sets an unprecedented state of the art in this task. We release our code and our models for research purposes at https://github.com/SapienzaNLP/sgl.
Unifying Cross-Lingual Semantic Role Labeling with Heterogeneous Linguistic Resources
Authors: Simone Conia, Andrea Bacciu, Roberto Navigli
Date: 2021-06-01 00:00:00
Proceeding: NAACL 2021 4
Tasks: Semantic Role Labeling, Sentence, Translation, Word Alignment

While cross-lingual techniques are finding increasing success in a wide range of Natural Language Processing tasks, their application to Semantic Role Labeling (SRL) has been strongly limited by the fact that each language adopts its own linguistic formalism, from PropBank for English to AnCora for Spanish and PDT-Vallex for Czech, inter alia. In this work, we address this issue and present a unified model to perform cross-lingual SRL over heterogeneous linguistic resources. Our model implicitly learns a high-quality mapping for different formalisms across diverse languages without resorting to word alignment and/or translation techniques. We find that, not only is our cross-lingual system competitive with the current state of the art but that it is also robust to low-data scenarios. Most interestingly, our unified model is able to annotate a sentence in a single forward pass with all the inventories it was trained with, providing a tool for the analysis and comparison of linguistic theories across different languages. We release our code and model at https://github.com/SapienzaNLP/unify-srl.
Text Generation from Discourse Representation Structures
Authors: Jiangming Liu, Shay B. Cohen, Mirella Lapata
Date: 2021-06-01 00:00:00
Proceeding: NAACL 2021 4
Tasks: Text Generation

We propose neural models to generate text from formal meaning representations based on Discourse Representation Structures (DRSs). DRSs are document-level representations which encode rich semantic detail pertaining to rhetorical relations, presupposition, and co-reference within and across sentences. We formalize the task of neural DRS-to-text generation and provide modeling solutions for the problems of condition ordering and variable naming which render generation from DRSs non-trivial. Our generator relies on a novel sibling treeLSTM model which is able to accurately represent DRS structures and is more generally suited to trees with wide branches. We achieve competitive performance (59.48 BLEU) on the GMB benchmark against several strong baselines.
Implicitly Abusive Language -- What does it actually look like and why are we not getting there?
Authors: Michael Wiegand, Josef Ruppenhofer, Elisabeth Eder
Date: 2021-06-01 00:00:00
Proceeding: NAACL 2021 4
Tasks: Abusive Language, Position

Abusive language detection is an emerging field in natural language processing which has received a large amount of attention recently. Still the success of automatic detection is limited. Particularly, the detection of implicitly abusive language, i.e. abusive language that is not conveyed by abusive words (e.g. dumbass or scum), is not working well. In this position paper, we explain why existing datasets make learning implicit abuse difficult and what needs to be changed in the design of such datasets. Arguing for a divide-and-conquer strategy, we present a list of subtypes of implicitly abusive language and formulate research tasks and questions for future research.
The Importance of Modeling Social Factors of Language: Theory and Practice
Authors: Dirk Hovy, Diyi Yang
Date: 2021-06-01 00:00:00
Proceeding: NAACL 2021 4

Natural language processing (NLP) applications are now more powerful and ubiquitous than ever before. With rapidly developing (neural) models and ever-more available data, current NLP models have access to more information than any human speaker during their life. Still, it would be hard to argue that NLP models have reached human-level capacity. In this position paper, we argue that the reason for the current limitations is a focus on information content while ignoring language{'}s social factors. We show that current NLP systems systematically break down when faced with interpreting the social factors of language. This limits applications to a subset of information-related tasks and prevents NLP from reaching human-level performance. At the same time, systems that incorporate even a minimum of social factors already show remarkable improvements. We formalize a taxonomy of seven social factors based on linguistic theory and exemplify current failures and emerging successes for each of them. We suggest that the NLP community address social factors to get closer to the goal of human-like language understanding.
On learning and representing social meaning in NLP: a sociolinguistic perspective
Authors: Dong Nguyen, Laura Rosseel, Jack Grieve
Date: 2021-06-01 00:00:00
Proceeding: NAACL 2021 4
Tasks: Representation Learning

The field of NLP has made substantial progress in building meaning representations. However, an important aspect of linguistic meaning, social meaning, has been largely overlooked. We introduce the concept of social meaning to NLP and discuss how insights from sociolinguistics can inform work on representation learning in NLP. We also identify key challenges for this new line of research.