pwc-archive/papers-with-abstracts huggingface dataset:
Interpreting character embeddings with perceptual representations: the case of shape, sound, and color
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11

Character-level information is included in many NLP models, but evaluating the information encoded in character representations is an open issue. We leverage perceptual representations in the form of shape, sound, and color embeddings to investigate their correlation to textual representations in five languages. This cross-lingual analysis shows that textual character representations correlate strongly with sound representations for languages using an alphabetic script, while shape correlates with featural scripts. We further develop a set of probing classifiers to intrinsically evaluate what phonological information is encoded in character embeddings. Our results suggest that information on features such as voiceness are embedded in both LSTM and transformer-based representations.
SOS: Systematic Offensive Stereotyping Bias in Word Embeddings
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR September 2021 9
Tasks: Hate Speech Detection, Word Embeddings

Hate speech detection models aim to provide a safe environment for marginalised social groups to express themselves. However, the bias in these models could lead to silencing those groups. In this paper, we introduce the systematic offensive stereotyping (SOS) bias. We propose a method to measure the SOS bias in different word embeddings and also investigate its influence on the downstream task of hate speech detection. Our results show that SOS bias against various groups exists in widely used word embeddings and that our SOS bias metric correlates positively with the statistics of published surveys on online abuse and \added[id=fa]{extremism}. However, we found that it is not easy to prove that bias in word embeddings influences downstream task performance. Finally, we show that SOS bias is more indicative of sexism and racism in the inspected word embeddings when used for sexism and racism detection than social biases.
Ensembling and Knowledge Distilling of Large Sequence Taggers for Grammatical Error Correction
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Grammatical Error Correction

In this paper, we investigate GEC sequence tagging architecture with focusing on ensembling of the recent cutting-edge Transformers’ encoders in their Large configurations. We encourage ensembling models by majority votes on span-level edits because it's tolerant to the model architecture and vocabulary size. Our best ensemble achieves a new SOTA result, the F_0.5 score of 76.05 on BEA-2019 (test), even without pre-training on synthetic datasets. Also, we perform model distillation of a trained ensemble to generate new training synthetic datasets, "Troy-Blogs" and "Troy-1BW". Our best single sequence tagging model that is pretrained on generated Troy- datasets in combination with publicly available synthetic PIE dataset achieves a near-SOTA result of the F_0.5 score of 73.21 on BEA-2019 (test). The code, datasets, and trained models are publicly available.
"That Is a Suspicious Reaction!": Interpreting Logits Variation to Detect NLP Adversarial Attacks
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Adversarial Text

Adversarial attacks are a major challenge faced by current machine learning research. These purposely crafted inputs fool even the most advanced models, precluding their deployment in safety-critical applications. Extensive research in computer vision has been carried to develop reliable defense strategies. However, the same issue remains less explored in natural language processing. Our work presents a model-agnostic detector of adversarial text examples. The approach identifies patterns in the logits of the target classifier when perturbing the input text. The proposed detector improves the current state-of-the-art performance in recognizing adversarial inputs and exhibits strong generalization capabilities across different NLP models, datasets, and word-level attacks.
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Decoder, Document Summarization, Multi-Document Summarization, Sentence

We introduce PRIMERA, a pre-trained model for multi-document representation with a focus on summarization that reduces the need for dataset-specific architectures and large amounts of fine-tuning labeled data. PRIMERA uses our newly proposed pre-training objective designed to teach the model to connect and aggregate information across documents. It also uses efficient encoder-decoder transformers to simplify the processing of concatenated input documents. With extensive experiments on 6 multi-document summarization datasets from 3 different domains on zero-shot, few-shot and full-supervised settings, PRIMERA outperforms current state-of-the-art dataset-specific and pre-trained models on most of these settings with large margins.
E-MMAD: Multimodal Advertising Caption Generation Based on Structured Information
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Caption Generation, valid, Video Captioning

With multimodal tasks increasingly getting popular in recent years, datasets with large scale and reliable authenticity are in urgent demand. Therefore, we present an e-commercial multimodal advertising dataset, E-MMAD, which contains 120 thousand valid data elaborately picked out from 1.3 million real product examples in both Chinese and English. Noticeably, it is one of the largest  video captioning datasets in this field, in which each example has its product video (around 30 seconds), title, caption and structured information table that is observed to play a vital role in practice. We also introduce a fresh task for vision-language research based on E-MMAD: e-commercial multimodal advertising generation, which requires to use aforementioned product multimodal information to generate textual advertisement. Accordingly, we propose a baseline method on the strength of structured information reasoning to solve the demand in reality on this dataset.
Sequence-to-Sequence Knowledge Graph Completion and Question Answering
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Decoder, Graph Embedding, Knowledge Graph Completion, Knowledge Graph Embedding, Link Prediction, Question Answering

Knowledge graph embedding (KGE) models represent each entity and relation of a knowledge graph (KG) with low-dimensional embedding vectors. These methods have recently been applied to KG link prediction and question answering over incomplete KGs (KGQA). KGEs typically create an embedding for each entity in the graph, which results in large model sizes on real-world graphs with millions of entities. Their atomic entity representation also necessitates a multi-stage approach to downstream tasks, which limits their utility. We show that an off-the-shelf encoder-decoder Transformer model can serve as a scalable and versatile KGE model obtaining state-of-the-art results for KG link prediction and KGQA. We achieve this by posing KG link prediction as a sequence-to-sequence task and exchange the triple scoring approach taken by prior KGE methods with a generative decoding approach. Such a simple but powerful method reduces the model size up to 90% compared to conventional KGE models and attains the best performance among small-sized models. An ensemble with a traditional KGE model even sets a new state-of-the-art. After finetuning this model on the task of KGQA over incomplete KGs, our approach outperforms baselines on multiple large-scale datasets without extensive hyperparameter tuning.
Comparative Opinion Summarization via Collaborative Decoding
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Opinion Summarization

Opinion summarization focuses on generating summaries that reflect popular opinions of multiple reviews for a single entity (e.g., a hotel or a product.) While generated summaries offer general and concise information about a particular entity, the information may be insufficient to help the user compare multiple entities. Thus, the user may still struggle with the question ``Which one should I pick?'' 
In this paper, we propose a {\em comparative opinion summarization} task, which is to generate two contrastive summaries and one common summary from two given sets of reviews from different entities. We develop a comparative summarization framework CoCoSum, which consists of two few-shot summarization models that jointly generate contrastive and common summaries.
Experimental results on a newly created benchmark CoCoTrip show that CoCoSum can produce higher-quality contrastive and common summaries than state-of-the-art opinion summarization models.
MILLIE: Modular & Iterative Multilingual Open Information Extraction
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Open Information Extraction

Open Information Extraction (OpenIE) is the task of extracting $(subject, predicate, object)$ triples from natural language sentences. Current OpenIE systems extract all triple slots independently. In contrast, we investigate the hypothesis that it may be beneficial to extract triple slots iteratively: first extract easy slots, followed by the difficult ones by conditioning on the easy slots, and therefore achieve a better overall extraction. Based on this hypothesis, we propose a neural OpenIE system, MILLIE, that operates in an iterative fashion.  Due to the iterative nature, the system is also modular: it is possible to seamlessly integrate rule based extraction systems with a neural end-to-end system, thereby allowing rule based systems to supply extraction slots which MILLIE can leverage for extracting the remaining slots. We confirm our hypothesis empirically: MILLIE outperforms SOTA systems on multiple languages ranging from Chinese to Arabic. Additionally, we are the first to provide an OpenIE test dataset for Arabic.
Make the Best of Cross-lingual Transfer: Evidence from POS Tagging with over 100 Languages
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Cross-Lingual Transfer, Part-Of-Speech Tagging, POS, POS Tagging, Transfer Learning

Cross-lingual transfer learning with large multilingual pre-trained models can be an effective approach for low-resource languages with no labeled training data. Existing evaluations of cross-lingual generalisability of large pre-trained models use datasets with English training data, and test data in a selection of target languages. We explore a more extensive transfer learning setup with 65 different source languages and 105 target languages for  part-of-speech tagging. Through our analysis, we show that pre-training of both source and target language, as well as matching language families, writing systems, word order systems, and lexical-phonetic distance significantly impact cross-lingual performance.
Vec2Node: Self-training with Tensor Augmentation for Text Classification with Few Labels
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Data Augmentation, Tensor Decomposition, text-classification, Text Classification, Word Embeddings

Recent  advances  in  state-of-the-art  machine learning  models  like  deep  neural  networks heavily rely on large amounts of labeled training data which is difficult to obtain for many applications.  To address label scarcity, recent work has focused on data augmentation techniques  to  create  synthetic  training  data.    In this  work,  we  propose  a  novel  approach  of data augmentation leveraging tensor decomposition to generate synthetic samples by exploiting local and global information in text and reducing concept drift. We develop Vec2Node that leverages self-training from in-domain un-labeled data augmented with tensorized word embeddings  that  significantly  improves  over state-of-the-art  models,   particularly  in  low-resource settings.  For instance, with only 1% of labeled training data,Vec2Node obtains a 21.5% improvement over the base model with augmentation.  Furthermore,Vec2Node generates  interpretable  explanations  for  the  augmented data leveraging tensor embeddings
Compositional Data Augmentation for Abstractive Conversation Summarization
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Conversation Summarization, Data Augmentation, Knowledge Distillation

Recent abstractive conversation summarization systems generally rely on large-scale annotated summaries. However, collecting conversations and annotating their corresponding summaries can be time-consuming and labor-intensive. To alleviate the data scarcity issue, in this work, we present a simple yet effective compositional data augmentation method, Compo, for generating diverse and high-quality pairs of conversations and summaries. Specifically, we generate novel conversation and summary pairs through first extracting conversation snippets and summary sentences based on conversation stages and then randomly composing them constrained by the temporal relation and semantic similarities. To deal with the noises in the augmented data, we further utilize knowledge distillation to learn concise representation from a teacher model trained on high-quality data. Extensive experiments on benchmark datasets demonstrate that Compo significantly outperforms prior state-of-the-art baselines in terms of both quantitative and qualitative evaluation, and exhibits a reasonable level of interpretability.
The Case of Imperfect Negation Cues: A Two-Step Approach for Automatic Negation Scope Resolution
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Negation, Negation Scope Resolution

Neural network-based methods are the state of the art in negation scope resolution.  However, they often use the unrealistic assumption that cue information is completely accurate.  Even if  this  assumption  holds,  there  remains  a  de-pendency  on  engineered  features  from  state-of-the-art machine learning methods. The cur-rent study adopted a two-step negation resolving approach to assess whether a bidirectional long short-term memory-based method can be used for cue detection as well, and how inaccurate cue predictions would affect the scope resolution performance.   Results suggest that the  scope  resolution  performance  is  most  robust against inaccurate information for models with  a  recurrent  layer  only,  compared  to  ex-tensions with a conditional random field layer or a post-processing algorithm.  We advocate for  more  research  into  the  application  of  automated deep learning on negation cue detection and the effect of imperfect information on scope resolution.
Modeling Multidimensional Language Matrices to Learn Predictive Text
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11

The predictive text in the tray bed of the Chinese typewriter presents “Radiating style” and other important patterns, which reflect the main properties of the Chinese language. For a robot to understand these patterns like human’s “once glanced, never forgotten”, we construct multidimensional language matrices (MLM) to present the characters and/or words of predictive text for Chinese Natural Language Processing (NLP). Using 2D LM, our approach identified the core character as the prefix of radiating outward words, and as the suffix of radiating inward words to show the best distribution of the characters in a nine-grid. Using 3D LM, our approach, for robots doing as human, recognized the meaning and location of the words in a nine-grid by “Once learning mechanism”. Even though these approaches are proposed for the Chinese language, their methods are extendable to other languages.
Invariant Language Modeling
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Domain Generalization, Language Modeling, Language Modelling

Modern pretrained language models are critical components of NLP pipelines. 
Yet, they suffer from spurious correlations, poor out-of-domain generalization, and biases.
Inspired by recent progress in causal machine learning, in particular the invariant risk minimization (IRM) paradigm,
we propose invariant language modeling, a framework for learning invariant representations that generalize better across multiple environments.
In particular, we adapt a game-theoretic implementation of IRM (IRM-games)
to language models, where the invariance emerges from a specific training schedule in which all the environments compete to optimize their own environment-specific loss by updating subsets of the model in a round-robin fashion.
In a series of controlled experiments, we demonstrate the ability of our method to (i) remove structured noise, (ii) ignore specific spurious correlations without affecting global performance, and (iii) achieve better out-of-domain generalization.
These benefits come with a negligible computational overhead compared to standard training, do not require changing the local loss, and can be applied to any language model architecture.
We believe this framework is promising to help mitigate spurious correlations and biases in language models.
Improved grammatical error correction by ranking elementary edits
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Decoder, Grammatical Error Correction

We offer a rescoring method for grammatical error correction which is based on two-stage procedure: the first stage model extracts local edits and the second classiifies them as correct or false. We show how to use an encoder-decoder or sequence labeling approach as the first stage of our model. We achieve state-of-the-art quality on BEA 2019 English dataset even with a weak BERT-GEC basic model. When using a state-of-the-art GECToR edit generator and the combined scorer, our model beats GECToR on BEA 2019 by $2-3\%$. Our model also beats previous state-of-the-art on Russian, despite using smaller models and less data than the previous approaches.
Sequence-to-sequence AMR Parsing with Ancestor Information
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: AMR Parsing, Decoder, Sentence

AMR parsing is the task that maps a sentence to an AMR semantic graph automatically. The difficulty comes from generating the complex graph structure. The previous state-of-the-art method translates the AMR graph into a sequence, then directly fine-tunes a pretrained sequence-to-sequence Transformer model (BART). However, purely treating the graph as a sequence does not take advantage of structural information about the graph. In this paper, we design several strategies to add the important \textit{ancestor information} into the Transformer Decoder. Our experiments show that we can improve the performance for both AMR 2.0 and AMR 3.0 dataset and achieve new state-of-the-art results.
Pre-training Pre-trained Models with Auxiliary Labels and Fine-tuning for Text Classification
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Classification, Sentence, Sentiment Analysis, Sentiment Classification, text-classification, Text Classification

With the development of pre-trained models, the performance of text classification has been continuously improved. However, we argue that directly employing features generated by pre-trained models for text classification might fail to fully capture discriminative features. For example, in sentiment classification, both the words
``very good'' and ``I would still choose'' are indicative of the positive sentiment. Most pre-trained model based approaches pay more attention to ``very good'' while ignoring the second when both are in the same sentence. To fully capture discriminative features, in this paper, we incorporate auxiliary labels to exploit the knowledge in the pre-trained model. In specific, we pre-train a pre-trained model incorporating auxiliary labels to effectively extract the discriminative textual semantic representation. Then, the classifier is fine tuned. Moreover, multiple pre-trained models are combined to further provide the textual semantic representation. Experiments were conducted on seven classification tasks and the results show that the proposed approach outperforms several baselines.
A Natural Diet: Towards Improving Naturalness of Machine Translation Output
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Diversity, Machine Translation, Sentence, Translation

Machine translation (MT) evaluation often focuses on accuracy and fluency, without paying much attention to translation style. This means that, even when considered accurate and fluent, MT output can still sound less natural than high quality human translations or text originally written in the target language. Machine translation output notably exhibits lower lexical diversity, and employs constructs that mirror those in the source sentence. In this work we propose a method for training MT systems to achieve a more natural style, i.e. mirroring the style of text originally written in the target language. Our method tags parallel training data according to the naturalness of the target side by contrasting language models trained on natural and translated data. Tagging data allows us to put greater emphasis on target sentences originally written in the target language. Automatic metrics show that the resulting models achieve lexical richness on par with human translations, mimicking a style much closer to sentences originally written in the target language. Furthermore, we find that their output is  preferred by human experts when compared to the baseline translations.
Making Transformers Solve Compositional Tasks
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR September 2021 9
Tasks: Semantic Parsing

Several studies have reported the inability of Transformer models to generalize compositionally, a key type of generalization in many NLP tasks such as semantic parsing. In this paper we explore the design space of Transformer models showing that the inductive biases given to the model by several design decisions significantly impact compositional generalization. We identified Transformer configurations that generalize compositionally significantly better than previously reported in the literature in many compositional tasks. We achieve state-of-the-art results in a semantic parsing compositional generalization benchmark (COGS), and a string edit operation composition benchmark (PCFG).
Identifying Moments of Change from Longitudinal User Text
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11

Identifying changes in individuals' behaviour and mood, as observed via content shared on online platforms, is increasingly gaining importance. Most research to-date on this topic focuses on either: (a) identifying individuals at risk or with a certain mental health condition given a batch of posts or (b) providing equivalent labels at the post level. A disadvantage of such work is the lack of a strong temporal component and the inability to make longitudinal assessments following an individual's trajectory and allowing timely interventions. Here we define a new task, that of identifying moments of change in individuals on the basis of their shared content online. The changes we consider are sudden shifts in mood (switches) or gradual mood progression (escalations). We have created detailed guidelines for capturing moments of change and a corpus of 500 manually annotated user timelines (18.7K posts). We have developed a variety of baseline models drawing inspiration from related tasks and show that the best performance is obtained through context aware sequential modelling. We also introduce new metrics for capturing rare events in temporal windows.