pwc-archive/papers-with-abstracts huggingface dataset:
UserIdentifier: Implicit User Representations for Simple and Effective Personalized Sentiment Analysis
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Few-Shot Learning, Meta-Learning, Sentiment Analysis

Conventionally trained classification models are trained to be as generalizable as possible, with user invariance considered desirable since the models are shared across multitudes of users.  As such, these models are often unable to produce personalized responses for individual users, based on their data.  Contrary to widely-used personalization techniques based on few-shot and meta learning, we  propose UserIdentifier, a novel scheme for training a single shared model for all users. Our approach produces personalized responses by prepending a fixed, user-specific non-trainable string (called ``user identifier'') to each user's input text. Unlike prior work, this method doesn't need any additional model parameters, any extra rounds of personal few-shot learning or any change made to the vocabulary. We empirically study different types of user identifiers (numeric, alphanumeric and also randomly generated) and demonstrate that, surprisingly, randomly generated user identifiers outperform the prefix-tuning based state-of-the-art approach by up to 13%, on a suite of sentiment analysis datasets.
Boosting coherence of language models
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Text Generation

Naturality of long-term information structure -- coherence -- remains a challenge in language generation. Large language models have insufficiently learned such structure, as their long-form generations differ from natural text in measures of coherence. To alleviate this divergence, we propose coherence boosting, an inference procedure that increases the effect of distant context on next-token prediction. We show the benefits of coherence boosting with pretrained models by distributional analyses of generated ordinary text and dialog responses. We also find that coherence boosting with state-of-the-art models for various zero-shot NLP tasks yields performance gains with no additional training.
Graph Refinement for Coreference Resolution
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: coreference-resolution, Coreference Resolution

The state-of-the-art models for coreference resolution are based on independent mention pair-wise decisions. We propose a modelling approach that learns coreference at the document-level and takes global decisions. For this purpose, we model coreference links in a graph structure where the nodes are tokens in the text, and the edges represent the relationship between them. Our model predicts the graph in a non-autoregressive manner, then iteratively refines it based on previous predictions, allowing global dependencies between decisions. The experimental results show improvements over various baselines, reinforcing the hypothesis that document-level information improves conference resolution.
A Generative Language Model for Few-shot Aspect-Based Sentiment Analysis
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Aspect-Based Sentiment Analysis, Aspect-Based Sentiment Analysis (ABSA), Intent Detection, Language Modeling, Language Modelling, Sentiment Analysis, SST-2, Text Generation

Sentiment analysis is an important task in natural language processing. In recent works, pre-trained language models are often used to achieve state-of-the-art results, especially when training data is scarce. It is common to fine-tune on the downstream task, usually by adding task-specific layers on top of the model. In this paper, we focus on aspect-based sentiment analysis, which involves extracting aspect term, category, and predicting their corresponding polarities. In particular, we are interested in few-shot settings. We propose to reformulate the extraction and prediction tasks into the sequence generation task, using a generative language model with unidirectional attention (GPT2 is used unless stated otherwise). This way, the model learns to accomplish the tasks via language generation without the need of training task-specific layers. Our evaluation results on the single-task polarity prediction show that our approach outperforms the previous state-of-the-art (based on BERT) on average performance by a large margins in few-shot and full-shot settings. More importantly, our generative approach significantly reduces the model variance caused by  low-resource data. We further demonstrate that the proposed generative language model can handle joint and multi-tasking settings, unlike previous work. We observe that the proposed sequence generation method achieves further improved performances on polarity prediction when the model is trained via joint and multi-tasking settings. Further evaluation on similar sentiment analysis datasets, SST-2, SST-5 and OOS intent detection validates the superiority and noise robustness of generative language model in few-shot settings.
Language Level Classification on German Texts using a Neural Approach
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Articles, Classification, Sentence

Studies on language level classification (LLC) for German are scarce. Of the few existing, most use a feature-engineered approach. To the best of our knowledge, there is no deep learning approach on German texts yet. This paper shows that LLC can also be successfully applied to German texts by exploiting different pre-existing neural network architectures. Seven diverse corpora represent the data basis for training the networks: a web-scraped corpus, a corpus  created  from  newspaper  articles, three second language learner corpora, a corpus created by a company that translates complex texts into incremental simplified versions, and a corpus  created  from  a collection of  written  examinations  covering  the whole CEFR  spectrum  (A1-C2). An approach based on the BERT architecture yielded the best results. The highest F1 score achieved was 1.0 and 0.83 on a document  and sentence level, respectively.
Temporal Knowledge-Aware Image Captioning
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Caption Generation, Image Captioning, World Knowledge

Contextualized image captioning is a task that extends beyond generating a purely visual description of the image content and aims to produce a caption that is influenced by the context and informed by the real world knowledge. In this paper, we present an approach to knowledge-aware image captioning, with a specific focus on the temporal domain. We propose a way to identify relevant information in external data sources, such as geographic databases and common knowledge bases, and then encode it in a way that is most useful for the captioning network. We develop an end-to-end caption generation system that incorporates external knowledge into the captioning process at several stages. The system is trained and tested on our novel temporal knowledge-aware captioning dataset, achieving significant improvements over multiple baselines across standardly used metrics. We demonstrate that our approach is effective for generating highly contextualized captions with both relevant and accurate temporal facts.
A Sentence is Worth 128 Pseudo Tokens: A Semantic-Aware Contrastive Learning Framework for Sentence Embeddings
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Contrastive Learning, Semantic Textual Similarity, Sentence, Sentence Embedding, Sentence-Embedding, Sentence Embeddings, STS

Contrastive learning has shown great potential in unsupervised sentence embedding tasks, e.g., SimCSE \citep{gao2021simcse}.
However, these existing solutions are heavily affected by superficial features like the length of sentences or syntactic structures. In this paper, we propose a semantic-aware contrastive learning framework for sentence embeddings, termed Pseudo-Token BERT (PT-BERT), which is able to explore the pseudo-token space (i.e., latent semantic space) representation of a sentence while eliminating the impact of superficial features such as sentence length and syntax. Specifically, we introduce an additional pseudo token embedding layer independent of the BERT encoder to map each sentence into a sequence of pseudo tokens in a fixed length. Leveraging these pseudo sequences, we are able to construct same-length positive and negative pairs based on the attention mechanism to perform contrastive learning. In addition, we utilize both the gradient-updating and momentum-updating encoders to encode instances while dynamically maintaining an additional queue to store the representation of sentence embeddings, enhancing the encoder's learning performance for negative examples. Experiments show that our model outperforms the state-of-the-art baselines on six standard semantic textual similarity (STS) tasks. Furthermore, experiments on alignments and uniformity losses, as well as hard examples with different sentence lengths and syntax, consistently verify the effectiveness of our method.
Probing Position-Aware Attention Mechanism in Long Document Understanding
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: document understanding, Natural Language Understanding, Position

Long document understanding is a challenging problem in natural language understanding. Most current transformer-based models only employ textual information for attention calculation due to high computation limit. To address those issues for long document understanding, we explore new approaches using different position-aware attention masks and investigate their performance on different benchmarks. Experimental results show that our models have the advantages on long document understanding based on various evaluation metrics. Furthermore, our approach makes changes only to the attention module in the transformer and thus can be flexibly detached and plugged into any other transformer-based solutions with ease.
FairLex: A Multilingual Benchmark for Evaluating Fairness in Legal Text Processing
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Fairness

We present a benchmark suite of four datasets for evaluating the fairness of pre-trained legal language models and the techniques used to fine-tune them for downstream tasks. Our benchmarks cover four jurisdictions (European Council, USA, Swiss, and Chinese), five languages (English, German, French, Italian and Chinese) and fairness across five attributes (gender, age, nationality/region, language, and legal area). In our experiments, we evaluate pre-trained language models using several group-robust fine-tuning techniques and show that none of these combinations guarantee fairness, nor consistently mitigate group disparities. Furthermore, we analyze what causes performance differences across groups, and how group-robust fine-tuning techniques fail to mitigate group disparities under both representation inequality and temporal distribution swift.
Self-Distilled Pruning of Neural Networks
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Knowledge Distillation, Language Modeling, Language Modelling

Pruning aims to reduce the number of parameters while maintaining performance close to the original network. This work proposes a novel \emph{self-distillation} based pruning strategy, whereby the representational similarity between the pruned and unpruned versions of the same network is maximized. Unlike previous approaches that treat distillation and pruning separately, we use distillation to inform the pruning criteria, without requiring a separate student network as in knowledge distillation.
We show that the proposed {\em cross-correlation objective for self-distilled pruning} implicitly encourages sparse solutions, naturally complementing magnitude-based pruning criteria. Experiments on the GLUE  and XGLUE benchmarks show that self-distilled pruning increases mono- and cross-lingual language model performance. Self-distilled pruned models also outperform smaller Transformers with an equal number of parameters and are competitive against (6 times) larger distilled networks. We also observe that self-distillation (1) maximizes class separability, (2) increases the signal-to-noise ratio, and (3) converges faster after pruning steps, providing further insights into why self-distilled pruning improves generalization.
Learning Methods for Solving Astronomy Course Problems
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Astronomy

This work trains a specialized machine learning model to solve university undergraduate level Introduction to Astronomy course problems. We use Transformers and graph neural networks to predict arithmetic expression trees with three key contributions: (i) a new dataset of formatted Introduction to Astrophysics course questions; (ii) support of new mathematical operators; and (iii) inclusion of tabular data consisting of known physical constants and units. Perhaps most importantly, this work introduces the concept of turning questions into programming tasks and using a Transformer trained on both text and code, namely OpenAI Codex, to solve these quantitative questions. Comparing our specialized model trained only on Astrophysics with the generic Codex model, across all topics taught in the course, we find a classical trade-off between accuracy on a specific course and the ability to scale to many courses. Our specialized model achieves 92% though does not scale to any other course; whereas the generic Codex achieves 67% on Astrophysics and is able to scale to many different courses.
On Vision Features in Multimodal Machine Translation
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Image Captioning, Machine Translation, Multimodal Machine Translation, object-detection, Object Detection, Translation

Previous work on multimodal machine translation (MMT) has focused on the way of incorporating vision features into translation but little attention is on the quality of vision models. In this work, we investigate the impact of vision models on MMT. Given the fact that Transformer is becoming popular in computer vision, we experiment with various strong models (such as Vision Transformer) and enhanced features (such as object-detection and image captioning). We develop a selective attention model to study the patch-level contribution of an image in MMT. On detailed probing tasks, we find that stronger vision models are helpful for learning translation from the vision modality. Our results also suggest the need of carefully examining MMT models, especially when current benchmarks are small-scale and biased.
CTM - A Model for Large-Scale Multi-View Tweet Topic Classification
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Classification, Topic Classification

Automatically associating social media posts with topics is an important prerequisite for effective search and recommendation on many social media platforms. However, topic classification of such posts is quite challenging because of (a) a large topic space (b) short text with weak topical cues, and (c) multiple topic associations per post. In contrast to most prior work which only focuses on post classification into a small number of topics ($10-20$), we consider the task of large-scale topic classification in the context of Twitter where the topic space is $10$ times larger with potentially multiple topic associations per Tweet. We address the challenges above and propose a novel neural model, CTM that (a) associates tweets from a large topic space of $300$ topics (b) takes a holistic approach to tweet content modeling -- leveraging multi-modal content, author context, and deeper semantic cues in the tweet.  We evaluate CTM quantitatively and show that our method offers an effective way to classify Tweets into topics at scale and is superior in performance to other approaches yielding a significant relative lift of $\mathbf{20}\%$.
Contextualized Sensorimotor Norms: multi-dimensional measures of sensorimotor strength for ambiguous English words, in context
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11

Most large language models are trained on linguistic input alone, yet humans appear to ground their understanding of words in sensorimotor experience. A natural solution is to augment LM representations with human judgments of a word's sensorimotor associations (e.g., the Lancaster Sensorimotor Norms), but this raises another challenge: most words are ambiguous, and judgments of words in isolation fail to account for this multiplicity of meaning (e.g., "wooden table" vs. "data table". We attempted to address this problem by building a new lexical resource of contextualized sensorimotor judgments for 112 English words, each rated in four different contexts (448 sentences total). We show that these ratings encode overlapping but distinct information from the Lancaster Sensorimotor Norms, and that they also predict other measures of interest (e.g., relatedness), above and beyond measures derived from BERT.
Tell me who you are and i'll tell you what to do: A Persona Grounded Task Oriented Dialogue Generation System
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Dialogue Generation

Modern dialogue agents can broadly be categorized as either chit-chat or task-oriented systems. While the purpose of a chit-chat agent is to entertain and engage the user- lubricating the conversation, so to say-, the task oriented chat-bot is dedicated to fulfilling specific requests (e.g., ticket booking). Current task-oriented agents produce precise but bland and uninteresting responses. While using such agents a user may interpose personal remarks, and the failure of the agent to process and respond to such statements could be a put-off for the user. In this paper we propose a system that is persona-specific, can handle chit-chat utterances, and produces responses that add a human element to the conversation, while always remaining grounded on the task. Since current task-oriented datasets do not have persona-profiles, and do not consist of personalized remarks in utterances, we modify an existing dataset (MultiWOZ 2.1) to suit our needs. We give a semi-automated dataset creation method that uses GPT-2 model trained on the PERSONA-CHAT dataset. A small subset of the obtained data is also manually crafted to acquire a gold standard data. Our framework is based on GPT-2, Graph Convolution Network (GCN) and Memory Network that is trained on this dataset to generate persona-grounded task-oriented responses. Both automatic and manual evaluation show the effectiveness of our model and dataset. Our proposed system achieves a BLEU score of 12.12 on this new dataset.
The Change that Matters in Discourse Parsing: Estimating the Impact of Domain Shift on Parser Error
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Discourse Parsing, Domain Adaptation, Sentence

Discourse analysis allows us to attain high-level inferences of a text document beyond the sentence-level. However, currently the performance of discourse models is very low on texts outside of the training distribution's coverage. There is need for a measure that can inform us to what extent our model generalizes from the training to the test sample when these samples may be drawn from distinct distributions. While this can be estimated via distribution shift, we argue that this does not directly correlate with change in the observed error of a classifier (i.e. error-gap). Thus, we propose to use a statistic from the theoretical domain adaptation literature which can be directly tied to error-gap. We study the bias of this statistic as an estimator of error-gap both theoretically and through a large-scale empirical study of over 2400 experiments on 6 discourse datasets from domains including, but not limited to: news, biomedical texts, TED talks, Reddit posts, and fiction. Our results not only motivate our proposal and help us to understand its limitations, but also provide insight on the properties of discourse models and datasets which improve performance in domain adaptation. For instance, we find that non-news datasets are slightly easier to transfer to than news datasets when the training and test sets are very different. We plan to release our code as a Python package to allow practitioners to make more informed model and dataset choices.
BORT: Back and Denoising Reconstruction for End-to-End Task-Oriented Dialog
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Denoising

A typical end-to-end task-oriented dialog system transfers context into dialog state, and upon which generates a response, which usually faces the problem of error propagation from both previously generated inaccurate dialog states and responses, especially in low-resource scenarios. To alleviate these issues, we propose BORT, a back and denoising reconstruction approach for end-to-end task-oriented dialog system. To improve the accuracy of dialog state that is essential for the task completion of dialog system, back reconstruction is used to reconstruct the original input context from the generated dialogue state since the inaccurate dialog state cannot recover its corresponding input context. To enhance the antinoise capability of the model, denosing reconstruction is used to reconstruct the corrupted dialog state and response. Extensive experiments conducted on MultiWOZ 2.0 and CamRest676 show the effectiveness of BORT which achieves state-of-the-art performance. Furthermore, BORT demonstrates its advanced capabilities in zero-shot domain scenarios and in low-resource scenarios.
Contrastive Conditional Masked Language Model for Non-autoregressive Neural Machine Translation
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Contrastive Learning, Language Modeling, Language Modelling, Machine Translation, Sentence, Translation

Inspired by the success of contrastive learning in natural language processing, we incorporate contrastive learning into the conditional masked language model which is extensively used in non-autoregressive neural machine translation (NAT) that we term Contrastive Conditional Masked Language Model (CCMLM). CCMLM optimizes the similarity of several different representations of the same token in the same sentence, resulting in a richer and more robust representation. We propose two methods to obtain various representations: Contrastive Common Mask and Contrastive Dropout. Positive pairs are various different representations of the same token, while negative pairs are representations of different tokens. In the feature space, the model with contrastive loss pulls positive pairs together and pushes negative pairs away. We conduct extensive experiments on four translation directions with different data sizes. The results demonstrate that CCMLM showed a consistent and significant improvement with margins ranging from 0.80-1.04 BLEU and is state-of-the-art on WMT'16 Ro-En (34.18 BLEU).