pwc-archive/papers-with-abstracts huggingface dataset:
Generating a Temporally Coherent Image Sequence for a Story by Multimodal Recurrent Transformers
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Image Generation, Sentence, Story Visualization, Text to Image Generation, Text-to-Image Generation

Story visualization is a challenging text-to-image generation task for the difficulty of rendering visual details from abstract text descriptions. 
Besides the difficulty of image generation, the generator also need to conform to the narrative of a multi-sentence story input. 
While prior arts in this domain has focused on improving semantic relevance between generated images and input text, controlling the generated images to be temporally consistent still remains as a challenge. To generate a semantically coherent image sequence, we propose an explicit memory controller which can augment the temporal coherence of images in the multi-modal autoregressive transformer, and call Story visualization by MultimodAl Recurrent Transformers or SMART for short. Our method generates high resolution high quality images, outperforming prior works by a significant margin across multiple evaluation metrics on PororoSV dataset.
MIMICause: Representation and automatic extraction of causal relation types from clinical notes
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Language Modeling, Language Modelling, Relation, Sentence

Understanding causal narratives communicated in clinical notes can help make strides towards personalized healthcare. Extracted causal information from clinical notes can be combined with structured EHR data such as patients' demographics, diagnoses, and medications. This will enhance healthcare providers' ability to identify aspects of a patient's story communicated in the clinical notes and help make more informed decisions.  

In this work, we propose annotation guidelines, develop an annotated corpus and provide baseline scores to identify types and direction of causal relations between a pair of biomedical concepts in clinical notes; communicated implicitly or explicitly, identified either in a single sentence or across multiple sentences. 

We annotate a total of 2714 de-identified examples sampled from the 2018 n2c2 shared task dataset and train four different language model based architectures. Annotation based on our guidelines achieved a high inter-annotator agreement i.e. Fleiss' kappa ($\kappa$) score of 0.72, and our model for identification of causal relations achieved a macro F1 score of 0.56 on the test data. The high inter-annotator agreement for clinical text shows the quality of our annotation guidelines while the provided baseline F1 score sets the direction for future research towards understanding narratives in clinical texts.
Understanding Gender Bias in Knowledge Base Embeddings
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11

Knowledge base (KB) embeddings have been shown to contain gender biases. In this paper, we study two questions regarding these biases: how to quantify them, and how to trace their origins in KB? Specifically, first, we develop two novel bias measures respectively for a group of person entities and an individual person entity. Evidence of their validity is observed by comparison with real-world census data. Second, we use the influence function to inspect the contribution of each triple in KB to the overall group bias. To exemplify the potential applications of our study, we also present two strategies (by adding and removing KB triples) to mitigate gender biases in KB embeddings.
PALBERT: Teaching ALBERT to Ponder
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11

Currently, pre-trained models can be considered the default choice for a wide range of NLP tasks. Despite their SoTA results, there is practical evidence that these models may require a different number of computing layers for different input sequences, since evaluating all layers leads to overconfidence on wrong predictions (namely overthinking). This problem can potentially be solved by implementing adaptive computation time approaches, which were first designed to improve inference speed.

Recently proposed PonderNet may be a promising solution for performing an early exit by treating the exit layer's index as a latent variable. However, the originally proposed exit criterion, relying on sampling from trained posterior distribution on the probability of exiting from $i$-th layer, introduces major variance in model outputs, significantly reducing the resulting model's performance.

In this paper, we propose Ponder ALBERT (PALBERT) – an improvement to PonderNet with a novel deterministic Q-exit criterion and a revisited model architecture. We compared PALBERT with recent methods for performing an early exit. We observed that the proposed changes can be considered significant improvements on the original PonderNet architecture and outperform PABEE on a wide range of GLUE tasks. In addition, we also performed an in-depth ablation study of the proposed architecture to further understand Lambda layers and their performance.
Unsupervised Dependency Graph Network
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Dependency Parsing, Language Modeling, Language Modelling, Masked Language Modeling, POS, Sentence, Unsupervised Dependency Parsing

Recent work has identified properties of pretrained self-attention models that mirror those of dependency parse structures. In particular, some self-attention heads correspond well to individual dependency types. Inspired by these developments, we propose a new competitive mechanism that encourages these attention heads to model different dependency relations. We introduce a new model, the Unsupervised Dependency Graph Network (UDGN), that can induce dependency structures from raw corpora and the masked language modeling task. Experiment results show that UDGN achieves very strong unsupervised dependency parsing performance without gold POS tags and any other external information. The competitive gated heads show a strong correlation with human-annotated dependency types. Furthermore, the UDGN can also achieve competitive performance on masked language modeling and sentence textual similarity tasks.
Can Neural Networks Understand Programs like Humans?
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Clone Detection, Code Summarization, Graph Neural Network

Program understanding is a fundamental task in program language processing. Despite the success, existing works fail to take human minds as reference in understanding programs. In this paper, we incorporate human minds and propose the PGNN-EK model that consists of two main components. On the one hand, inspired by the “divide-and-conquer” reading behaviours of humans, we present a partitioning-based graph neural network model PGNN on the upgraded AST of codes. On the other hand, to characterize human minds of resorting to other resources to help code comprehension, we transform raw codes with external knowledge and apply pre-training techniques for information extraction. Finally, we combine the two embeddings generated from the two components to output code embeddings. We conduct extensive experiments to show the superior performance of PGNN-EK on the code summarization and code clone detection tasks. In particular, to show the generalization ability of our model, we release a new dataset that is more challenging for code clone detection and could advance the development of the community. Our codes and data are publicly available at https://github.com/anonymousforpaper1997/PGNN-EK.
MarCQAp: Effective Context Modeling for Conversational Question Answering
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Conversational Question Answering, Question Answering

State-of-the-art models for Document-grounded Conversational Question Answering (DCQA) are based on the Transformer architecture. This raises two open issues: (a) Is it sufficient to concatenate the dialog history and the grounding document and perform cross-attention via a Transformer in order to capture the document/dialogue relationships? and (b) What is the best way to cope with the Transformers’ quadratic complexity, given the long inputs in DCQA? We address these issues in two dimensions. First, we introduce MarCQAp, a new modeling approach which encodes the historic answers by adding textual markups in the grounding document text, and then answers the question conditioned on the marked document. Second, we show that sparse self-attention architectures, such as the Longformer, can replace the Transformer, resolving the input length limitation. Our results demonstrate the effectiveness of each approach and their combination for explicit representation of dialogue/document relationships, significantly improving over state-of-the-art DCQA models.
Slot Filling for Biomedical Information Extraction
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER), Passage Retrieval, Reading Comprehension, Relation, Relation Extraction, Retrieval, slot-filling, Slot Filling

Information Extraction (IE) from text refers to the task of extracting structured knowledge from unstructured text. The task typically consists of a series of sub-tasks such as Named Entity Recognition and Relation Extraction. Sourcing entity and relation type specific training data is a major bottleneck in domains with limited resources such as biomedicine. In this work we present a slot filling approach to the task of biomedical IE, effectively replacing the need for entity and relation-specific training data, allowing us to deal with zero-shot settings. We follow the recently proposed paradigm of coupling a Tranformer-based bi-encoder, Dense Passage Retrieval, with a Transformer-based reading comprehension model to extract relations from biomedical text. We assemble a biomedical slot filling dataset for both retrieval and reading comprehension and conduct a series of experiments demonstrating that our approach outperforms a number of simpler baselines. We also evaluate our approach end-to-end for standard as well as zero-shot settings. Our work provides a fresh perspective on how to solve biomedical IE tasks, in the absence of relevant training data. Our code, models and datasets are available at https://github.com/tba.
A Novel End-to-End CAPT System for L2 Children Learners
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), Decoder, Phoneme Recognition, Sentence, speech-recognition, Speech Recognition

Recently, Conformer-based model shows promising results in automatic speech recognition (ASR) task. There still is a dearth of research on Conformer based model for computer-assisted pronunciation learning (CAPT) system. In this paper, a Conformer-based CAPT system is introduced to provide the mispronunciation detection and diagnosis.  We apply the Conformer as the main pronunciation error detection model in phoneme level since superior phoneme recognition performance. Then, the features, including the Log Phone Posterior (LPP), the Log Posterior Ratio (LPR) and some other features, extracted from the Conformer decoder, are trained by a XGBoost model to predict phoneme and sentence level scores labeled by experts. Both results on open datasets and our internal Chinese children data demonstrate that the Conformer-based system, which has smaller model size and detailed diagnosis, achieves better performance compared with neutral network (NN)-based system.
More Than Words: Collocation Retokenization for Latent Dirichlet Allocation Models
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11

Traditionally, Latent Dirichlet Allocation (LDA) ingests words in a collection of documents to discover their latent topics using word-document co-occurrences. Previous studies show that representing bigrams collocations in the input can improve topic coherence in English. However, it is unclear how to achieve the best results for languages without marked word boundaries such as Chinese and Thai. Here, we explore the use of retokenization based on chi-squared measures, $t$-statistics, and raw frequency to merge frequent token ngrams into collocations when preparing input to the LDA model. Based on the goodness of fit and the coherence metric, we show that topics trained with merged tokens result in topic keys that are clearer, more coherent, and more effective at distinguishing topics than those of unmerged models.
A New Search Paradigm for Natural Language Code Search
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Code Search

Code search can accelerate the efficiency of software development by finding code snippets for the given query. The dominant code search paradigm is to learn the semantic matching between code snippets and queries by neural networks. However, this search paradigm causes the gap transferring and expansion between code snippets and queries because researchers utilize pairs of code snippets and code descriptions (e.g., comments and documentation) to train their models and evaluate the trained models on the query which is different from the code description in writing style and application scenario. To remedy the issue, we propose a new simple but effective search paradigm, Query2Desc, which entirely depends on natural language and conducts code search by performing the semantic matching between code descriptions and queries. Experimental results on dataset CoSQA show that the state-of-the-art model CodeBERT gets improvement of 17.48\% in terms of the average MRR when applying it on Query2Desc. Moreover, baseline models on Query2Desc can return the right results in top-$10$ search results for at least 95\% of queries in the test set of CoSQA.
A simple log-based loss function for ordinal text classification
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Classification, text-classification, Text Classification

The cross-entropy loss function is widely used and generally considered the default loss function for text classification. When it comes to ordinal text classification where there is an ordinal relationship between labels, the cross-entropy is not optimal as it does not incorporate the ordinal character into its feedback. In this paper, we propose a new simple loss function called ordinal log-loss (OLL). We show that this loss function outperforms state-of-the-art previously introduced losses on four benchmark text classification datasets.
A Comparative Study of Faithfulness Metrics for Model Interpretability Methods
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Decision Making

Interpretable methods to reveal the internal reasoning processes behind machine learning models have attracted increasing attention in recent years. To quantify the extent to which the identified interpretations truly reflect the intrinsic decision-making mechanisms, various faithfulness evaluation metrics have been proposed. However, we find that different faithfulness metrics show conflicting preferences when comparing different interpretations. Motivated by this observation, we aim to conduct a comprehensive and comparative study of the widely adopted faithfulness metrics. In particular, we introduce two assessment dimensions, namely diagnosticity and complexity. Diagnosticity refers to the degree to which the faithfulness metric favors relatively faithful interpretations over randomly generated ones, and complexity is measured by the average number of model forward passes. According to the experimental results, we find that sufficiency and comprehensiveness metrics have higher diagnosticity and lower complexity than the other faithfulness metrics.
Hyperlink-induced Pre-training for Passage Retrieval of Open-domain Question Answering
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Open-Domain Question Answering, Passage Retrieval, Question Answering, Retrieval

To alleviate the data scarcity problem in training question answering systems, recent works propose additional intermediate pre-training for dense passage retrieval (DPR). However, there still remains a large discrepancy between the provided upstream signals and the downstream question-passage relevance, which leads to less improvement. To bridge this gap, we propose the Hyperlink-induced Pre-training (HLP), a method to pre-train the dense retriever with the text relevance induced by hyperlink-based topology within Web documents. We demonstrate that the hyperlink-based structures of dual-link and co-mention can provide effective relevance signals for large-scale pre-training that better facilitate downstream passage retrieval. We investigate the effectiveness of our approach across a wide range of open-domain QA datasets under zero-shot, few-shot, multi-hop, and out-of-domain scenarios. The experiments show our HLP outperforms the BM25 by up to 7 points as well as other pre-training methods by up to 30 points in terms of top-20 retrieval accuracy under the zero-shot scenario. Furthermore, HLP significantly outperforms other pre-training methods under the other scenarios.
Effective Unsupervised Constrained Text Generation based on Perturbed Masking
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Position, Sentence, Text Generation

Unsupervised constrained text generation aims to generate text under a given set of constraints without any supervised data. Current state-of-the-art methods stochastically sample edit positions which may cause unnecessary search steps. In this paper, we propose PMCTG to improve effectiveness by searching for the best position and action in each step. Specifically, PMCTG extends the perturbed masking technique to effectively search for the best edit position. Then it uses proposed multi-aspect scoring functions to select edit action to further reduce search difficulty. Since PMCTG does not require supervised data, it can extend to different generation tasks. We show PMCTG achieves state-of-the-art results in keywords-to-sentence generation and paraphrasing.
Crossword: Estimating Unknown Embeddings using Cross Attention and Alignment Strategies
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Word Embeddings

Word embedding methods like word2vec and GloVe have been shown to learn strong representations of words.  However, these methods only learn representations for words in the training corpus.  This is problematic, as models using these representations need ways to handle unknown and new words, known as out-of-vocabulary (OOV) words.  As a result, there have been multiple attempts to learn OOV word representations in a similar fashion to how humans learn new words, using surrounding words (``context clues") and word roots/subwords.  However, most current approaches suffer from two problems.  First, these models calculate context clue estimates and subword estimates separately and then combine them shallowly for a final estimate, therefore ignoring potentially important information each type can learn from the other.  Secondly, although subword embeddings are trained to estimate word vectors, we find these embeddings don't occupy the same space as word embeddings.  Current models do not take this into account, and do not align the spaces before combining them.  In response to this, we propose Crossword, a transformer based OOV estimation model that combines context and subwords at the attention level, allowing each type to influence the other for a stronger final estimate.  Crossword successfully combines these different sources of information using cross attention, along with strategies to align subword and context spaces.
As Little as Possible, as Much as Necessary: Detecting Over- and Undertranslations with Contrastive Conditioning
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Machine Translation, Translation

Omission and addition of content is a typical issue in neural machine translation. We propose a method for detecting such phenomena with off-the-shelf translation models. Using contrastive conditioning, we compare the likelihood of a full sequence under a translation model to the likelihood of its parts, given the corresponding source or target sequence. This allows to pinpoint superfluous words in the translation and untranslated words in the source even in the absence of a reference translation. The accuracy of our method is comparable to a supervised method that requires a custom quality estimation model.
Learning from Missing Relations: Contrastive Learning with Commonsense Knowledge Graphs for Commonsense Inference
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Contrastive Learning, Diversity, Knowledge Graphs

Commonsense inference poses a unique challenge to reason and generate the physical, social, and causal conditions of a given event. Existing approaches to commonsense inference utilize commonsense transformers, which are large-scale language models that learn commonsense knowledge graphs. However, they suffer from a lack of coverage and expressive diversity of the graphs, resulting in a degradation of the representation quality. In this paper, we focus on addressing missing relations in commonsense knowledge graphs, and propose a novel contrastive learning framework called SOLAR. Our framework contrasts sets of semantically similar and dissimilar events, learning richer inferential knowledge compared to existing approaches. Empirical results demonstrate the efficacy of SOLAR in commonsense inference of diverse commonsense knowledge graphs. Specifically, SOLAR outperforms the state-of-the-art commonsense transformer on commonsense inference with ConceptNet by 1.84% on average among 8 automatic evaluation metrics. In-depth analysis of SOLAR sheds light on the effects of the missing relations utilized in learning commonsense knowledge graphs.
Semantic Matching from Different Perspectives
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Sentence, Text Matching, text similarity

In this paper, we pay attention to the issue which is usually overlooked, i.e., similarity should be determined from different perspectives. To explore this issue, we release a Multi-Perspective Text Similarity (MPTS) dataset, in which sentence similarities are labeled from twelve perspectives. Furthermore, we conduct a series of experimental analysis on this task by retrofitting some famous text matching models. Finally, we obtain several conclusions and baseline models, laying the foundation for the following investigation of this issue. The dataset and code are publicly available at Github.