pwc-archive/papers-with-abstracts huggingface dataset:
MDERank: A Masked Document Embedding Rank Approach for Unsupervised Keyphrase Extraction
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Contrastive Learning, Document Embedding, Information Retrieval, Keyphrase Extraction, Retrieval

Keyphrase extraction (KPE) automatically extracts phrases in a document that provide a concise summary of the core content, which benefits downstream information retrieval and NLP tasks. Previous state-of-the-art methods select candidate keyphrases based on the similarity between learned representations of the candidates and the document. They suffer performance degradation on long documents due to discrepancy between sequence lengths which causes mismatch between representations of keyphrase candidates and the document. In this work, we propose a novel unsupervised embedding-based KPE approach, Masked Document Embedding Rank (MDERank), to address this problem by leveraging a mask strategy and ranking candidates by the similarity between embeddings of the source document and the masked document. We further develop a KPE-oriented BERT (KPEBERT) model by proposing a novel self-supervised contrastive learning method, which is more compatible to MDERank than vanilla BERT. Comprehensive evaluations on six KPE benchmarks demonstrate that the proposed MDERank outperforms state-of-the-art unsupervised KPE approach by average 1.80 $F1@15$ improvement. MDERank further benefits from KPEBERT and overall achieves average 3.53 $F1@15$ improvement over SIFRank.
Answering Open-Domain Multi-Answer Questions via a Recall-then-Verify Framework
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: valid

Open-domain questions are likely to be open-ended and ambiguous, leading to multiple valid answers. Existing approaches typically adopt the rerank-then-read framework, where a reader reads top-ranking evidence to predict answers. According to our empirical analysis, this framework faces three problems: first, to leverage a large reader under a memory constraint, the reranker should select only a few relevant passages to cover diverse answers, while balancing relevance and diversity is non-trivial; second, the small reading budget prevents the reader from accessing valuable retrieved evidence filtered out by the reranker; third, when using a generative reader to predict answers all at once based on all selected evidence, whether a valid answer will be predicted also pathologically depends on evidence of some other valid answer(s). To address these issues, we propose to answer open-domain multi-answer questions with a recall-then-verify framework, which separates the reasoning process of each answer so that we can make better use of retrieved evidence while also leveraging large models under the same memory constraint. Our framework achieves state-of-the-art results on two multi-answer datasets, and predicts significantly more gold answers than a rerank-then-read system that uses an oracle reranker.
Break it - Message it - Fix it : Learning to Repair Python Programs using Error Messages without Labelled Data
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Code Repair

In recent years there is an increasing demand to reduce the gap in development to deployment. It has been estimated that developers spend almost 20\% of their time in fixed problems with their code. Therefore tools which can automatically repair code can help accelerate the DevOps cycles. In this work we build upon recent success of deploying neuro-symbolic approaches for automatic code repair. In our approach, we use a dataset of python code, viz, CodeNet, which represents data distribution for human generated code. We train two neural modules a breaker and a fixer, which are trained iteratively, along with a symbolic module Pylint. The breaker learns to introduce errors in the code, the symbolic module acts as a Critic and is able to fragment the error by identifying the line, as well provide the error type with a specific exception message. The Fixer utilizes the exception message to repair the erroneous line in the code. We are able to cover 32 different syntax errors, and iterative training based on back translation actually helps improve the performance of the Fixer.
ShrinkNAS : Single-Path One-Shot Operator Exploratory Training for Transformer with Dynamic Space Shrinking
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Neural Architecture Search

Neural Architecture Search (NAS) for Transformer has shown its growing capabilities in exploiting the benefits of various Transformer architecture configurations. Recent studies envision the diverse potential of introducing unprecedented Transformer operators (OPs, such as Convolution) to its structure, yet the existing methods of doing so are all time-consuming. Traditionally, Single-Path One-Shot (SPOS) models enable efficient search over a vast set of OPs. However, existing SPOS methods on Transformer focus only on dimensional configurations of the vanilla Transformer OP (e.g., Multi-head Attention), and did not consider introducing other OPs. This paper explores the possibility of including OPs in the Transformer-based SPOS architecture search to discover better Transformer structures with the high efficiency facilitated in the SPOS category. To achieve that, we propose Dynamic Space Shrinking  (DSS), a novel method that resolves problems brought from newly added OPs by dynamically keeping the current sample space containing subnets with good configurations and performance. 
We implemented DSS in ShrinkNAS, the first SPOS one-shot inter-OP model for Transformer. Our evaluation shows that ShrinkNAS is of much higher elasticity by finding a better structure beating the human-designed ones under tight constraint (<10M parameters), while existing intra-OP SPOS methods are not even close.
BanglaBERT: Language Model Pretraining and Benchmarks for Low-Resource Language Understanding Evaluation in Bangla
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Language Modeling, Language Modelling, Natural Language Inference, Natural Language Understanding, text-classification, Text Classification

In this paper, we introduce 'BanglaBERT', a BERT-based Natural Language Understanding (NLU) model pretrained in Bangla, a widely spoken yet low-resource language in the NLP literature. To pretrain BanglaBERT, we collect 27.5 GB of Bangla pretraining data (dubbed 'Bangla2B+') by crawling 110 popular Bangla sites. We introduce a new downstream task dataset on Natural Language Inference (NLI) and benchmark on four diverse NLU tasks covering text classification, sequence labeling, and span prediction. In the process, we bring them under the first-ever Bangla Language Understanding Evaluation (BLUE) benchmark. BanglaBERT achieves state-of-the-art results outperforming multilingual and monolingual models. We will make the BanglaBERT model, the new datasets, and a leaderboard publicly available to advance Bangla NLP.
Improving Unsupervised Sentence Simplification Using Fine-Tuned Masked Language Models
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Language Modeling, Language Modelling, Sentence

Simple word suggestion in unsupervised sentence simplification (SS) methods is mostly done independently of the context. The idea of adapting and fine-tuning a context-aware model on simple data for improving performance has been discussed but not practiced. In this paper, we propose a framework involving fine-tuning a pre-trained BERT masked language model on simple English corpora to aid SS. Our analysis of public test data shows that fine-tuning on any set of simple sentences do not necessarily yield better simplifications but generally makes improvements. To tackle this issue, we propose a self-supervised framework which is composed of a labeling method that conducts an estimate about the \textit{usefulness} of each training sample, paired with a simple linear classifier that decides the inclusion of a given sentence in the fine-tuning process. The fine-tuned BERT will be used in an iterative edit-based unsupervised SS model to provide contextual word suggestions. The results show that our data selection approach can improve simplifications as much as having a simple-to-complex parallel corpus.
Two Front-Ends, One Model : Fusing Heterogeneous Speech Features for Low Resource ASR with Multilingual Pre-Training
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), speech-recognition, Speech Recognition, Transfer Learning

Transfer learning is widely applied in various deep learning-based speech tasks, especially for tasks with a limited amount of data. Recent studies in transfer learning mainly focused on either supervised or self-supervised perspectives. This work, however, seeks to incorporate the two schemes together towards low-resource automatic speech recognition (ASR) for minor and endangered language (EL) communities. We propose a general framework to use learned transformations to resolve time resolution differences between any speech features, allowing for fusion of any self-supervised representations or spectral features used in multilingual pre-training. Our experiments over two low-resource languages and three ELs demonstrate that the proposed framework can significantly improve the absolute average word error rate from 45.4% to 35.5%.
Probing the Prompting of CLIP on Human Faces
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11

Large-scale multimodal models such as CLIP have caught great attention due to their generalization capability. CLIP can take free-form text prompts, but the performance varies with different text prompt manipulations, which is considered unpredictable. In this paper, we conduct a controlled study to understand how CLIP perceives images with different forms of text prompts, particularly on human facial attributes. We find that (1) using the prompt starter "a photo of" can guide the model to allocate higher attention weights to human faces, leading to better classification performance; (2) CLIP model is better at aligning information from shorter text prompts, as additional textual details shift away the attention from key words; (3) properly adding punctuation or removing stop words in the text prompt can shift attention to target information. Our practice on facial attributes shed light on the design of reliable text prompts for CLIP in other tasks.
A Well-Composed Text is Half Done! Semantic Composition Sampling for Diverse Conditional Generation
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Question Generation, Question-Generation, Semantic Composition

We propose Composition Sampling, a simple but effective method to generate higher quality diverse outputs for conditional generation tasks, compared to previous stochastic decoding strategies. It builds on recently proposed planning-based neural generation models that are trained to first create a composition of the output using an entity chain and then continue to generate conditioned on the entity chain and the input \cite{frost}. Our approach avoids text degeneration by first sampling a composition in the form of an entity chain and then using beam search to generate the best possible text grounded to the entity chain. Experiments  on CNN/DailyMail and XSum  using a variety of automatic metrics and human-based evaluation demonstrate that Composition Sampling is currently the best available decoding strategy for generating diverse meaningful summaries. We further outperform state-of-the-art approaches for question generation in terms of BLEU.
Exploiting Dialogue Act for Knowledge Selection and Response Generation
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Multi-Task Learning, Response Generation

Dialogue act (DA) is the description of the intention or function of a dialogue utterance. In document-grounded dialogue, correctly understanding the dialogue context is crucial for models to select knowledge and inject knowledge into responses. Leveraging dialogue act can help to understand the dialogue context and consequently assist the utilization of document information. In this paper, we propose a novel framework leveraging two different kinds of DAs (model-annotated and human-annotated) for \textbf{Knowledge Selection} (KS) and \textbf{Response Generation} (RG). The framework consists of two modules: the prediction module is trained with multi-task learning and learns to select knowledge and predict the next DA; the generation module uses the selected knowledge and the predicted DA for the RG. Our model achieves new state-of-the-art performance on three public datasets and the results verify that leveraging DA can help KS and RG. Our code and data will be released on github.com.
Uncertainty-based Visual Question Answering: Estimating Semantic Inconsistency between Image and Knowledge Base
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Question Answering, Semantic Similarity, Semantic Textual Similarity, Visual Question Answering, Visual Question Answering (VQA)

Knowledge-based visual question answering (KVQA) task aims to answer questions that require additional external knowledge as well as an understanding of images and questions. Recent studies on KVQA inject an external knowledge in a multi-modal form, and as more knowledge is used, irrelevant information may be added and can confuse the question answering. In order to properly use the knowledge, this study proposes the following: 1) We introduce a novel semantic inconsistency measure using caption uncertainty and semantic similarity. 2) We suggest a new external knowledge assimilation method based on the semantic inconsistency measure and apply it to integrate explicit knowledge and implicit knowledge for KVQA. 3) The proposed method is evaluated on the OK-VQA dataset and achieves the state-of-the-art performance.
Real-Time Visual Feedback to Guide Benchmark Creation: A Human-and-Metric-in-the-Loop Workflow
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11

Recent research has shown that language models exploit 'artifacts' in benchmarks to solve tasks, rather than truly learning them, leading to inflated model performance. In pursuit of creating better benchmarks, we propose VAIDA, a novel benchmark creation paradigm for NLP, that focuses on guiding crowdworkers, an under-explored facet of addressing benchmark idiosyncrasies. VAIDA facilitates sample correction by providing realtime visual feedback and recommendations to improve sample quality. Our approach is domain, model, task, and metric agnostic, and constitutes a paradigm shift for robust, validated, and dynamic benchmark creation via human-and-metric-in-the-loop workflows. We evaluate via expert review and a user study with NASA TLX. We find that VAIDA decreases effort, frustration, mental, and temporal demands of crowdworkers and analysts, simultaneously increasing the performance of both user groups with a 45.8% decrease in the level of artifacts in created samples. As a by product of our user study, we observe that created samples are adversarial across models, leading to decreases of 31.3% (BERT), 22.5% (RoBERTa), 14.98% (GPT-3 fewshot) in performance.
Multi-Granularity Contrastive Knowledge Distillation for Multimodal Named Entity Recognition
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Knowledge Distillation, Multi-modal Named Entity Recognition, named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER)

It is very valuable to recognize named entities from short and informal multimodal posts in this age of information explosion. Despite existing methods success in multi-modal named entity recognition (MNER), they rely on the well aligned text and image pairs, while a lot of noises exist in the datasets. And the representation of text and image with internal correlations is difficult to establish a deep connection, because of the mismatched semantic levels of the text encoder and image encoder.  In this paper, we propose multi-granularity contrastive knowledge distillation (MGC) to build a unified joint representation space of two modalities. By leveraging multi-granularity contrastive loss, our approach pushes representations of matched image-text pairs or image-entity pairs together while pushing those unrelated image-text or image-entity pairs apart. By utilizing CLIP model for knowledge distillation, we can obtain a more fine-grained visual concept. Experimental results on two benchmark datasets prove the effectiveness of our method.
Integrating Vectorized Lexical Constraints for Neural Machine Translation
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Machine Translation, NMT, Translation

Lexically constrained neural machine translation (NMT), which controls the generation of NMT models with pre-specified constraints, is important in many practical scenarios. Due to the representation gap between discrete constraints and continuous vectors of NMT models, most existing works propose to construct synthetic data or modify the decoding algorithm to impose lexical constraints, treating the NMT model as a black box. In this work, we directly integrate the constraints into NMT models through vectorizing discrete constraints into continuous keys and values that can be utilized by the attention modules of NMT models. The proposed integration method is based on the assumption that the correspondence between the keys and values in attention modules is naturally suitable for modeling constraint pairs. Experimental results show that our method consistently outperforms several representative baselines on four language pairs, demonstrating the necessity of integrating vectorized lexical constraints.
CREATE: A Benchmark for Chinese Short Video Retrieval and Title Generation
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Retrieval, Video Captioning, Video Retrieval

Previous works of video captioning aim to objectively describe the video's actual content, lack of subjective and attractive expression, limiting its practical application scenarios. Video titling is intended to achieve this goal, but there is a lack of a proper benchmark. In this paper, we propose CREATE, the first large-scale Chinese shoRt vidEo retrievAl and Title gEneration benchmark, to facilitate research and application in video titling and video retrieval in Chinese. CREATE consists of a high-quality labeled 210K dataset and two large-scale 3M/10M pre-training datasets, covering 51 categories, 50K+ tags, 537K manually annotated titles and captions, and 10M+ short videos. Based on CREATE, we propose a novel model ALWIG which combines video retrieval and video titling tasks to achieve the purpose of multi-modal ALignment WIth Generation with the help of video tags and GPT pre-trained model. CREATE opens new directions for facilitating future research and applications on video titling and video retrieval in the field of Chinese short videos.
EventBERT
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Natural Language Understanding

Pre-trained language models (PrLMs) have shown impressive performance in natural language understanding. However, they mainly rest on extracting context-sensitive statistical patterns without explicit modeling of linguistic information such as semantic relationships entailed in natural language. In this work, we propose EventBERT, an event-based semantic representation model that takes BERT as the backbone and refines with event-based structural semantics in terms of graph convolution network. EventBERT benefits simultaneously from rich event-based structures embodied in the graph and contextual semantics learned in pre-trained model BERT. Experimental results on the GLUE benchmark show the effectiveness.
Tokenization on the Number Line is All You Need
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: All, Language Modeling, Language Modelling

Despite the recent breakthroughs in language modeling, their ability to represent numbers is insufficient. Subword tokenization, the standard choice for number representation, breaks down a number into arbitrary chunks thereby failing to explicitly capture the relationship between two numbers on on the number-line. To alleviate this shortcoming, alternate approaches have been proposed that modify numbers at various stages of the language modeling pipeline. These methods can be broadly classified into three categories that make changes to a) the notation (\eg scientific vs decimal) b) vocabulary (\eg introduce a new token for numbers in range $10-100$) and c) architectural changes to directly regress to a desired number. The contributions of this work are three fold -- firstly, we propose vocabulary level changes in the decoding stage and study its behavior. Next, we study the performance of both the proposed approach and existing number representation schemes in the context of masked number presentation. We find that a carefully designed tokenization scheme is both the simplest to implement and sufficient \ie with similar performance to the state-of-the-art approach that requires making significant architectural changes. Finally, we evaluate the various number representation schemes on the downstream task of numerical fact estimation (for fermi problems) in a zero-shot setting and find similar trends \ie changes at the tokenization level achieve near state-of-the-art results while requiring minimal resources compared to other number representation schemes.
Unsupervised Domain Adaptation with Contrastive Learning for Cross-domain Chinese NER
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Articles, Contrastive Learning, Domain Adaptation, NER, Sentence, Unsupervised Domain Adaptation, Word Embeddings

Understanding and recognizing the entities of Chinese articles highly relies on the fully supervised learning based on the domain-specific annotation corpus. However, this paradigm fails to generalize over other unlabeled domain data which consists of different entities semantics and domain knowledge. To address this domain shift issue, we propose the framework of unsupervised Domain Adaptation with Contrastive learning for Chinese NER (DAC-NER). We follow Domain Separation Network (DSN) framework to leverage private-share pattern to capture domain-specific and domain-invariant knowledge. Specifically, we enhance the Chinese word by injecting external lexical knowledge base into the context-aware word embeddings, and then combine with sentence-level semantics to represent the domain knowledge. To learn the domain-invariant knowledge, we replace the conventional adversarial method with novel contrastive regularization to further improve the generalization abilities. Extensive experiments conducted over the labeled source domain MSRA and the unlabeled target domain Social Media and News show that our approach outperforms state-of-the-arts, and achieves the improvement of F1 score by 8.7% over the baseline.