pwc-archive/papers-with-abstracts huggingface dataset:
Error-Correcting Codes For Approximate Neural Sequence Prediction
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Language Modeling, Language Modelling, Prediction, Text Generation

We propose a novel neural sequence prediction method based on \textit{error-correcting codes} that avoids exact softmax normalization and allows for a tradeoff between speed and performance. Error-correcting codes represent predictions and targets as a binary code where each bit is represented by a logit. The codebook is arranged such that similar tokens are close to each other using word embedding similarity, ensuring that incorrect predictions are at least semantically close to the target. We also address the well-established problem of compounding errors by mixing the latent codes of past predictions and past targets in one of two ways: (1) according to a predefined sampling schedule or (2) a differentiable sampling procedure that replaces the argmax operation. Low dimensional codes show similar performance to models that use the full softmax and outperform alternative approximate methods for language modeling and text generation, while generation further benefits from our mixture sampling.
Pinyin-bert: A new solution to Chinese pinyin to character conversion task
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Language Modeling, Language Modelling

Pinyin to Character conversion (P2C) task is the key task of Input Method Engine (IME) in commercial input software for Asian languages, such as Chinese, Japanese, Thai language, and so on. The dominant technique is Ngram language model together with smoothing technique. However, Ngram model's low capacity limits its performance. Under the trend of deep learning, this paper choose the powerful bert network architecture and propose Pinyin-bert to solve the P2C task, which achieves substantial performance improvement from Ngram model. Furthermore, we combine Pinyin-bert with Ngram model under Markov model's framework and improve performance further. Lastly, we design a way to incorporate external lexicon into Pinyin-bert so as to adapt to the out of domain.
Multimodal Learning: Are Captions All You Need?
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: All, Image Captioning

In today's digital world, it is increasingly common for information to be multimodal: images or videos often accompany text. Sophisticated multimodal architectures such as ViLBERT, VisualBERT, and LXMERT have achieved state-of-the-art performance in vision-and-language tasks. However, existing vision models cannot represent contextual information and semantics like transformer-based language models can. Fusing the semantic-rich information coming from text becomes a challenge. In this work, we study the alternative of first transforming images into text using image captioning. We then use transformer-based methods to combine the two modalities in a simple but effective way. We perform an empirical analysis on different multimodal tasks, describing the benefits, limitations, and situations where this simple approach can replace large and expensive handcrafted multimodal models.
Sparsity Regularization for Chinese Spelling Check
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Sentence

The Chinese Spelling Check (CSC) research objective is to detect and correct the spelling errors in the input. Generally, the number of incorrect characters in the input is far less than the correct, so the error probability sequence of input sentence predicted by the detection module should be sparse and sharp. However, all existing work has ignored this problem. In this paper, we add a sparsity regularization item to the objective function to make the output of the detection module close to sparse and sharp. We study two kinds of regularization: L1  regularization and minimum entropy regularization. Extensive experiments on the SIGHAN show that the sparsity regularization proposed in this paper can effectively improve the performance of the CSC model while without increasing the computational complexity. In addition, the robustness experiment results show that our method is robust.
Perturbation-based Self-supervised Attention for Text Classification
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Classification, Sentence, text-classification, Text Classification

For text classification, the traditional attention mechanism usually pays too much attention to words that appear frequently and needs a lot of labeled data for learning a good distribution. Introducing human attention is a classical method, but it needs a high cost of manual labeling. This paper proposes a perturbation-based self-supervised attention approach to guide attention learning without any annotation overheads. Specifically, we add as much noise as possible to all the words in the sentence simultaneously while without changing their semantics and predictions. According to the words that tolerate more noise are supposed to be less significant, we can obtain attention supervision information and utilize it to refine the attention distribution. Experimental results on three text classification tasks show that our approach can significantly promote the performance of current attention-based models and is more effective than existing self-supervised methods. We also provide visualization analysis to verify the effectiveness of our approach.
A Generative Approach for Mitigating Structural Biases in Natural Language Inference
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Natural Language Inference

Many natural language inference (NLI) datasets contain biases that allow models to perform well by only using a biased subset of the input, without considering the remainder features. For instance, models are able to classify samples by only using the hypothesis, without learning the true relationship between it and the premise.  These structural biases lead discriminative models to learn unintended superficial features and generalize poorly out of the training distribution. In this work, we reformulate the NLI task as a generative task, where a model is conditioned on the biased subset of the input and the label and generates the remaining subset of the input. We show that by imposing a uniform prior, we obtain a provably unbiased model. Through synthetic experiments, we find that this approach is highly robust to large amounts of bias. We then demonstrate empirically on two types of natural bias that this approach leads to fully unbiased models in practice. However, we find that generative models are difficult to train and generally perform worse than discriminative baselines. We highlight the difficulty of the generative modeling task in the context of NLI as a cause for this worse performance. Finally, by fine-tuning the generative model with a discriminative objective, we reduce the performance gap between the generative model and the discriminative baseline, while allowing for a small amount of bias.
Speciesist Language and Nonhuman Animal Bias in English Masked Language Models
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11

Various existing studies have analyzed what social biases are inherited by NLP models. These biases may directly or indirectly harm people, therefore previous studies have focused only on human attributes. If the social biases in NLP models can be indirectly harmful to humans involved, then the models can also indirectly harm nonhuman animals. However, no research on social biases in NLP regarding nonhumans exists. In this paper, we analyze biases to nonhuman animals, i.e. speciesist bias, inherent in English Masked Language Models. We analyze this bias using template-based and corpus-extracted sentences which contain speciesist (or non-speciesist) language, to show that these models tend to associate harmful words with nonhuman animals. Our code for reproducing the experiments will be made available on GitHub.
Night Owls and Majestic Whales: Modeling Metaphor Comprehension as a Rational Speech Act over Vector Representations of Lexical Semantics
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Attribute

While they are one of the few computational models that directly capture some of the pragmatic processes underlying language reasoning, current Rational Speech Act (RSA) models of metaphor do not align well with contemporary accounts of metaphor comprehension. The following research project leverages GloVe word vectors to capture pragmatic language reasoning in metaphoric utterances using an updated RSA framework. The model yields higher posterior probabilities for the same attributes that humans deem relevant in metaphoric utterances over erroneous ones in 89% of all cases, validating the use of word vectors to generate prior probabilities in an RSA framework. When presented with biased priors like listeners are in many naturalistic conversations, the model accurately matches human judgements of the top-most relevant attribute of a topic/target indicated by a metaphoric utterance 90% of the time.
Multi-Stage Framework with Refinement based Point Set Registration for Unsupervised Bi-Lingual Word Alignment
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Machine Translation, Sentence, Transfer Learning, Translation, Word Alignment, Word Embeddings, Word Similarity

Cross-lingual alignment of word embeddings play an important role in knowledge transfer across languages, for improving machine translation and other multi-lingual applications. Current unsupervised approaches rely on learning structure preserving linear transformations using adversarial networks and refinement strategies. However, such techniques, tend to suffer from instability and convergence issues, requiring tedious fine-tuning of parameter setting. This paper proposes BioSpere, a novel multi-stage framework for unsupervised mapping of bi-lingual word embeddings onto a shared vector space, by combining adversarial initialization, refinement procedure and point set registration algorithm. We show that our framework alleviates the above shortcomings, and is robust against variable adversarial learning performance and parameter choices. Experiments for parallel dictionary induction, sentence translation and word similarity demonstrate state-of-the-art results for BioSpere on diverse language pairs.
PicTalky: Augmentative and Alternative Communication Software for Language Developmental Disabilities
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11

Children with language disabilities face communication difficulties in social life. They are often deprived of the opportunity to participate in social activities due to their difficulty in understanding or using natural language. In this regard, Augmentative and Alternative Communication (AAC) is a practical means of communication for children with language disabilities. In this study, we propose PicTalky, which is an AI-based AAC system that helps children with language developmental disabilities to improve their communication skills and language comprehension abilities. PicTalky can process both text and pictograms more accurately by connecting a series of neural-based NLP modules. Moreover, we perform quantitative and qualitative analyses on the essential features of PicTalky. It is expected that those suffering from language problems will be able to express their intentions or desires more easily and improve their quality of life by using this service. We have made the models freely available alongside a demonstration of the Web interface. Furthermore, we implemented robotics AAC for the first time by applying PicTalky to the NAO robot.
Relation Extraction from Tables using Artificially Generated Metadata
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Relation, Relation Extraction

Relation Extraction (RE) from tables is the task of identifying relations between pairs of columns of a table. Generally, RE models for this task require labelled tables for training. These labelled tables can also be generated artificially from a Knowledge Graph (KG), which makes the cost to acquire them much lower in comparison to manual annotations. However, unlike real tables, these synthetic tables lack associated metadata, such as, column-headers, captions, etc; this is because synthetic tables are created out of KGs that do not store such metadata. Meanwhile, previous works have shown that metadata is important for accurate RE from tables. To address this issue, we propose methods to artificially create some of this metadata for synthetic tables. Afterward, we experiment with a BERT-based model, in line with recently published works, that takes as input a combination of proposed artificial metadata and table content. Our empirical results show that this leads to improvements of 9\%-45\% in F1 score, in absolute terms, over 2 tabular datasets.
Salient Phrase Aware Dense Retrieval: Can a Dense Retriever Imitate a Sparse One?
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Open-Domain Question Answering, Passage Retrieval, Question Answering, Retrieval

Despite their recent popularity and well known advantages, dense retrievers still lag behind sparse methods such as BM25 in their ability to reliably match salient phrases and rare entities in the query. It has been argued that this is an inherent limitation of dense models. We disprove this claim by introducing the Salient Phrase Aware Retriever (SPAR), a dense retriever with the lexical matching capacity of a sparse model. In particular, we show that a dense retriever Λ can be trained to imitate a sparse one, and SPAR is built by augmenting a standard dense retriever with Λ. When evaluated on five open-domain question answering datasets and the MS MARCO passage retrieval task, SPAR sets a new state of the art for dense and sparse retrievers and can match or exceed the performance of more complicated dense-sparse hybrid systems.
SHCT: A Successively Hierarchical Conditional Transformer for Controllable Paraphrase Generation
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Diversity, Paraphrase Generation

Paraphrase generation has consistently been a challenging area in the field of NLP. Despite the considerable achievements made by previous work, existing methods lack a flexible way to include multiple controllable attributes to enhance the diversity of paraphrased sentences. To overcome this challenge, we propose a Successively Hierarchical Conditional Transformer(SHCT) to tackle this task. SHCT is based on a combination of Conditional Variational AutoEncoder(CVAE) with Transformer framework to benefit from the advantage of generating diversified words. More specifically, our SHCT deploys multi-head attention and dynamic memory mechanism to keep the interaction between each of the attributes and the corresponding encoder layer hidden state. To address the problem of absorbing flexible attributes, we apply a hierarchical structure to our SHCT which enables the framework to couple the CVAE latent variables with encoder layer hidden states successively. In addition, Our SHCT is trained by minimizing a tailor-designed loss for producing paraphrased sentences as required.  Finally, We conduct extensive experiments to substantiate the validity and effectiveness of our proposed model. The results show that SHCT significantly outperforms the existing state-of-the-art approaches and generates more diverse paraphrased sentences.
Accurate Online Posterior Alignments for Principled Lexically-Constrained Decoding
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR September 2021 9
Tasks: Machine Translation, Translation

Online alignment in machine translation refers to the task of aligning a target word to a source word when the target sequence has only been partially decoded. Good online alignments facilitate important applications such as lexically constrained translation where user-defined dictionaries are used to inject lexical constraints into the translation model. We propose a novel posterior alignment technique that is truly online in its execution and superior in terms of alignment error rates compared to existing methods. Our proposed inference technique jointly considers alignment and token probabilities in a principled manner and can be seamlessly integrated within existing constrained beam-search decoding algorithms. On five language pairs, including two distant language pairs, we achieve consistent drop in alignment error rates. When deployed on seven lexically constrained translation tasks, we achieve significant improvements in BLEU specifically around the constrained positions.
Learning to Teach with Student Feedback
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Knowledge Distillation

Knowledge distillation (KD) has gained much attention due to its effectiveness in compressing large-scale pre-trained models. In typical KD methods, the small student model is trained to match the soft targets generated by the big teacher model. However, the interaction between student and teacher is one-way. The teacher is usually fixed once trained, resulting in static soft targets to be distilled. This one-way interaction leads to the teacher's inability to perceive the characteristics of the student and its training progress. To address this issue, we propose Interactive Knowledge Distillation (IKD), which also allows the teacher to learn to teach from the feedback of the student. In particular, IKD trains the teacher model to generate specific soft target at each training step for a certain student. Joint optimization for both teacher and student is achieved by two iterative steps: a course step to optimize student with the soft target of teacher, and an exam step to optimize teacher with the feedback of student. IKD is a general framework that is orthogonal to most existing knowledge distillation methods. Experimental results show that IKD outperforms traditional KD methods on various NLP tasks.
A Multilingual Bag-of-Entities Model for Zero-Shot Cross-Lingual Text Classification
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Classification, Entity Typing, Language Modeling, Language Modelling, text-classification, Text Classification, Topic Classification

We present a multilingual bag-of-entities model that effectively boosts the performance of zero-shot cross-lingual text classification by extending a multilingual pre-trained language model (e.g., M-BERT). It leverages the multilingual nature of Wikidata: entities in multiple languages representing the same concept are defined with a unique identifier. This enables entities described in multiple languages to be represented using shared embeddings.
A model trained on entity features in a resource-rich language can thus be directly applied to other languages. Our experimental results on cross-lingual topic classification (using the MLDoc and TED-CLDC datasets) and entity typing (using the SHINRA2020-ML dataset) show that the proposed model consistently outperforms state-of-the-art models.
Exploring the Influence of Dialog Input Format for Unsupervised Clinical Questionnaire Filling
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Natural Language Inference

In the medical field, we have seen the emergence of health-bots that interact with patients to gather data and track their state. One of the downstream application is automatic questionnaire filling, where the content of the dialog is used to automatically fill a pre-defined medical questionnaire. Answering questions from the dialog context can be cast as a Natural Language Inference (NLI) task and therefore benefit from current pre-trained NLI models. However, these models have not been generally trained on dialog input format, which may have an influence on their performance. In this paper, we study the influence of dialog input format on the task. Our results demonstrate that dialog pre-processing and content selection can significantly improve performance of zero-shot models.
On Isotropy Calibration of Transformer Models
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11

Different studies of the embedding space of transformer models suggest that the distribution of contextual representations is highly anisotropic - the embeddings are distributed in a narrow cone. Meanwhile, static word representations (e.g., Word2Vec or GloVe) have been shown to benefit from isotropic spaces. Therefore, previous work has developed methods to calibrate the embedding space of transformers in order to ensure isotropy. However, a recent study (Cai et al. 2021) shows that the embedding space of transformers is locally isotropic, which suggests that these models are already capable of exploiting the expressive capacity of their embedding space. In this work, we conduct an empirical evaluation of state-of-the-art methods for isotropy calibration on transformers and find that they do not provide consistent improvements across models and tasks. These results support the thesis that, given the local isotropy, transformers do not benefit from additional isotropy calibration.
A System to Filter out Unwanted Social Media Content in Real-time on iPhones
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Sentiment Analysis

Social media users are often harassed. This paper presents a patented system to filter out harassing content before it reaches the recipient. Our first version is for the iPhone. To detect harassment, we adopted sentiment analysis with a supervised learning approach that combines Machine Learning (ML) text classifiers with a lexicon approach that provides a feedback loop to retrain the ML model with unknown terms. Because good data is essential to obtain the best output of any system, we focused on validating our labeled data. Our results on static and real-time data have an accuracy of, respectively, 90% and 94%. Our labeled data validation allows us to correct labels; we also realized the need to increase the number of sets in our lexicons. Our prototype demonstrates that we are able to build an AI infrastructure to filter out harassment on an iPhone in real-time with good results.
We need to talk about random seeds
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR September 2021 9
Tasks: Position

Modern neural network libraries all take as a hyperparameter a random seed, typically used to determine the initial state of the model parameters. This position piece argues that there are some safe uses for random seeds: as part of the hyperparameter search to select a good model, creating an ensemble of several models, or measuring the sensitivity of the training algorithm to the random seed hyperparameter. It argues that some uses for random seeds are risky: using a fixed random seed for ``replicability'' and varying only the random seed to create score distributions for performance comparison. An analysis of 85 recent publications from the ACL Anthology shows that more than 50% contain risky uses of random seeds.