pwc-archive/papers-with-abstracts huggingface dataset:
Eye Gaze and Self-attention: How Humans and Transformers Attend Words in Sentences
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11

Attention mechanisms are used to describe human reading processes and natural language processing by transformer neural networks. On the surface, attention appears to be very different under these two contexts. However, this paper presents evidence that there are links between the two during reading tasks.

During reading, the dwell times of human eye movements were strongly correlated with the attention patterns occurring in the early layers of pre-trained transformers such as BERT. Furthermore, we explored what factors lead to variations in these correlations and observed that data were more correlated when humans read for comprehension than when they were searching for specific information. Additionally, the strength of a correlation was not related to number of parameters within a transformer.
Modeling speech recognition and synthesis simultaneously: Encoding and decoding lexical and sublexical semantic information into speech with no access to speech data
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: speech-recognition, Speech Recognition, Speech Synthesis

Human speakers encode information into raw speech which is then decoded by the listeners. This complex relationship between encoding (production) and decoding (perception) is often modeled separately. Here, we test how decoding of lexical and sublexical semantic information can emerge automatically from raw speech in unsupervised generative deep convolutional networks that combine both the production and perception principle. We introduce, to our knowledge, the most challenging objective in unsupervised lexical learning: an unsupervised network that must learn to assign unique representations for lexical items with no direct access to training data. We train several models (ciwGAN and fiwGAN by Beguš 2021) and test how the networks classify raw acoustic lexical items in the unobserved test data. Strong evidence in favor of lexical learning emerges. The architecture that combines the production and perception principles is thus able to learn to decode unique information from raw acoustic data in an unsupervised manner without ever accessing real training data. We propose a technique to explore lexical and sublexical learned representations in the classifier network. The results bear implications for both unsupervised speech synthesis and recognition as well as for unsupervised semantic modeling as language models increasingly bypass text and operate from raw acoustics.
One General Teacher for Multi-Data Multi-Task: A New Knowledge Distillation Framework for Discourse Relation Analysis
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Knowledge Distillation, Multi-Task Learning, Reading Comprehension, Relation, Relation Classification

Automatically identifying the  discourse relations can help many downstream NLP tasks such as reading comprehension. It can be categorized into explicit and implicit discourse relation recognition (EDRR and IDRR). Due to the lack of connectives, IDRR remains to be a big challenge. A good number of methods have been developed to combine explicit data with implicit ones under the multi-task learning framework. However, the difference in linguistic property and class distribution makes it hard to directly optimize EDRR and IDRR with multi-task learning.

In this paper, we take the first step to exploit the knowledge distillation (KD) technique for  discourse relation analysis. Our target is to train a focused single-data single-task student with the help of a general multi-data multi-task teacher. Specifically, we first train one teacher for both the top and second level relation classification tasks with explicit and implicit data. We then transfer the feature embeddings and soft labels from the teacher network to the student network. Extensive experimental results on the popular PDTB dataset proves that our model achieves a new state-of-the-art performance. We also show the effectiveness of our proposed KD architecture through detailed analysis.
ARCNN: A Semantic Enhanced Relation Detection Model for Knowledge Base Question Answering
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Knowledge Base Question Answering, Question Answering, Relation

Relation detection plays an important role in knowledge base question answering (KBQA), and it is critical for the final performance of KBQA systems. The previous works mainly focused on enriching the information representations of questions and relations, and neglected the interaction information of questions and relations and different tokens within the relation. In this paper, we propose a semantic enhanced relation detection model called ARCNN, which is carefully designed by combining BiGRU, multi-scale semantic extracted CNN, and different attention mechanisms in a seamless way. Moreover, we combine four levels of relation abstractions to ensure the integrity of relation information and hence to enrich the relation representation. The experimental results on two benchmarks show that our ARCNN model achieves new state-of-the-art accuracies of 96.42% for SimpleQuestions and 90.4% for WebQuestions. Moreover, it helps our KBQA system to yield the accuracy of 81.5% and the F1 score of 72.0% on two benchmarks, respectively.
Zero-shot Cross-lingual Conversational Semantic Role Labeling
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Response Generation, Semantic Role Labeling, Translation

While conversational semantic role labeling (CSRL) has shown its usefulness on Chinese conversational tasks, it is still under-explored in non-Chinese languages due to the lack of multilingual CSRL annotations for the parser training. To avoid expensive data collection and error-propagation of translation-based methods, we present a simple but effective approach to perform zero-shot cross-lingual CSRL. Our model implicitly learns language-agnostic, conversational structure-aware and semantically rich representations with the hierarchical encoders and elaborately designed pre-training objectives. Experimental results show that our cross-lingual model not only outperforms baselines by large margins but it is also robust to low-resource scenarios. More importantly, we confirm the usefulness of CSRL to English conversational tasks such as question-in-context rewriting and multi-turn dialogue response generation by incorporating the CSRL information into the downstream conversation-based models. We believe this finding is significant and will facilitate the research of English dialogue tasks which suffer the problems of ellipsis and anaphora.
Weight Squeezing: Reparameterization for Knowledge Transfer and Model Compression
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Model Compression, text-classification, Text Classification, Transfer Learning

In this work, we present a novel approach to simultaneous knowledge transfer and model compression called \textbf{Weight Squeezing}. With this method, we perform knowledge transfer from a teacher model \textbf{by learning the mapping from its weights to smaller student model weights}.

We applied Weight Squeezing to a pre-trained text classification model based on a BERT-Medium model. We compared our method to various other knowledge transfer and model compression methods using the GLUE multitask benchmark. We observed that our approach produces better results while being significantly faster than other methods for training student models.

We also proposed a variant of Weight Squeezing called Gated Weight Squeezing, in which we combined fine-tuning a small BERT model and learning mapping from larger BERT weights. We showed that, in most cases, fine-tuning a BERT model with Gated Weight Squeezing can outperform plain fine-tuning.
First Bilingual Word Embeddings for te reo Māori and English: Towards Code-switching Detection in a Low-resourced setting
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Word Embeddings

Māori speakers are bilingual, where  Māori is code-switched with English. With Māori being low-resourced for technology development, there are minimal resources available for Māori-English code-switch detection. This research collected the Māori-English Words corpus containing more than 71M words, and developed the first open-sourced Māori-English bilingual word embeddings model. We provide experimental evidence to show an improvement of atleast 5% in F1-scores when the bilingual embeddings we developed is used for code-switch detection compared to the already-available English-only embeddings trained on a relatively large database. This study is the first one providing resources and exploring deep learning for Māori-English code-switch detection.
Cross-Lingual UMLS Named Entity Linking using UMLS Dictionary Fine-Tuning
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Descriptive, Entity Linking, Language Modelling

We study cross-lingual UMLS named entity linking, where mentions in a given source language are mapped to UMLS concepts, most of which are labeled in English. We propose a general solution that can be easily adapted to any source language and demonstrate the method on Hebrew documents. 
Our cross-lingual framework includes an offline unsupervised construction of a bilingual UMLS dictionary and a per-document pipeline which identifies UMLS candidate mentions and uses a fine-tuned pretrained transformer language model to filter candidates according to context.
Our method exploits a small dataset of manually annotated UMLS mentions in the source language and uses this supervised data in two ways: to extend the unsupervised UMLS dictionary and to fine-tune the contextual filtering of candidate mentions in full documents. Our method addresses cross-lingual UMLS NEL in a low resource setting, where the ontology is large, there is a lack of descriptive text defining most entities, and labeled data can only cover a small portion of the ontology. We demonstrate results of our approach on both Hebrew and English. We achieve new state-of-the-art results on the Hebrew Camoni corpus, +8.9 F1 on average across three communities in the dataset. We also achieve new SOTA on the English dataset MedMentions with +7.3 F1.
Question-Led Semantic Structure Enhanced Attentions for VQA
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Question Answering, Visual Question Answering, Visual Question Answering (VQA)

The exploit of the semantic structure in the visual question answering (VQA) task is a trending topic where researchers are interested in leveraging internal semantics and bringing in external knowledge to tackle more complex questions. The prevailing approaches either encode the external knowledge separately from the local context, which magnificently increases the complexity of the ensemble system, or use graph neural networks to model the semantic structure in the context, which suffers from the limited reasoning capability due to the relatively shallow network. In this work, we propose a question-led structure extraction scheme using external knowledge and explore multiple training methods, including direct attention supervision, SGHMC-EM Bayesian multitask learning, and masking strategies, to aggregate the structural knowledge into deep models without changing the architectures.  We conduct extensive experiments on two domain-specific but challenging sub-tasks of VrR-VG dataset and demonstrate that our proposed methods achieve significant improvements over strong baselines, showing the promising potentials of applicability.
Investigating Logic Tensor Networks for Neural-Symbolic Argument Mining
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR September 2021 9
Tasks: Argument Mining, Tensor Networks

We present an application of neural-symbolic learning to argument mining.
We use Logic Tensor Networks to train neural models to jointly fit the data and satisfy specific domain rules.
Our experiments on a corpus of scientific abstracts indicate that including symbolic rules during the training process improves classification performance, compliance with the rules, and robustness of the results.
WeTS: A Benchmark for Translation Suggestion
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Machine Translation, Translation

Translation suggestion (TS), which provides alternatives for specific words or phrases given the entire documents generated by machine translation (MT), has been proven to play a significant role in post-editing (PE). There are two main pitfalls for previous researches in this line. First, most conventional works only focus on the overall performance of PE but ignore the exact performance of TS, which makes the progress of PE sluggish and less explainable; Second, as no publicly available golden dataset to support in-depth research for TS, almost all of the previous works conduct experiments on their in-house datasets or the noisy datasets built automatically, which makes their experiments hard to be reproduced and compared. To break these limitations mentioned above and spur the research in TS,  we create a benchmark dataset, called \emph{WeTS}, which is a golden corpus annotated by expert translators on four translation directions. Apart from the golden corpus, we also propose several methods to generate synthetic corpus which can be used to improve the performance substantially through pre-training. As for the model, we propose the segment-aware self-attention based Transformer for TS. Experimental results show that our approach achieves state-of-the-art results on all four directions, including English-to-German, German-to-English, Chinese-to-English, and English-to-Chinese.
A Neural Approach to KGQA via SPARQL Silhouette Generation
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Graph Question Answering, Machine Translation, NMT, Question Answering, Relation, Relation Linking, Semantic Parsing

Semantic  parsing  is  a  predominant  approach  to  solve  the Knowledge Graph Question Answering (KGQA) task where, natural language question is translated into a logic form such as  SPARQL.  Semantic  parsing  based  solutions  are  mostly modular/pipelined where, noise introduced by the upstream modules  for  entity/relation  linking  makes  it  hard  to  solve the  complex  questions.  Recently,  Neural  Machine  Translation (NMT) based  approaches have emerged  that are capable of handling complex questions. However, NMT-based approaches struggle with handling the large number of test entities and relations that are unseen during training. In this work,we propose a modular two-stage neural approach which combines best of both the worlds - NMT and semantic parsing pipeline. Stage-I of our approach comprises an NMT-based seq2seq module that translates a question into a sketch of the desired  SPARQL,  called  as SPARQL silhouette.  This  stage also contains a noise simulator which combines the masking scheme  with  an  entity/relation  linker  in  a  novel  manner  so as to take care of unseen entities/relation without blowing up the vocabulary of seq2seq module.
Stage-II of our approach comprises a Neural Graph Search (NGS) module which aims to distil the SPARQL silhouette in order to reduce the entity/relation linking noise.  
Experimental results show that, the quality of generated SPARQL silhouette is impressive for an ideal scenario where entity/relation linker is noise-free. For the realistic scenario (i.e. noisy linker), the quality of the SPARQL silhouette drops but  our NGS module recovers it considerably. We show that, our proposed approach improves state-of-the-art on LC-QuAD-1 dataset by an absolute margin of $3.72 \%$ $F_1$.
Improving Robustness of Language Models from a Geometry-aware Perspective
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Data Augmentation

Recent studies have found that removing the norm-bounded projection and increasing search steps in adversarial training can significantly improve robustness. However, we observe that a too large number of search steps can hurt accuracy. We aim to obtain strong robustness efficiently using fewer steps. Through a toy experiment, we find that perturbing the clean data to the decision boundary but not crossing it does not degrade the test accuracy. Inspired by this, we propose friendly adversarial data augmentation (FADA) to generate ``friendly'' adversarial data. On top of FADA, we propose geometry-aware adversarial training (GAT) to perform adversarial training (e.g., FGM) on friendly adversarial data so that we can save a large number of search steps. Comprehensive experiments across two widely used datasets and three pre-trained language models demonstrate that GAT can obtain stronger robustness via less steps. In addition, we provide extensive empirical results and in-depth analyses on robustness to facilitate future studies.
Lot or Not: Identifying Multi-Quantity Offerings in E-Commerce
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Friction

The term \textit{lot} in \ecom is defined to mean an offering that contains a collection of multiple identical items for sale. In a large online marketplace, lot offerings play an important role, allowing buyers and sellers to set price levels to optimally balance supply and demand needs. In spite of their central role, \ecom platforms often struggle to identify lot offerings, since explicit lot status identification is frequently not provided by sellers. The ability to identify lot offerings plays a key role in many fundamental \ecom tasks, from matching offerings to catalog products, through ranking \ecom search results, to providing effective pricing guidance. In this work, we seek to determine the lot status (and lot size) of each offering, in order to facilitate an improved buyer experience, while reducing the friction for sellers posting new offerings. We demonstrate experimentally the ability to accurately classify offerings as lots (and predict their lot size) using only the offer title, by adapting state-of-the-art natural language techniques to the lot identification  problem.
Generating Authentic Adversarial Examples beyond Meaning-preserving with Doubly Round-trip Translation
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Machine Translation, NMT, Translation

Generating adversarial examples for Neural Machine Translation (NMT) with single Round-Trip Translation (RTT) has achieved promising results by releasing the meaning-preserving restriction. However, a potential pitfall for this approach is that we cannot decide whether the generated examples are adversarial to the target NMT model or the auxiliary backward one, as the reconstruction error through the RTT can be related to either. To remedy this problem, we propose a new definition for NMT adversarial examples based on the Doubly Round-Trip Translation (DRTT). Specifically, apart from the source-target-source RTT, we also consider the target-source-target one, which is utilized to pick out the authentic adversarial examples for the target NMT model. Additionally, to enhance the robustness of the NMT model, we introduce the masked language models to construct bilingual adversarial pairs based on DRTT, which are used to train the NMT model directly. Extensive experiments on both the clean and noise test sets (including the artificial and natural noise) show that our approach substantially improves the robustness of NMT models.
Making Small Language Models Better Few-Shot Learners
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Few-Shot Learning, Knowledge Distillation, Sentence, Sentence Classification

Large-scale language models coupled with prompts have shown remarkable performance on few-shot learning. However, through systematic experiments, we find that the few-shot performance of small language models is poor, and using prompts on them brings fewer improvements than on larger ones. In this paper, we propose \textbf{SMASH}, an approach to improve \textbf{SMA}ll language models' few-\textbf{SH}ot ability by training on intermediate tasks before prompt-based fine-tuning on downstream tasks. We design intermediate tasks for sentence-pair tasks and single-sentence classification tasks by creating training examples with prompt templates similar to downstream tasks using sentences sampled from a large-scale unsupervised corpus, and apply knowledge distillation to distill from outputs of larger pre-trained models as training objective. We conduct extensive experiments and show that SMASH can make a 6-layer DistilRoBRETa-base achieve comparable performance on few-shot datasets to a 12-layer RoBERTa-base at a low cost.
Few-shot Controllable Style Transfer for Low-Resource Multilinugal Settings
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Attribute, Sentence, Style Transfer, Text Anonymization, Text Simplification

Style transfer is the task of rewriting an input sentence into a target style while approximately preserving its content. While most prior literature assumes access to large style-labelled corpora, recent work (Riley et al. 2021) has attempted "few-shot" style transfer using only 3-10 sentences at inference for extracting the target style. In this work we study a relevant low-resource setting: style transfer for languages where no style-labelled corpora are available. We find that existing few-shot methods perform this task poorly, with a strong tendency to copy inputs verbatim. We push the state-of-the-art for few-shot style transfer with a new method modeling the stylistic difference between paraphrases. When compared to prior work using automatic and human evaluations, our model achieves 2-3x better performance and output diversity in formality transfer and code-mixing addition across seven languages. Moreover, our method is better able to control the amount of style transfer using an input scalar knob. We report promising qualitative results for several attribute transfer directions, including sentiment transfer, text simplification, gender neutralization and text anonymization, all without retraining the model. Finally we found model evaluation to be difficult due to the lack of evaluation datasets and metrics for many languages. To facilitate further research in formality transfer for Indic languages, we crowdsource annotations for 4000 sentence pairs in four languages, and use this dataset to design our automatic evaluation suite.