pwc-archive/papers-with-abstracts huggingface dataset:
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11

The performance of multilingual pretrained models is highly dependent on the availability of monolingual or parallel text present in a target language. Thus, the majority of the world’s languages cannot benefit from recent progress in NLP as they have no or limited textual data. To expand possibilities of using NLP technology in these under-represented languages, we systematically study strategies that relax the reliance on conventional language resources through the use of bilingual lexicons, an alternative resource with much better language coverage. We analyze different strategies to synthesize textual or labeled data using lexicons, and how this data can be combined with monolingual or parallel text when available. For 19 under-represented languages across 3 tasks, our methods lead to consistent improvements of up to 5 and 15 points with and without extra monolingual text respectively.  Overall, our study highlights how NLP methods can be adapted to thousands more languages that are under-served by current technology.
One Country, 700+ Languages: NLP Challenges for Underrepresented Languages and Dialects in Indonesia
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11

NLP research is impeded by a lack of resources and awareness of the challenges presented by under-represented languages and dialects. Focusing on the languages spoken in Indonesia, the second most linguistically diverse and the fourth most populous nation of the world, we provide an overview of the current state of NLP research for Indonesia's 700+ languages. We highlight challenges in Indonesian NLP and how these affect the performance of current NLP systems. Finally, we provide general recommendations to help develop NLP technology not only for languages of Indonesia, but also other underrepresented languages.
Challenges in Region-Specific Image Captioning: A Deep Learning Approach
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Deep Learning, Image Captioning

Region-specific image captioning is the task of generating a caption from an image such that the caption is about the specific region in that image.
This paper describes the challenges involved in region-specific image captioning and provides several methods to utilize the region-specific features to enhance the quality of the captions in addition to utilizing the features from the whole image. Our experiments on real-world data sets demonstrate that generating region-specific captions is challenging even after utilizing the information specific to the region.
We analyze the variables impacting the quality of the captions which include the bounding box size and the region-specific feature extractor.
Predictive text for agglutinative and polysynthetic languages
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Language Modelling, Segmentation

This paper presents a set of experiments in the area of morphological modelling and 
predictioning. We examine the tasks of segmentation
and predictive text entry for two under-resourced and indigenous languages, K'iche'
and Chukchi. We use different segmentation methods to make datasets for language modelling and then train models of different types: single-way segmented, which are trained using data from one segmentor; two-way segmented, which are trained using concatenated data from two segmentors; and finetuned, which are trained on two datasets from different segmentors. We 
measure word and character level perplexities of the language models and find that single-way segmented models trained using morphologically segmented data and finetuned models work the best.
Finally, we test the language models on the task of predictive text entry using gold standard data and measure
the average number of clicks per character and keystroke savings rate. We find that the models trained using morphologically segmented data work better,
although with substantial room for improvement. At last, we propose the usage of morphological segmentation in order to improve the end-user experience while using predictive text and we plan on testing this assumption by training other models and experimenting on more languages.
On Spoken Language Understanding Systems for Low Resourced Languages
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), intent-classification, Intent Classification, speech-recognition, Speech Recognition, Spoken Language Understanding

Spoken dialog systems are slowly becoming and integral part of the human experience due to their various advantages over textual interfaces. Spoken language understanding (SLU) systems are fundamental building blocks of spoken dialog systems. But creating SLU systems for low resourced languages is still a challenge. In a large number of low resourced settings we don't have access to enough data to build automatic speech recognition (ASR) technologies, which are fundamental to any SLU system. Also, ASR based SLU systems do not generalize to unwritten languages. In this paper, we present a series of experiments to explore an extremely low resourced setting - something we refer to as a true k-shot setting, where we perform intent classification with systems trained on different values of k. We test our system on English and Flemish and find that even in such granular settings and no language specific ASR technology, we can create SLU systems that can be deployed in the real world.
Alleviating the Sparsity of Open Knowledge Graphs with Pretrained Contrastive Learning
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Contrastive Learning, Knowledge Graphs

Due to the sparsity of formal knowledge and the roughness of non-ontological construction methods, relevant facts are often missing in Open Knowledge Graphs (OpenKGs). Although existing completion methods have achieved promising performance, they do not alleviate the sparsity problem of OpenKGs. Owing to fewer training chances caused by sparse links, many few-shot and zero-shot entities cannot fully learn high-dimensional features. In this paper, we propose a new OpenKG Contrastive Learning (OKGCL) model to alleviate the sparsity with contrastive entities and relations. OKGCL designs (a) negative entities to discriminate different entities with the same relation, (b) negative relations to discriminate different relations with the same entity-pair, and (c) \emph{self} positive samples to give zero-shot and few-shot entities chances to learn discriminative representations. Extensive experiments on benchmark datasets show the superiority of OKGCL over state-of-the-art models.
A Scalable Holistic approach for Age and Gender inference of Twitter Users
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Contrastive Learning

Numerous studies have focused on inference of age and gender. We consider a new approach that takes advantage of contrastive learning methods by using both text and image content for this prediction task. We also consider the case where only text or image data is available. Under both of these conditions, we show that our model achieves better performance than the state-of-the-art ones, and still performs well with text/images only. Moreover, because demographic datasets can be small, we also consider combining different datasets to understand when augmentation is valuable and when it is not.
Towards Interpretable Math Word Problem Solving with Grounded Linguistic Logic Reasoning
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Math, Math Word Problem Solving

Automatically math word problem (MWP) solving is a challenging artificial intelligence task since a machine should be able to not only understand problem comprehensively on linguistics but also the  grounded math logic entailed in problem. Recently, lots of deep learning models have made great progress in MWP solving on answer accuracy, they rely on shallow heuristics to achieve high performance, lacking of grounded math logic reasoning, which makes them uninterpretable. To address this issue and push the research boundary of MWPs to interpretable MWP solving, we construct a large-scale and high-quality MWP dataset named InterMWP which consists of 11507 MWP data and annotates interpretable algebraic knowledge formulas as the grounded linguistic logic of each solving equation and asks for a solver to output the formulas when it decides current predicted node is a inner-node (operator) during expression reasoning. We further propose a strong baseline called InterSolver to show the effectiveness of our constructed dataset and show how to harvest these logic knowledge by fusing logic knowledge with semantic representation to improve problem solving and make a step towards providing interpretability. Experimental results shows that our InterSolver has strong logical formula-based interpretability while achieving high answer accuracy simultaneously.
Aligned Weight Regularizers for Pruning Pretrained Neural Networks
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Knowledge Distillation, Language Modeling, Language Modelling

Pruning aims to reduce the number of parameters while maintaining performance close to the original network. This work proposes a novel \emph{self-distillation} based pruning strategy, whereby the representational similarity between the pruned and unpruned versions of the same network is maximized. Unlike previous approaches that treat distillation and pruning separately, we use distillation to inform the pruning criteria, without requiring a separate student network as in knowledge distillation. We show that the proposed {\em cross-correlation objective for self-distilled pruning} implicitly encourages sparse solutions, naturally complementing magnitude-based pruning criteria. Experiments on the GLUE and XGLUE benchmarks show that self-distilled pruning increases mono- and cross-lingual language model performance. Self-distilled pruned models also outperform smaller Transformers with an equal number of parameters and are competitive against (6 times) larger distilled networks. We also observe that self-distillation (1) maximizes class separability, (2) increases the signal-to-noise ratio, and (3) converges faster after pruning steps, providing further insights into why self-distilled pruning improves generalization.
DialogConv: A Lightweight Fully Convolutional Network for Multi-view Response Selection
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Retrieval

Current end-to-end retrieval-based dialogue systems are primarily based on Recurrent Neural Networks or Transformers with attention mechanisms. Despite promising results have been achieved, these models usually suffer from slow inference speed or an enormous amount of parameters. In this paper, we propose a novel lightweight fully convolutional architecture called DialogConv for the response selection. DialogConv is built exclusively on convolutions for distilling the matching features of context and response. The dialogue is modeled in a 3D view, where DialogConv conducts convolution operations on embedding dimension, word dimension and utterance dimension iteratively to capture richer semantic information from a multi-view of context. On four benchmark datasets, DialogConv is approximately 4.0x smaller and up to 27x faster in inference compared with strong baselines. Moreover, DialogConv can achieve competitive performance results on four public datasets.
TextMosaic: A New Data Augmentation Method for Named Entity Recognition Using Document-Level Contexts
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Data Augmentation, GPU, named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER), NER, object-detection, Object Detection

Named Entity Recognition (NER) often faces the problem of lacking massive and diverse annotation data, recent advances of pre-training techniques have shown great power on such low-resource tasks.  However, the robustness of NER models is still insufficient which motivates us for efficient text enhancement method.  Inspired by the mosaic augmentation method for object detection, this paper puts forward a novel data augmentation method named TextMosaic for NER through span sampling,over-sampling, and random sampling, which takes full consideration of the context-sensitive relevance. Meanwhile, sliding window is leveraged in the sampling to effectively capture rich document-level information and solve the problem of label imbalance. Our proposed method won the Top 1 in the robustness evaluation of CCIR Cup 2021.  We also conduct extensive experiments on OntoNote 4.0 dataset, on which our method achieves higher accuracy and robustness for NER simultaneously.  Besides, it consumes less computing resources and makes the model capable of running in 1080ti GPU efficiently. The code will be open-sourced on Github.
Sparse Progressive Distillation: Resolving Overfitting under Pretrain-and-Finetune Paradigm
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Knowledge Distillation

Conventional wisdom in pruning Transformer-based language models is that pruning reduces the model expressiveness and thus is more likely to underfit rather than overfit. However, under the trending pretrain-and-finetune paradigm, we postulate a counter-traditional hypothesis, that is: pruning increases the risk of overfitting when performed at the fine-tuning phase. In this paper, 
we aim to address the overfitting problem and improve pruning performance via progressive knowledge distillation with error-bound properties. We show for the first time that reducing the risk of overfitting can help the effectiveness of pruning under the pretrain-and-finetune paradigm. Ablation studies and experiments on the GLUE benchmark show that our method outperforms the leading competitors across different tasks.
A Self-Adaptive Learning Rate and Curriculum Learning Based Framework for Few-Shot Text Classification
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Few-Shot Learning, Few-Shot Text Classification, Meta-Learning, text-classification, Text Classification

Due to the lack of labeled data in many realistic scenarios, a number of few-shot learning methods for text classification have been proposed, among which the meta learning based ones have recently attracted much attention. Such methods usually consist of a learner as the classifier and a meta learner for specializing the learner to tasks. For the learner, learning rate is crucial to its performance. However, existing methods treat it as a hyper parameter and adjust it manually, which is time-consuming and laborious. Intuitively, for different tasks and neural network layers, the learning rates should be different and self-adaptive. For the meta learner, it requires a good generalization ability so as to quickly adapt to new tasks. Therefore, we propose a novel meta learning framework, called MetaCLSLR, for few-shot text classification. Specifically, we present a novel meta learning mechanism to obtain different learning rates for different tasks and neural network layers so as to enable the learner to quickly adapt to new training data. Moreover, we propose a task-oriented curriculum learning mechanism to help the meta learner achieve a better generalization ability by learning from different tasks with increasing difficulties. Extensive experiments on three benchmark datasets demonstrate the effectiveness of MetaCLSLR.
Combating Spurious Features by Distribution Difference Regularization
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: coreference-resolution, Coreference Resolution

Prior studies show that spurious features are inevitable to avoid in the data collection process. These spurious features cause a shortcut for a model making bad prediction in real world test data due to ignoring the real features. In this work, we focus ondesigning a learning scheme to hinder the model from leveraging spurious features. To achieve this, prior studies usually make strong assumptions about the spurious features and identify them purely by manipulating the training data. In contrast, we make weaker assumptions and purpose a new framework for combating spurious features by observing the distribution shift between training and auxiliary data. In particular, with the help of unlabeled auxiliary data, we design a regularization technique based on the embedding distribution difference between training and auxiliary data to mitigate the effect of spurious features. Experimental results on NLI and coreference resolution tasks demonstrate that we improve the models on out-of-domain test data and reduce the contribution of spurious features in model predictions.
Remove Noise and Keep Truth: A Noisy Channel Model for Semantic Role Labeling
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Semantic Role Labeling

Semantic role labeling usually models structures using sequences, trees, or graphs. Past works focused on researching novel modeling methods and neural structures and integrating more features. In this paper, we re-examined the noise in neural semantic role labeling models, a problem that has been long-ignored. By proposing a noisy channel model structure, we effectively eliminate the noise in the labeling flow and thus improve performance. Without relying on additional features, our proposed novel model significantly outperforms a strong baseline on multiple popular semantic role labeling benchmarks, which demonstrates the effectiveness and robustness of our proposed model.
Enhancing Neural Machine Translation with Syntactic Ambiguities
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Machine Translation, Translation

Benefiting from the data-driven end-to-end model architecture, neural machine translation has obvious performance advantages over statistical machine translation, but its demand for data is also significantly greater, including monolingual and parallel corpus. Most of the past studies have focused on reducing the demand for parallel corpus or making more effective use of limited parallel corpus. In this work, we have studied a method of using ambiguity of syntactic structure to achieve more effective use of monolingual corpus. Experiments conducted on multiple benchmarks for various languages show that our method has a greater improvement than the method using back-translation only, demonstrating the effectiveness of our proposed method.
Counting What Deserves to be Counted for Graph Parsing
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Semantic Parsing

   Graph parsers rely on scoring every subgraphs for building a complete graph. In real syntactic parsing or semantic parsing, every types of subgraphs in terms of syntactic or semantic roles may generate quite unbalanced distribution, which seems not well captured by the current graph paring models. Thus we propose an enhanced model design to let the parser explicitly capture such kind of unbalanced distribution. In detail, we introduce Accumulative Operation-based Induction (AOI) attention mechanism to assign accumulative  scores for words. AOI scorer successfully approximates word-level unbalanced distribution. With conceptually simple but general-purpose design, our proposed AOI attention enhancement indeed leads to better parsing performance on a wide range of datasets of different parsing tasks, which verifies the scalability and robustness of capturing diverse subgraph distribution.
Learn to Adapt for Generalized Zero-Shot Text Classification
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Classification, Generalized Zero-Shot Learning, Meta-Learning, text-classification, Text Classification, Zero-Shot Learning, Zero-Shot Text Classification

Generalized zero-shot text classification aims to classify textual instances from both previously seen classes and incrementally emerging unseen classes. Most existing methods generalize poorly since the learned parameters are only optimal for seen classes rather than for both classes, and the parameters keep stationary in predicting procedures. To address these challenges, we propose a novel Learn to Adapt (LTA) network using a variant meta-learning framework. Specifically, LTA trains multiple meta-learners by using both seen classes and virtual unseen classes to simulate a generalized zero-shot learning (GZSL) scenario in accordance with the test time, and simultaneously learns to calibrate the class prototypes and sample representations to make the learned parameters adaptive to incoming unseen classes. We claim that the proposed model is capable of representing all prototypes and samples from both classes to a more consistent distribution in the global space. Extensive experiments on five text classification datasets show that our model outperforms several competitive previous approaches by large margins. The code and the whole datasets will be available after paper publication.
SiSP: Japanese Situation-dependent Sentiment Polarity Dictionary
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11

In order to deal with the variety of meanings and contexts of words, we created a Japanese Situation-dependent Sentiment Polarity Dictionary (SiSP) of sentiment values labeled for 20 different situations. This dictionary was annotated by crowdworkers with 25,520 Japanese words, and consists of 10 responses for each situation of each word. Using our SiSP, we predicted the polarity of each word in the dictionary and that of dictionary words in sentences considering the context. In both experiments, situation-dependent prediction showed superior results in determining emotional polarity.
There’s a Time and Place for Reasoning Beyond the Image
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: 16k, Articles, Image Clustering, Retrieval, World Knowledge

Images are often more significant than only the pixels to human eyes, as we can infer, associate, and reason with contextual information from other sources to establish a more complete picture. For example, in Figure 1, we can find a way to identify the news articles related to the picture through segment-wise understandings on the signs, the buildings, the crowds, and more. This tells us the time when and the location where the image is taken, which will help us in subsequent tasks, such as evidence retrieval for criminal activities, automatic storyline construction, and upper-stream processing such as image clustering. 

In this work, we formulate this problem and introduce TARA: a dataset with 16k images with their associated news, time and location automatically extracted from New York Times (NYT), and an additional 61k examples as distant supervision from WIT. On top of the extractions, we present a crowdsourced subset in which images are believed to be feasible to find their spatio-temporal information for evaluation purpose. We show that there exists a $70\%$ gap between a state-of-the-art joint model and human performance, which is slightly filled by our proposed model that uses segment-wise reasoning, motivating higher-level vision-language joint models that can conduct open-ended reasoning with world knowledge.