pwc-archive/papers-with-abstracts huggingface dataset:
Exploring the Boundaries of Low-Resource BERT Distillation
Authors: Moshe Wasserblat, Oren Pereg, Peter Izsak
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (sustainlp) 2020 11
Tasks: Model Compression

In recent years, large pre-trained models have demonstrated state-of-the-art performance in many of NLP tasks. However, the deployment of these models on devices with limited resources is challenging due to the models’ large computational consumption and memory requirements. Moreover, the need for a considerable amount of labeled training data also hinders real-world deployment scenarios. Model distillation has shown promising results for reducing model size, computational load and data efficiency. In this paper we test the boundaries of BERT model distillation in terms of model compression, inference efficiency and data scarcity. We show that classification tasks that require the capturing of general lexical semantics can be successfully distilled by very simple and efficient models and require relatively small amount of labeled training data. We also show that the distillation of large pre-trained models is more effective in real-life scenarios where limited amounts of labeled training are available.
Don’t Read Too Much Into It: Adaptive Computation for Open-Domain Question Answering
Authors: Yuxiang Wu, Pasquale Minervini, Pontus Stenetorp, Sebastian Riedel
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (sustainlp) 2020 11
Tasks: Open-Domain Question Answering, Question Answering

Most approaches to Open-Domain Question Answering consist of a light-weight retriever that selects a set of candidate passages, and a computationally expensive reader that examines the passages to identify the correct answer. Previous works have shown that as the number of retrieved passages increases, so does the performance of the reader. However, they assume all retrieved passages are of equal importance and allocate the same amount of computation to them, leading to a substantial increase in computational cost. To reduce this cost, we propose the use of adaptive computation to control the computational budget allocated for the passages to be read. We first introduce a technique operating on individual passages in isolation which relies on anytime prediction and a per-layer estimation of an early exit probability. We then introduce SKYLINEBUILDER, an approach for dynamically deciding on which passage to allocate computation at each step, based on a resource allocation policy trained via reinforcement learning. Our results on SQuAD-Open show that adaptive computation with global prioritisation improves over several strong static and adaptive methods, leading to a 4.3x reduction in computation while retaining 95% performance of the full model.
A Two-stage Model for Slot Filling in Low-resource Settings: Domain-agnostic Non-slot Reduction and Pretrained Contextual Embeddings
Authors: Cennet Oguz, Ngoc Thang Vu
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (sustainlp) 2020 11
Tasks: slot-filling, Slot Filling, Spoken Language Understanding

Learning-based slot filling - a key component of spoken language understanding systems - typically requires a large amount of in-domain hand-labeled data for training. In this paper, we propose a novel two-stage model architecture that can be trained with only a few in-domain hand-labeled examples. The first step is designed to remove non-slot tokens (i.e., O labeled tokens), as they introduce noise in the input of slot filling models. This step is domain-agnostic and therefore, can be trained by exploiting out-of-domain data. The second step identifies slot names only for slot tokens by using state-of-the-art pretrained contextual embeddings such as ELMO and BERT. We show that our approach outperforms other state-of-art systems on the SNIPS benchmark dataset.
Early Exiting BERT for Efficient Document Ranking
Authors: Ji Xin, Rodrigo Nogueira, YaoLiang Yu, Jimmy Lin
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (sustainlp) 2020 11
Tasks: Document Ranking

Pre-trained language models such as BERT have shown their effectiveness in various tasks. Despite their power, they are known to be computationally intensive, which hinders real-world applications. In this paper, we introduce early exiting BERT for document ranking. With a slight modification, BERT becomes a model with multiple output paths, and each inference sample can exit early from these paths. In this way, computation can be effectively allocated among samples, and overall system latency is significantly reduced while the original quality is maintained. Our experiments on two document ranking datasets demonstrate up to 2.5x inference speedup with minimal quality degradation. The source code of our implementation can be found at https://github.com/castorini/earlyexiting-monobert.
Keyphrase Generation with GANs in Low-Resources Scenarios
Authors: Giuseppe Lancioni, Saida S.Mohamed, Beatrice Portelli, Giuseppe Serra, Carlo Tasso
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (sustainlp) 2020 11
Tasks: Keyphrase Generation

Keyphrase Generation is the task of predicting Keyphrases (KPs), short phrases that summarize the semantic meaning of a given document. Several past studies provided diverse approaches to generate Keyphrases for an input document. However, all of these approaches still need to be trained on very large datasets. In this paper, we introduce BeGanKP, a new conditional GAN model to address the problem of Keyphrase Generation in a low-resource scenario. Our main contribution relies in the Discriminator’s architecture: a new BERT-based module which is able to distinguish between the generated and humancurated KPs reliably. Its characteristics allow us to use it in a low-resource scenario, where only a small amount of training data are available, obtaining an efficient Generator. The resulting architecture achieves, on five public datasets, competitive results with respect to the state-of-the-art approaches, using less than 1% of the training data.
Quasi-Multitask Learning: an Efficient Surrogate for Obtaining Model Ensembles
Authors: Norbert Kis-Szabó, Gábor Berend
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (sustainlp) 2020 11

We propose the technique of quasi-multitask learning (Q-MTL), a simple and easy to implement modification of standard multitask learning, in which the tasks to be modeled are identical. With this easy modification of a standard neural classifier we can get benefits similar to an ensemble of classifiers with a fraction of the resources required.We illustrate it through a series of sequence labeling experiments over a diverse set of languages, that applying Q-MTL consistently increases the generalization ability of the applied models. The proposed architecture can be regarded as a new regularization technique that encourages the model to develop an internal representation of the problem at hand which is beneficial to multiple output units of the classifier at the same time. Our experiments corroborate that by relying on the proposed algorithm, we can approximate the quality of an ensemble of classifiers at a fraction of computational resources required. Additionally, our results suggest that Q-MTL handles the presence of noisy training labels better than ensembles.
A Little Bit Is Worse Than None: Ranking with Limited Training Data
Authors: Xinyu Zhang, Andrew Yates, Jimmy Lin
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (sustainlp) 2020 11
Tasks: Passage Retrieval, Retrieval

Researchers have proposed simple yet effective techniques for the retrieval problem based on using BERT as a relevance classifier to rerank initial candidates from keyword search. In this work, we tackle the challenge of fine-tuning these models for specific domains in a data and computationally efficient manner. Typically, researchers fine-tune models using corpus-specific labeled data from sources such as TREC. We first answer the question: How much data of this type do we need? Recognizing that the most computationally efficient training is no training, we explore zero-shot ranking using BERT models that have already been fine-tuned with the large MS MARCO passage retrieval dataset. We arrive at the surprising and novel finding that “some” labeled in-domain data can be worse than none at all.
Predictive Model Selection for Transfer Learning in Sequence Labeling Tasks
Authors: Parul Awasthy, Bishwaranjan Bhattacharjee, John Kender, Radu Florian
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (sustainlp) 2020 11
Tasks: Model Selection, Transfer Learning

Transfer learning is a popular technique to learn a task using less training data and fewer compute resources. However, selecting the correct source model for transfer learning is a challenging task. We demonstrate a novel predictive method that determines which existing source model would minimize error for transfer learning to a given target. This technique does not require learning for prediction, and avoids computational costs of trail-and-error. We have evaluated this technique on nine datasets across diverse domains, including newswire, user forums, air flight booking, cybersecurity news, etc. We show that it per-forms better than existing techniques such as fine-tuning over vanilla BERT, or curriculum learning over the largest dataset on top of BERT, resulting in average F1 score gains in excess of 3%. Moreover, our technique consistently selects the best model using fewer tries.
Load What You Need: Smaller Versions of Mutililingual BERT
Authors: Amine Abdaoui, Camille Pradel, Grégoire Sigel
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (sustainlp) 2020 11

Pre-trained Transformer-based models are achieving state-of-the-art results on a variety of Natural Language Processing data sets. However, the size of these models is often a drawback for their deployment in real production applications. In the case of multilingual models, most of the parameters are located in the embeddings layer. Therefore, reducing the vocabulary size should have an important impact on the total number of parameters. In this paper, we propose to extract smaller models that handle fewer number of languages according to the targeted corpora. We present an evaluation of smaller versions of multilingual BERT on the XNLI data set, but we believe that this method may be applied to other multilingual transformers. The obtained results confirm that we can generate smaller models that keep comparable results, while reducing up to 45% of the total number of parameters. We compared our models with DistilmBERT (a distilled version of multilingual BERT) and showed that unlike language reduction, distillation induced a 1.7% to 6% drop in the overall accuracy on the XNLI data set. The presented models and code are publicly available.
Analysis of Resource-efficient Predictive Models for Natural Language Processing
Authors: Raj Pranesh, Ambesh Shekhar
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (sustainlp) 2020 11
Tasks: BIG-bench Machine Learning, Emotion Recognition, News Classification

In this paper, we presented an analyses of the resource efficient predictive models, namely Bonsai, Binary Neighbor Compression(BNC), ProtoNN, Random Forest, Naive Bayes and Support vector machine(SVM), in the machine learning field for resource constraint devices. These models try to minimize resource requirements like RAM and storage without hurting the accuracy much. We utilized these models on multiple benchmark natural language processing tasks, which were sentimental analysis, spam message detection, emotion analysis and fake news classification. The experiment results shows that the tree-based algorithm, Bonsai, surpassed the rest of the machine learning algorithms by achieve higher accuracy scores while having significantly lower memory usage.
A comparison between CNNs and WFAs for Sequence Classification
Authors: Ariadna Quattoni, Xavier Carreras
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (sustainlp) 2020 11
Tasks: Classification

We compare a classical CNN architecture for sequence classification involving several convolutional and max-pooling layers against a simple model based on weighted finite state automata (WFA). Each model has its advantages and disadvantages and it is possible that they could be combined. However, we believe that the first research goal should be to investigate and understand how do these two apparently dissimilar models compare in the context of specific natural language processing tasks. This paper is the first step towards that goal. Our experiments with five sequence classification datasets suggest that, despite the apparent simplicity of WFA models and training algorithms, the performance of WFAs is comparable to that of the CNNs.
Label-Efficient Training for Next Response Selection
Authors: Seungtaek Choi, Myeongho Jeong, Jinyoung Yeo, Seung-won Hwang
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (sustainlp) 2020 11
Tasks: counterfactual

This paper studies label augmentation for training dialogue response selection. The existing model is trained by “observational” annotation, where one observed response is annotated as gold. In this paper, we propose “counterfactual augmentation” of pseudo-positive labels. We validate that the effectiveness of augmented labels are comparable to positives, such that ours outperform state-of-the-arts without augmentation.
Do We Need to Create Big Datasets to Learn a Task?
Authors: Swaroop Mishra, Bhavdeep Singh Sachdeva
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (sustainlp) 2020 11
Tasks: Zero-shot Generalization

Deep Learning research has been largely accelerated by the development of huge datasets such as Imagenet. The general trend has been to create big datasets to make a deep neural network learn. A huge amount of resources is being spent in creating these big datasets, developing models, training them, and iterating this process to dominate leaderboards. We argue that the trend of creating bigger datasets needs to be revised by better leveraging the power of pre-trained language models. Since the language models have already been pre-trained with huge amount of data and have basic linguistic knowledge, there is no need to create big datasets to learn a task. Instead, we need to create a dataset that is sufficient for the model to learn various task-specific terminologies, such as ‘Entailment’, ‘Neutral’, and ‘Contradiction’ for NLI. As evidence, we show that RoBERTA is able to achieve near-equal performance on 2% data of SNLI. We also observe competitive zero-shot generalization on several OOD datasets. In this paper, we propose a baseline algorithm to find the optimal dataset for learning a task.
Overview of the SustaiNLP 2020 Shared Task
Authors: Alex Wang, Thomas Wolf
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (sustainlp) 2020 11

We describe the SustaiNLP 2020 shared task: efficient inference on the SuperGLUE benchmark (Wang et al., 2019). Participants are evaluated based on performance on the benchmark as well as energy consumed in making predictions on the test sets. We describe the task, its organization, and the submitted systems. Across the six submissions to the shared task, participants achieved efficiency gains of 20× over a standard BERT (Devlin et al., 2019) baseline, while losing less than an absolute point in performance.
Making Sense of schema.org with WordNet
Authors: Csaba Veres
Date: 2019-07-01 00:00:00
Proceeding: GWC 2019 7

The schema.org initiative was designed to introduce machine readable metadata into the World Wide Web. This paper investigates conceptual biases in the schema through a mapping exercise between schema.org types and WordNet synsets. We create a mapping ontology which establishes the relationship between schema metadata types and the corresponding everyday concepts. This in turn can be used to enhance metadata annotation to include a more complete description of knowledge on the Web of data.
Leaving No Stone Unturned When Identifying and Classifying Verbal Multiword Expressions in the Romanian Wordnet
Authors: Verginica Mititelu, Maria Mitrofan
Date: 2019-07-01 00:00:00
Proceeding: GWC 2019 7

We present here the enhancement of the Romanian wordnet with a new type of information, very useful in language processing, namely types of verbal multi-word expressions. All verb literals made of two or more words are attached a label specific to the type of verbal multi-word expression they correspond to. These labels were created in the PARSEME Cost Action and were used in the version 1.1 of the shared task they organized. The results of this annotation are compared to those obtained in the annotation of a Romanian news corpus with the same labels. Given the alignment of the Romanian wordnet to the Princeton WordNet, this type of annotation can be further used for drawing comparisons between equivalent verbal literals in various languages, provided that such information is annotated in the wordnets of the respective languages and their wordnets are aligned to Princeton WordNet, and thus to the Romanian wordnet.
Thesaurus Verification Based on Distributional Similarities
Authors: Natalia Loukachevitch, Ekaterina Parkhomenko
Date: 2019-07-01 00:00:00
Proceeding: GWC 2019 7

In this paper we consider an approach to verification of large lexical-semantic resources as WordNet. The method of verification procedure is based on the analysis of discrepancies of corpus-based and thesaurus-based word similarities. We calculated such word similarities on the basis of a Russian news collection and Russian wordnet (RuWordNet). We applied the procedure to more than 30 thousand words and found some serious errors in word sense description, including incorrect or absent relations or missed main senses of ambiguous words.
Including Swiss Standard German in GermaNet
Authors: Eva Huber, Erhard Hinrichs
Date: 2019-07-01 00:00:00
Proceeding: GWC 2019 7

GermaNet (Henrich and Hinrichs, 2010; Hamp and Feldweg, 1997) is a comprehensive wordnet of Standard German spoken in the Federal Republic of Germany. The GermaNet team aims at modelling the basic vocabulary of the language. German is an official language or a minority language in many countries. It is an official language in Austria, Germany and Switzerland, each with its own codified standard variety (Auer, 2014, p. 21), and also in Belgium, Liechtenstein, and Luxemburg. German is recognized as a minority language in thirteen additional countries, including Brasil, Italy, Poland, and Russia. However, the different standard varieties of German are currently not represented in GermaNet. With this project, we make a start on changing this by including one variety, namely Swiss Standard German, into GermaNet. This shall give a more inclusive perspective on the German language. We will argue that Swiss Standard German words, Helvetisms, are best included into the already existing wordnet GermaNet, rather than creating them as a separate wordnet.
Danish in Wikidata lexemes
Authors: Finn Årup Nielsen
Date: 2019-07-01 00:00:00
Proceeding: GWC 2019 7

Wikidata introduced support for lexicographic data in 2018. Here we describe the lexicographic part of Wikidata as well as experiences with setting up lexemes for the Danish language. We note various possible annotations for lexemes as well as discuss various choices made.
Using Thesaurus Data to Improve Coreference Resolution for Russian
Authors: Ilya Azerkovich
Date: 2019-07-01 00:00:00
Proceeding: GWC 2019 7
Tasks: coreference-resolution, Coreference Resolution, Semantic Similarity, Semantic Textual Similarity

Semantic information about entities, specifically, how close in meaning two mentions are to each other, can become very useful for the task of co-reference resolution. One of the most well-researched and widely used forms of presenting this information are measures of semantic similarity and semantic relatedness. These metrics are often computed, relying upon the structure of a thesaurus, but it is also possible to use alternative resources. One such source is Wikipedia, which possesses the category structure similar to that of a thesaurus. In this work we describe an attempt to use semantic relatedness measures, calculated on thesaurus and Wikipedia data, to improve the quality of a co-reference resolution system for Russian language. The results show that this is a viable solution and that combining the two sources yields the most gain in quality.
The Extended Arabic WordNet: a Case Study and an Evaluation using a Word Sense Disambiguation System
Authors: Mohamed Ali Batita, Mounir Zrigui
Date: 2019-07-01 00:00:00
Proceeding: GWC 2019 7
Tasks: Word Sense Disambiguation

Arabic WordNet (AWN) represents one of the best-known lexical resources for the Arabic language. However, it contains various issues that affect its use in different Natural Language Processing (NLP) applications. Due to resources deficiency, the update of Arabic WordNet requires much effort. There have only been only two updates it was first published in 2006. The most significant of those being in 2013, which represented a significant development in the usability and coverage of Arabic WordNet. This paper provides a study case on the updates of the Arabic WordNet and the development of its contents. More precisely, we present the new content in terms of relations that have been added to the extended version of Arabic WordNet. We also validate and evaluate its contents at different levels. We use its different versions in a Word Sense Disambiguation system. Finally, we compare the results and evaluate them. Results show that newly added semantic relations can improve the performance of a Word Sense Disambiguation system.
On Hidden Semantic Relations between Nouns in WordNet
Authors: Tsvetana Dimitrova, Valentina Stefanova
Date: 2019-07-01 00:00:00
Proceeding: GWC 2019 7
Tasks: POS

The paper presents an effort on transferability of noun–verb and noun–adjective derivative and semantic relations to noun-noun relations. The approach relies on information from semantic classes and existing inter-POS derivative and (morpho)semantic relations between noun and verb, and noun and adjective synsets. We have added semantic relations between nouns in WordNet that are indirectly linked via verbs and adjectives. Observations on the combination between the relations and semantic classes of nouns they link, may facilitate further efforts in assigning semantic properties to nouns pointing to their abilities to participate in predicate-argument structures.
Linking Russian Wordnet RuWordNet to WordNet
Authors: Natalia Loukachevitch, Anastasia Gerasimova
Date: 2019-07-01 00:00:00
Proceeding: GWC 2019 7
Tasks: Specificity

In this paper we consider the linking procedure of Russian wordnet (RuWordNet) to Wordnet. The specificity of the procedure in our case is based on the fact that a lot of bilingual (Russian and English) lexical data have been gathered in another Russian thesaurus RuThes, which has a different structure than WordNet. Previously, RuThes has been semi-automatically transformed into RuWordNet, having the WordNet-like structure. Now, the RuThes English data are utilized to establish matching from the RuWordNet synsets to the WordNet synsets.
Fast developing of a Natural Language Interface for a Portuguese WordNet: Leveraging on Sentence Embeddings
Authors: Hugo Gonçalo Oliveira, Alexandre Rademaker
Date: 2019-07-01 00:00:00
Proceeding: GWC 2019 7
Tasks: Natural Language Queries, Sentence, Sentence Embeddings

We describe how a natural language interface can be developed for a wordnet with a small set of handcrafted templates, leveraging on sentence embeddings. The proposed approach does not use rules for parsing natural language queries but experiments showed that the embeddings model is tolerant enough for correctly predicting relation types that do not match known patterns exactly. It was tested with OpenWordNet-PT, for which this method may provide an alternative interface, with benefits also on the curation process.