pwc-archive/papers-with-abstracts huggingface dataset:
Composing Noun Phrase Vector Representations
Authors: Aikaterini-Lida Kalouli, Valeria de Paiva, Richard Crouch
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Word Embeddings

Vector representations of words have seen an increasing success over the past years in a variety of NLP tasks. While there seems to be a consensus about the usefulness of word embeddings and how to learn them, it is still unclear which representations can capture the meaning of phrases or even whole sentences. Recent work has shown that simple operations outperform more complex deep architectures. In this work, we propose two novel constraints for computing noun phrase vector representations. First, we propose that the semantic and not the syntactic contribution of each component of a noun phrase should be considered, so that the resulting composed vectors express more of the phrase meaning. Second, the composition process of the two phrase vectors should apply suitable dimensions{'} selection in a way that specific semantic features captured by the phrase{'}s meaning become more salient. Our proposed methods are compared to 11 other approaches, including popular baselines and a neural net architecture, and are evaluated across 6 tasks and 2 datasets. Our results show that these constraints lead to more expressive phrase representations and can be applied to other state-of-the-art methods to improve their performance.
An Empirical Study on Pre-trained Embeddings and Language Models for Bot Detection
Authors: Andres Garcia-Silva, Cristian Berrio, Jos{\'e} Manuel G{\'o}mez-P{\'e}rez
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Language Modeling, Language Modelling

Fine-tuning pre-trained language models has significantly advanced the state of art in a wide range of NLP downstream tasks. Usually, such language models are learned from large and well-formed text corpora from e.g. encyclopedic resources, books or news. However, a significant amount of the text to be analyzed nowadays is Web data, often from social media. In this paper we consider the research question: How do standard pre-trained language models generalize and capture the peculiarities of rather short, informal and frequently automatically generated text found in social media? To answer this question, we focus on bot detection in Twitter as our evaluation task and test the performance of fine-tuning approaches based on language models against popular neural architectures such as LSTM and CNN combined with pre-trained and contextualized embeddings. Our results also show strong performance variations among the different language model approaches, which suggest further research.
Learning Multilingual Meta-Embeddings for Code-Switching Named Entity Recognition
Authors: Genta Indra Winata, Zhaojiang Lin, Pascale Fung
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Language Identification, MME, named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER)

In this paper, we propose Multilingual Meta-Embeddings (MME), an effective method to learn multilingual representations by leveraging monolingual pre-trained embeddings. MME learns to utilize information from these embeddings via a self-attention mechanism without explicit language identification. We evaluate the proposed embedding method on the code-switching English-Spanish Named Entity Recognition dataset in a multilingual and cross-lingual setting. The experimental results show that our proposed method achieves state-of-the-art performance on the multilingual setting, and it has the ability to generalize to an unseen language task.
Investigating Sub-Word Embedding Strategies for the Morphologically Rich and Free Phrase-Order Hungarian
Authors: B{\'a}lint D{\"o}br{\"o}ssy, M{\'a}rton Makrai, Bal{\'a}zs Tarj{\'a}n, Gy{\"o}rgy Szasz{\'a}k
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Lemmatization, Morphological Analysis, Word Embeddings

For morphologically rich languages, word embeddings provide less consistent semantic representations due to higher variance in word forms. Moreover, these languages often allow for less constrained word order, which further increases variance. For the highly agglutinative Hungarian, semantic accuracy of word embeddings measured on word analogy tasks drops by 50-75{\%} compared to English. We observed that embeddings learn morphosyntax quite well instead. Therefore, we explore and evaluate several sub-word unit based embedding strategies {--} character n-grams, lemmatization provided by an NLP-pipeline, and segments obtained in unsupervised learning (morfessor) {--} to boost semantic consistency in Hungarian word vectors. The effect of changing embedding dimension and context window size have also been considered. Morphological analysis based lemmatization was found to be the best strategy to improve embeddings{'} semantic accuracy, whereas adding character n-grams was found consistently counterproductive in this regard.
A Self-Training Approach for Short Text Clustering
Authors: Amir Hadifar, Lucas Sterckx, Thomas Demeester, Chris Develder
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Clustering, Deep Clustering, Sentence, Sentence Embedding, Sentence-Embedding, Short Text Clustering, Text Clustering

Short text clustering is a challenging problem when adopting traditional bag-of-words or TF-IDF representations, since these lead to sparse vector representations of the short texts. Low-dimensional continuous representations or embeddings can counter that sparseness problem: their high representational power is exploited in deep clustering algorithms. While deep clustering has been studied extensively in computer vision, relatively little work has focused on NLP. The method we propose, learns discriminative features from both an autoencoder and a sentence embedding, then uses assignments from a clustering algorithm as supervision to update weights of the encoder network. Experiments on three short text datasets empirically validate the effectiveness of our method.
Improving Word Embeddings Using Kernel PCA
Authors: Vishwani Gupta, Sven Giesselbach, Stefan R{\"u}ping, Christian Bauckhage
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Articles, Sentence, Sentence Classification, Word Embeddings, Word Similarity

Word-based embedding approaches such as Word2Vec capture the meaning of words and relations between them, particularly well when trained with large text collections; however, they fail to do so with small datasets. Extensions such as fastText reduce the amount of data needed slightly, however, the joint task of learning meaningful morphology, syntactic and semantic representations still requires a lot of data. In this paper, we introduce a new approach to warm-start embedding models with morphological information, in order to reduce training time and enhance their performance. We use word embeddings generated using both word2vec and fastText models and enrich them with morphological information of words, derived from kernel principal component analysis (KPCA) of word similarity matrices. This can be seen as explicitly feeding the network morphological similarities and letting it learn semantic and syntactic similarities. Evaluating our models on word similarity and analogy tasks in English and German, we find that they not only achieve higher accuracies than the original skip-gram and fastText models but also require significantly less training data and time. Another benefit of our approach is that it is capable of generating a high-quality representation of infrequent words as, for example, found in very recent news articles with rapidly changing vocabularies. Lastly, we evaluate the different models on a downstream sentence classification task in which a CNN model is initialized with our embeddings and find promising results.
On Committee Representations of Adversarial Learning Models for Question-Answer Ranking
Authors: Sparsh Gupta, Vitor Carvalho
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8

Adversarial training is a process in Machine Learning that explicitly trains models on adversarial inputs (inputs designed to deceive or trick the learning process) in order to make it more robust or accurate. In this paper we investigate how representing adversarial training models as committees can be used to effectively improve the performance of Question-Answer (QA) Ranking. We start by empirically probing the effects of adversarial training over multiple QA ranking algorithms, including the state-of-the-art Multihop Attention Network model. We evaluate these algorithms on several benchmark datasets and observe that, while adversarial training is beneficial to most baseline algorithms, there are cases where it may lead to overfitting and performance degradation. We investigate the causes of such degradation, and then propose a new representation procedure for this adversarial learning problem, based on committee learning, that not only is capable of consistently improving all baseline algorithms, but also outperforms the previous state-of-the-art algorithm by as much as 6{\%} in NDCG.
Meta-Learning Improves Lifelong Relation Extraction
Authors: Abiola Obamuyide, Andreas Vlachos
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Lifelong learning, Meta-Learning, Relation, Relation Extraction

Most existing relation extraction models assume a fixed set of relations and are unable to adapt to exploit newly available supervision data to extract new relations. In order to alleviate such problems, there is the need to develop approaches that make relation extraction models capable of continuous adaptation and learning. We investigate and present results for such an approach, based on a combination of ideas from lifelong learning and optimization-based meta-learning. We evaluate the proposed approach on two recent lifelong relation extraction benchmarks, and demonstrate that it markedly outperforms current state-of-the-art approaches.
Effective Dimensionality Reduction for Word Embeddings
Authors: Vikas Raunak, Vivek Gupta, Florian Metze
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Dimensionality Reduction, Word Embeddings

Pre-trained word embeddings are used in several downstream applications as well as for constructing representations for sentences, paragraphs and documents. Recently, there has been an emphasis on improving the pretrained word vectors through post-processing algorithms. One improvement area is reducing the dimensionality of word embeddings. Reducing the size of word embeddings can improve their utility in memory constrained devices, benefiting several real world applications. In this work, we present a novel technique that efficiently combines PCA based dimensionality reduction with a recently proposed post-processing algorithm (Mu and Viswanath, 2018), to construct effective word embeddings of lower dimensions. Empirical evaluations on several benchmarks show that our algorithm efficiently reduces the embedding size while achieving similar or (more often) better performance than original embeddings. We have released the source code along with this paper.
Learning Word Embeddings without Context Vectors
Authors: Alexey Zobnin, Evgenia Elistratova
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Learning Word Embeddings, Word Embeddings, Word Similarity

Most word embedding algorithms such as word2vec or fastText construct two sort of vectors: for words and for contexts. Naive use of vectors of only one sort leads to poor results. We suggest using indefinite inner product in skip-gram negative sampling algorithm. This allows us to use only one sort of vectors without loss of quality. Our {``}context-free{''} cf algorithm performs on par with SGNS on word similarity datasets
Leveraging Pre-Trained Embeddings for Welsh Taggers
Authors: Ignatius Ezeani, Scott Piao, Steven Neale, Paul Rayson, Dawn Knight
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8

While the application of word embedding models to downstream Natural Language Processing (NLP) tasks has been shown to be successful, the benefits for low-resource languages is somewhat limited due to lack of adequate data for training the models. However, NLP research efforts for low-resource languages have focused on constantly seeking ways to harness pre-trained models to improve the performance of NLP systems built to process these languages without the need to re-invent the wheel. One such language is Welsh and therefore, in this paper, we present the results of our experiments on learning a simple multi-task neural network model for part-of-speech and semantic tagging for Welsh using a pre-trained embedding model from FastText. Our model{'}s performance was compared with those of the existing rule-based stand-alone taggers for part-of-speech and semantic taggers. Despite its simplicity and capacity to perform both tasks simultaneously, our tagger compared very well with the existing taggers.
The many dimensions of algorithmic fairness in educational applications
Authors: Anastassia Loukina, Nitin Madnani, Klaus Zechner
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: BIG-bench Machine Learning, Fairness

The issues of algorithmic fairness and bias have recently featured prominently in many publications highlighting the fact that training the algorithms for maximum performance may often result in predictions that are biased against various groups. Educational applications based on NLP and speech processing technologies often combine multiple complex machine learning algorithms and are thus vulnerable to the same sources of bias as other machine learning systems. Yet such systems can have high impact on people{'}s lives especially when deployed as part of high-stakes tests. In this paper we discuss different definitions of fairness and possible ways to apply them to educational applications. We then use simulated and real data to consider how test-takers{'} native language backgrounds can affect their automated scores on an English language proficiency assessment. We illustrate that total fairness may not be achievable and that different definitions of fairness may require different solutions.
Predicting the Difficulty of Multiple Choice Questions in a High-stakes Medical Exam
Authors: Le An Ha, Victoria Yaneva, Peter Baldwin, Janet Mee
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Multiple-choice, Question Answering

Predicting the construct-relevant difficulty of Multiple-Choice Questions (MCQs) has the potential to reduce cost while maintaining the quality of high-stakes exams. In this paper, we propose a method for estimating the difficulty of MCQs from a high-stakes medical exam, where all questions were deliberately written to a common reading level. To accomplish this, we extract a large number of linguistic features and embedding types, as well as features quantifying the difficulty of the items for an automatic question-answering system. The results show that the proposed approach outperforms various baselines with a statistically significant difference. Best results were achieved when using the full feature set, where embeddings had the highest predictive power, followed by linguistic features. An ablation study of the various types of linguistic features suggested that information from all levels of linguistic processing contributes to predicting item difficulty, with features related to semantic ambiguity and the psycholinguistic properties of words having a slightly higher importance. Owing to its generic nature, the presented approach has the potential to generalize over other exams containing MCQs.
An Intelligent Testing Strategy for Vocabulary Assessment of Chinese Second Language Learners
Authors: Wei Zhou, Renfen Hu, Feipeng Sun, Ronghuai Huang
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8

Vocabulary is one of the most important parts of language competence. Testing of vocabulary knowledge is central to research on reading and language. However, it usually costs a large amount of time and human labor to build an item bank and to test large number of students. In this paper, we propose a novel testing strategy by combining automatic item generation (AIG) and computerized adaptive testing (CAT) in vocabulary assessment for Chinese L2 learners. Firstly, we generate three types of vocabulary questions by modeling both the vocabulary knowledge and learners{'} writing error data. After evaluation and calibration, we construct a balanced item pool with automatically generated items, and implement a three-parameter computerized adaptive test. We conduct manual item evaluation and online student tests in the experiments. The results show that the combination of AIG and CAT can construct test items efficiently and reduce test cost significantly. Also, the test result of CAT can provide valuable feedback to AIG algorithms.
Computationally Modeling the Impact of Task-Appropriate Language Complexity and Accuracy on Human Grading of German Essays
Authors: Zarah Weiss, Anja Riemenschneider, Pauline Schr{\"o}ter, Detmar Meurers
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8

Computational linguistic research on the language complexity of student writing typically involves human ratings as a gold standard. However, educational science shows that teachers find it difficult to identify and cleanly separate accuracy, different aspects of complexity, contents, and structure. In this paper, we therefore explore the use of computational linguistic methods to investigate how task-appropriate complexity and accuracy relate to the grading of overall performance, content performance, and language performance as assigned by teachers. Based on texts written by students for the official school-leaving state examination (Abitur), we show that teachers successfully assign higher language performance grades to essays with higher task-appropriate language complexity and properly separate this from content scores. Yet, accuracy impacts teacher assessment for all grading rubrics, also the content score, overemphasizing the role of accuracy. Our analysis is based on broad computational linguistic modeling of German language complexity and an innovative theory- and data-driven feature aggregation method inferring task-appropriate language complexity.
Analysing Rhetorical Structure as a Key Feature of Summary Coherence
Authors: Jan {\v{S}}najder, Tamara Sladoljev-Agejev, Svjetlana Koli{\'c} Vehovec
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: regression, Relation

We present a model for automatic scoring of coherence based on comparing the rhetorical structure (RS) of college student summaries in L2 (English) against expert summaries. Coherence is conceptualised as a construct consisting of the rhetorical relation and its arguments. Comparison with expert-assigned scores shows that RS scores correlate with both cohesion and coherence. Furthermore, RS scores improve the accuracy of a regression model for cohesion score prediction.
The BEA-2019 Shared Task on Grammatical Error Correction
Authors: Christopher Bryant, Mariano Felice, {\O}istein E. Andersen, Ted Briscoe
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Grammatical Error Correction

This paper reports on the BEA-2019 Shared Task on Grammatical Error Correction (GEC). As with the CoNLL-2014 shared task, participants are required to correct all types of errors in test data. One of the main contributions of the BEA-2019 shared task is the introduction of a new dataset, the Write{\&}Improve+LOCNESS corpus, which represents a wider range of native and learner English levels and abilities. Another contribution is the introduction of tracks, which control the amount of annotated data available to participants. Systems are evaluated in terms of ERRANT F{\_}0.5, which allows us to report a much wider range of performance statistics. The competition was hosted on Codalab and remains open for further submissions on the blind test set.
A Benchmark Corpus of English Misspellings and a Minimally-supervised Model for Spelling Correction
Authors: Michael Flor, Michael Fried, Alla Rozovskaya
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Spelling Correction

Spelling correction has attracted a lot of attention in the NLP community. However, models have been usually evaluated on artificiallycreated or proprietary corpora. A publiclyavailable corpus of authentic misspellings, annotated in context, is still lacking. To address this, we present and release an annotated data set of 6,121 spelling errors in context, based on a corpus of essays written by English language learners. We also develop a minimallysupervised context-aware approach to spelling correction. It achieves strong results on our data: 88.12{\%} accuracy. This approach can also train with a minimal amount of annotated data (performance reduced by less than 1{\%}). Furthermore, this approach allows easy portability to new domains. We evaluate our model on data from a medical domain and demonstrate that it rivals the performance of a model trained and tuned on in-domain data.
Artificial Error Generation with Fluency Filtering
Authors: Mengyang Qiu, Jungyeul Park
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Grammatical Error Correction

The quantity and quality of training data plays a crucial role in grammatical error correction (GEC). However, due to the fact that obtaining human-annotated GEC data is both time-consuming and expensive, several studies have focused on generating artificial error sentences to boost training data for grammatical error correction, and shown significantly better performance. The present study explores how fluency filtering can affect the quality of artificial errors. By comparing artificial data filtered by different levels of fluency, we find that artificial error sentences with low fluency can greatly facilitate error correction, while high fluency errors introduce more noise.
Regression or classification? Automated Essay Scoring for Norwegian
Authors: Stig Johan Berggren, Taraka Rama, Lilja {\O}vrelid
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Automated Essay Scoring, BIG-bench Machine Learning, Classification, General Classification, Language Identification, Multi-Task Learning, Native Language Identification, regression

In this paper we present first results for the task of Automated Essay Scoring for Norwegian learner language. We analyze a number of properties of this task experimentally and assess (i) the formulation of the task as either regression or classification, (ii) the use of various non-neural and neural machine learning architectures with various types of input representations, and (iii) applying multi-task learning for joint prediction of essay scoring and native language identification. We find that a GRU-based attention model trained in a single-task setting performs best at the AES task.
How to account for mispellings: Quantifying the benefit of character representations in neural content scoring models
Authors: Brian Riordan, Michael Flor, Robert Pugh
Date: 2019-08-01 00:00:00
Proceeding: WS 2019 8
Tasks: Spelling Correction

Character-based representations in neural models have been claimed to be a tool to overcome spelling variation in in word token-based input. We examine this claim in neural models for content scoring. We formulate precise hypotheses about the possible effects of adding character representations to word-based models and test these hypotheses on large-scale real world content scoring datasets. We find that, while character representations may provide small performance gains in general, their effectiveness in accounting for spelling variation may be limited. We show that spelling correction can provide larger gains than character representations, and that spelling correction improves the performance of models with character representations. With these insights, we report a new state of the art on the ASAP-SAS content scoring dataset.