NYTWIT : A Dataset of Novel Words in the N ew Y ork T imes (2020)

Abstract: We present the New York Times Word Innovation Types dataset, or NYTWIT, a collection of over 2,500 novel English words published in the New York Times between November 2017 and March 2019, manually annotated for their class of novelty (such as lexical derivation, dialectal variation, blending, or compounding). We present baseline results for both uncontextual and contextual prediction of novelty class, showing that there is room for improvement even for state-of-the-art NLP systems. We hope this resource will prove useful for linguists and NLP practitioners by providing a real-world environment of novel word appearance.

## introduction novel words, or out-of-vocabulary words (oovs), are a pervasive problem in modern natural language processing @xcite . a common scenario in which this problem appears is that of a pre-trained model containing a word representation component such as an embedding table, encountering a previously-unseen word in a downstream task such as question answering or natural language inference. multiple lines of work attempt to alleviate the downstream effect of oov words @xcite , but most tend to focus on individual categories of oovs: typographical errors @xcite , domain-specific terminology @xcite , stylistic variability @xcite , morphological productivity @xcite , or novel named entities @xcite . in reality, unseen texts contain all these classes of novelty, and more. oovs are a typically presented as a significant challenge for generalization or understanding in noisy user-generated text (e.g. twitter) and/or domain-specific content. nevertheless, even large corpora that are narrow in domain (edited news stories) contain linguistic innovations, including but not limited to novel morphological processes, typographical errors, and loan words. in this paper, we present a dataset of novel words in english relative to the corpus of articles published by the new york times (nyt), as collected automatically in real time by a twitter bot. we name it the new york times word innovation types corpus, or nytwit for short. we annotated each word for one of eighteen linguistically-informed categories of novelty within the context of the nyt corpus, as well as for its date of publication and a retrieval document identifier to enable context extraction. 1 to our knowledge, this is the first resource to include novel words along with their contextual information in addition to linguistically-informed annotation, a method that enables expansion beyond dictionary-based methods @xcite @xcite and decontextualized neologisms @xcite . in contrast with resources which provide examples and attestations to lexical forms, nytwit was constructed in a corpus-comprehensive manner where novelty guides curation and not vice versa. in addition, we provide results for the task of classifying words into their categories based on word form and contextual information, a task which can both provide data for linguistic analysis of lexical enrichment and serve as a processing step for nlp systems which may work better if different modules are applied to different classes of novel words. we show that both character-level models and large pre-trained sentence encoders struggle on this task, illustrating the challenges of modeling language innovation. we release the data at https://github.com/yuvalpinter/nytwit under the gnu general public license v3.0. the project is ongoing, and this document pertains to version 1.1. an associated context bot replies to the tweets with links to the original articles. 5 we used the urls from this bot's posts as the main reference for the words' contexts. for 17 words, the article url was retrieved manually by searching for the target article directly. 6 as the articles are subject to edits long after publication, there is an increasing but small portion of articles which no longer contain the context, although at time of publication these mostly include the removal of typographical errors from the stories and which are ultimately filtered by our annotation process (see below). ## oov classification task the task of classifying oovs, i.e. assigning a novel word with a label from the taxonomy we defined above, can be beneficial from both an analytical linguistic standpoint, and from an nlp standpoint concerned with model performance on downstream language understanding tasks. to get a sense of the predictability of the various oov classes in the dataset, we present several baselines for this straightforward task. the uniqueness of our dataset allows us to apply both type-level and context-dependent systems, the latter operating in the real-world scenario of encountering a word for the first time in the actual context of its introduction to the corpus. first, our majority class baseline assumes all oovs are the result of affixation. for all following models we trained a ridge classifier with default regularization parameters in scikit-learn. scores for all supervised models are reported via 10-fold cross-validation using the same folds for all systems. due to the class imbalance, we chose to implement training in such a way that upsampled rare classes with replacement at each iteration to equal frequency as the most common class. we report accuracy (acc) and macro f1 scores. contextless features. we compare and contrast several input features to our classifier that only have access to the form of the oov, without consideration of the context: ## conclusion we presented a novel dataset of oovs along with their contexts and linguistic novelty class annotations. we showed that contextual information in the form of other parts of the sentence provides some signal, but simple models relying on character n-gram information alone achieve high performance. the availability of broader document contexts in which these neologisms occur enables many linguistic and technical applications. from the perspective of the study of language growth and formation, the dataset may be used to assess the morphological productivity of different affixes and roots, or the prevalence of the different word formation processes in a realistic setting; or perform in-depth analysis on any of the specific types of innovations we identified. in addition, the in-vivo nature of the dataset provides a reference for neologisms which may or may not be later adopted into everyday use, allowing diachronic studies anchored in the time of word introduction. analysis of the phonological, morphological, and discourse-level properties of these words may provide insight into lexical adoption dynamics. for nlp researchers, an important component of text applications is proper normalization and segmentation of word forms. our experiment shows that popular word form encoders, such as elmo or bert's wordpiece, still have a long way to go in terms of recognizing the origins of a novel form. errors at this stage might lead to inability to handle morphologically complex oovs in downstream semantic applications @xcite , although further study of such effects and of the utility of oov classification in alleviating them is still necessary. properly leveraging context for morphological decomposition of complex forms also remains an open problem. the resource is an ongoing project; the repository includes plans for the next versions, including increasing the dataset size by including newer words from the bot, and annotating additional information such as part-of-speech tags.