# Natural Language Toolkit

## Summary


Natural Language Toolkit (NLTK) is a Python package to perform natural language processing (NLP). It was created mainly as a tool for learning NLP via a hands-on approach. It was not designed to be used in production. 

The growth of unstructured data via social media, online reviews, blogs, and voice-based human-computer interaction are some reasons why NLP has become important in the late 2010s. NLTK is a useful toolkit for many of these NLP applications. 

NLTK is composed of sub-packages and modules. A typical processing pipeline will call modules in sequence. Python data structures are passed from one module to another. Beyond the algorithms, NLTK gives quick access to many text corpora and datasets.

## Discussion

### Which are the fundamental NLP tasks that can be performed using NLTK?

NLTK can be used in wide range of applications for NLP. For basic understanding, let's try to analyze a paragraph using NLTK. It can be pre-processed using sentence segmentation, removing stopwords, removing punctuation and special symbols, and word tokenization. After pre-processing the corpus, it can be analyzed sentence-wise using parts of speech (POS) to extract nouns and adjectives. Subsequent tasks can include named entity recognition (NER), coreference resolution, constituency parsing and dependency parsing. The goal is to find insights and context about the corpus. 

Further downstream tasks, more pertaining to application areas, could be emotion detection, sentiment analysis or text summarization. Tasks such as text classification and topic modeling typically require large amounts of text for better results.


### Which are the modules available in NLTK?

NLTK's architecture is modular. Functionality is organized into sub-packages and modules. NLTK is used for its simplicity, consistency and extensibility of its modules and functions. It's better explained in the tabular list of modules. 

A complete module index is available as part of NLTK documentation.


### How is NLTK package split into sub-packages and modules?

NLTK is divided into different sub-packages and modules for text analysis using various methods. Figure depicts an example of `text` sub-package and the modules within it. Each module fulfils a specific function. 


### Which are the natural languages supported in NLTK?

Languages supported by NLTK depends on the task being implemented. For **stemming**, we have RSLPStemmer (Portuguese), ISRIStemmer (Arabic), and SnowballStemmer (Danish, Dutch, English, Finnish, French, German, Hungarian, Italian, Norwegian, Portuguese, Romanian, Russian, Spanish, Swedish). 

For **sentence tokenization**, PunktSentenceTokenizer is capable of multilingual processing. 

**Stopwords** are also available in multiple languages. After importing `stopwords`, we can obtain a list of languages by running `print(stopwords.fileids())`. 

Although most **taggers** in NLTK support only English, `nltk.tag.stanford` module allows us to use StanfordPOSTagger, which has multilingual support. This is only an interface to the Stanford tagger, which must be running on the machine. 


### What datasets are available in NLTK for practice?

NLTK corpus is a natural dump for all kinds of NLP datasets that can be used for practice or maybe combined for generating models. For example, to import the Inaugural Address address, the statement to execute is `from nltk.corpus import inaugural`. 

Out of dozens of corpora, some popular ones are Brown, Name Genders, Penn Treebank, and Inaugural Address. NLTK makes it easy to read a corpus via the package `nltk.corpus`. This package has a reader object for each corpus. 


### What are the disadvantages or limitations of NLTK?

It's been mentioned that NLTK is "a complicated solution with a harsh learning curve and a maze of internal limitations". 

For sentence tokenization, NLTK doesn't apply semantic analysis. Unlike *Gensim*, NLTK lacks neural network models or word embeddings. 

NLTK is slow, whereas *spaCy* is said to be the fastest alternative. In fact, since NLTK was created educational purpose, optimized runtime performance was never a goal. However, it's possible to speed up execution using Python's `multiprocessing` module. 

Matthew Honnibal, the creator of *spaCy*, noted that NTLK has lots of modules but very few (tokenization, stemming, visualization) are actually useful. Often NLTK has wrappers to external libraries and this leads to slow execution. The POS tagger was terrible, until Honnibal's averaged perceptron tagger was merged into NLTK in September 2015. 

In general, NLP is evolving so fast that maintainers need to curate often and throw away old things. 


### For beginners, what are some useful resources to learn NLTK?

The official website includes documentation, Wiki, and index of all modules. There are Google Groups for users and developers.

For basic usage of NLTK, you can read a tutorial by Bill Chambers. This also shows some text classification examples using *Scikit-learn*. Another basic tutorial from Harry Howard includes examples from *Pattern* library as well. 

Often specific processing is implemented in external libraries. Benjamin Bengfort shows in a blog post how to call CoreNLP from inside NLTK for syntactic parsing. 

There's a handy cheat sheet by murenei. Another one from 2017 is published at Northwestern University.

A list of recommended NLTK books appears on BookAuthority. You can start by reading Natural Language Processing with Python (Bird et al. 2009). Those who wish to learn via videos can look up a playlist of 21 videos from sentdex. 

## Milestones

Jul  
2001

The first downloadable version of NLTK appears on SourceForge. Created at the University of Pennsylvania, the aim is to have a set of open source software, tutorials and problem sets to aid the teaching of computational linguistics. Before NLTK, a project might require students to learn multiple programming languages and toolkits. Lack of visualizations also made it difficult to have class demonstrations. NLTK is meant to solve these problems. 

Jul  
2005

**NLTK-Lite 0.1** is released. Steven Bird, one of the creators of NLTK, explains that NLTK 1.4 introduced Python's dictionary-based architecture for storing tokens. This created overhead for programmers. With NLTK-Lite, programmers can use simpler data structures. For better performance, iterators are used instead of lists. Taggers use backoff by default. Method names are shorter. Since then, regular releases are made till NLTK-Lite 0.9 in October 2007. NLTK-Lite eventually becomes NLTK. 

Apr  
2008

Two NLTK projects are accepted for **Google Summer of Code**: dependency parsing and natural language generation. The dependency parser becomes part of NLTK version 0.9.6 (December 2008). 

Jun  
2009

Book titled *Natural Language Processing with Python* by Bird et al. is published by O'Reilly Media. Since October 2013, the authors release online revised versions of the book updated for Python 3 and NLTK 3. 

Apr  
2011

**Version 2.0.1rc1** becomes the first release available via GitHub, although till July 2014 releases are also made via SourceForge. 

Jul  
2013

Over a five-year period from January 2008 to July 2013, NLTK gets more than half a million downloads. This excludes downloads via GitHub. 

Sep  
2014

**NLTK 3.0.0** is released, making this the first stable release supporting Python 3. Alpha release, version 3.0a0 (alpha), supporting Python 3 can be traced to January 2013.