Exploring NLP concepts using Apache OpenNLP
Tags: nlp, java, graalvm, machinelearning
Published: 2019-11-21 00:58:46

---
title: Exploring NLP concepts using Apache OpenNLP
published: true
description: A post showing NLP concepts with examples with the help of the Java NLP library called Apache OpenNLP
canonical_url: https://blog.valohai.com/exploring-nlp-concepts-using-apache-opennlp-1
cover_image: https://thepracticaldev.s3.amazonaws.com/i/9zlxmy8i9baec5ootnaw.jpeg
tags: nlp, java, graalvm, machine learning
---


# Introduction

After looking at a lot of [Java/JVM based NLP libraries](https://github.com/neomatrix369/awesome-ai-ml-dl/blob/master/natural-language-processing/java-jvm.md#javajvm) listed on [Awesome AI/ML/DL](https://github.com/neomatrix369/awesome-ai-ml-dl/) I decided to pick the [Apache OpenNLP](https://opennlp.apache.org/) library. One of the reasons comes from the fact another developer (who had a look at it previously) recommended it. Besides, it’s an [Apache](https://apache.org) project, they have been great supporters of F/OSS Java projects for the last two decades or so (see [Wikipedia](https://en.wikipedia.org/wiki/The_Apache_Software_Foundation)). It also goes without saying that [Apache OpenNLP](https://opennlp.apache.org/) is backed by the [Apache 2.0 license](https://www.apache.org/licenses/LICENSE-2.0.html).

In addition, this tweet from an NLP researcher added some more confidence to the matter:


{% twitter 1101890668283215872 %}


I’ll like to say my personal experience has been similar with [Apache OpenNLP](https://opennlp.apache.org/) so far and I echo the simplicity and user-friendly API and design. You will see as we explore it further, that being the case.

# Exploring NLP using Apache OpenNLP

## Java bindings

We won’t be covering the Java API to [Apache OpenNLP](https://opennlp.apache.org/) tool in this post but you can find a number of examples in [their docs](https://opennlp.apache.org/docs/1.9.1/manual/opennlp.html). A bit later you will also need some of the resources enlisted in the **Resources** section at the bottom of this post in order to progress further.


## Command-line Interface

I was drawn to the simplicity of the CLI available and it just worked out-of-the-box, for instances where a model was needed, and when it was provided. It would just work without additional configuration.

To make it easier to use and also not have to remember all the CLI parameters it supports I have put together some [shell scripts](https://github.com/valohai/nlp-java-jvm-example#scripts-provided). Have a look at the [README](https://github.com/valohai/nlp-java-jvm-example#nlp-javajvm-) to get more insight into what they are and how to use them.


## Getting started 

You will need the following from this point forward:


- Git client 2.x or higher (an account on GitHub to fork the repo)
- Java 8 or higher (suggest install [GraalVM CE](https://www.graalvm.org) 19.x or higher)
- Docker CE 19.x or higher and check it is running before going further
- Ability to run [shell scripts](https://github.com/valohai/nlp-java-jvm-example#scripts-provided) from the CLI
- Understand reading/writing shell scripts (optional)

*Note: At the time of the writing version 1.9.1 of* [*Apache OpenNLP*](https://opennlp.apache.org/) *was available.*

We have put [together scripts](https://github.com/valohai/nlp-java-jvm-example#scripts-provided) to make these steps easy for everyone:

```
    $ git clone git@github.com:valohai/nlp-java-jvm-example.git
    or 
    $ git clone https://github.com/valohai/nlp-java-jvm-example.git
    $ cd nlp-java-jvm-example
```

This will lead us to the folder with the following files in it:

```
    LICENSE.txt      
    README.md        
    docker-runner.sh     <=== only this one concerns us at startup
    images
    shared               <=== created just when you run the container
```

*Note: a* [*docker image*](https://hub.docker.com/r/neomatrix369/nlp-java) *has been provided to be able to run a docker container that would contain all the tools you need to go further. You can see the* `*shared*` *folder has been created, which is a volume mounted into your container but it’s actually* *a* *directory created on your local machine and mapped to this volume. So anything created or downloaded there will be available even after you exit out of your container!*

Have a quick read of the main [README](https://github.com/valohai/nlp-java-jvm-example#nlp-javajvm-) file to get an idea of how to go about using the [docker-runner.sh shell script](https://github.com/valohai/nlp-java-jvm-example/blob/master/docker-runner.sh), and take a quick glance at the [Usage section](https://github.com/valohai/nlp-java-jvm-example#usage) ****as well**.** Thereafter also take a look into the [Apache OpenNLP README](https://github.com/valohai/nlp-java-jvm-example/blob/master/images/java/opennlp/README.md) file to see the [usages](https://github.com/valohai/nlp-java-jvm-example/tree/master/images/java/opennlp#exploring-nlp-concepts) of the [scripts](https://github.com/valohai/nlp-java-jvm-example/tree/master/images/java/opennlp#scripts-provided) provided there in.

## Run the NLP Java/JVM docker container

At your local machine command prompt while at the root of the project, do this:

```
    $ ./docker-runner.sh --runContainer
```

There is a chance you get this first, before you get the prompt:

```
    Unable to find image 'neomatrix369/nlp-java:0.1' locally
    0.1: Pulling from neomatrix369/nlp-java
    f476d66f5408: ...
    .
    .
    .
    Digest: sha256:53b89b166d42ddfba808575731f0a7a02f06d7c47ee2bd3622e980540233dcff
    Status: Downloaded newer image for neomatrix369/nlp-java:0.1
```

And then you will be presented with prompt inside the container:

```
    Running container neomatrix369/nlp-java:0.1
    
    ++ pwd
    + time docker run --rm --interactive --tty --workdir /home/nlp-java --env JDK_TO_USE= --env JAVA_OPTS=<--snipped>
    nlp-java@cf9d493f0722:~$
```

The container is packed with all the [Apache OpenNLP scripts/tools](https://github.com/valohai/nlp-java-jvm-example/tree/master/images/java/opennlp) you need to get started with exploring various NLP solutions. 

## Installing Apache OpenNLP inside the container

Here is how we go further from here when you are inside the container, at the container command-prompt:

```
    nlp-java@cf9d493f0722:~$ cd opennlp


    nlp-java@cf9d493f0722:~$ ./opennlp.sh
```

You will see the `apache-opennlp-1.9.1-bin.tar.gz` artifact being downloaded and expanded into the `shared` folder:

```
    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100 10.6M  100 10.6M    0     0  4225k      0  0:00:02  0:00:02 --:--:-- 4225k
    apache-opennlp-1.9.1/
    apache-opennlp-1.9.1/NOTICE
    apache-opennlp-1.9.1/LICENSE
    apache-opennlp-1.9.1/README.html
    .
    .
    .
    apache-opennlp-1.9.1/lib/jackson-jaxrs-json-provider-2.8.4.jar
    apache-opennlp-1.9.1/lib/jackson-module-jaxb-annotations-2.8.4.jar
```

## Viewing and accessing the shared folder

Just as you run the container,  a shared folder is created, it may be empty in the beginning but as we go along we will find it fill up with different files and folders.

It’s also where you will find the downloaded models and the [Apache OpenNLP](https://opennlp.apache.org/) binary exploded into its own directory (by the name `apache-opennlp-1.9.1`). 

You can access and see the contents of it from the command-prompt (outside the container) as well:

```
    ### Open a new command prompt
    $ cd nlp-java-jvm-example
    $ cd images/java/opennlp
    $ ls ..
    Dockerfile       corenlp.sh       opennlp          reverb.sh        word2vec.sh
    cogcomp-nlp.sh   mallet.sh        openregex.sh     shared
    common.sh        nlp4j.sh         rdrposttagger.sh version.txt
    
    $ ls ../shared
    apache-opennlp-1.9.1   en-ner-date.bin        en-sent.bin
    en-chunker.bin         en-parser-chunking.bin langdetect-183.bin
    
    ### In your case the contents of the shared folder may vary but the way to get to the folder is above.
```

From inside the container this is what you see:

```
    nlp-java@cf9d493f0722:~$ ls 
    cogcomp-nlp.sh   corenlp.sh  nlp4j.sh  openregex.sh        reverb.sh  word2vec.sh
    common.sh        mallet.sh   opennlp   rdrposttagger.sh        shared
    
    nlp-java@cf9d493f0722:~$ ls shared
    MyFirstJavaNotebook.ipynb      en-ner-date.bin           en-pos-maxent.bin          
    langdetect-183.bin
    apache-opennlp-1.9.1           en-ner-time.bin           en-pos-perceptron.bin  
    notebooks
    en-chunker.bin                 en-parser-chunking.bin    en-token.bin

    ### In your case the contents of the shared folder may vary but the way to get to the folder is above.
```

## Performing NLP actions inside the container

The good thing is without ever leaving your current folder you can perform these NLP actions (check out the [Exploring NLP Concepts](https://github.com/valohai/nlp-java-jvm-example/tree/master/images/java/opennlp#exploring-nlp-concepts) section in the [README](https://github.com/valohai/nlp-java-jvm-example/tree/master/images/java/opennlp#apache-opennlp--)):

Usage help of any of the scripts: at any point in time you can always query the scripts by calling them this way:

```
    nlp-java@cf9d493f0722:~$ ./[script-name.sh] --help
```

For e.g.

```
    nlp-java@cf9d493f0722:~$ ./detectLanguage.sh --help
```

gives us this usage text as output:

```
           Detecting language in a single-line text or article
    
           Usage: ./detectLanguage.sh --text [text]
                     --file [path/to/filename]
                     --help
    
           --text      plain text surrounded by quotes
           --file      name of the file containing text to pass as command arg
           --help      shows the script usage help text
```

- Detecting language in a single-line text or article (see [legend of language abbreviations](https://www.apache.org/dist/opennlp/models/langdetect/1.8.3/README.txt) used)

```
    nlp-java@cf9d493f0722:~$ ./detectLanguage.sh --text "This is an english sentence"

    eng This is an english sentence
```

See [Detecting languages](https://github.com/valohai/nlp-java-jvm-example/blob/master/images/java/opennlp/README-detecting-language.md) section in the [README](https://github.com/valohai/nlp-java-jvm-example#nlp-javajvm-) for more examples and detailed output.


- Detecting sentences in a single line text or article.

```
    nlp-java@cf9d493f0722:~$ ./detectSentence.sh --text "This is an english sentence. And this is another sentence."


    This is an english sentence.
    And this is another sentence.
```

See [Detecting sentences](https://github.com/valohai/nlp-java-jvm-example/blob/master/images/java/opennlp/README-detecting-sentences.md) section in the [README](https://github.com/valohai/nlp-java-jvm-example#nlp-javajvm-) for more examples and detailed output.


- Finding person name, organisation name, date, time, money, location, 
percentage information in a single line text or article.

```
    nlp-java@cf9d493f0722:~$ ./nameFinder.sh --method person  --text "My name is John"


    My name is <START:person> John <END>
```

See [Finding names](https://github.com/valohai/nlp-java-jvm-example/blob/master/images/java/opennlp/README-finding-names.md) section in the [README](https://github.com/valohai/nlp-java-jvm-example#nlp-javajvm-) for more examples and detailed output. There are a number of types of name finder examples in this section.


- Tokenize a line of text or an article into its smaller components (i.e. words, punctuation, numbers).

```
    nlp-java@cf9d493f0722:~$ ./tokenizer.sh --method simple --text "this-is-worth,tokenising.and,this,is,another,one"


    this - is - worth , tokenising . and , this , is , another , one
```

See [Tokenise](https://github.com/valohai/nlp-java-jvm-example/blob/master/images/java/opennlp/README-tokenise.md) section in the [README](https://github.com/valohai/nlp-java-jvm-example#nlp-javajvm-) for more examples and detailed output. 


- Parse a line of text or an article and identify groups of words or phrases that go together (see [Penn Treebank tag set](https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html) for the legend of token types), also see https://nlp.stanford.edu/software/lex-parser.shtml.

```
    nlp-java@cf9d493f0722:~$ ./parser.sh --text "The quick brown fox jumps over the lazy dog ."


    (TOP (NP (NP (DT The) (JJ quick) (JJ brown) (NN fox) (NNS jumps)) (PP (IN over) (NP (DT the) (JJ lazy) (NN dog))) (. .)))
```

See [Parser](https://github.com/valohai/nlp-java-jvm-example/blob/master/images/java/opennlp/README-parser.md) section in the [README](https://github.com/valohai/nlp-java-jvm-example#nlp-javajvm-) for more examples and detailed output.


- Tag parts of speech of each token in a line of text or an article (see [Penn Treebank tag set](https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html) for the legend of token types), also see https://nlp.stanford.edu/software/tagger.shtml.

```
    nlp-java@cf9d493f0722:~$ ./posTagger.sh --method maxent --text "This is a simple text to tag"


    This_DT is_VBZ a_DT simple_JJ text_NN to_TO tag_NN
```

See [Tag Parts of Speech](https://github.com/valohai/nlp-java-jvm-example/blob/master/images/java/opennlp/README-tag-parts-of-speech.md) section in the [README](https://github.com/valohai/nlp-java-jvm-example#nlp-javajvm-) for more examples and detailed output.


- Text chunking by dividing a text or an article into syntactically correlated parts of words, like noun groups, verb groups. You apply this feature on the tagged parts of speech text or article. *Apply chunking on a text already tagged by PoS tagger* (see [Penn Treebank tag set](https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html) for the legend of token types, also see https://nlpforhackers.io/text-chunking/).

```
    nlp-java@cf9d493f0722:~$ ./chunker.sh --text "This_DT is_VBZ a_DT simple_JJ text_NN to_TO tag_NN"


    \[NP This_DT \] [VP is_VBZ ] \[NP a_DT simple_JJ text_NN \] [PP to_TO ] [NP tag_NN]
```

See [Chunking](https://github.com/valohai/nlp-java-jvm-example/blob/master/images/java/opennlp/README-chunking.md) section in the [README](https://github.com/valohai/nlp-java-jvm-example#nlp-javajvm-) for more examples and detailed output.


## Exiting from the  NLP Java/JVM docker container

It is as simple as this:

```
    nlp-java@f8562baf983d:~/opennlp$ exit
    exit
           67.41 real         0.06 user         0.05 sys
```

And you are back to your local machine prompt.

# Benchmarking

One of the salient features of this tool is, it’s recording and reporting metrics of its actions at different execution points - time taken at micro and macro levels, here’s a sample output to illustrate this feature:

```
    Loading Token Name Finder model ... done (1.200s)
    My name is <START:person> John <END>
    
    
    Average: 24.4 sent/s
    Total: 1 sent
    Runtime: 0.041s
    Execution time: 1.845 seconds
```

From the above I have come across 5 metrics that are useful for me as a scientist or an analyst or even as an engineer:

```
    Took 1.200s to load the model into memory
    
    (Average) Processed at an average rate of 24.4 sentences per second
    (Total) Processed 1 sentence
    (Runtime) It took 0.040983606557377 (0.041 seconds) to process this 1 sentence
    (Execution time) The whole process ran for 1.845 seconds (startup, processing sentence(s) and shutdown)
```

Information like this is invaluable when it comes to making performance comparisons like:

- between two or more models (load-time and run-time performance)
- between two or more environments or configurations
- between applications doing the same NLP, action put together using different tech stacks
    - also includes different languages
- finding co-relations between different corpora of text data processed (quantitative and qualitative comparisons)


## Empirical example

[*BetterNLP*](https://bit.ly/better-nlp-launch) *library written in python is doing something similar, see* [*Kaggle*](https://kaggle.com) *kernels:* [*Better NLP Notebook*](https://www.kaggle.com/neomatrix369/better-nlp-class-notebook) *and* [*Better NLP Summarisers Notebook*](https://www.kaggle.com/neomatrix369/better-nlp-summarisers-notebook) *(search for* *time_in_secs* *inside both the notebooks to see the metrics reported).*

![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/ks0v2tkihh3l994ppfuy.png)
![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/ppch7gn9v6f8hu0oghb0.png)

Personally, it’s quite inspiring and also validates that this is a useful feature (or action) to offer to the end-user.

# Other concepts, libraries and tools 

There are other [Java/JVM based NLP libraries](https://github.com/valohai/nlp-java-jvm-example#libraries--frameworks-provided) mentioned in the **Resources** section below, for brevity we won’t cover them. The links provided will lead to further information for your own pursuit.

Within the [Apache OpenNLP](https://opennlp.apache.org/) tool itself, we have only covered the command-line access part of it and not the Java Bindings. In addition, we haven’t gone through all the NLP concepts or features of the tool again for brevity have only covered a handful of them. But the [documentation](https://opennlp.apache.org/docs/) and [resources](https://github.com/apache/opennlp#useful-links) on the [GitHub repo](https://github.com/apache/opennlp) should help in further exploration.

You can also find out how to build the [docker image](https://hub.docker.com/r/neomatrix369/nlp-java) for yourself, by examining the [docker-runner script](https://github.com/valohai/nlp-java-jvm-example/blob/master/docker-runner.sh).

# Conclusion
![](https://thepracticaldev.s3.amazonaws.com/i/bsrca9odbf87282z68to.jpeg)


After going through the above, we can conclude the following about the [Apache OpenNLP](https://opennlp.apache.org/) tool by exploring its pros and cons:

Pros

- It’s an easy to use API and understand
- Shallow learning curve and detailed documentation with lots of examples
- Covers a lot of NLP functionality, there’s more in the [docs](https://opennlp.apache.org/docs/) to explore than we did above
- Easy [shell scripts](https://github.com/valohai/nlp-java-jvm-example#scripts-provided) and [Apache OpenNLP scripts](https://github.com/valohai/nlp-java-jvm-example/tree/master/images/java/opennlp) have been provided to play with the tool
- Lots of resources available below to learn more about NLP (See the **Resources** section below)
- Resources provided to quickly get started and explore the [Apache OpenNLP](https://opennlp.apache.org/) tool

Cons

- Looking at the [GitHub](https://github.com/apache/opennlp) repo, it seems the development is slow or has been stagnated (last two commits have a wide gap i.e. May 2019 and Oct 15, 2019)
- A few models are missing when going through the examples in the [documentation (manual)](https://opennlp.apache.org/docs/1.9.1/manual/opennlp.html)
- The current models provided may need further training as per your use case(s), see this tweet:

{% twitter 1193626559439134721 %}

# Resources
## Apache OpenNLP
- [nlp-java-jvm-example](https://github.com/valohai/nlp-java-jvm-example) GitHub project 
- [Apache OpenNLP](https://opennlp.apache.org/) | [GitHub](https://github.com/apache/opennlp) | [Mailing list](https://opennlp.apache.org/mailing-lists.html) | [@apacheopennlp](https://twitter.com/@apacheopennlp)
- Docs
    - [Documentation resources](https://opennlp.apache.org/docs/)
    - [Manual](https://opennlp.apache.org/docs/1.9.1/manual/opennlp.html)
    - [Apache OpenNLP Tools Javadoc](https://opennlp.apache.org/docs/1.9.1/apidocs/opennlp-tools/index.html)
- Download
    - [Apache OpenNLP Jar/binary](https://opennlp.apache.org/download.html)
    - Model Zoo
        - [Models page](https://opennlp.apache.org/models.html)
        - [Language Detect model](http://www.mirrorservice.org/sites/ftp.apache.org/opennlp/models/langdetect/1.8.3/langdetect-183.bin)
        - [Older models to support the examples in the docs](http://opennlp.sourceforge.net/models-1.5/)
- Legends to support the examples in the docs
    - [List of languages](https://www.apache.org/dist/opennlp/models/langdetect/1.8.3/README.txt)
    - [Penn Treebank tag set](https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html)
- Find more in the [Resources](https://github.com/valohai/nlp-java-jvm-example/blob/master/images/java/opennlp/README.md#resources) section in the [README](https://github.com/valohai/nlp-java-jvm-example/blob/master/images/java/opennlp/README.md#apache-opennlp--)
## Other related posts
    - [How to do Deep Learning for Java on the Valohai Platform?](https://blog.valohai.com/nlp_with_dl4j_in_java_all_from_the_command-line?from=3oxenia9mtr6)
    - [NLP with DL4J in Java, all from the command-line](https://blog.valohai.com/nlp_with_dl4j_in_java_all_from_the_command-line?from=3oxenia9mtr6)

## About me

Mani Sarkar is a passionate developer mainly in the Java/JVM space, currently strengthening teams and helping them accelerate when working with small teams and startups, as a freelance software/data/ml engineer, [more….](https://neomatrix369.wordpress.com/about)

**Twitter:** [@theNeomatrix369](https://twitter.com/@theNeomatrix369) | **GitHub:** [@neomatrix369](https://github.com/neomatrix369)


Originally published at [https://blog.valohai.com](https://blog.valohai.com/exploring-nlp-concepts-using-apache-opennlp-1?from=3oxenia9mtr6).