How to easily add sentiment analysis to any Java application
Tags: sentimentanalysis, nlp, chatbots, java
Published: 2020-03-26 06:38:39

You probably want your bot to <a href="https://xatkit.com/two-simple-ways-make-chatbots-more-human/" target="_blank" rel="noopener noreferrer">look as human as possible</a>. And a key characteristic of (many) humans is their empathy, i.e. the ability to understand the feeling of the person we are talking to and to adapt our response to those feelings.

<strong>I want my chatbots to show the same level of empathy</strong>. And for that, the key ingredient is a sentiment analysis feature that helps the bot detect whether the user is happy or not. Or, in more technical terms, whether the sentences she is writing have a positive or a negative tone. It would be even better if we could detect also whether he is sad, or joking or .... but, as usual in any type of classification problem, the more categories you define the more difficult is the classification.

The same applies to a large variety of software applications that need some degree of user interaction. We will use chatbots in the rest of the post as the most obvious case of heavy text-based interaction but everything we explain can also be useful in any other scenario where you need to process some kind of input text and respond based on the sentiment you can detect in it. 

We have now <strong>added sentiment analysis capabilities to [Xatkit](https://xatkit.com/)</strong>. The goal of this post is to explain how we did it, hoping our experience will be useful to any of you thinking of doing the same. If I manage to convince you of how easy the process was, I'll be happy enough!

In Xatkit, sentiment analysis it's implemented as one of our brand new <a href="https://github.com/xatkit-bot-platform/xatkit/wiki/Processors" target="_blank" rel="noopener noreferrer">Processor</a> mechanism. Processors are additional pieces of logic that can be plugged to tune the intent recognition process. <em>Pre-processors</em> operate on the user input to optimize it before intent extraction. <em>Post-processors</em> are designed to operate <em>after</em> the intent recognition process, usually to set additional context parameters. Sentiment analysis is one of such post-processors (we'll talk about other processors in future posts). This allows us <strong>to tune the chatbot response to how the user is feeling. </strong>
<blockquote>Keep your visitors engaged in the conversation by adapting your response to their emotional state</blockquote>
But we didn't want to implement such a complex feature from scratch.  Instead, as <a href="https://github.com/xatkit-bot-platform/xatkit" target="_blank" rel="noopener noreferrer">Xatkit's core</a> is written in Java, we searched for existing Java libraries that we could reuse. And here is where the wonderful <a href="https://stanfordnlp.github.io/CoreNLP/download.html" target="_blank" rel="noopener noreferrer">Stanford CoreNLP</a> project, and in particular the <a href="https://stanfordnlp.github.io/CoreNLP/sentiment.html" target="_blank" rel="noopener noreferrer">SentimentAnnnotator</a> component,  came to the rescue. Stanford NLP annotates an input text with 5 classes of sentiment classification: <span class="scoreOne">very negative</span>, <span class="scoreTwo">negative</span>, <span class="scoreThree">neutral</span>, <span class="scoreFour">positive</span>, and <span class="scoreFive">very positive</span>. To do so, it relies on a <a href="https://nlp.stanford.edu/sentiment/index.html" target="_blank" rel="noopener noreferrer">Recursive Neural Network pre-trained with its own sentiment treebank</a>.

This may sound complicated but I promise it's not. All the code showing how we integrated Stanford NLP in Xatkit is available in <a href="https://github.com/xatkit-bot-platform/xatkit-runtime/blob/master/core/src/main/java/com/xatkit/core/recognition/processor/EnglishSentimentPostProcessor.java" target="_blank" rel="noopener noreferrer">this Java class</a>. Let's see the key code excerpts so that you can build a similar integration for your own tool.

The following snippet shows how we create an <code>Annotation</code> object from the input text string (provided by <code>getMatchedInput</code> in the example; this and the <code>session</code> objects are internal Xatkit elements storing the user utterance and session data, respectively). From the <code>annotation</code> instance, we then get the set of annotated sentences (Stanford assumes we want to identify the sentiment in a block of text, not just single sentences as there may not be enough context there, though in our tests, it behaved pretty impressive even with short sentences). Finally, with <code>get(SentimentCoreAnnotations.SentimentClass.class)</code>, we filter out other types of annotations and keep only the sentiment analysis one. This value is then added to the <code>session</code> information so that we can use it to decide the response action later on.

{% gist https://gist.github.com/jcabot/2fe8638191aeb37da532c1c993598c29 %}

The <code>getAnnotation</code> method is the most important one as it's here where we are actually asking Stanford NLP to run its analysis on our input data. In our code, we have wrapped the call in our own <a href="https://github.com/xatkit-bot-platform/xatkit-runtime/blob/master/core/src/main/java/com/xatkit/core/recognition/processor/StanfordNLPPostProcessor.java" target="_blank" rel="noopener noreferrer">StanfordNLPPostProcessor class</a> for performance reasons. Annotating a text takes time and we wanted to make sure that we were doing it only once even if we had several processors running on the text for different reasons. As shown in the following code snippet, we make sure to call this method if and only if there is no other <code>Annotation</code> object already available for the same text.

{% gist https://gist.github.com/jcabot/2ea340282feddeaa85fb70ead68c61b4 %}

Finally, the result of the analysis can be accessed through the <code>context</code> variable in the bot execution language, that contains a special <code>nlp</code> context holding the results of the different StanfordNLP-related post-processors (sentiment analysis, but also whether a question is a yes/no question, etc). The code below is actually a real-world example from our <a href="https://www.xatkit.com">own bot</a>!

{% gist https://gist.github.com/gdaniel/39af9dbb284e940cf651f3744b8bbe70 %}

Don't forget you can give it a try by just moving to our home page. See an animation of the sentiment test in our demo:

<img class="wp-image-119094 aligncenter" src="https://xatkit.com/wp-content/uploads/2020/03/sentiment.gif" alt="Sentiment Analysis test in our own chatbot" width="301" height="441" />

I hope we managed to convince you that adding a Sentiment Analysis capability to your tool is worth your time. And this is just the beginning, beyond sentiment analysis (and the other available <a href="https://github.com/xatkit-bot-platform/xatkit/wiki/Processors" target="_blank" rel="noopener noreferrer">Processors</a>), we believe there are plenty of other opportunities that could make chatbots much better in understanding the users. Even to prevent the bot from trying to match the utterance at all. I'm thinking, for instance, in a <strong>pre-processor for toxic comments</strong> (e.g. see <a href="https://seriouswp.com/machine-learning-wordpress-artificial-intelligence/" target="_blank" rel="noopener noreferrer">one for WordPress comments</a>) or <strong>a troll detector</strong>. This can also prevent you from wasting valuable resources (including money if you're using a cloud-based API for the matching phase) by NOT processing user utterances that can just be thrown away.

<small>Featured image by <a href="https://unsplash.com/@jtylernix?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Tyler Nix</a> on <a href="https://unsplash.com/s/photos/love?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></small>