How to build your own chatbot NLP engine
Tags: chatbot, nlp, tutorial, python
Published: 2022-03-28 09:23:57

This post goes over our <a href="https://github.com/xatkit-bot-platform/xatkit-nlu-server" target="_blank" rel="noopener">new NLU chatbot intent classifier engine for chatbots</a> to show how you can use it to:
<ol>
 	<li>(obviously) Create your own chatbots (pairing it up with <a href="https://github.com/xatkit-bot-platform/xatkit" target="_blank" rel="noopener">Xatkit </a>or any other chatbot platform for all the front-end and behaviour processing components)</li>
 	<li>Learn about natural language processing by playing with the code and executing it with different parameter combinations</li>
 	<li>Demystify the complexity of building a NLU engine (thanks to the myriad of wonderful open-source libraries and frameworks available) and provide you with a starting point you can use to build your own.</li>
</ol>

Ready for some NLP fun? We'll first give some context about the project and then we'll take a deep dive into the NLU engine code.

<h2>What is an intent classifier?</h2>
An intent or <em>intention</em> is a user question / request a chatbot should be able to recognize. For instance, the questions in a FAQ-like chatbot. To be able to answer the user input request (called user <em>utterance</em>) the bot needs to first understands what the user is asking about.

An intent classifier (also known as intent recognition, intent matching, intent detector,...) is then the function that, given a set of intents the bot can understand and an utterance from the user, returns the probability that the user is asking about a certain intent.

Nowadays, intent classifiers are typically implemented as <a href="https://en.wikipedia.org/wiki/Multiclass_classification#Neural_networks" target="_blank" rel="noopener">multi-class classifier</a> neural network.
<h2>Does the world really needs yet another NLU chatbot engine?</h2>
Probably not. In fact, in Xatkit we aim to be a <a href="https://xatkit.com/chatbot-orchestration-platform-open-source/" target="_blank" rel="noopener">chatbot orchestration platform</a> exactly to avoid reinventing the wheel and the <a href="https://en.wikipedia.org/wiki/Not_invented_here" target="_blank" rel="noopener">non-invented here syndrome</a>. So, in most cases, other existing platform (like DialogFlow or <a href="https://github.com/axa-group/nlp.js/" target="_blank" rel="noopener">nlp.js</a>) will work just fine. But we have also realized that there are always some particularly tricky bots for which you really need to be able to customize your engine to the specific chatbot semantics to get the results you want.

And this is sometimes impossible (for platforms where the source code is not available, which, by the way, is the norm) or difficult (e.g. if you're not ready to invest lots of time trying to understand how the platform works or if the platform is not created in a way to easily allow for customizations.

And of course, if your goal is to learn natural language processing techniques, there is nothing better to write some NLP code :-). Ours is simple enough to give you a kick-start.
<h2>What makes our NLU engine different?</h2>
<a href="https://github.com/xatkit-bot-platform/xatkit-nlu-server" target="_blank" rel="noopener">Xatkit's NLU Engine</a> is (or better said, it will be, as what we're releasing now is still an alpha version with limited functionality, which is good to play with and to learn, not so much to use it on production ;-) ) a flexible and pragmatic chatbot engine. A couple of examples of this flexible and pragmatic approach.

<h3 dir="auto">Xatkit lets you configure <em>almost</em> everything</h3>
So that the data processing, the training of the network, the behaviour of the classifier itself, etc can be configured. Every time we make a design decision we create a parameter for it so that you can choose whether to follow our strategy or not.
<h3 dir="auto">Xatkit creates a <em>separate neural network</em> for each bot context</h3>
<p dir="auto">We see bots as having different conversation contexts (e.g. as part of a <a href="https://xatkit.com/chatbot-dsl-state-machines-xatkit-language/" rel="nofollow">bot state machine</a>). When in a given context, only the intents that make sense in that context should be evaluated when considering possible matches.</p>
<p dir="auto">A Xatkit bot is composed of contexts where each context may include a number of intents (see the <code>dsl</code> package). During the training phase, a NLP model is trained on those intents' training sentences and attached to the context for future predictions).</p>

<h3 dir="auto"><a id="user-content-xatkit-understands-that-a-neural-network-is-not-always-the-ideal-solution-for-intent-matching" class="anchor" href="https://github.com/xatkit-bot-platform/xatkit-nlu-server#xatkit-understands-that-a-neural-network-is-not-always-the-ideal-solution-for-intent-matching" aria-hidden="true"></a>Xatkit understands that <em>a neural network is not always the ideal solution</em> for intent matching</h3>
<p dir="auto">What if the user input text is full of words the NN has never seen before? It's safe to assume that we can directly determine there is no matching and trigger a bot move to the a <a href="https://github.com/xatkit-bot-platform/xatkit/wiki/Default-and-Local-Fallback">default fallback</a> state.</p>
<p dir="auto">Or what if the input text is a perfect literal match to one of the training sentences? Shouldn't we assume that's the intent to be returned with maximum confidence?</p>
<p dir="auto">This type of pragmatic decisions are at the core of Xatkit to make it a really useful chatbot-specific intent matching project.</p>

<h2>Show me code!</h2>
At the core of the intent classifier we have <strong>a <a href="https://keras.io/" target="_blank" rel="noopener">Keras</a> / <a href="https://www.tensorflow.org/" target="_blank" rel="noopener">Tensorflow </a>model</strong>. But before training the network and using it for prediction we need to make sure we <strong>process the training data</strong> (the pairs <training sentence, intent> in this case). And to make the NLP Engine useful we expose it through an <strong>easy-to-use REST API</strong> that makes it easy to integrate it in external platforms or libraries.

So, overall, the project structure is the following:<img class=" wp-image-120162 aligncenter" src="https://xatkit.com/wp-content/uploads/2022/03/projectstructure.png" alt="NLU Server Structure of the python project" width="518" height="435" />

<code> Main.py</code> holds the API definition (thanks to the <a href="https://fastapi.tiangolo.com/" target="_blank" rel="noopener">FastAPI</a> framework). The <code>dsl</code> package has the internal data structures storing the bot definition. The <code>dto</code> is a simplified version of the dsl classes to facilitate the API calls. Finally, the <code>core</code> package includes the configuration options and the core prediction and training functions.

<h3>Core Neural Network definition</h3>
At the core of the Xatkit NLU engine we have a <a href="https://keras.io/" target="_blank" rel="noopener">Keras</a>/<a href="https://www.tensorflow.org/" target="_blank" rel="noopener">Tensorflow</a> model.

{% embed https://gist.github.com/jcabot/0674804e0dd8f7126a27aeec9dc4fb5c.js %}

The layers and parameters are rather standard for a classifier network. Two aspects worth mentioning:
<ul>
 	<li>The number of classes depends on the value of <code>len(context.intents)</code>. We have as many classes as intents are defined in a given bot context.
 	<li>We use <a href="https://en.wikipedia.org/wiki/Sigmoid_function" target="_blank" rel="noopener"><em>sigmoid </em></a>function in the last layer as the intents are not always mutually exclusive so we want to get the probabilities for each of them independently</li>
</ul>

<h3>Preparing the data for the training</h3>
Before training the chatbot using the above ML model, we need to prepare the data for training. 

Keep in mind that bots are defined as a set of context where for each context we have a number of intents and for each intent a number of training sentences.

{% embed https://gist.github.com/jcabot/0beb16be016a3be726c97aae73c59b22.js %}

To get the training started, we take the training sentences and link them with their corresponding intents to create the labeled data for the bot.

{% embed https://gist.github.com/jcabot/873bf02c7b0919f2678ec34b376ccec3.js %}

Some details on the code listing above:
<ul>
	<li> We assign a numeric value to each intent and we use that number when populating the <code> total_labels_training_sentences </code> list
        <li> We use a tokenizer to create a <em>word index</em> for the words in the training sentences by calling <code>fit_on_texts</code>. This word index is then used (<code>texts_to_sequences</code> to transform words into their index value. At this point training sentences are a set of numeric values, which we call <code>training_sequences</code>
       <li> Padding ensures that all sequences have the same length. This length, as always, is part of the NLP configuration.
       <li> The model is finally trained by calling the <code>fit</code>
</ul>

<h3> Predicting the best intent matches </h3>

Once the ML model has been trained, prediction is straightforward. The only important thing to keep in mind is to process the user utterance with the same process we prepared the training data. Note that we also implement one of the optimizations discussed above to directly return a zero match prediction for sentences where all tokens have never seen before by the model.

{% embed  https://gist.github.com/jcabot/cce2c1897235cd5a14db94a1d9c5b9a2.js  %}

<h3> Exposing the chatbot intent classifier as a REST API </h3>
The <code> main.py </code> module is in charge of exposing our FastAPI methods. As an example, this is the method for training a bot. It relies on <a href="https://pydantic-docs.helpmanual.io/">Pydantic</a> to facilitate the processing of the JSON input and output parameters. Parameter types are the <code>dto</code> version of the <code>dsl</code> classes.  

{% embed https://gist.github.com/jcabot/b0d5b3963c395251d998dff364398efb.js %}

<h2>Ready to give it a try?</h2>
Great. Go over to <a href="https://github.com/xatkit-bot-platform/xatkit-nlu-server" target="_blank" rel="noopener">https://github.com/xatkit-bot-platform/xatkit-nlu-server</a>  and follow the installation instructions. Even better if you watch/star it to follow our new developments. And even, even better if you decide to contribute to the project in any way, shape or form you can!