Step by step guide to create customized chatbot by using spaCy (Python NLP library)
Tags: docker, json, machinelearning, python
Published: 2024-03-23 17:02:33

<p><img alt="image" src="https://github.com/mwaseem75/iris-GenLab/assets/18219467/e71ba8d2-f379-4d56-a7b3-770005fe08c8"></p>
<p>Hi Community,<br>In this article, I will demonstrate below steps to create your own chatbot by using <a href="https://spacy.io/" rel="nofollow"><strong>spaCy</strong></a><span>&nbsp;(</span>spaCy is an open-source software library for advanced natural language processing, written in the programming languages Python and Cython<span>)</span>:</p>
<ul>
<li>
<h3>Step1: Install required libraries</h3></li>
<li>
<h3>Step2: Create patterns and responses file</h3></li>
<li>
<h3>Step3: Train the Model</h3></li>
<li>
<h3>Step4: Create ChatBot Application based on the trained model</h3></li>
</ul>
<p>So Let us start.</p>
<h2><!--break-->Step1: Install required libraries</h2>
<p>First of all, we need to install the required python libraries, which we can achieve by running the below command:</p>
<pre class="codeblock-container" idlang="11" lang="Bash" tabsize="4"><code class="language-bash hljs">pip3 install spacy nltk
pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
</code></pre>
<h2><!--break-->Step2: Create patterns and responses file<be></be></h2>
<p>We need to create intents.json&nbsp;file containing question patterns and responses, below is an example of some of the question patterns and responses</p>
<pre class="codeblock-container" idlang="3" lang="JSON" tabsize="4"><code class="language-json hljs">{
  <span class="hljs-attr">"intents"</span>: [
    {
      <span class="hljs-attr">"tag"</span>: <span class="hljs-string">"greeting"</span>,
      <span class="hljs-attr">"patterns"</span>: [
        <span class="hljs-string">"Hi"</span>,
        <span class="hljs-string">"Hey"</span>,
        <span class="hljs-string">"How are you"</span>,
        <span class="hljs-string">"Is anyone there?"</span>,
        <span class="hljs-string">"Hello"</span>,
        <span class="hljs-string">"Good day"</span>
      ],
      <span class="hljs-attr">"responses"</span>: [
        <span class="hljs-string">"Hey :-)"</span>,
        <span class="hljs-string">"Hello, thanks for visiting"</span>,
        <span class="hljs-string">"Hi there, what can I do for you?"</span>,
        <span class="hljs-string">"Hi there, how can I help?"</span>
      ]
    },
    {
      <span class="hljs-attr">"tag"</span>: <span class="hljs-string">"goodbye"</span>,
      <span class="hljs-attr">"patterns"</span>: [<span class="hljs-string">"Bye"</span>, <span class="hljs-string">"See you later"</span>, <span class="hljs-string">"Goodbye"</span>],
      <span class="hljs-attr">"responses"</span>: [
        <span class="hljs-string">"See you later, thanks for visiting"</span>,
        <span class="hljs-string">"Have a nice day"</span>,
        <span class="hljs-string">"Bye! Come back again soon."</span>
      ]
    },
    {
      <span class="hljs-attr">"tag"</span>: <span class="hljs-string">"thanks"</span>,
      <span class="hljs-attr">"patterns"</span>: [<span class="hljs-string">"Thanks"</span>, <span class="hljs-string">"Thank you"</span>, <span class="hljs-string">"That's helpful"</span>, <span class="hljs-string">"Thank's a lot!"</span>],
      <span class="hljs-attr">"responses"</span>: [<span class="hljs-string">"Happy to help!"</span>, <span class="hljs-string">"Any time!"</span>, <span class="hljs-string">"My pleasure"</span>]
    },
    {
      <span class="hljs-attr">"tag"</span>: <span class="hljs-string">"items"</span>,
      <span class="hljs-attr">"patterns"</span>: [
        <span class="hljs-string">"tell me about this app"</span>,
        <span class="hljs-string">"What kinds of technology used?"</span>,
        <span class="hljs-string">"What do you have?"</span>
      ],
      <span class="hljs-attr">"responses"</span>: [
        <span class="hljs-string">"Write something about the app."</span>
      ]
    },        
    {
      <span class="hljs-attr">"tag"</span>: <span class="hljs-string">"funny"</span>,
      <span class="hljs-attr">"patterns"</span>: [
        <span class="hljs-string">"Tell me a joke!"</span>,
        <span class="hljs-string">"Tell me something funny!"</span>,
        <span class="hljs-string">"Do you know a joke?"</span>
      ],
      <span class="hljs-attr">"responses"</span>: [
        <span class="hljs-string">"Why did the hipster burn his mouth? He drank the coffee before it was cool."</span>,
        <span class="hljs-string">"What did the buffalo say when his son left for college? Bison."</span>
      ]
    }
  ]
}
</code></pre>
<h2><!--break-->Step3: Train the Model</h2>
<p>3.1- Create a NeuralNet model.py file which will be used to train the model</p>
<pre class="codeblock-container" idlang="2" lang="Python" tabsize="4"><code class="language-python hljs"><span class="hljs-comment">#model.py file</span>
<span class="hljs-keyword">import</span> torch.nn <span class="hljs-keyword">as</span> nn

<span class="hljs-class"><span class="hljs-keyword">class</span> <span class="hljs-title">NeuralNet</span><span class="hljs-params">(nn.Module)</span>:</span>
    <span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">__init__</span><span class="hljs-params">(self, input_size, hidden_size, num_classes)</span>:</span>
        super(NeuralNet, self).__init__()
        self.l1 = nn.Linear(input_size, hidden_size) 
        self.l2 = nn.Linear(hidden_size, hidden_size) 
        self.l3 = nn.Linear(hidden_size, num_classes)
        self.relu = nn.ReLU()
    
    <span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">forward</span><span class="hljs-params">(self, x)</span>:</span>
        out = self.l1(x)
        out = self.relu(out)
        out = self.l2(out)
        out = self.relu(out)
        out = self.l3(out)
        <span class="hljs-comment"># no activation and no softmax at the end</span>
        <span class="hljs-keyword">return</span> out
</code></pre>
<p>3.2 - We need nltk_utils.py file which uses "nltk" python library (a platform used for&nbsp;building Python programs that work with human language data for applying in statistical natural language processing<b> (NLP))</b>.<span>&nbsp;</span></p>
<pre class="codeblock-container" idlang="2" lang="Python" tabsize="4"><code class="language-python hljs"><span class="hljs-comment">#nltk_utils.py</span>
<span class="hljs-comment">#nltk library used for building Python programs that work with human language data for applying in </span>
<span class="hljs-comment">#statistical natural language processing (NLP)</span>
<span class="hljs-keyword">import</span> numpy <span class="hljs-keyword">as</span> np
<span class="hljs-keyword">import</span> nltk
<span class="hljs-keyword">from</span> nltk.stem.porter <span class="hljs-keyword">import</span> PorterStemmer
stemmer = PorterStemmer()

<span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">tokenize</span><span class="hljs-params">(sentence)</span>:</span>
    <span class="hljs-string">"""
    split sentence into array of words/tokens
    a token can be a word or punctuation character, or number
    """</span>
    <span class="hljs-keyword">return</span> nltk.word_tokenize(sentence)


<span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">stem</span><span class="hljs-params">(word)</span>:</span>
    <span class="hljs-string">"""
    stemming = find the root form of the word
    examples:
    words = ["organize", "organizes", "organizing"]
    words = [stem(w) for w in words]
    -&gt; ["organ", "organ", "organ"]
    """</span>
    <span class="hljs-keyword">return</span> stemmer.stem(word.lower())


<span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">bag_of_words</span><span class="hljs-params">(tokenized_sentence, words)</span>:</span>
    <span class="hljs-string">"""
    return bag of words array:
    1 for each known word that exists in the sentence, 0 otherwise
    example:
    sentence = ["hello", "how", "are", "you"]
    words = ["hi", "hello", "I", "you", "bye", "thank", "cool"]
    bog   = [  0 ,    1 ,    0 ,   1 ,    0 ,    0 ,      0]
    """</span>
    <span class="hljs-comment"># stem each word</span>
    sentence_words = [stem(word) <span class="hljs-keyword">for</span> word <span class="hljs-keyword">in</span> tokenized_sentence]
    <span class="hljs-comment"># initialize bag with 0 for each word</span>
    bag = np.zeros(len(words), dtype=np.float32)
    <span class="hljs-keyword">for</span> idx, w <span class="hljs-keyword">in</span> enumerate(words):
        <span class="hljs-keyword">if</span> w <span class="hljs-keyword">in</span> sentence_words: 
            bag[idx] = <span class="hljs-number">1</span>
    <span class="hljs-keyword">return</span> bag
</code></pre>
<p>3.3 - We need to train the model, We will use the train.py file to create our model</p>
<pre class="codeblock-container" idlang="2" lang="Python" tabsize="4"><code class="language-python hljs"><span class="hljs-comment">#train.py</span>
<span class="hljs-keyword">import</span> numpy <span class="hljs-keyword">as</span> np
<span class="hljs-keyword">import</span> json,torch
<span class="hljs-keyword">import</span> torch.nn <span class="hljs-keyword">as</span> nn
<span class="hljs-keyword">from</span> torch.utils.data <span class="hljs-keyword">import</span> Dataset, DataLoader
<span class="hljs-keyword">from</span> nltk_utils <span class="hljs-keyword">import</span> bag_of_words, tokenize, stem
<span class="hljs-keyword">from</span> model <span class="hljs-keyword">import</span> NeuralNet
 
<span class="hljs-keyword">with</span> open(<span class="hljs-string">'intents.json'</span>, <span class="hljs-string">'r'</span>) <span class="hljs-keyword">as</span> f:
    intents = json.load(f)

all_words = []
tags = []
xy = []
<span class="hljs-comment"># loop through each sentence in our intents patterns</span>
<span class="hljs-keyword">for</span> intent <span class="hljs-keyword">in</span> intents[<span class="hljs-string">'intents'</span>]:
    tag = intent[<span class="hljs-string">'tag'</span>]
    <span class="hljs-comment"># add to tag list</span>
    tags.append(tag)
    <span class="hljs-keyword">for</span> pattern <span class="hljs-keyword">in</span> intent[<span class="hljs-string">'patterns'</span>]:
        <span class="hljs-comment"># tokenize each word in the sentence</span>
        w = tokenize(pattern)
        <span class="hljs-comment"># add to our words list</span>
        all_words.extend(w)
        <span class="hljs-comment"># add to xy pair</span>
        xy.append((w, tag))

<span class="hljs-comment"># stem and lower each word</span>
ignore_words = [<span class="hljs-string">'?'</span>, <span class="hljs-string">'.'</span>, <span class="hljs-string">'!'</span>]
all_words = [stem(w) <span class="hljs-keyword">for</span> w <span class="hljs-keyword">in</span> all_words <span class="hljs-keyword">if</span> w <span class="hljs-keyword">not</span> <span class="hljs-keyword">in</span> ignore_words]
<span class="hljs-comment"># remove duplicates and sort</span>
all_words = sorted(set(all_words))
tags = sorted(set(tags))

print(len(xy), <span class="hljs-string">"patterns"</span>)
print(len(tags), <span class="hljs-string">"tags:"</span>, tags)
print(len(all_words), <span class="hljs-string">"unique stemmed words:"</span>, all_words)

<span class="hljs-comment"># create training data</span>
X_train = []
y_train = []
<span class="hljs-keyword">for</span> (pattern_sentence, tag) <span class="hljs-keyword">in</span> xy:
    <span class="hljs-comment"># X: bag of words for each pattern_sentence</span>
    bag = bag_of_words(pattern_sentence, all_words)
    X_train.append(bag)
    <span class="hljs-comment"># y: PyTorch CrossEntropyLoss needs only class labels, not one-hot</span>
    label = tags.index(tag)
    y_train.append(label)

X_train = np.array(X_train)
y_train = np.array(y_train)

<span class="hljs-comment"># Hyper-parameters </span>
num_epochs = <span class="hljs-number">1000</span>
batch_size = <span class="hljs-number">8</span>
learning_rate = <span class="hljs-number">0.001</span>
input_size = len(X_train[<span class="hljs-number">0</span>])
hidden_size = <span class="hljs-number">8</span>
output_size = len(tags)
print(input_size, output_size)

<span class="hljs-class"><span class="hljs-keyword">class</span> <span class="hljs-title">ChatDataset</span><span class="hljs-params">(Dataset)</span>:</span>
    <span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">__init__</span><span class="hljs-params">(self)</span>:</span>
        self.n_samples = len(X_train)
        self.x_data = X_train
        self.y_data = y_train

    <span class="hljs-comment"># support indexing such that dataset[i] can be used to get i-th sample</span>
    <span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">__getitem__</span><span class="hljs-params">(self, index)</span>:</span>
        <span class="hljs-keyword">return</span> self.x_data[index], self.y_data[index]

    <span class="hljs-comment"># we can call len(dataset) to return the size</span>
    <span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">__len__</span><span class="hljs-params">(self)</span>:</span>
        <span class="hljs-keyword">return</span> self.n_samples

dataset = ChatDataset()
train_loader = DataLoader(dataset=dataset,
                          batch_size=batch_size,
                          shuffle=<span class="hljs-keyword">True</span>,
                          num_workers=<span class="hljs-number">0</span>)

device = torch.device(<span class="hljs-string">'cuda'</span> <span class="hljs-keyword">if</span> torch.cuda.is_available() <span class="hljs-keyword">else</span> <span class="hljs-string">'cpu'</span>)
model = NeuralNet(input_size, hidden_size, output_size).to(device)
<span class="hljs-comment"># Loss and optimizer</span>
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
<span class="hljs-comment"># Train the model</span>
<span class="hljs-keyword">for</span> epoch <span class="hljs-keyword">in</span> range(num_epochs):
    <span class="hljs-keyword">for</span> (words, labels) <span class="hljs-keyword">in</span> train_loader:
        words = words.to(device)
        labels = labels.to(dtype=torch.long).to(device)        
        <span class="hljs-comment"># Forward pass</span>
        outputs = model(words)
        <span class="hljs-comment"># if y would be one-hot, we must apply</span>
        <span class="hljs-comment"># labels = torch.max(labels, 1)[1]</span>
        loss = criterion(outputs, labels)        
        <span class="hljs-comment"># Backward and optimize</span>
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()        
    <span class="hljs-keyword">if</span> (epoch+<span class="hljs-number">1</span>) % <span class="hljs-number">100</span> == <span class="hljs-number">0</span>:
        <span class="hljs-keyword">print</span> (<span class="hljs-string">f'Epoch [<span class="hljs-subst">{epoch+<span class="hljs-number">1</span>}</span>/<span class="hljs-subst">{num_epochs}</span>], Loss: <span class="hljs-subst">{loss.item():<span class="hljs-number">.4</span>f}</span>'</span>)

print(<span class="hljs-string">f'final loss: <span class="hljs-subst">{loss.item():<span class="hljs-number">.4</span>f}</span>'</span>)
data = {
<span class="hljs-string">"model_state"</span>: model.state_dict(),
<span class="hljs-string">"input_size"</span>: input_size,
<span class="hljs-string">"hidden_size"</span>: hidden_size,
<span class="hljs-string">"output_size"</span>: output_size,
<span class="hljs-string">"all_words"</span>: all_words,
<span class="hljs-string">"tags"</span>: tags
}
FILE = <span class="hljs-string">"chatdata.pth"</span>
torch.save(data, FILE)
print(<span class="hljs-string">f'training complete. file saved to <span class="hljs-subst">{FILE}</span>'</span>)
</code></pre>
<p>3.4 - Run the above train.py file to train the model</p>

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xp94s643krflkyim8uar.png)

<p>Train.py file&nbsp;will create&nbsp;chatdata.pth file which will be used in our chat.py file<br><!--break--></p>
<h2>Step4: Create ChatBot Application based on the trained model</h2>
<p>As Our model is created. In the final step, we will create a chat.py file which we can use in our chatbot.</p>
<pre class="codeblock-container" idlang="2" lang="Python" tabsize="4"><code class="language-python hljs"><span class="hljs-comment">#chat.py</span>
<span class="hljs-keyword">import</span> os,random,json,torch
<span class="hljs-keyword">from</span> model <span class="hljs-keyword">import</span> NeuralNet
<span class="hljs-keyword">from</span> nltk_utils <span class="hljs-keyword">import</span> bag_of_words, tokenize

device = torch.device(<span class="hljs-string">'cuda'</span> <span class="hljs-keyword">if</span> torch.cuda.is_available() <span class="hljs-keyword">else</span> <span class="hljs-string">'cpu'</span>)

<span class="hljs-keyword">with</span> open(<span class="hljs-string">"intents.json"</span>) <span class="hljs-keyword">as</span> json_data:
    intents = json.load(json_data)

data_dir = os.path.join(os.path.dirname(__file__))
FILE = os.path.join(data_dir, <span class="hljs-string">'chatdata.pth'</span>)
data = torch.load(FILE)

input_size = data[<span class="hljs-string">"input_size"</span>]
hidden_size = data[<span class="hljs-string">"hidden_size"</span>]
output_size = data[<span class="hljs-string">"output_size"</span>]
all_words = data[<span class="hljs-string">'all_words'</span>]
tags = data[<span class="hljs-string">'tags'</span>]
model_state = data[<span class="hljs-string">"model_state"</span>]

model = NeuralNet(input_size, hidden_size, output_size).to(device)
model.load_state_dict(model_state)
model.eval()

bot_name = <span class="hljs-string">"iris-NLP"</span>
<span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">get_response</span><span class="hljs-params">(msg)</span>:</span>
    sentence = tokenize(msg)
    X = bag_of_words(sentence, all_words)
    X = X.reshape(<span class="hljs-number">1</span>, X.shape[<span class="hljs-number">0</span>])
    X = torch.from_numpy(X).to(device)

    output = model(X)
    _, predicted = torch.max(output, dim=<span class="hljs-number">1</span>)

    tag = tags[predicted.item()]

    probs = torch.softmax(output, dim=<span class="hljs-number">1</span>)
    prob = probs[<span class="hljs-number">0</span>][predicted.item()]
    <span class="hljs-keyword">if</span> prob.item() &gt; <span class="hljs-number">0.75</span>:
        <span class="hljs-keyword">for</span> intent <span class="hljs-keyword">in</span> intents[<span class="hljs-string">'intents'</span>]:
            <span class="hljs-keyword">if</span> tag == intent[<span class="hljs-string">"tag"</span>]:
                <span class="hljs-keyword">return</span> random.choice(intent[<span class="hljs-string">'responses'</span>])
    <span class="hljs-keyword">return</span> <span class="hljs-string">"I do not understand..."</span>
<span class="hljs-keyword">if</span> __name__ == <span class="hljs-string">"__main__"</span>:
    print(<span class="hljs-string">"Let's chat! (type 'quit' to exit)"</span>)
    <span class="hljs-keyword">while</span> <span class="hljs-keyword">True</span>:
        sentence = input(<span class="hljs-string">"You: "</span>)
        <span class="hljs-keyword">if</span> sentence == <span class="hljs-string">"quit"</span>:
            <span class="hljs-keyword">break</span>
        resp = get_response(sentence)
        print(resp)

</code></pre>
<p>Run chat.py file and ask some questions which already defined above&nbsp; in our intents.json file</p>

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/b3cjcoj3kj1y7bviyr6t.png)

<p>For more details please visit <a href="https://openexchange.intersystems.com/package/IRIS-GenLab" rel="nofollow"><strong>IRIS-GenLab</strong></a>&nbsp;open exchange application page<br><br>Thanks</p>