LSTMs: How they work and how to make one
Published: 2023-08-29 22:36:29

## Introduction

A Long Short-Term Memory (LSTM) is an improved Recurrent Neural Network (RNN) with the goal being to improve long and short-term information retention. LSTMs are great for tasks that require contextual information or for tasks that have variable input or output sizes, like language translation. LSTMs are essential due to the fact that they are great at tasks like Natural Language Processing (NLP) and image processing which are major objectives of machine learning. By the end of this article, you will not be equipped with the knowledge of a college major in this field, but rather you will learn about the LSTM: how it functions and how to implement one in code.

LSTMs tend to be intimidating to look at mathematically, however, it is important to keep one thing in mind to alleviate that anxiety, the math is important to know **only** for understanding the functionality on a deep level. When coding LSTMs, tools like TensorFlow or Pytorch do the hard math for you. With that, let's get learning!

## A Mathematical View of LSTMs

![](https://www.researchgate.net/publication/338905482/figure/fig1/AS:852908019048459@1580360526797/The-general-schema-of-a-RNN-unit-versus-a-LSTM-one-adapted-from-Olah-2015.png)

The first thing that you will notice is that the LSTM looks like a more complicated version of an RNN, one with extra functions. But those functions are what make them so powerful.  They help filter out non-important information and they mitigate vanishing gradients. (Vanishing gradients occur when the the amount that the model is tuned becomes infinity small and does not improve the model.) 

### Crucial Functions in LSTMs

![](https://miro.medium.com/v2/resize:fit:1400/1*ahafyNt0Ph_J6Ed9_2hvdg.png)

The first function is the **Forget Gate**.

{% katex %} f_t = \sigma (W_{t} \cdot [h_{t-1},X_{t}]+b_{f}) {% endkatex %}
The forget gate is essentially a filter for numerical representations of words and their importance. For example, if we had an autocomplete model that used LSTMs, it would be trained to know that word **x** should go after word **y**.

---

Now that the "filter" has been applied, now we can uses an actual **Input Gate**.

{% katex %}
i_{t} = \sigma (W_{i} \cdot [h_{t-1},X_{t}]+b_i)
{% endkatex %}
The input gate is fairly standard and is seen in almost every kind of machine learning model. It is not always exactly the same but the typical scheme is an activation function like $\sigma$ that is applied to a vector consisting of previous inputs with a bias added inside the activation function.

---

The **Output Gate** also known as the **New Memory Network** is the function responsible for a part of the output. The functions defined thus far are the building blocks for the outputs which consist of two more function and the output gate is the penultimate block.

{% katex %}
\hat{C_{t}}= \tanh (W_{c} \cdot [h_{t-1},X_{t}]+b_{\hat{c}})
{% endkatex %}
---

The final pieces of the LSTM are the outputs! There are two outputs and one use used twice (that is why there are three out arrows in the diagrams). The first output is the **Hidden State** which is defined as $h_t$ and uses $o_t$ . This is important for the model so it has underlying connections about the data that are not necessary for users to know.

{% katex %}
o_{t}=\sigma (W_{o}\cdot [h_{t-1},X_t]+b_{o})
{% endkatex %}
{% katex %}
h_{t}=o_{t} \cdot \tanh (C_{t})
{% endkatex %}
The final output is the **Cell State**. This is the part that is useful the output of the model. The model will need to use the Cell State output from all the LSTM cells. 
{% katex %}
C_{t}=i_{t} \cdot \hat C_{t} + f_{t} \cdot C_{t-1}  
{% endkatex %}

---

## How to Code One

While the way that a machine learning model is coded will vary depending on its intended use case, I can still share a fairly simple example of how to make one. It is important to know that once you get into the realm of Natural Language Processing (NLP), you **should** know more about Embeddings, Tokenizing, Lemmatization and other important aspects of NLP before delving into it. While LSTMs shine with NLP, you are equipped with the knowledge to utilize an LSTM. For example, you can make a bot to play the game mankala. [Mancala](https://en.wikipedia.org/wiki/Mancala) is a great first example because you must make plays based on information from you and your opponent's board relative to your own, which is perfect for a LSTM. To start, install tensorflow, numpy, and pandas for python via pip/pip3:

```bash
pip3 install tensorflow numpy pandas scikit-learn
```

(I always recommend doing machine learning in a Jupyter Notebook for convenience sake.)

Firstly, import the dependencies that you just installed.
```python
import numpy as np
import pandas as pd
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
```

Now we have to make a few variables that will hold the training data.

```python
data = pd.read_csv("mancala_data.csv")

# Split the data into input (board states) and target (optimal moves) columns
X = data.drop(columns=["optimal_move"]).values  # Board states
y = data["optimal_move"].values  # Optimal moves


# Split the data into training and validation sets
# sklearn is not needed but it makes this step clean and easy
from sklearn.model_selection import train_test_split

X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)
```

Now it is time to make the actual model!

```python
def create_lstm_model(board_size, hidden_units):
    model = keras.Sequential([
        layers.Input(shape=(None, board_size)),
        layers.LSTM(hidden_units, return_sequences=True),
        layers.Dense(board_size, activation='softmax')
    ])
    return model

# Define parameters
board_size = len(X_train[0])  # Automatically determine the board size from the data
hidden_units = 64

# Create the LSTM model
lstm_model = create_lstm_model(board_size, hidden_units)

# Compile the model
lstm_model.compile(loss='sparse_categorical_crossentropy', optimizer='adam')
```

Finally, the model has to be trained or fitted to the data

```python
# Define training parameters
batch_size = 32
epochs = 50

# Train the model
history = lstm_model.fit(
    X_train, y_train,
    validation_data=(X_val, y_val),
    batch_size=batch_size,
    epochs=epochs
)
```

Congratulations! You just made a machine learning model to make predictions and actually use your model!

```python
lstm_model.predict()
```

The intent of this section was not to teach you the ins and outs of using tensorflow, but rather to get a feel for coding a simple model. To make that model, we had to have the data, load it into our code, make a model, and finally use the tools that tensorflow provides us with to train that model on the data.

## Conclusion

LSTMs are powerful models that can be the engine behind natural language processing, sequence base data analysis, and more complex problems like image labeling or speech recognition. While this article did not cover those complex implications of the model, it has primed you with the knowledge of how they work under the hood and equally importantly how to code one for individual use. You have to walk before you run and the analogy is no different in machine learning: you must learn about a model before diving head first into it. And this is your anchor point on your journey to mastering the LSTM.