Guide to Custom Recurrent Modeling in Keras
Tags: machinelearning, deeplearning, datascience, python
Published: 2021-01-07 19:03:28

### Introduction
The initial set of layers for recurrent neural operations universally begins with LSTM, GRU and RNN. But with an increase in the complexity of the task, we should use more complex models. That said, before moving directly to different and relatively complex models like attention or transformers, we should first ask a simple question - can we still do something quickly with the basic recurrent layers? In this article, I will focus on the same philosophy that - we should first exhaust the simple solutions before going for the more complex ones. In the next sections, we will explore the same old set of recurrent layers (albeit with some interesting arrangements) to get better inference on the data. In total, we will discuss 12 such arrangements (including the original set). Finally, we will conclude with testing all of our models on a Text Sentiment Detection task to compare their performance. Let's get started!

### Prerequisite
Before going further, let's discuss a few topics which we should know to completely appreciate this article. One more thing, I use arrangement and model interchangeably, because all the discussed models are nothing but compositions of basic recurrent layers, which acts as the smallest building blocks.

### Recurrent layer output types
Any recurrent layer has the option to return two types of outputs - (1) Last state output (1 output) and (2) All state output (N outputs). Actually, the 2nd type is more generic as it also contains the last state output (i.e. the 1st type), but usually, popular deep learning library's implementation provides options to return both the types of outputs. In case of Keras, the default is the 1st type and you can set the parameter `return_sequence=True` to shift to type 2. Note here, by "state" I mean hidden state of the recurrent layer and the N number of output is usually the number of time steps in your dataset. To know more about the internals of the recurrent layers and about the hidden states, I would suggest this excellent article. Now coming back to the topic, what's interesting is that while the 2nd type contains more info (which is usually a good thing), it could also be overwhelming as we don't know what exactly to do with it. As most of the downstream application still needs one output, we end up consolidating all state's output to a single final output (possible techniques - using `AveragePooling` or `MaxPooling` ). And in a way, LSTM or GRU does the same, they use the previous state's output (and current input) to create the next state's output. Then the interesting question is, "What is better - trusting LSTM/GRU to consolidate the state outputs or applying our own logic?". We will try to answer this in later sections, for now, let's move forward.


![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1610043418945/Z8lS5w2OX.png)

### Stacking recurrent layers
A recurrent layer takes sequential input and processes them to return one or many outputs (state vectors). Now as the output (if we return all state's output) also follow the sense of sequence, they can be thought of as some transformed original input and can be passed on to another layer of LSTM/GRU to be further processed. This is called stacking and here the main intuition is that - just like vanilla Deep Neural Network where you add more dense layers or with CNNs where you add multiple convolution layers one after the another, in recurrent layering you can stack multiple layers one in top of another. Ideally more layers give you more tunable parameters hence more learning power. But be wary of adding too many stacks, as it may lead to overfitting (too complex network for too simple learning requirement). The main question we can ask here is "Does adding stacking layers results in better performance?".


![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1610043551417/ISXelLMxH.png)


### Bidirectional recurrent layers
One interesting arrangement is when you have two recurrent layers (they are not stacked), and in one layer data is passed left-to-right for training and this direction is reversed for the other layer. This is the bidirectional recurrent layer and the intuition is that - in contrast to a normal layer with only forward training and left context, having both left and right context while preparing the state vector output could lead to higher performance. Usually, this is true for tasks where you expect the complete data (bidirectional context) to be present, so this is valid for sentiment detection - as we have the complete sentence string available, but not true for time-series forecasting - as we don't have future date's data if we want to today's value (say temperature). As bidirectional systems use 2 recurrent layers, we compare it against the stacking architectures and ask, "Which is better - a 2 layer stacked recurrent layer or a bi-directional recurrent layer?".


![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1610043577764/y3t-8qTBM.png)


### Methodology
Moving forward, now let's try to define and group the different arrangements we can create using the models we knew (basic recurrent layers) and the techniques we learned above. At the higher layer, we group all of our arrangements into two broad classes -
1. **Single stack (SS)**: wherein we only use one recurrent layer
2. **Multiple stacks (MS)**: wherein we use multiple recurrent layers. To keep analysis simple, I will only use 2 layered arrangements.

Also, as RNNs have  [given way to "better"](https://stats.stackexchange.com/questions/222584/difference-between-feedback-rnn-and-lstm-gru#:~:text=We%20can%20say%20that,%20when,Inputs%20as%20per%20trained%20Weights.&text=So,%20LSTM%20gives%20us%20the,more%20Complexity%20and%20Operating%20Cost.)  recurrent layers, we will only consider LSTM and GRU in our analysis. Now in case of SS, we can use either LSTM or GRU, also we can add flavour by taking only 1 output or N outputs for each. This gives us `2x2=4` different arrangements we may want to consider. Moving forward, in case of MS (only 2 for this article), lets first start with 1 and N outputs. For each such setting, we can either use a 2 layer stacked or a bidirectional LSTM and GRU. This gives us `2x2x2=8` different arrangements. The grouping of all 12 arrangement is shown below. Note, by "Return Sequence False" I mean 1 output as you are only returning one output and not the complete sequence.


![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1610043682437/nwAnx2joj.png)


### Code
Now let us see how we can code up the different arrangements in Keras. For the sake of brevity, I will only show the code for the LSTM recurrent layer. Shifting to GRU is as easy as replacing LSTM with GRU in the code (don't forget to import GRU though). I will also only cover 4 different varieties as rest can be easily built after minor modifications. Getting started with SS_RSF_LSTM i.e. single stacked - return sequence false - LSTM layer 😉

**SS_RSF_LSTM**

```python
# import 
from tensorflow.keras import layers
from tensorflow import keras
# model
inputs = keras.Input(shape=(99, )) # input layer - shape should be defined by user.
embedding = layers.Embedding(num_words, 64)(inputs) # embedding layer
rl = layers.LSTM(128)(embedding) # our LSTM layer - default return sequence is False
dense = layers.Dense(64)(rl) # a dense layer on the output of LSTM
output = layers.Dense(1, activation='sigmoid')(dense) # final layer which gives classification result
self.model = keras.Model(inputs=inputs, outputs=output, name="SS_RSF_LSTM") # stitching everything together
```

Few things to note,
- The code has been created with the sentiment classification task in mind (next section). Hence the final output layer is of size=1 with a sigmoid activation to return a probability as output.
- We define an input where we say shape=99 i.e. the timestep size we expect in our data. Here this means the number of words in each sentence.
- We use the embedding layer to transform words into a vector representation. We could have used ready-made word embeddings like Glove or Word2Vec, but we initialized a new one which will train specifically for this task. The size of each vector is 64 and num_words is the vocabulary size of the dataset we are using.
- We use one LSTM layer with state output of size=128. Note, as per default return sequence is False, so we only get one output i.e. of the last state of the LSTM.
- We connect the last state output with a dense layer of size=64. This is used to enhance the complex thresholding on the output of LSTM.

**SS_RST_LSTM**

The only difference here is that we return all state's output from LSTM. We can do this as follows,
```python
inputs = keras.Input(shape=(99, ))
embedding = layers.Embedding(num_words, 64)(inputs)
rl = layers.LSTM(128, return_sequences=True)(embedding)
avg = tf.keras.layers.AveragePooling1D(pool_size=99)(rl)
dense = layers.Dense(64)(avg)
output = layers.Dense(1, activation='sigmoid')(dense)
self.model = keras.Model(inputs=inputs, outputs=output, name="SS_RST_LSTM")
```

Points to note,
- The first change is the addition of return_sequence=True in LSTM definition.
- Next, we need to consolidate our 99x128 matrix into 1x128 sized vector. While there are many ways to do so we will pick AveragePooling1D which simply takes the average of each 99 timestep vectors to return 1 vector output. Here pool_size is to specify that we are taking an average of each 99 timesteps.

**MS_RSF_biLSTM**

To have a bidirectional layer, all we need to do is add a Bidirectional function on top of LSTM.
```python
inputs = keras.Input(shape=(99, ))
embedding = layers.Embedding(num_words, 64)(inputs)
rl = layers.Bidirectional(layers.LSTM(128))(embedding)
dense = layers.Dense(64)(rl)
output = layers.Dense(1, activation='sigmoid')(dense)
self.model = keras.Model(inputs=inputs, outputs=output, name="MS_RSF_biLSTM")
```

Note to move to **MS_RST_biLSTM** you need to do 2 things - (1) add `return_sequence=True` inside the LSTM layer, and (2) add the `AveragePooling1D` logic after the bidirectional LSTM layer as done above.

**MS_RST_LSTM**

```python
inputs = keras.Input(shape=(99, ))
embedding = layers.Embedding(num_words, 64)(inputs)
rl = layers.LSTM(128, return_sequences=True)(embedding)
rl = layers.LSTM(128, return_sequences=True)(rl)
avg = tf.keras.layers.AveragePooling1D(pool_size=99)(rl)
dense = layers.Dense(64)(avg)
output = layers.Dense(1, activation='sigmoid')(dense)
self.model = keras.Model(inputs=inputs, outputs=output, name="MS_RST_LSTM")
```

Points to note,
- To stack multiple LSTMs, it is mandatory for all the lower LSTMs to have `return_sequence=True` , as they will be fed as input to the next LSTM.
- For the topmost LSTM, its user's choice. As we are seeing an RST example, the code is returning all of the state sequences which is later averaged.
- To stack more layers in this fashion, all we need to do is copy-paste the `rl = layers.LSTM(128, return_sequences=True)(rl)` line again and again. Here as we have 2 such lines, we have 2 layers stacked LSTM.

The remaining arrangements can be coded up pretty easily by following the hints shared above. I leave this as an exercise and also share the complete  [code here ](https://gist.github.com/imohitmayank/757a2d878a1510180f134a8c7f45d6dc) for reference.

### Experiment

**Dataset**

For testing our models, I have selected  [IMDB sentiment classification ](https://www.tensorflow.org/api_docs/python/tf/keras/datasets/imdb) datasets which contain 25,000 highly polar movie reviews with binary sentiment represented by labels 0 and 1. To be little easier on my laptop, I have further trimmed the data by considering only the top 5000 words by frequency and truncating sentence over 100 words. As the data API is exposed by Keras, all of this can be done by,


```
# import
import tensorflow as tf
# set parameters
num_words = 5000
maxlen = 100
# fetch data
(x_train, y_train), (x_test, y_test) = tf.keras.datasets.imdb.load_data(num_words=num_words, maxlen=maxlen)
``` 

Also, Keras has already transformed the words into integer representation, making it easier to be fed to our models. All that's remaining is to make sure all of the sentences are of the same size, as there could be some sentence with size less than maxlen. This can be done by padding the smaller sentences with a dummy number (say 0). You can add padding on the left or right of the short sentences, I have chosen left. The code to do so is,


```
## perform padding
x_train = tf.keras.preprocessing.sequence.pad_sequences(x_train, padding="pre")
x_test = tf.keras.preprocessing.sequence.pad_sequences(x_test, padding="pre")
``` 

**Results**

To compare the performance score across all models, we will train each model for a specified number of epochs on the prepared data and report accuracy. To handle variation in case of training, we will repeat the training process multiple times (runs) and also report the variation in accuracy score observed in each run along with max and min scores. Lastly, to make sure that the variation in our model performance is not introduced by random weight initialization, we will use the same initial weights for each run. Then the major difference introduced is in model fitting and its internal train-validation data split.
Each experiment (for one model) ran 5 times and each time we ran for 5 epochs. In total, we ran 12 such experiments (for all of our models). The consolidated performance report card is shown below,


![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1610044105151/ttrSC5uS1.png)

Behold the winner MS_RST_LSTM which is way better than simple SS_RSF_LSTM (the vanilla LSTM). Now, let us try to answer the questions we raised before,


![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1610044123717/ho-le2G0F.png)

- **Q1: Does different recurrent output types matter? **- Short answer: Yes! To understand why, let's compare each RSF model with its counterpart RST models. All such possible comparisons and the difference (right minus left) is shown above. As evident, 4/6 models perform better. In fact, our best model is MS_RST_LSTM, which has over ~3% improvement over its RSF version.
- **Q2: Does stacking multiple layers matters?** - Yes again! See the second comparison table with all positive difference. Adding more layers does results in additional tunable parameters and hence more learning power. But be vary, this doesn't mean you can stack 100 layers and expect it to work better than before - at some point, it will converge i.e. adding more layers will not help. But stacking few layers does help, as shown here.
- **Q3: Does bidirectional layering performs better against normal stacking? **- Interestingly no! 3/4 times stacking layers perform better. This may be because multiple stack layers have more training parameters, hence better inference on data.

### Conclusion
Let me start with a disclaimer - while I am (or rather the experiment results are) saying that one arrangement works better than another, I am not implying that you should only train "good" models and ignore the rest. Please remember, every task and its data is different and may result in different result from the ones reported above. Take Bi-direction as an example, usually, I find bi-direction competing toe-to-toe with stacking layers and in some cases even perform better. This is in contrast to what we saw before and that's going on to prove that every data is different. That said, the purpose of this article was two folds - (1) to showcase the different varieties of layering arrangements that can be made by just using basic recurrent layers, and (2) based on experiments to help prioritize arrangements rather than ignoring them. So next time you have a text classification project, you can start with MS_RST_LSTM model and with time (if you have) experiments on others.

Cheers!

----

For more articles visit [my website](https://mohitmayank.com) or follow me on [Linkedin](https://www.linkedin.com/in/imohitmayank/).