Tokenization and Sequencing in TensorFlow
Tags: python, nlp, machinelearning
Published: 2021-04-20 13:50:16

In this blog post, we shall seek to learn how to implement tokenization and sequencing, important text pre-processing steps, in Tensorflow.

## Outline
- Introduction to Tokenizer
- Understanding Sequencing

## Introduction to Tokenizer
Tokenization is the process of splitting the text into smaller units such as sentences, words or subwords. In this section, we shall see how we can pre-process the text corpus by tokenizing text into words in TensorFlow. We shall use the Keras API with TensorFlow backend; The code snippet below shows the necessary imports.

``` python
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras.preprocessing.text import Tokenizer
```
And voila!🎉 we have all modules imported! Let’s initialize a list of sentences that we shall tokenize.
```python
sentences = [
'Life is so beautiful',
'Hope keeps us going',
'Let us celebrate life!'
]
```
The next step is to instantiate the Tokenizer and call the `fit_on_texts` method.
```python
tokenizer = Tokenizer()
tokenizer.fit_on_texts(sentences)
```
Well, when the text corpus is very large, we can specify an additional `num_words` argument to get the most frequent words. For example, if we’d like to get the 100 most frequent words in the corpus, then `tokenizer = Tokenizer(num_words=100)` does just that! 

To know how these tokens have been created and the indices assigned to words, we can use the `word_index` attribute.
```python
word_index = tokenizer.word_index
print(word_index)
 # Here’s the output:

{‘life’: 1, ‘us’: 2, ‘is’: 3, ‘so’: 4, ‘beautiful’: 5, ‘hope’: 6, ‘keeps’: 7, ‘going’: 8, ‘let’: 9, ‘celebrate’: 10}
```

Well, so far so good! But what happens when the test data contains words that we’ve not accounted for in the vocabulary?🤔
``` python
test_data = [
'Our life is to celebrate',
'Hoping for the best!',
'Let peace prevail everywhere'
]
```
We have introduced sentences in `test_data` which contain words that are not in our earlier vocabulary.

How do we account for such words which are not in vocabulary? We can define an argument `oov_token` to account for such Out Of Vocabulary (OOV) tokens.

```python
tokenizer = Tokenizer(oov_token=”<OOV>”)
The word_index now returns the following output:

{‘<OOV>’: 1, ‘life’: 2, ‘us’: 3, ‘is’: 4, ‘so’: 5, ‘beautiful’: 6, ‘hope’: 7, ‘keeps’: 8, ‘going’: 9, ‘let’: 10, ‘celebrate’: 11}
```

***

## Understanding Sequencing
In this section, we shall build on the tokenized text, using these generated tokens to convert the text into a sequence.

We can get a sequence by calling the `texts_to_sequences` method.

```python
sequences = tokenizer.texts_to_sequences(sentences)
#Here’s the output:

[[2, 4, 5, 6], [7, 8, 3, 9], [10, 3, 11, 2]]
```
Let’s now take a step back. What happens when the sentences are of different lengths?Then, we will have to convert all of them to the same length.

We shall import `pad_sequences` function to pad our sequences and look at the padded sequences.
```python
from tensorflow.keras.preprocessing.sequence import pad_sequences
padded = pad_sequences(sequences)
print("\nPadded Sequences:")
print(padded)
# Output
Padded Sequences:
 [[ 2  4  5  6]  
  [ 7  8  3  9] 
  [10  3 11  2]]
```
By default, the `length of the padded sequence = length of the longest sentence`. However, we can limit the maximum length by explicitly setting the `maxlen` argument.
```python
padded = pad_sequences(sequences,maxlen=5)
print("\nPadded Sequences:")
print(padded)
# Output
Padded Sequences: 
[[ 0  2  4  5  6] 
 [ 0  7  8  3  9] 
 [ 0 10  3 11  2]]
```
Now, let’s pad our test sequences after converting them to sequences.
```python
test_seq = tokenizer.texts_to_sequences(test_data)
print("\nTest Sequence = ", test_seq)
padded = pad_sequences(test_seq, maxlen=10)
print("\nPadded Test Sequence: ")
print(padded)
And here’s our output.

# Output 
Test Sequence =  [[1, 2, 4, 1, 11], [1, 1, 1, 1], [10, 1, 1, 1]]  
Padded Test Sequence:  
[[ 0  0  0  0  0  1  2  4  1 11] 
 [ 0  0  0  0  0  0  1  1  1  1] 
 [ 0  0  0  0  0  0 10  1  1  1]]
```
We see that all the padded sequences are of length `maxlen` and are padded with 0s at the beginning. What if we would like to add trailing zeros instead of at the beginning? We only need to specify `padding='post'`
```python
padded = pad_sequences(test_seq, maxlen=10, padding='post')
print("\nPadded Test Sequence: ")
print(padded)
# Output
Padded Test Sequence: 
 [[ 1  2  4  1 11  0  0  0  0  0] 
 [ 1  1  1  1  0  0  0  0  0  0]  
 [10  1  1  1  0  0  0  0  0  0]]
```
So far, none of the sentences have length exceeding `maxlen`, but in practice, we may have sentences that are much longer than `maxlen`. In that case, we have to truncate the sentences and can set the argument `truncating='post'` or `'pre'` to drop the first few or the last few words that exceed the specified `maxlen`.

Happy learning and coding!🎈✨🎉👩🏽‍💻

## Reference
[Natural Language Processing in TensorFlow on Coursera]
(https://www.coursera.org/learn/natural-language-processing-tensorflow)

Cover Image: Photo by <a href="https://unsplash.com/@syinq?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Susan Q Yin</a> on <a href="https://unsplash.com/s/photos/books?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Unsplash</a>