How to fine tune BERT for real time sentiment analysis
Published: 2023-03-11 19:30:00

---
title: How to fine tune BERT for real time sentiment analysis
published: true
description: How to fine tune BERT for real time sentiment analysis. Explaining how to find, clean, train, test and validate a machine learning model
tags: 
cover_image: https://blog-files-prod.s3.us-east-2.amazonaws.com/public/how-to-fine-tune-bert-for-real-time-sentiment-analysis/how-to-fine-tune-bert-for-real-time-sentiment-analysis.jpg
# Use a ratio of 100:42 for best results.
published_at: 2023-03-11 19:30 +0000
---

Post Series

1. How to fine tune BERT for real time sentiment analysis(this post)
2. [How to run BERT on AWS](https://cryptoclamor.com/blog/how-to-run-bert-on-aws)

Table of Contents

1. [Introduction](#introduction)
2. [BERT Summary](#bert-summary)
3. [Overview of Hugging Face](#overview-of-hugging-face)
4. [Setting up the remote environment](#setting-up-the-remote-environment)
5. [Introduction to Jupyter Notebook](#introduction-to-jupyter-notebook)
6. [Training BERT for social media sentiment analysis](#training-bert-for-social-media-sentiment-analysis)
    1. [Finding and cleaning data for fine-tuning BERT](#finding-and-cleaning-data-for-fine-tuning-bert)
    2. [Train, validation and testing method of BERT fine-tuning](#train-validation-and-testing-method-of-bert-fine-tuning)
        1. [Training of train, validation and testing method](#training-of-train-validation-and-testing-method)
        2. [Validation of train, validation and testing method](#validation-of-train-validation-and-testing-method)
        3. [Testing of train, validation and testing method](#testing-of-train-validation-and-testing-method)
    3. [Saving the model](#saving-the-model)
7. [Conclusion](#conclusion)

## Introduction

In this blog post series (this is part one), I’ll be covering how to fine tune a pre-trained BERT model. And to give you enough information to build your own data mining cluster with BERT. To begin with, let’s briefly summarize BERT.

## BERT Summary

BERT (**B**idirectional **E**ncoder **R**epresentations from **T**ransformers) is an open source natural language processing (NLP) model developed by Google in 2018. It's one of the first NLP models to use Transformers. The reason why this is important is because of it's newly developed capability to bi-directionally process the words in a sentence to more fully capture the context that humans understanding languages like English take for granted and has allowed BERT to reach previously unprecedented levels of accuracy for NLP tasks like sentiment analysis. The BERT model has made it's way into Hugging Face, a company maintaining a library of freely available, open source machine learning models.

## Overview of Hugging Face

[Hugging Face](https://huggingface.co/) exposes machine learning models with Python APIs for users to train and use for production. They offer many different base models as well fine-tuned models for more specific tasks. as In this blog post, we're going to be focusing on Hugging Face's [bert-base-uncased](https://huggingface.co/bert-base-uncased) version of the BERT NLP model ("uncased" means the model makes no distinction between words like "Car" and "car"). The value in using pre-trained models like from Hugging Face is to allow users to "stand on the shoulders of giants", and focus on satisfying their needs that these open source, freely available models open doors for meeting.

## Setting up the remote environment

So, we now know what model we are going to fine-tune with. What's next? We need to look at setting up a remote environment. The reason why is because fine-tuning a model often takes hours (if not days) and you don't want to tie up your own machine, even if it has the specs to make fine-tuning feasible. One of the easier options we found for setting up a remote environment is [Lambda Labs](https://lambdalabs.com/). At least currently, we've found that Lambda Labs offers cheaper GPUs than on AWS for model fine-tuning - but for everything else, we use AWS. We'll be explaining our architecture in an upcoming blog post!

The process to setup an account is pretty simple: you'll just need a payment method to complete signup. Afterwards, you'll be taken to the dashboard and from there you can select which GPU-based instances you want to spin up. We used the `gpu.2x.a6000` GPU instance type. Once you have selected an instance to spin up, and once it finishes booting, you'll have the option to open up the "cloud IDE" for the instance. This will spin up a Jupyter Notebook to ultimately run Python code on the GPU instance for fine-tuning our base pre-trained BERT model.

## Introduction to Jupyter Notebook

[Jupyter Notebook](https://jupyter.org/) is a widely used tool for use in data science and machine learning applications. [The Jupyer docs](https://docs.jupyter.org/en/latest/) provide a ton of helpful resources for getting up to speed with Jupyter, but in this section we'll go over the basic concepts to help you get oriented when working with Jupyter Notebooks.

[From a very high level architectural standpoint](https://docs.jupyter.org/en/latest/projects/architecture/content-architecture.html) Jupyter Notebook is an interactive GUI to execute Python code underneath with [IPython](https://github.com/ipython/ipython). While there a ton of features that are worth checking out, we'll focus on the "building blocks" of any notebook: cells.

Notebook Cells allow you to put separate blocks of Python code independently runnable from other cells (but can also be run "all together" sequentially). The current Notebook will persist the results of running a particular cell, which is especially useful when experimenting data cleanup transformation functions - you can keep running slight modifications of a particular cell, and it's latest output will be the output that is saved and can be used by the next cell (if that cell's code block uses a particular variable created by a previous cell). To clarify, every notebook cell is part of the same IPython environment, that's why re-running cells that change variable values can be seen in running subsequent cells - it's reassiging variables when cells that assign variables change the values that are assigned to those variables.

Knowing about Jupyter Notebook Cells, and how all the cells of a given notebook (saved as *.ipynb files) are executed within the same IPython environment (as well as knowing that the Jupyter Notebook interface is a GUI for the underlying IPython shell to execute code from notebook cells) is enough to start working with Jupyter Notebook for training a machine learning model.

## Training BERT for social media sentiment analysis

Once we have a GPU instance running on a Lambda Labs virual machine, and are able to navigate around the installed IDE for working with Jupyter Notebooks, now we can get to the most interesting part of the blog post: how to fine tune a pre-trained BERT model for analyzing sentiment from social media.

### Finding and cleaning data for fine-tuning BERT

Before we get to fine-tuning BERT, we need to figure out what kind of data we can use for fine-tuning. In this tutorial, we are focused on social media sentiment analysis, and we wanted to find a dataset that ideally had previously labeled social media posts for sentiment — both positive and negative sentiment, on some kind of binary classification, a “1” for positive sentiment, and a “0” for negative sentiment.

As luck would have it, such a dataset exists. The famous [Sentiment140 dataset](https://www.kaggle.com/datasets/kazanova/sentiment140) has been often used in Kaggle competitions, and it's also great for fine-tuning BERT, with a little bit of cleanup. The issue is that negative sentiment has a "0" score, which is great, but the positive sentiment has a score of "4". We need a simpler binary classification, so with a bit of cleanup from such a bash script:

```bash
#!/usr/bin/env bash

# "shuffles" the rows of the csv file
command shuf training.1600000.processed.noemoticon.csv -o shuffled_output.csv && \
# sleep 1 second just to guarantee the "shuffled_output.csv" has finished writing to disk
command sleep 1 && \
# find every "4" label value, and replace with "1"
command awk -F ',' -v OFS=',' '$1 == "\"4\"" { $1 = "\"1\"" }1' shuffled_output.csv > output.csv
```

We get two main things. One, we go ahead and "shuffle" the initial csv file just to make sure that there aren't any related / similar tweets next to each other in the Sentiment140 dataset. Granted, we are also going to be telling the BERT model to shuffle the dataset, but in this context, more shuffling doesn't hurt, since we are saving this `output.csv` file for use in fine-tuning BERT. The `output.csv` is now in a good shape to create a Tensorflow `dataset` from like so:

```python
import tensorflow as tf
from transformers import BertTokenizer, TFBertForSequenceClassification, InputFeatures

CSV_PATH = './output.csv'

NUM_EPOCHS = 10
# Reducing the size of the dataset to keep GPU hosting costs down as well as allowing this whole dataset to be held in this python process memory for simpler, initial fine-tuning
DATASET_SIZE = 2500
BATCH_SIZE = 25
AUTOTUNE = tf.data.experimental.AUTOTUNE

dataset = tf.data.experimental.make_csv_dataset(
    CSV_PATH,
    batch_size=BATCH_SIZE,
    column_names=['score','timestamp', 'datestring', 'N/A', 'user', 'tweet'],
    label_name='score',
    select_columns=['score', 'tweet'],
    num_epochs=NUM_EPOCHS,
    header=False,
    shuffle_seed=0,
    shuffle=True,
    num_rows_for_inference=1600000,
    ignore_errors=True,).prefetch(AUTOTUNE)

train_dataset = dataset.take(DATASET_SIZE)
validation_dataset = dataset.skip(DATASET_SIZE).take(DATASET_SIZE)
```

This creates a Tensorflow Dataset from our csv data file, as well as our `train_dataset` and our `validation_dataset` which does not include any data from our `train_dataset`, which is the goal here.

We are not done yet with transforming our dataset. We need to take our now usable dataset (which still has just raw tweet strings) and tokenize all the raw tweets before passing to our model for fine-tuning. Next, we need to create a tokenizer, that is designed to work with BERT, called the [BertTokenizer](https://huggingface.co/docs/transformers/model_doc/bert#transformers.BertTokenizer). This is another class from the Hugging Face transformers package specifically for BERT models' input tokenizing. We can create the tokenizer like so:

```python
tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
```

At this point, we have our raw input training dataset, our raw input validation dataset, and a tokenizer instance. We have one final data cleaning step, which is to tokenize all the input of our datasets for both the training and validation datasets. No shame, inspired by this [post about sentiment analysis with BERT](https://towardsdatascience.com/sentiment-analysis-in-10-minutes-with-bert-and-hugging-face-294e8a04b671):

```python
def convert_examples_to_tf_dataset(batches, tokenizer, max_length=200):
    datasetArr = []
    for batch in batches:
      batchFeatures = [] # -> will hold InputFeatures to be converted later
      index = 0
      textArr = batch['text']
      labelArr = batch['label']
      for textItem in textArr:
        input_dict = tokenizer.encode_plus(
            textItem,
            add_special_tokens=True,
            max_length=max_length, # truncates if len(s) > max_length
            return_token_type_ids=True,
            return_attention_mask=True,
            pad_to_max_length=True, # pads to the right by default
            truncation=True
        )

        input_ids, token_type_ids, attention_mask = (input_dict["input_ids"],
            input_dict["token_type_ids"], input_dict['attention_mask'])

        batchFeatures.append(
            InputFeatures(
                  input_ids=input_ids, attention_mask=attention_mask, token_type_ids=token_type_ids, label=int(labelArr[index])
              )
        )
        index += 1
      datasetArr.append(batchFeatures)

    def gen():
        for batch in datasetArr:
          for f in batch:
            yield ({
                    "input_ids": f.input_ids,
                    "attention_mask": f.attention_mask,
                    "token_type_ids": f.token_type_ids,
                }, f.label)
                
    return tf.data.Dataset.from_generator(
        gen,
        ({"input_ids": tf.int32, "attention_mask": tf.int32, "token_type_ids": tf.int32}, tf.int64),
        (
            {
                "input_ids": tf.TensorShape([None]),
                "attention_mask": tf.TensorShape([None]),
                "token_type_ids": tf.TensorShape([None]),
            },
            tf.TensorShape(None),
        ),
    ).batch(BATCH_SIZE)

cleaned_batch = []
validation_cleaned_batch = []
for batch in train_dataset:
  rawTextArr = list(map( lambda tweet: str(tweet), batch[0]['tweet'].numpy()))
  rawLabel = list(map( lambda tweet: int(tweet), batch[1]))
  cleaned_batch.append({ 'text': rawTextArr, 'label': rawLabel })

for batch in validation_dataset:
  rawTextArr = list(map( lambda tweet: str(tweet), batch[0]['tweet'].numpy()))
  rawLabel = list(map( lambda tweet: int(tweet), batch[1]))
  validation_cleaned_batch.append({ 'text': rawTextArr, 'label': rawLabel })

# our cleaned datasets, ready to be consumed by pre-trained BERT model instance
convert_dataset = convert_examples_to_tf_dataset(cleaned_batch, tokenizer)
convert_validation_dataset = convert_examples_to_tf_dataset(validation_cleaned_batch, tokenizer)
```

Once we have the "cleaned" and tokenized dataset, we can move onto the actual setting up of the fine-tuning of BERT through what's called the "train, validation and testing method" of model fine-tuning.

### Train, validation and testing method of BERT fine-tuning

There are a variety of methods for fine-tuning machine learning models, but to keep both explaining (and defending!) how we fine-tuned BERT, we are going to go over what's called the [train, validation and test split](https://blog.roboflow.com/train-test-split/) method. Train validation, and test split involves splitting your initial dataset (here, our `cleaned_output.csv` dataset) into three independent datasets. For (big surprise) training, validation, and testing.

#### Training of train, validation and testing method

We've now cleaned our dataset and are ready to select which base model to fine tune from for our task which is sentiment analysis of social media. This is where Hugging Face really shines: we can pick the [TFBertForSequenceClassification](https://huggingface.co/docs/transformers/model_doc/bert#transformers.TFBertForSequenceClassification).

This transformers class is very cool because of how transformer models (like BERT) are tailored for a specific task. This tailoring is accomplished by adding another layer on top of a base model. This layer added on top is called the "classifier" which creates a special token (for sentiment analysis, a "sentiment token") that BERT tries to guess at it's value, between 0 and 1. The base task of BERT is actually sequence prediction, so a "sentiment classifier" basically asks BERT to predict the value of the "sentiment token". The value of this token is our desired prediction value.

The reason this needs to be mentioned is because the `TFBertForSequenceClassification` transformer class is the classifier that Hugging Face provides for us, which makes our fine tuning task all that much simpler. We don't have to create a classifier, and we only have to worry about creating good data for our train, validate and test datasets. and working with `TFBertForSequenceClassification`'s API. Specifically, the [compile](https://www.tensorflow.org/api_docs/python/tf/keras/Model#compile) and [fit](https://www.tensorflow.org/api_docs/python/tf/keras/Model#fit) methods that are part of it's parent class, the Keras Model class.

##### Keras model compile method

The `compile` method first. Before we can call compile, we need to create the model instance with that method on it. For BERT sequence classification (for sentiment analysis) we can create a model instance like so:

```python
model = TFBertForSequenceClassification.from_pretrained("bert-base-uncased")
```

This creates a Keras model instance using the pretrained "bert-base-uncased" model with a classifier layer on top. Now, we can look at what's involved for compiling the model:

```python
model.compile(
    optimizer=tf.keras.optimizers.Adam(learning_rate=3e-5, epsilon=1e-08, clipnorm=1.0), 
    loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True), 
    metrics=[tf.keras.metrics.SparseCategoricalAccuracy('accuracy')]
)
```

There's a lot to unpack here, so we're going to go over each argument that we need to compile a model for BERT sequence classification (our sentiment analysis task).

First, the `optimizer` argument. Takes a `tf.keras.optimizers` type. A keras optimizer is an optimization algorithm used for minimizing the loss of a predictive model with regard to a training dataset (see the [difference between back-propagation and optimization](https://machinelearningmastery.com/difference-between-backpropagation-and-stochastic-gradient-descent/)). The particular optimization algorithm we are using is called the [Adam algorithm](https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/Adam). The Adam algorithm is pretty common and works well here for fine-tuning BERT. The arguments we in turn passed to the Adam optimizer are the `learning_rate`, the `epsilon`, and the `clipnorm`.

###### Adam optimizer learning_rate argument

The `learning_rate` (also known as the "step size"), used to determine the degree of which to "change" a model with respect to the gradient. [A gradient simply measures the change in all weights with regard to the change in error](https://builtin.com/data-science/gradient-descent)

###### Adam optimizer epsilon argument

The `epsilon` prevents the "divide by zero" error when the gradient is close to zero. Awesome Stack Overflow [answer on what is the epsilon for the Adam Optimizer](https://stackoverflow.com/questions/43221065/how-does-the-epsilon-hyperparameter-affect-tf-train-adamoptimizer)

###### Adam optimizer clipnorm argument

The `clipnorm` "clips the gradient" to prevent the [exploding gradient problem](https://cnvrg.io/gradient-clipping/).

Now, the `model.compile()` `loss` argument. The loss function to use. The purpose of loss functions is to compute the quantity that a model should seek to minimize during training (taken from Keras [Losses](https://keras.io/api/losses/) docs). We are using the [SparseCategoricalCrossentropy](https://keras.io/api/losses/probabilistic_losses/#sparsecategoricalcrossentropy-class) class, which "computes the crossentropy loss between the labels and predictions" (see previous link). We need a function to in order to measure the performance between the predicted probability (from 0, "negative" sentiment to "1" for positive sentiment), and we are using the `SparseCategoricalCrossentropy` Keras class to provide the loss function. More reading on [what cross-entropy is](https://ml-cheatsheet.readthedocs.io/en/latest/loss_functions.html).

Finally, the `metrics` argument. A metric is a [function that is used to judge the performance of your model](https://keras.io/api/metrics/). We are using the `SparseCategoricalAccuracy` class. The `"accuracy"` value we pass to the `SparseCategoricalAccuracy` class tells the model to use the base metric `Accuracy` class from the keras metrics package, which, to quote it's docstring, [Calculates how often predictions equal labels](https://github.com/keras-team/keras/blob/07e13740fd181fc3ddec7d9a594d8a08666645f6/keras/metrics/metrics.py#L132). More readable documentation on [the Accuracy class](https://keras.io/api/metrics/accuracy_metrics/).

There are a few more arguments optionally accepted by `compile` with explanation provied by the [compile method docs](https://www.tensorflow.org/api_docs/python/tf/keras/Model#compile).

##### Keras model fit method

Now onto the final method we need for fine-tuning the BERT pre-trained model, the `fit` method, that actually peforms the work of fine-tuning the model:

```python
history = model.fit(convert_dataset, epochs=NUM_EPOCHS, validation_data=convert_test_dataset)
```

The `fit` method takes at least three arguments. First positional argument is the training data argument, for which we are passing the `convert_dataset` that we converted in an earlier step to fit into the Dataset type expected by `model.fit()`. The second argument (that we've named here) is called `epochs`. `Epochs` is the number of times the model makes a complete pass over of all the training data that is used to fine-tune the model in the training step. We're passing `NUM_EPOCHS` set to `10`, so we are telling the model to iterate over the entire training dataset ten times, using the `validation_data` (the third argument we are passing to `model.fit()`, the `convert_test_dataset`) after each pass of the training data to "correct" the model using the error produced by the validation step occurring right after the training step.

We want to create a `history` variable set to the result of `model.fit()` so we can perform analyses of how training impacted the model such as with plots with [matplotlib](https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.html):

```python
import matplotlib.pyplot as plt

# ...

# summarize history for accuracy
plt.figure()
plt.plot(history.history['accuracy'])
plt.plot(history.history['val_accuracy'])
plt.title('model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')
plt.savefig('training_results_accuracy.pdf')
# summarize history for loss
plt.figure()
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')
plt.savefig('training_results_loss.pdf')

plt.close()
```

#### Validation of train, validation and testing method

Tensorflow keras model API for `fit` handles accepting the validation dataset for us. So for the sake of this blog post we won't need to go into further detail about this step. We just need to make sure that the validation dataset (and later, the test dataset as well as future data once the model is determined to be viable for production) is structured in the exact same format as the training data, and very importantly, does not contain any input found in the training data.

#### Testing of train, validation and testing method

Once we've fine-tuned the BERT model using both the training data and the validation data, the final step is to perform end stage testing using previously unused data (data not used by neither the training dataset nor the validation dataset) to check for overfitting. Overfitting occurs when a model has been overly trained on the training and validation data, and thus fails to maintain a level of accuracy of predictions from new input it hasn't seen before.

What we did for final testing, is taking a very very small number of tweets to see the actual model output afterwards. This final test set has no direct impact on training (which uses the validation set to "correct" itself) but rather to evaluate this model's output post training makes sense). Basically, a "sanity check" to make sure the output makes sense and the accuracy is acceptable before deploying the model into a production environment.

### Saving the model

We need to save the model into a format we can download from Lamda Labs (or whatever GPU remote hosting you are using). Something like this works for creating a tar archive and compressing it:

```python
def make_tarfile(output_filename, source_dir):
    with tarfile.open(output_filename, "w:gz") as tar:
        tar.add(source_dir, arcname=os.path.basename(source_dir))

# Need to "save" model output configuration
model.save_pretrained('fine_tuned_model')
tokenizer.save_pretrained('fine_tuned_model_tokenizer')

# tar and compress "saved" model
make_tarfile('fine_tuned_model.tar.gz', 'fine_tuned_model')
make_tarfile('fine_tuned_model_tokenizer.tar.gz', 'fine_tuned_model_tokenizer')
```

## Conclusion

Hopefully this blog post has demystified how to fine tune BERT for sentiment analysis. This how to blog post describes the exact same process we used for creating our real time social media mining with fine-tuned BERT, that powers both our homepage's charts as well as the data provided by our REST-based API.

Our next blog post (in the works!) will explain how we setup our data mining cluster, using the weights created from this fine-tuning approach to cost-effectively run a data mining cluster running 24/7, collecting relevant raw social media posts once a minute.

Like this post? Share on social media and connect with me on [Twitter](https://twitter.com/stephenc1webdev)! And check back for updates as we go into further detail and our next blog post in this series: how to deploy a BERT fine-tuned model into production.

Congratulations, you made it to the end! Here's all the code in one snippet:

Put this in a bash script (after downloading the [Sentiment140 dataset](https://www.kaggle.com/datasets/kazanova/sentiment140)):

```bash
#!/usr/bin/env bash

# "shuffles" the rows of the csv file
command shuf training.1600000.processed.noemoticon.csv -o shuffled_output.csv && \
# sleep 1 second just to guarantee the "shuffled_output.csv" has finished writing to disk
command sleep 1 && \
# find every "4" label value, and replace with "1"
command awk -F ',' -v OFS=',' '$1 == "\"4\"" { $1 = "\"1\"" }1' shuffled_output.csv > output.csv
```

Upload `output.csv` to a GPU instance (at least as big as a `gpu.2x.a6000`) on Lambda Labs (or another GPU remote hosting service) and copy and paste this in a Jupyter Notebook:

```python
import matplotlib.pyplot as plt
import tensorflow as tf
from transformers import BertTokenizer, TFBertForSequenceClassification, InputFeatures

CSV_PATH = './output.csv'
NUM_EPOCHS = 10
# Reducing the size of the dataset to keep GPU hosting costs down as well as allowing this whole dataset to be held in this python process memory for simpler, initial fine-tuning
DATASET_SIZE = 2500
BATCH_SIZE = 25
AUTOTUNE = tf.data.experimental.AUTOTUNE

tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")

dataset = tf.data.experimental.make_csv_dataset(
    CSV_PATH,
    batch_size=BATCH_SIZE,
    column_names=['score','timestamp', 'datestring', 'N/A', 'user', 'tweet'],
    label_name='score',
    select_columns=['score', 'tweet'],
    num_epochs=NUM_EPOCHS,
    header=False,
    shuffle_seed=0,
    shuffle=True,
    num_rows_for_inference=1600000,
    ignore_errors=True,).prefetch(AUTOTUNE)

train_dataset = dataset.take(DATASET_SIZE)
validation_dataset = dataset.skip(DATASET_SIZE).take(DATASET_SIZE)

def convert_examples_to_tf_dataset(batches, tokenizer, max_length=200):
    datasetArr = []
    for batch in batches:
      batchFeatures = [] # -> will hold InputFeatures to be converted later
      index = 0
      textArr = batch['text']
      labelArr = batch['label']
      for textItem in textArr:
        input_dict = tokenizer.encode_plus(
            textItem,
            add_special_tokens=True,
            max_length=max_length, # truncates if len(s) > max_length
            return_token_type_ids=True,
            return_attention_mask=True,
            pad_to_max_length=True, # pads to the right by default
            truncation=True
        )

        input_ids, token_type_ids, attention_mask = (input_dict["input_ids"],
            input_dict["token_type_ids"], input_dict['attention_mask'])

        batchFeatures.append(
            InputFeatures(
                  input_ids=input_ids, attention_mask=attention_mask, token_type_ids=token_type_ids, label=int(labelArr[index])
              )
        )
        index += 1
      datasetArr.append(batchFeatures)

    def gen():
        for batch in datasetArr:
          for f in batch:
            yield ({
                    "input_ids": f.input_ids,
                    "attention_mask": f.attention_mask,
                    "token_type_ids": f.token_type_ids,
                }, f.label)
                
    return tf.data.Dataset.from_generator(
        gen,
        ({"input_ids": tf.int32, "attention_mask": tf.int32, "token_type_ids": tf.int32}, tf.int64),
        (
            {
                "input_ids": tf.TensorShape([None]),
                "attention_mask": tf.TensorShape([None]),
                "token_type_ids": tf.TensorShape([None]),
            },
            tf.TensorShape(None),
        ),
    ).batch(BATCH_SIZE)

cleaned_batch = []
validation_cleaned_batch = []
for batch in train_dataset:
  rawTextArr = list(map( lambda tweet: str(tweet), batch[0]['tweet'].numpy()))
  rawLabel = list(map( lambda tweet: int(tweet), batch[1]))
  cleaned_batch.append({ 'text': rawTextArr, 'label': rawLabel })

for batch in validation_dataset:
  rawTextArr = list(map( lambda tweet: str(tweet), batch[0]['tweet'].numpy()))
  rawLabel = list(map( lambda tweet: int(tweet), batch[1]))
  validation_cleaned_batch.append({ 'text': rawTextArr, 'label': rawLabel })

# our cleaned datasets, ready to be consumed by pre-trained BERT model instance
convert_dataset = convert_examples_to_tf_dataset(cleaned_batch, tokenizer)
convert_validation_dataset = convert_examples_to_tf_dataset(validation_cleaned_batch, tokenizer)


model.compile(
    optimizer=tf.keras.optimizers.Adam(learning_rate=3e-5, epsilon=1e-08, clipnorm=1.0), 
    loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True), 
    metrics=[tf.keras.metrics.SparseCategoricalAccuracy('accuracy')]
)

history = model.fit(convert_dataset, epochs=NUM_EPOCHS, validation_data=convert_test_dataset)

# summarize history for accuracy
plt.figure()
plt.plot(history.history['accuracy'])
plt.plot(history.history['val_accuracy'])
plt.title('model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')
plt.savefig('training_results_accuracy.pdf')
# summarize history for loss
plt.figure()
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')
plt.savefig('training_results_loss.pdf')

plt.close()

def make_tarfile(output_filename, source_dir):
    with tarfile.open(output_filename, "w:gz") as tar:
        tar.add(source_dir, arcname=os.path.basename(source_dir))

# Need to "save" model output configuration
model.save_pretrained('fine_tuned_model')
tokenizer.save_pretrained('fine_tuned_model_tokenizer')

# tar and compress "saved" model
make_tarfile('fine_tuned_model.tar.gz', 'fine_tuned_model')
make_tarfile('fine_tuned_model_tokenizer.tar.gz', 'fine_tuned_model_tokenizer')
```