{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# SCC0633/SCC5908 - Processamento de Linguagem Natural\n",
        "> **Docente:** Thiago Alexandre Salgueiro Pardo \\\\\n",
        "> **Estagiário PAE:** Germano Antonio Zani Jorge\n",
        "\n",
        "\n",
        "# Integrantes do Grupo: GPTrouxas\n",
        "> André Guarnier De Mitri - 11395579 \\\\\n",
        "> Daniel Carvalho - 10685702 \\\\\n",
        "> Fernando - 11795342 \\\\\n",
        "> Lucas Henrique Sant'Anna - 10748521 \\\\\n",
        "> Magaly L Fujimoto - 4890582 \\\\\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Abordagem Neural usando BERT\n",
        "![alt text](../imagens/BERT_TDIDF.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "###"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6yecpJR0feeQ"
      },
      "source": [
        "## Importando bibliotecas"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "FAIvyZwodEtm"
      },
      "outputs": [],
      "source": [
        "import torch\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "import math\n",
        "from tqdm.notebook import tqdm\n",
        "import pandas as pd"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {},
      "outputs": [],
      "source": [
        "#!pip install transformers seaborn nltk"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Carregando dados"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "LYgXl3RIfgfo",
        "outputId": "eb496faf-7826-44f7-fa88-3b21fb6e7cbf"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>review</th>\n",
              "      <th>sentiment</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>One of the other reviewers has mentioned that ...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>A wonderful little production. &lt;br /&gt;&lt;br /&gt;The...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>I thought this was a wonderful way to spend ti...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Basically there's a family where a little boy ...</td>\n",
              "      <td>negative</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Petter Mattei's \"Love in the Time of Money\" is...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                              review sentiment\n",
              "0  One of the other reviewers has mentioned that ...  positive\n",
              "1  A wonderful little production. <br /><br />The...  positive\n",
              "2  I thought this was a wonderful way to spend ti...  positive\n",
              "3  Basically there's a family where a little boy ...  negative\n",
              "4  Petter Mattei's \"Love in the Time of Money\" is...  positive"
            ]
          },
          "execution_count": 3,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "df_reviews = pd.read_csv('imdb_reviews.csv')\n",
        "df_reviews.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Mapeando as classes\n",
        "- Sentimento positivo recebe label 1\n",
        "- Sentimento negativo recebe label 0"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "D-5n8XzJbWOO",
        "outputId": "cef630cc-b0cc-4598-c53f-d32636bfcd86"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>review</th>\n",
              "      <th>sentiment</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>One of the other reviewers has mentioned that ...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>A wonderful little production. &lt;br /&gt;&lt;br /&gt;The...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>I thought this was a wonderful way to spend ti...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Basically there's a family where a little boy ...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Petter Mattei's \"Love in the Time of Money\" is...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                              review  sentiment\n",
              "0  One of the other reviewers has mentioned that ...          1\n",
              "1  A wonderful little production. <br /><br />The...          1\n",
              "2  I thought this was a wonderful way to spend ti...          1\n",
              "3  Basically there's a family where a little boy ...          0\n",
              "4  Petter Mattei's \"Love in the Time of Money\" is...          1"
            ]
          },
          "execution_count": 4,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "def map_sentiments(sentiment):\n",
        "    if sentiment == 'positive':\n",
        "        return 1\n",
        "    return 0\n",
        "\n",
        "df_reviews['sentiment'] = df_reviews['sentiment'].apply(map_sentiments)\n",
        "df_reviews.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Funções para limpeza do texto\n",
        "**lowercase_text(text)** Converte o texto para letras minúsculas para uniformizar o texto.\n",
        "\n",
        "\n",
        "**remove_html(text)**  Remove quaisquer tags HTML do texto para limpar dados provenientes de fontes HTML.\n",
        "\n",
        "\n",
        " **remove_url(text)** Remove URLs do texto para eliminar links que podem não ser relevantes para a análise de texto.\n",
        "\n",
        "\n",
        "**remove_punctuations(text)** Remove pontuações do texto para simplificar a estrutura do texto, mantendo apenas palavras.\n",
        "\n",
        "**remove_emojis(text)** Remove emojis do texto para evitar caracteres não verbais que podem interferir na análise textual.\n",
        "\n",
        "**remove_stop_words(text)** Remove stop words (palavras comuns como \"e\", \"de\", \"o\") que geralmente não adicionam valor significativo à análise de texto.\n",
        "\n",
        "**stem_words(text)**  Aplica stemming nas palavras do texto, reduzindo-as à sua raiz (por exemplo, \"running\" vira \"run\") para normalizar as variações das palavras.\n",
        "\n",
        "**preprocess_text(text)**  Aplica todas as funções acima em sequência para pré-processar o texto de forma completa, tornando-o mais adequado para análise de texto ou modelagem.\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 241
        },
        "id": "PnFHO62rnWn-",
        "outputId": "17fb6619-fab9-4395-de5d-4c5199e7e45e"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "[nltk_data] Downloading package stopwords to\n",
            "[nltk_data]     C:\\Users\\andre\\AppData\\Roaming\\nltk_data...\n",
            "[nltk_data]   Package stopwords is already up-to-date!\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>review</th>\n",
              "      <th>sentiment</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>one review mention watch 1 oz episod hook righ...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>wonder littl product film techniqu unassum old...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>thought wonder way spend time hot summer weeke...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>basic famili littl boy jake think zombi closet...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>petter mattei love time money visual stun film...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                              review  sentiment\n",
              "0  one review mention watch 1 oz episod hook righ...          1\n",
              "1  wonder littl product film techniqu unassum old...          1\n",
              "2  thought wonder way spend time hot summer weeke...          1\n",
              "3  basic famili littl boy jake think zombi closet...          0\n",
              "4  petter mattei love time money visual stun film...          1"
            ]
          },
          "execution_count": 5,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "import re\n",
        "import nltk\n",
        "from nltk.corpus import stopwords\n",
        "from nltk.stem import PorterStemmer\n",
        "\n",
        "\n",
        "def lowercase_text(text):\n",
        "    return text.lower()\n",
        "\n",
        "def remove_html(text):\n",
        "    return re.sub(r'<[^<]+?>', '', text)\n",
        "\n",
        "def remove_url(text):\n",
        "    return re.sub(r'http[s]?://\\S+|www\\.\\S+', '', text)\n",
        "\n",
        "def remove_punctuations(text):\n",
        "    tokens_list = '!\"#$%&\\'()*+,-./:;<=>?@[\\\\]^_`{|}~'\n",
        "    for char in text:\n",
        "        if char in tokens_list:\n",
        "            text = text.replace(char, ' ')\n",
        "\n",
        "    return text\n",
        "\n",
        "def remove_emojis(text):\n",
        "    emojis = re.compile(\"[\"\n",
        "                        u\"\\U0001F600-\\U0001F64F\"\n",
        "                        u\"\\U0001F300-\\U0001F5FF\"\n",
        "                        u\"\\U0001F680-\\U0001F6FF\"\n",
        "                        u\"\\U0001F1E0-\\U0001F1FF\"\n",
        "                        u\"\\U00002500-\\U00002BEF\"\n",
        "                        u\"\\U00002702-\\U000027B0\"\n",
        "                        u\"\\U00002702-\\U000027B0\"\n",
        "                        u\"\\U000024C2-\\U0001F251\"\n",
        "                        u\"\\U0001f926-\\U0001f937\"\n",
        "                        u\"\\U00010000-\\U0010ffff\"\n",
        "                        u\"\\u2640-\\u2642\"\n",
        "                        u\"\\u2600-\\u2B55\"\n",
        "                        u\"\\u200d\"\n",
        "                        u\"\\u23cf\"\n",
        "                        u\"\\u23e9\"\n",
        "                        u\"\\u231a\"\n",
        "                        u\"\\ufe0f\"\n",
        "                        u\"\\u3030\"\n",
        "                        \"]+\", re.UNICODE)\n",
        "\n",
        "    text = re.sub(emojis, '', text)\n",
        "    return text\n",
        "\n",
        "def remove_stop_words(text):\n",
        "    stop_words = stopwords.words('english')\n",
        "    new_text = ''\n",
        "    for word in text.split():\n",
        "        if word not in stop_words:\n",
        "            new_text += ''.join(f'{word} ')\n",
        "\n",
        "    return new_text.strip()\n",
        "\n",
        "def stem_words(text):\n",
        "    stemmer = PorterStemmer()\n",
        "    new_text = ''\n",
        "    for word in text.split():\n",
        "        new_text += ''.join(f'{stemmer.stem(word)} ')\n",
        "\n",
        "    return new_text\n",
        "\n",
        "def preprocess_text(text):\n",
        "    text = lowercase_text(text)\n",
        "    text = remove_html(text)\n",
        "    text = remove_url(text)\n",
        "    text = remove_punctuations(text)\n",
        "    text = remove_emojis(text)\n",
        "    text = remove_stop_words(text)\n",
        "    text = stem_words(text)\n",
        "\n",
        "    return text\n",
        "\n",
        "nltk.download('stopwords')\n",
        "df_reviews['review'] = df_reviews['review'].apply(preprocess_text)\n",
        "df_reviews.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Visualizando balancemento da classes"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 452
        },
        "id": "Gdi_L0HWfntv",
        "outputId": "bce77594-f662-4b3f-c8eb-27d8a188b4f2"
      },
      "outputs": [
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjkAAAGzCAYAAADNKAZOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy80BEi2AAAACXBIWXMAAA9hAAAPYQGoP6dpAAA0bElEQVR4nO3deVxVdf7H8TegXHABNAVECRXLfSktJLdKfqKS5mS5NQ46pi3YpDSmpqNmM2NjizppOuUkzYyW1pTlMijhwqhYk8mUlpZ7peCWXFdQ+P7+mAenbuACscS31/PxuI+83/M553zOV+2+vfd7Ll7GGCMAAADLeFd0AwAAAGWBkAMAAKxEyAEAAFYi5AAAACsRcgAAgJUIOQAAwEqEHAAAYCVCDgAAsBIhBwAAWImQA6BCDRs2TA0bNqzoNhwHDhyQl5eXkpKSnLFp06bJy8urXM5/++236/bbb3eeb9iwQV5eXnrrrbfK5fw/td8P4Mcg5AClxMvL65oeGzZsqOhWPWzZskXTpk3TqVOnKroVqxw+fFjTpk1TRkZGRbdSyE+5N6A0VanoBgBb/P3vf/d4/re//U0pKSmFxps3b16ebV3Vli1b9NRTT2nYsGEKCgqq6HZ+kiZPnqwJEyYUa5/Dhw/rqaeeUsOGDdWuXbtr3m/t2rXF7K74rtTbK6+8ovz8/DLvASgPhByglPzyl7/0eL5161alpKQUGi8JY4wuXLggf3//H30sFF+VKlVUpUrZ/u/y3Llzqlatmnx9fcv0PFdTtWrVCj0/UJr4uAooR4sWLdKdd96p4OBguVwutWjRQvPnzy9U17BhQ911111as2aNOnToIH9/f/3lL3+RJB08eFB9+/ZV9erVFRwcrLFjx2rNmjVFfhT2wQcfqGfPngoMDFS1atXUrVs3bd682dk+bdo0jRs3TpLUqFEj5yO1AwcOFNn/6NGjVaNGDZ07d67QtsGDBys0NFR5eXmSpHfffVdxcXEKCwuTy+VSZGSknn76aWf75RSsQfnhtRS1VkaSdu3apXvvvVe1a9eWn5+fOnTooPfee++K5yhw6tQpDRs2TIGBgQoKClJ8fHyRH9sVtSYnJSVFnTt3VlBQkGrUqKGmTZvqySefdK7hlltukSQNHz7cmdeC3m+//Xa1atVK27ZtU9euXVWtWjVn3x+uySmQl5enJ598UqGhoapevbr69u2rr776yqOmYcOGGjZsWKF9v3/Mq/VW1Jqcs2fP6vHHH1d4eLhcLpeaNm2q5557TsYYjzovLy+NHj1ay5cvV6tWreRyudSyZUslJycX6gkoD7yTA5Sj+fPnq2XLlurbt6+qVKmiFStW6JFHHlF+fr4SEhI8anfv3q3BgwfrwQcf1MiRI9W0aVOdPXtWd955p44cOaLHHntMoaGhWrJkidavX1/oXOvWrVOvXr3Uvn17TZ06Vd7e3k7I+ve//61bb71V99xzj7744gu9/vrrmjVrlurUqSNJqlu3bpH9Dxw4UPPmzdOqVat03333OePnzp3TihUrNGzYMPn4+EiSkpKSVKNGDSUmJqpGjRpat26dpkyZIrfbrWeffbZU5nPnzp3q1KmT6tevrwkTJqh69epatmyZ+vXrp3/+85/6xS9+cdl9jTG6++67tWnTJj300ENq3ry53nnnHcXHx1/Tee+66y61adNG06dPl8vl0p49e5wA2bx5c02fPl1TpkzRqFGj1KVLF0nSbbfd5hzjxIkT6tWrlwYNGqRf/vKXCgkJueI5//CHP8jLy0vjx4/X0aNHNXv2bMXExCgjI6NY7/BdS2/fZ4xR3759tX79eo0YMULt2rXTmjVrNG7cOH3zzTeaNWuWR/2mTZv09ttv65FHHlHNmjX15z//Wf3799ehQ4d03XXXXXOfQKkwAMpEQkKC+eFfsXPnzhWqi42NNY0bN/YYi4iIMJJMcnKyx/jzzz9vJJnly5c7Y+fPnzfNmjUzksz69euNMcbk5+ebG264wcTGxpr8/HyP8zdq1Mj83//9nzP27LPPGklm//79V72m/Px8U79+fdO/f3+P8WXLlhlJJi0t7YrX+uCDD5pq1aqZCxcuOGPx8fEmIiLCeb5+/XqPaymwf/9+I8ksWrTIGevevbtp3bq1x/Hy8/PNbbfdZm644YYrXsvy5cuNJDNz5kxn7NKlS6ZLly6FzjN16lSP38tZs2YZSebYsWOXPf5//vOfQscp0K1bNyPJLFiwoMht3bp1c54XzEf9+vWN2+12xgvmfM6cOc5YRESEiY+Pv+oxr9TbD38/Cubp97//vUfdvffea7y8vMyePXucMUnG19fXY+y///2vkWRefPHFQucCyhofVwHl6Pv/4s7Oztbx48fVrVs37du3T9nZ2R61jRo1UmxsrMdYcnKy6tevr759+zpjfn5+GjlypEddRkaGvvzySw0ZMkQnTpzQ8ePHdfz4cZ09e1bdu3dXWlpaiRaXenl56b777tPq1at15swZZ3zp0qWqX7++OnfuXOS1nj59WsePH1eXLl107tw57dq1q9jn/qGTJ09q3bp1GjBggHP848eP68SJE4qNjdWXX36pb7755rL7r169WlWqVNHDDz/sjPn4+OjRRx+96rkLFmi/++67JV6k63K5NHz48Guu/9WvfqWaNWs6z++9917Vq1dPq1evLtH5r9Xq1avl4+Oj3/zmNx7jjz/+uIwx+te//uUxHhMTo8jISOd5mzZtFBAQoH379pVpn0BRCDlAOdq8ebNiYmJUvXp1BQUFqW7dus5ajKJCzg8dPHhQkZGRhdaHNGnSxOP5l19+KUmKj49X3bp1PR4LFy5UTk5OofNdq4EDB+r8+fPOupczZ85o9erVuu+++zz62rlzp37xi18oMDBQAQEBqlu3rrMIu6Tn/r49e/bIGKPf/e53ha5x6tSpkqSjR49edv+DBw+qXr16qlGjhsd406ZNr3rugQMHqlOnTnrggQcUEhKiQYMGadmyZcUKPPXr1y/WIuMbbrjB47mXl5eaNGly2fVTpeXgwYMKCwvzCFjSd3cJHjx40GP8+uuvL3SMWrVq6dtvvy27JoHLYE0OUE727t2r7t27q1mzZnrhhRcUHh4uX19frV69WrNmzSr0Avlj7qQqONazzz572duXf/jifq06duyohg0batmyZRoyZIhWrFih8+fPa+DAgU7NqVOn1K1bNwUEBGj69OmKjIyUn5+fPv74Y40fP/6KYeByX7r3wwXLBcf47W9/W+gdrwI/DH+lxd/fX2lpaVq/fr1WrVql5ORkLV26VHfeeafWrl3rrEu62jFK25Xm7lp6Kg2XO4/5wSJloDwQcoBysmLFCuXk5Oi9997z+NduUYuGLyciIkKfffaZjDEeL2h79uzxqCv4uCAgIEAxMTFXPGZJvsl3wIABmjNnjtxut5YuXaqGDRuqY8eOzvYNGzboxIkTevvtt9W1a1dnfP/+/Vc9dq1atSSp0F1OP3zHoHHjxpL+d8vz1a6xKBEREUpNTdWZM2c8At/u3buvaX9vb291795d3bt31wsvvKA//vGPmjRpktavX6+YmJhS/4bkgnfnChhjtGfPHrVp08YZq1WrVpF3hx08eNCZL6l4v+cRERF6//33dfr0aY93cwo+coyIiLjmYwHljY+rgHJS8C/c7/+LNjs7W4sWLbrmY8TGxuqbb77xuEX6woULeuWVVzzq2rdvr8jISD333HMea2cKHDt2zPl19erVJRUOFVcycOBA5eTk6LXXXlNycrIGDBjgsb2oa83NzdVLL7101WNHRETIx8dHaWlpHuM/3Dc4OFi33367/vKXv+jIkSOFjvP9ayxK7969denSJY9b+PPy8vTiiy9etceTJ08WGit4xywnJ0dSyeb1Sv72t7/p9OnTzvO33npLR44cUa9evZyxyMhIbd26Vbm5uc7YypUrC91qXpzeevfurby8PM2dO9djfNasWfLy8vI4P/BTwzs5QDnp0aOHfH191adPHz344IM6c+aMXnnlFQUHBxf5Il2UBx98UHPnztXgwYP12GOPqV69elq8eLH8/PwkffcvdG9vby1cuFC9evVSy5YtNXz4cNWvX1/ffPON1q9fr4CAAK1YsULS/wKRJE2aNEmDBg1S1apV1adPH+eFsCg333yzmjRpokmTJiknJ8fjoyrpf7cj16pVS/Hx8frNb34jLy8v/f3vf7+mjywCAwN133336cUXX5SXl5ciIyO1cuXKItfXzJs3T507d1br1q01cuRINW7cWFlZWUpPT9fXX3+t//73v5c9T58+fdSpUydNmDBBBw4cUIsWLfT2229f03qh6dOnKy0tTXFxcYqIiNDRo0f10ksvqUGDBs7i68jISAUFBWnBggWqWbOmqlevrqioqCLXWl2L2rVrq3Pnzho+fLiysrI0e/ZsNWnSxGPR+QMPPKC33npLPXv21IABA7R371794x//8FgIXNze+vTpozvuuEOTJk3SgQMH1LZtW61du1bvvvuuxowZU+jYwE9Kxd3YBditqFvI33vvPdOmTRvj5+dnGjZsaP70pz+ZV199tdAt3BERESYuLq7I4+7bt8/ExcUZf39/U7duXfP444+bf/7zn0aS2bp1q0ft9u3bzT333GOuu+4643K5TEREhBkwYIBJTU31qHv66adN/fr1jbe39zXfTj5p0iQjyTRp0qTI7Zs3bzYdO3Y0/v7+JiwszDzxxBNmzZo1hW4P/+Ety8YYc+zYMdO/f39TrVo1U6tWLfPggw+aHTt2FHnb8969e82vfvUrExoaaqpWrWrq169v7rrrLvPWW29d9RpOnDhhhg4dagICAkxgYKAZOnSo2b59+1VvIU9NTTV33323CQsLM76+viYsLMwMHjzYfPHFFx7Hf/fdd02LFi1MlSpVPI7ZrVs307JlyyJ7utwt5K+//rqZOHGiCQ4ONv7+/iYuLs4cPHiw0P7PP/+8qV+/vnG5XKZTp07mo48+KnTMK/VW1O/H6dOnzdixY01YWJipWrWqueGGG8yzzz7r8fUExvzvFvKEhIRCPV3u1nagrHkZw2owoLKbPXu2xo4dq6+//lr169ev6HYA4CeBkANUMufPn/e4M+fChQu66aablJeXpy+++KICOwOAnxbW5ACVzD333KPrr79e7dq1U3Z2tv7xj39o165dWrx4cUW3BgA/KYQcoJKJjY3VwoULtXjxYuXl5alFixZ64403Ci3+BYCfOz6uAgAAVuJ7cgAAgJUIOQAAwEo/6zU5+fn5Onz4sGrWrFnqX8EOAADKhjFGp0+fVlhYmLy9L/9+zc865Bw+fFjh4eEV3QYAACiBr776Sg0aNLjs9p91yCn4YXNfffWVAgICKrgbAABwLdxut8LDwz1+aGxRftYhp+AjqoCAAEIOAACVzNWWmrDwGAAAWImQAwAArETIAQAAViLkAAAAKxFyAACAlQg5AADASoQcAABgJUIOAACwEiEHAABYiZADAACsVKyQM2PGDN1yyy2qWbOmgoOD1a9fP+3evduj5vbbb5eXl5fH46GHHvKoOXTokOLi4lStWjUFBwdr3LhxunTpkkfNhg0bdPPNN8vlcqlJkyZKSkoq1M+8efPUsGFD+fn5KSoqSh9++GFxLgcAAFisWCFn48aNSkhI0NatW5WSkqKLFy+qR48eOnv2rEfdyJEjdeTIEecxc+ZMZ1teXp7i4uKUm5urLVu26LXXXlNSUpKmTJni1Ozfv19xcXG64447lJGRoTFjxuiBBx7QmjVrnJqlS5cqMTFRU6dO1ccff6y2bdsqNjZWR48eLelcAAAAi3gZY0xJdz527JiCg4O1ceNGde3aVdL/3slp166dZs+eXeQ+//rXv3TXXXfp8OHDCgkJkSQtWLBA48eP17Fjx+Tr66vx48dr1apV2rFjh7PfoEGDdOrUKSUnJ0uSoqKidMstt2ju3LmSpPz8fIWHh+vRRx/VhAkTrql/t9utwMBAZWdn8wM6AQCoJK719ftHrcnJzs6WJNWuXdtjfPHixapTp45atWqliRMn6ty5c8629PR0tW7d2gk4khQbGyu3262dO3c6NTExMR7HjI2NVXp6uiQpNzdX27Zt86jx9vZWTEyMU1OUnJwcud1ujwcAALBTlZLumJ+frzFjxqhTp05q1aqVMz5kyBBFREQoLCxMn3zyicaPH6/du3fr7bffliRlZmZ6BBxJzvPMzMwr1rjdbp0/f17ffvut8vLyiqzZtWvXZXueMWOGnnrqqZJecrE0nLCqXM5Tmg48E1fRLQAALoPXleIrcchJSEjQjh07tGnTJo/xUaNGOb9u3bq16tWrp+7du2vv3r2KjIwseaelYOLEiUpMTHSeu91uhYeHV2BHAACgrJQo5IwePVorV65UWlqaGjRocMXaqKgoSdKePXsUGRmp0NDQQndBZWVlSZJCQ0Od/xaMfb8mICBA/v7+8vHxkY+PT5E1BccoisvlksvluraLBAAAlVqx1uQYYzR69Gi98847WrdunRo1anTVfTIyMiRJ9erVkyRFR0fr008/9bgLKiUlRQEBAWrRooVTk5qa6nGclJQURUdHS5J8fX3Vvn17j5r8/HylpqY6NQAA4OetWO/kJCQkaMmSJXr33XdVs2ZNZw1NYGCg/P39tXfvXi1ZskS9e/fWddddp08++URjx45V165d1aZNG0lSjx491KJFCw0dOlQzZ85UZmamJk+erISEBOddloceekhz587VE088oV//+tdat26dli1bplWrvvs8MjExUfHx8erQoYNuvfVWzZ49W2fPntXw4cNLa24AAEAlVqyQM3/+fEn/u038+xYtWqRhw4bJ19dX77//vhM4wsPD1b9/f02ePNmp9fHx0cqVK/Xwww8rOjpa1atXV3x8vKZPn+7UNGrUSKtWrdLYsWM1Z84cNWjQQAsXLlRsbKxTM3DgQB07dkxTpkxRZmam2rVrp+Tk5EKLkQEAwM/Tj/qenMquLL8nh1XwAIDSxOvKd8rle3IAAAB+qgg5AADASoQcAABgJUIOAACwEiEHAABYiZADAACsRMgBAABWIuQAAAArEXIAAICVCDkAAMBKhBwAAGAlQg4AALASIQcAAFiJkAMAAKxEyAEAAFYi5AAAACsRcgAAgJUIOQAAwEqEHAAAYCVCDgAAsBIhBwAAWImQAwAArETIAQAAViLkAAAAKxFyAACAlQg5AADASoQcAABgJUIOAACwEiEHAABYiZADAACsRMgBAABWIuQAAAArEXIAAICVCDkAAMBKhBwAAGAlQg4AALASIQcAAFiJkAMAAKxEyAEAAFYi5AAAACsRcgAAgJUIOQAAwEqEHAAAYCVCDgAAsBIhBwAAWImQAwAArETIAQAAViLkAAAAKxFyAACAlQg5AADASoQcAABgJUIOAACwEiEHAABYiZADAACsRMgBAABWIuQAAAArEXIAAICVCDkAAMBKhBwAAGAlQg4AALBSsULOjBkzdMstt6hmzZoKDg5Wv379tHv3bo+aCxcuKCEhQdddd51q1Kih/v37Kysry6Pm0KFDiouLU7Vq1RQcHKxx48bp0qVLHjUbNmzQzTffLJfLpSZNmigpKalQP/PmzVPDhg3l5+enqKgoffjhh8W5HAAAYLFihZyNGzcqISFBW7duVUpKii5evKgePXro7NmzTs3YsWO1YsUKvfnmm9q4caMOHz6se+65x9mel5enuLg45ebmasuWLXrttdeUlJSkKVOmODX79+9XXFyc7rjjDmVkZGjMmDF64IEHtGbNGqdm6dKlSkxM1NSpU/Xxxx+rbdu2io2N1dGjR3/MfAAAAEt4GWNMSXc+duyYgoODtXHjRnXt2lXZ2dmqW7eulixZonvvvVeStGvXLjVv3lzp6enq2LGj/vWvf+muu+7S4cOHFRISIklasGCBxo8fr2PHjsnX11fjx4/XqlWrtGPHDudcgwYN0qlTp5ScnCxJioqK0i233KK5c+dKkvLz8xUeHq5HH31UEyZMuKb+3W63AgMDlZ2drYCAgJJOQ5EaTlhVqscrDweeiavoFgAAl8Hryneu9fX7R63Jyc7OliTVrl1bkrRt2zZdvHhRMTExTk2zZs10/fXXKz09XZKUnp6u1q1bOwFHkmJjY+V2u7Vz506n5vvHKKgpOEZubq62bdvmUePt7a2YmBinpig5OTlyu90eDwAAYKcSh5z8/HyNGTNGnTp1UqtWrSRJmZmZ8vX1VVBQkEdtSEiIMjMznZrvB5yC7QXbrlTjdrt1/vx5HT9+XHl5eUXWFByjKDNmzFBgYKDzCA8PL/6FAwCASqHEISchIUE7duzQG2+8UZr9lKmJEycqOzvbeXz11VcV3RIAACgjVUqy0+jRo7Vy5UqlpaWpQYMGznhoaKhyc3N16tQpj3dzsrKyFBoa6tT88C6ogruvvl/zwzuysrKyFBAQIH9/f/n4+MjHx6fImoJjFMXlcsnlchX/ggEAQKVTrHdyjDEaPXq03nnnHa1bt06NGjXy2N6+fXtVrVpVqampztju3bt16NAhRUdHS5Kio6P16aefetwFlZKSooCAALVo0cKp+f4xCmoKjuHr66v27dt71OTn5ys1NdWpAQAAP2/FeicnISFBS5Ys0bvvvquaNWs6618CAwPl7++vwMBAjRgxQomJiapdu7YCAgL06KOPKjo6Wh07dpQk9ejRQy1atNDQoUM1c+ZMZWZmavLkyUpISHDeZXnooYc0d+5cPfHEE/r1r3+tdevWadmyZVq16ruV5YmJiYqPj1eHDh106623avbs2Tp79qyGDx9eWnMDAAAqsWKFnPnz50uSbr/9do/xRYsWadiwYZKkWbNmydvbW/3791dOTo5iY2P10ksvObU+Pj5auXKlHn74YUVHR6t69eqKj4/X9OnTnZpGjRpp1apVGjt2rObMmaMGDRpo4cKFio2NdWoGDhyoY8eOacqUKcrMzFS7du2UnJxcaDEyAAD4efpR35NT2fE9OZ74nhwA+OnideU75fI9OQAAAD9VhBwAAGAlQg4AALASIQcAAFiJkAMAAKxEyAEAAFYi5AAAACsRcgAAgJUIOQAAwEqEHAAAYCVCDgAAsBIhBwAAWImQAwAArETIAQAAViLkAAAAKxFyAACAlQg5AADASoQcAABgJUIOAACwEiEHAABYiZADAACsRMgBAABWIuQAAAArEXIAAICVCDkAAMBKhBwAAGAlQg4AALASIQcAAFiJkAMAAKxEyAEAAFYi5AAAACsRcgAAgJUIOQAAwEqEHAAAYCVCDgAAsBIhBwAAWImQAwAArETIAQAAViLkAAAAKxFyAACAlQg5AADASoQcAABgJUIOAACwEiEHAABYiZADAACsRMgBAABWIuQAAAArEXIAAICVCDkAAMBKhBwAAGAlQg4AALASIQcAAFiJkAMAAKxEyAEAAFYi5AAAACsRcgAAgJUIOQAAwEqEHAAAYCVCDgAAsFKxQ05aWpr69OmjsLAweXl5afny5R7bhw0bJi8vL49Hz549PWpOnjyp+++/XwEBAQoKCtKIESN05swZj5pPPvlEXbp0kZ+fn8LDwzVz5sxCvbz55ptq1qyZ/Pz81Lp1a61evbq4lwMAACxV7JBz9uxZtW3bVvPmzbtsTc+ePXXkyBHn8frrr3tsv//++7Vz506lpKRo5cqVSktL06hRo5ztbrdbPXr0UEREhLZt26Znn31W06ZN08svv+zUbNmyRYMHD9aIESO0fft29evXT/369dOOHTuKe0kAAMBCVYq7Q69evdSrV68r1rhcLoWGhha57fPPP1dycrL+85//qEOHDpKkF198Ub1799Zzzz2nsLAwLV68WLm5uXr11Vfl6+urli1bKiMjQy+88IIThubMmaOePXtq3LhxkqSnn35aKSkpmjt3rhYsWFDcywIAAJYpkzU5GzZsUHBwsJo2baqHH35YJ06ccLalp6crKCjICTiSFBMTI29vb33wwQdOTdeuXeXr6+vUxMbGavfu3fr222+dmpiYGI/zxsbGKj09/bJ95eTkyO12ezwAAICdSj3k9OzZU3/729+UmpqqP/3pT9q4caN69eqlvLw8SVJmZqaCg4M99qlSpYpq166tzMxMpyYkJMSjpuD51WoKthdlxowZCgwMdB7h4eE/7mIBAMBPVrE/rrqaQYMGOb9u3bq12rRpo8jISG3YsEHdu3cv7dMVy8SJE5WYmOg8d7vdBB0AACxV5reQN27cWHXq1NGePXskSaGhoTp69KhHzaVLl3Ty5ElnHU9oaKiysrI8agqeX63mcmuBpP+tFQoICPB4AAAAO5V5yPn666914sQJ1atXT5IUHR2tU6dOadu2bU7NunXrlJ+fr6ioKKcmLS1NFy9edGpSUlLUtGlT1apVy6lJTU31OFdKSoqio6PL+pIAAEAlUOyQc+bMGWVkZCgjI0OStH//fmVkZOjQoUM6c+aMxo0bp61bt+rAgQNKTU3V3XffrSZNmig2NlaS1Lx5c/Xs2VMjR47Uhx9+qM2bN2v06NEaNGiQwsLCJElDhgyRr6+vRowYoZ07d2rp0qWaM2eOx0dNjz32mJKTk/X8889r165dmjZtmj766CONHj26FKYFAABUdsUOOR999JFuuukm3XTTTZKkxMRE3XTTTZoyZYp8fHz0ySefqG/fvrrxxhs1YsQItW/fXv/+97/lcrmcYyxevFjNmjVT9+7d1bt3b3Xu3NnjO3ACAwO1du1a7d+/X+3bt9fjjz+uKVOmeHyXzm233aYlS5bo5ZdfVtu2bfXWW29p+fLlatWq1Y+ZDwAAYAkvY4yp6CYqitvtVmBgoLKzs0t9fU7DCatK9Xjl4cAzcRXdAgDgMnhd+c61vn7zs6sAAICVCDkAAMBKhBwAAGAlQg4AALASIQcAAFiJkAMAAKxEyAEAAFYi5AAAACsRcgAAgJUIOQAAwEqEHAAAYCVCDgAAsBIhBwAAWImQAwAArETIAQAAViLkAAAAKxFyAACAlQg5AADASoQcAABgJUIOAACwEiEHAABYiZADAACsRMgBAABWIuQAAAArEXIAAICVCDkAAMBKhBwAAGAlQg4AALASIQcAAFiJkAMAAKxEyAEAAFYi5AAAACsRcgAAgJUIOQAAwEqEHAAAYCVCDgAAsBIhBwAAWImQAwAArETIAQAAViLkAAAAKxFyAACAlQg5AADASoQcAABgJUIOAACwEiEHAABYiZADAACsRMgBAABWIuQAAAArEXIAAICVCDkAAMBKhBwAAGAlQg4AALASIQcAAFiJkAMAAKxEyAEAAFYi5AAAACsRcgAAgJUIOQAAwEqEHAAAYCVCDgAAsFKxQ05aWpr69OmjsLAweXl5afny5R7bjTGaMmWK6tWrJ39/f8XExOjLL7/0qDl58qTuv/9+BQQEKCgoSCNGjNCZM2c8aj755BN16dJFfn5+Cg8P18yZMwv18uabb6pZs2by8/NT69attXr16uJeDgAAsFSxQ87Zs2fVtm1bzZs3r8jtM2fO1J///GctWLBAH3zwgapXr67Y2FhduHDBqbn//vu1c+dOpaSkaOXKlUpLS9OoUaOc7W63Wz169FBERIS2bdumZ599VtOmTdPLL7/s1GzZskWDBw/WiBEjtH37dvXr10/9+vXTjh07intJAADAQl7GGFPinb289M4776hfv36S/vcuTlhYmB5//HH99re/lSRlZ2crJCRESUlJGjRokD7//HO1aNFC//nPf9ShQwdJUnJysnr37q2vv/5aYWFhmj9/viZNmqTMzEz5+vpKkiZMmKDly5dr165dkqSBAwfq7NmzWrlypdNPx44d1a5dOy1YsKDIfnNycpSTk+M8d7vdCg8PV3Z2tgICAko6DUVqOGFVqR6vPBx4Jq6iWwAAXAavK99xu90KDAy86ut3qa7J2b9/vzIzMxUTE+OMBQYGKioqSunp6ZKk9PR0BQUFOQFHkmJiYuTt7a0PPvjAqenatasTcCQpNjZWu3fv1rfffuvUfP88BTUF5ynKjBkzFBgY6DzCw8N//EUDAICfpFINOZmZmZKkkJAQj/GQkBBnW2ZmpoKDgz22V6lSRbVr1/aoKeoY3z/H5WoKthdl4sSJys7Odh5fffVVcS8RAABUElUquoHy5HK55HK5KroNAABQDkr1nZzQ0FBJUlZWlsd4VlaWsy00NFRHjx712H7p0iWdPHnSo6aoY3z/HJerKdgOAAB+3ko15DRq1EihoaFKTU11xtxutz744ANFR0dLkqKjo3Xq1Clt27bNqVm3bp3y8/MVFRXl1KSlpenixYtOTUpKipo2bapatWo5Nd8/T0FNwXkAAMDPW7FDzpkzZ5SRkaGMjAxJ/1tsnJGRoUOHDsnLy0tjxozR73//e7333nv69NNP9atf/UphYWHOHVjNmzdXz549NXLkSH344YfavHmzRo8erUGDBiksLEySNGTIEPn6+mrEiBHauXOnli5dqjlz5igxMdHp47HHHlNycrKef/557dq1S9OmTdNHH32k0aNH//hZAQAAlV6x1+R89NFHuuOOO5znBcEjPj5eSUlJeuKJJ3T27FmNGjVKp06dUufOnZWcnCw/Pz9nn8WLF2v06NHq3r27vL291b9/f/35z392tgcGBmrt2rVKSEhQ+/btVadOHU2ZMsXju3Ruu+02LVmyRJMnT9aTTz6pG264QcuXL1erVq1KNBEAAMAuP+p7ciq7a73PviT4PgMAQGnideU7FfI9OQAAAD8VhBwAAGAlQg4AALASIQcAAFiJkAMAAKxEyAEAAFYi5AAAACsRcgAAgJUIOQAAwEqEHAAAYCVCDgAAsBIhBwAAWImQAwAArETIAQAAViLkAAAAKxFyAACAlQg5AADASoQcAABgJUIOAACwEiEHAABYiZADAACsRMgBAABWIuQAAAArEXIAAICVCDkAAMBKhBwAAGAlQg4AALASIQcAAFiJkAMAAKxEyAEAAFYi5AAAACsRcgAAgJUIOQAAwEqEHAAAYCVCDgAAsBIhBwAAWImQAwAArETIAQAAViLkAAAAKxFyAACAlQg5AADASoQcAABgJUIOAACwEiEHAABYiZADAACsRMgBAABWIuQAAAArEXIAAICVCDkAAMBKhBwAAGAlQg4AALASIQcAAFiJkAMAAKxEyAEAAFYi5AAAACsRcgAAgJUIOQAAwEqEHAAAYCVCDgAAsFKph5xp06bJy8vL49GsWTNn+4ULF5SQkKDrrrtONWrUUP/+/ZWVleVxjEOHDikuLk7VqlVTcHCwxo0bp0uXLnnUbNiwQTfffLNcLpeaNGmipKSk0r4UAABQiZXJOzktW7bUkSNHnMemTZucbWPHjtWKFSv05ptvauPGjTp8+LDuueceZ3teXp7i4uKUm5urLVu26LXXXlNSUpKmTJni1Ozfv19xcXG64447lJGRoTFjxuiBBx7QmjVryuJyAABAJVSlTA5apYpCQ0MLjWdnZ+uvf/2rlixZojvvvFOStGjRIjVv3lxbt25Vx44dtXbtWn322Wd6//33FRISonbt2unpp5/W+PHjNW3aNPn6+mrBggVq1KiRnn/+eUlS8+bNtWnTJs2aNUuxsbFlcUkAAKCSKZN3cr788kuFhYWpcePGuv/++3Xo0CFJ0rZt23Tx4kXFxMQ4tc2aNdP111+v9PR0SVJ6erpat26tkJAQpyY2NlZut1s7d+50ar5/jIKagmNcTk5Ojtxut8cDAADYqdRDTlRUlJKSkpScnKz58+dr//796tKli06fPq3MzEz5+voqKCjIY5+QkBBlZmZKkjIzMz0CTsH2gm1XqnG73Tp//vxle5sxY4YCAwOdR3h4+I+9XAAA8BNV6h9X9erVy/l1mzZtFBUVpYiICC1btkz+/v6lfbpimThxohITE53nbreboAMAgKXK/BbyoKAg3XjjjdqzZ49CQ0OVm5urU6dOedRkZWU5a3hCQ0ML3W1V8PxqNQEBAVcMUi6XSwEBAR4PAABgpzIPOWfOnNHevXtVr149tW/fXlWrVlVqaqqzfffu3Tp06JCio6MlSdHR0fr000919OhRpyYlJUUBAQFq0aKFU/P9YxTUFBwDAACg1EPOb3/7W23cuFEHDhzQli1b9Itf/EI+Pj4aPHiwAgMDNWLECCUmJmr9+vXatm2bhg8frujoaHXs2FGS1KNHD7Vo0UJDhw7Vf//7X61Zs0aTJ09WQkKCXC6XJOmhhx7Svn379MQTT2jXrl166aWXtGzZMo0dO7a0LwcAAFRSpb4m5+uvv9bgwYN14sQJ1a1bV507d9bWrVtVt25dSdKsWbPk7e2t/v37KycnR7GxsXrppZec/X18fLRy5Uo9/PDDio6OVvXq1RUfH6/p06c7NY0aNdKqVas0duxYzZkzRw0aNNDChQu5fRwAADi8jDGmopuoKG63W4GBgcrOzi719TkNJ6wq1eOVhwPPxFV0CwCAy+B15TvX+vrNz64CAABWIuQAAAArEXIAAICVCDkAAMBKhBwAAGAlQg4AALASIQcAAFiJkAMAAKxEyAEAAFYi5AAAACsRcgAAgJUIOQAAwEqEHAAAYCVCDgAAsBIhBwAAWImQAwAArETIAQAAViLkAAAAKxFyAACAlQg5AADASoQcAABgJUIOAACwEiEHAABYiZADAACsRMgBAABWIuQAAAArEXIAAICVCDkAAMBKhBwAAGAlQg4AALASIQcAAFiJkAMAAKxEyAEAAFYi5AAAACsRcgAAgJUIOQAAwEqEHAAAYCVCDgAAsBIhBwAAWImQAwAArETIAQAAViLkAAAAKxFyAACAlQg5AADASoQcAABgJUIOAACwEiEHAABYiZADAACsRMgBAABWIuQAAAArEXIAAICVCDkAAMBKhBwAAGAlQg4AALASIQcAAFiJkAMAAKxEyAEAAFYi5AAAACsRcgAAgJUqfciZN2+eGjZsKD8/P0VFRenDDz+s6JYAAMBPQKUOOUuXLlViYqKmTp2qjz/+WG3btlVsbKyOHj1a0a0BAIAKVqlDzgsvvKCRI0dq+PDhatGihRYsWKBq1arp1VdfrejWAABABatS0Q2UVG5urrZt26aJEyc6Y97e3oqJiVF6enqR++Tk5CgnJ8d5np2dLUlyu92l3l9+zrlSP2ZZK4t5AACUDl5XCh/XGHPFukobco4fP668vDyFhIR4jIeEhGjXrl1F7jNjxgw99dRThcbDw8PLpMfKJnB2RXcAALBJWb+unD59WoGBgZfdXmlDTklMnDhRiYmJzvP8/HydPHlS1113nby8vErtPG63W+Hh4frqq68UEBBQaseFJ+a5/DDX5YN5Lh/Mc/koy3k2xuj06dMKCwu7Yl2lDTl16tSRj4+PsrKyPMazsrIUGhpa5D4ul0sul8tjLCgoqKxaVEBAAH+BygHzXH6Y6/LBPJcP5rl8lNU8X+kdnAKVduGxr6+v2rdvr9TUVGcsPz9fqampio6OrsDOAADAT0GlfSdHkhITExUfH68OHTro1ltv1ezZs3X27FkNHz68olsDAAAVrFKHnIEDB+rYsWOaMmWKMjMz1a5dOyUnJxdajFzeXC6Xpk6dWuijMZQu5rn8MNflg3kuH8xz+fgpzLOXudr9VwAAAJVQpV2TAwAAcCWEHAAAYCVCDgAAsBIhBwAAWImQAwAArETIKaF58+apYcOG8vPzU1RUlD788MMr1r/55ptq1qyZ/Pz81Lp1a61evbqcOq3cijPPr7zyirp06aJatWqpVq1aiomJuervC/6nuH+eC7zxxhvy8vJSv379yrZBixR3rk+dOqWEhATVq1dPLpdLN954I///uAbFnefZs2eradOm8vf3V3h4uMaOHasLFy6UU7eVU1pamvr06aOwsDB5eXlp+fLlV91nw4YNuvnmm+VyudSkSRMlJSWVbZMGxfbGG28YX19f8+qrr5qdO3eakSNHmqCgIJOVlVVk/ebNm42Pj4+ZOXOm+eyzz8zkyZNN1apVzaefflrOnVcuxZ3nIUOGmHnz5pnt27ebzz//3AwbNswEBgaar7/+upw7r1yKO88F9u/fb+rXr2+6dOli7r777vJptpIr7lzn5OSYDh06mN69e5tNmzaZ/fv3mw0bNpiMjIxy7rxyKe48L1682LhcLrN48WKzf/9+s2bNGlOvXj0zduzYcu68clm9erWZNGmSefvtt40k884771yxft++faZatWomMTHRfPbZZ+bFF180Pj4+Jjk5ucx6JOSUwK233moSEhKc53l5eSYsLMzMmDGjyPoBAwaYuLg4j7GoqCjz4IMPlmmflV1x5/mHLl26ZGrWrGlee+21smrRCiWZ50uXLpnbbrvNLFy40MTHxxNyrlFx53r+/PmmcePGJjc3t7xatEJx5zkhIcHceeedHmOJiYmmU6dOZdqnTa4l5DzxxBOmZcuWHmMDBw40sbGxZdYXH1cVU25urrZt26aYmBhnzNvbWzExMUpPTy9yn/T0dI96SYqNjb1sPUo2zz907tw5Xbx4UbVr1y6rNiu9ks7z9OnTFRwcrBEjRpRHm1YoyVy/9957io6OVkJCgkJCQtSqVSv98Y9/VF5eXnm1XemUZJ5vu+02bdu2zflIa9++fVq9erV69+5dLj3/XFTEa2Gl/rEOFeH48ePKy8sr9KMjQkJCtGvXriL3yczMLLI+MzOzzPqs7Eoyzz80fvx4hYWFFfpLhe+UZJ43bdqkv/71r8rIyCiHDu1Rkrnet2+f1q1bp/vvv1+rV6/Wnj179Mgjj+jixYuaOnVqebRd6ZRknocMGaLjx4+rc+fOMsbo0qVLeuihh/Tkk0+WR8s/G5d7LXS73Tp//rz8/f1L/Zy8kwMrPfPMM3rjjTf0zjvvyM/Pr6Lbscbp06c1dOhQvfLKK6pTp05Ft2O9/Px8BQcH6+WXX1b79u01cOBATZo0SQsWLKjo1qyyYcMG/fGPf9RLL72kjz/+WG+//bZWrVqlp59+uqJbw4/EOznFVKdOHfn4+CgrK8tjPCsrS6GhoUXuExoaWqx6lGyeCzz33HN65pln9P7776tNmzZl2WalV9x53rt3rw4cOKA+ffo4Y/n5+ZKkKlWqaPfu3YqMjCzbpiupkvyZrlevnqpWrSofHx9nrHnz5srMzFRubq58fX3LtOfKqCTz/Lvf/U5Dhw7VAw88IElq3bq1zp49q1GjRmnSpEny9ub9gNJwudfCgICAMnkXR+KdnGLz9fVV+/btlZqa6ozl5+crNTVV0dHRRe4THR3tUS9JKSkpl61HyeZZkmbOnKmnn35aycnJ6tChQ3m0WqkVd56bNWumTz/9VBkZGc6jb9++uuOOO5SRkaHw8PDybL9SKcmf6U6dOmnPnj1OkJSkL774QvXq1SPgXEZJ5vncuXOFgkxBsDT8DOtSUyGvhWW2pNlib7zxhnG5XCYpKcl89tlnZtSoUSYoKMhkZmYaY4wZOnSomTBhglO/efNmU6VKFfPcc8+Zzz//3EydOpVbyK9Bcef5mWeeMb6+vuatt94yR44ccR6nT5+uqEuoFIo7zz/E3VXXrrhzfejQIVOzZk0zevRos3v3brNy5UoTHBxsfv/731fUJVQKxZ3nqVOnmpo1a5rXX3/d7Nu3z6xdu9ZERkaaAQMGVNQlVAqnT58227dvN9u3bzeSzAsvvGC2b99uDh48aIwxZsKECWbo0KFOfcEt5OPGjTOff/65mTdvHreQ/1S9+OKL5vrrrze+vr7m1ltvNVu3bnW2devWzcTHx3vUL1u2zNx4443G19fXtGzZ0qxataqcO66cijPPERERRlKhx9SpU8u/8UqmuH+ev4+QUzzFnestW7aYqKgo43K5TOPGjc0f/vAHc+nSpXLuuvIpzjxfvHjRTJs2zURGRho/Pz8THh5uHnnkEfPtt9+Wf+OVyPr164v8f27B3MbHx5tu3boV2qddu3bG19fXNG7c2CxatKhMe/QyhvfiAACAfViTAwAArETIAQAAViLkAAAAKxFyAACAlQg5AADASoQcAABgJUIOAACwEiEHAABYiZADAACsRMgBAABWIuQAAAAr/T9WUkYIDg5fygAAAABJRU5ErkJggg==",
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "plt.title('Target value distribution')\n",
        "plt.hist(df_reviews['sentiment'])\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Modelo BERT"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EDkjlPDakskM"
      },
      "source": [
        "## Instalando Bibliotecas"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lk7m_1xvmWvz",
        "outputId": "ce842053-b261-4768-d9d7-fe9c65c9f6aa"
      },
      "outputs": [],
      "source": [
        "#pip install transformers\n",
        "#pip install accelerate -U\n",
        "#pip install transformers[torch]\n",
        "#pip install datasets evaluate"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Carregando o modelo treinado e tokenizador"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GlyrkK52zMcc",
        "outputId": "a938653b-92c3-4b4e-802c-eacc3f1b6ecf"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "c:\\Users\\andre\\1JUPYTER\\dt_labs\\.venv\\Lib\\site-packages\\tqdm\\auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
            "  from .autonotebook import tqdm as notebook_tqdm\n"
          ]
        }
      ],
      "source": [
        "from transformers import AutoTokenizer\n",
        "from transformers import BertForSequenceClassification\n",
        "\n",
        "pre_trained_base = \"bert-base-uncased\"\n",
        "tokenizer = AutoTokenizer.from_pretrained(pre_trained_base)\n",
        "model = BertForSequenceClassification.from_pretrained(pre_trained_base, num_labels = 2, output_attentions=False, output_hidden_states=False)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Tokenização das Sentenças e Cálculo do Tamanho dos Tokens"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "id": "LKEjDZCHpk4e"
      },
      "outputs": [],
      "source": [
        "token_lens = []\n",
        "\n",
        "for sentence in df_reviews['review']:\n",
        "    tokens = tokenizer.encode(sentence, max_length=200, truncation=True)\n",
        "    token_lens.append(len(tokens))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Divisão dos Dados em Conjunto de Treinamento e Validação:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "id": "H7PfXaVVp2uQ"
      },
      "outputs": [],
      "source": [
        "SEED=42\n",
        "MAX_LEN = 200\n",
        "from sklearn.model_selection import train_test_split\n",
        "df_train, df_val = train_test_split(df_reviews, test_size=0.2, random_state=SEED)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Processando os dados\n",
        "A função process_data recebe uma linha de um dataframe contendo uma revisão de texto e sua respectiva classificação de sentimento. Ela começa extraindo e limpando o texto da revisão, removendo quaisquer espaços extras. Em seguida, utiliza o tokenizer BERT para tokenizar o texto, aplicando padding e truncamento para garantir que todas as sequências tenham um comprimento fixo definido pela variável MAX_LEN. A função então adiciona a etiqueta de sentimento original e o texto limpo às codificações geradas, retornando um dicionário que contém os tokens do texto, a etiqueta de sentimento e o texto original."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "id": "v7EZ6wd-qDfd"
      },
      "outputs": [],
      "source": [
        "def process_data(row):\n",
        "\n",
        "    text = row['review']\n",
        "    text = str(text)\n",
        "    text = ' '.join(text.split())\n",
        "\n",
        "    encodings = tokenizer(text, padding=\"max_length\", truncation=True, max_length=MAX_LEN)\n",
        "\n",
        "    encodings['label'] = row['sentiment']\n",
        "    encodings['text'] = text\n",
        "\n",
        "    return encodings"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "id": "d9VgrXNSqIYL"
      },
      "outputs": [],
      "source": [
        "# Treino\n",
        "processed_data_tr = []\n",
        "for i in range(df_train.shape[0]):\n",
        "    processed_data_tr.append(process_data(df_train.iloc[i]))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "id": "p0NLQxoKqJ_k"
      },
      "outputs": [],
      "source": [
        "# Validação\n",
        "processed_data_val = []\n",
        "for i in range(df_val.shape[0]):\n",
        "    processed_data_val.append(process_data(df_val.iloc[i]))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {
        "id": "ac76Rb6fqP_G"
      },
      "outputs": [],
      "source": [
        "# Dataframes de Treino e Validação\n",
        "df_train = pd.DataFrame(processed_data_tr)\n",
        "df_val = pd.DataFrame(processed_data_val)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "RdbHaVy_fd64",
        "outputId": "a9aed834-81b7-4223-da42-6289799c2e1e"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>attention_mask</th>\n",
              "      <th>input_ids</th>\n",
              "      <th>label</th>\n",
              "      <th>text</th>\n",
              "      <th>token_type_ids</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...</td>\n",
              "      <td>[101, 2921, 3198, 23624, 2954, 6978, 2674, 841...</td>\n",
              "      <td>0</td>\n",
              "      <td>kept ask mani fight scream match swear gener m...</td>\n",
              "      <td>[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...</td>\n",
              "      <td>[101, 3422, 4372, 3775, 2099, 9587, 5737, 2071...</td>\n",
              "      <td>0</td>\n",
              "      <td>watch entir movi could watch entir movi stop d...</td>\n",
              "      <td>[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...</td>\n",
              "      <td>[101, 3543, 2293, 2358, 10050, 2128, 25300, 11...</td>\n",
              "      <td>1</td>\n",
              "      <td>touch love stori reminisc ‘in mood love draw h...</td>\n",
              "      <td>[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...</td>\n",
              "      <td>[101, 3732, 2154, 11865, 15472, 2072, 8040, 73...</td>\n",
              "      <td>0</td>\n",
              "      <td>latter day fulci schlocker total abysm concoct...</td>\n",
              "      <td>[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...</td>\n",
              "      <td>[101, 2034, 3813, 3669, 19337, 2666, 2615, 504...</td>\n",
              "      <td>0</td>\n",
              "      <td>first firmli believ norwegian movi continu get...</td>\n",
              "      <td>[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                      attention_mask  \\\n",
              "0  [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...   \n",
              "1  [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...   \n",
              "2  [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...   \n",
              "3  [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...   \n",
              "4  [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...   \n",
              "\n",
              "                                           input_ids  label  \\\n",
              "0  [101, 2921, 3198, 23624, 2954, 6978, 2674, 841...      0   \n",
              "1  [101, 3422, 4372, 3775, 2099, 9587, 5737, 2071...      0   \n",
              "2  [101, 3543, 2293, 2358, 10050, 2128, 25300, 11...      1   \n",
              "3  [101, 3732, 2154, 11865, 15472, 2072, 8040, 73...      0   \n",
              "4  [101, 2034, 3813, 3669, 19337, 2666, 2615, 504...      0   \n",
              "\n",
              "                                                text  \\\n",
              "0  kept ask mani fight scream match swear gener m...   \n",
              "1  watch entir movi could watch entir movi stop d...   \n",
              "2  touch love stori reminisc ‘in mood love draw h...   \n",
              "3  latter day fulci schlocker total abysm concoct...   \n",
              "4  first firmli believ norwegian movi continu get...   \n",
              "\n",
              "                                      token_type_ids  \n",
              "0  [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...  \n",
              "1  [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...  \n",
              "2  [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...  \n",
              "3  [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...  \n",
              "4  [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...  "
            ]
          },
          "execution_count": 20,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "df_train.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0lTWT8JwkRic"
      },
      "source": [
        "## Fine Tunning do Modelo\n",
        "Ajuste fino do BERT para tarefas específica de classificação de sentimento para o dataset do IMDB"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import torch\n",
        "import pyarrow as pa\n",
        "from datasets import Dataset\n",
        "import evaluate\n",
        "import numpy as np"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kW53p7VQqUDD",
        "outputId": "8231f3ba-37d5-4546-c4d0-6b4ff317ecf3"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "device(type='cuda', index=0)"
            ]
          },
          "execution_count": 21,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
        "device"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "metadata": {
        "id": "68OdbTv5rLrm"
      },
      "outputs": [],
      "source": [
        "train_hg = Dataset(pa.Table.from_pandas(df_train))\n",
        "valid_hg = Dataset(pa.Table.from_pandas(df_val))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Metricas de avaliação F1 Score e Acc"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "`compute_metrics` calcula tanto a acurácia quanto o F1-score para avaliar um modelo de classificação. Primeiramente, são carregadas as métricas de acurácia e F1-score usando evaluate.load. Em seguida, a função compute_metrics recebe um par de arrays eval_pred, contendo as previsões do modelo e os rótulos verdadeiros. Utilizando as previsões, a função calcula a acurácia e o F1-score ponderado, onde a acurácia é obtida através da comparação das previsões com os rótulos utilizando a métrica de acurácia previamente carregada, e o F1-score é calculado utilizando a métrica de F1 previamente carregada, com ponderação \"weighted\". Os resultados de ambas as métricas são então combinados em um dicionário e retornados como um único objeto contendo as métricas de avaliação calculadas."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 25,
      "metadata": {
        "id": "lUNhDPs0ry4m"
      },
      "outputs": [],
      "source": [
        "\n",
        "# Load both accuracy and f1 metrics\n",
        "accuracy_metric = evaluate.load(\"accuracy\")\n",
        "f1_metric = evaluate.load(\"f1\")\n",
        "\n",
        "# Metric helper method\n",
        "def compute_metrics(eval_pred):\n",
        "    predictions, labels = eval_pred\n",
        "    predictions = np.argmax(predictions, axis=1)\n",
        "\n",
        "    # Compute accuracy\n",
        "    accuracy = accuracy_metric.compute(predictions=predictions, references=labels)\n",
        "\n",
        "    # Compute F1 score\n",
        "    f1 = f1_metric.compute(predictions=predictions, references=labels, average=\"weighted\")\n",
        "\n",
        "    # Combine the metrics into a single dictionary\n",
        "    combined_metrics = {\n",
        "        'accuracy': accuracy['accuracy'],\n",
        "        'f1': f1['f1']\n",
        "    }\n",
        "\n",
        "    return combined_metrics"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 26,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9jJYTWsHjnEc",
        "outputId": "fe45691a-4476-4978-89b8-15f36465c37c"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Name: accelerateNote: you may need to restart the kernel to use updated packages.\n",
            "\n",
            "Version: 0.31.0\n",
            "Summary: Accelerate\n",
            "Home-page: https://github.com/huggingface/accelerate\n",
            "Author: The HuggingFace team\n",
            "Author-email: zach.mueller@huggingface.co\n",
            "License: Apache\n",
            "Location: c:\\Users\\andre\\1JUPYTER\\dt_labs\\.venv\\Lib\\site-packages\n",
            "Requires: huggingface-hub, numpy, packaging, psutil, pyyaml, safetensors, torch\n",
            "Required-by: \n",
            "---\n",
            "Name: transformers\n",
            "Version: 4.41.2\n",
            "Summary: State-of-the-art Machine Learning for JAX, PyTorch and TensorFlow\n",
            "Home-page: https://github.com/huggingface/transformers\n",
            "Author: The Hugging Face team (past and future) with the help of all our contributors (https://github.com/huggingface/transformers/graphs/contributors)\n",
            "Author-email: transformers@huggingface.co\n",
            "License: Apache 2.0 License\n",
            "Location: c:\\Users\\andre\\1JUPYTER\\dt_labs\\.venv\\Lib\\site-packages\n",
            "Requires: filelock, huggingface-hub, numpy, packaging, pyyaml, regex, requests, safetensors, tokenizers, tqdm\n",
            "Required-by: \n"
          ]
        }
      ],
      "source": [
        "pip show accelerate transformers"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Treinamento do modelo"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 27,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QlaLCwf7rLtp",
        "outputId": "7e10e82a-8bc7-478b-851e-c7b628b46c41"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "c:\\Users\\andre\\1JUPYTER\\dt_labs\\.venv\\Lib\\site-packages\\transformers\\training_args.py:1474: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead\n",
            "  warnings.warn(\n"
          ]
        }
      ],
      "source": [
        "from transformers import TrainingArguments, Trainer\n",
        "\n",
        "EPOCHS = 1\n",
        "\n",
        "training_args = TrainingArguments(output_dir=\"./result\",\n",
        "                                  evaluation_strategy=\"epoch\",\n",
        "                                  num_train_epochs= EPOCHS,\n",
        "                                  per_device_train_batch_size=16,\n",
        "                                  per_device_eval_batch_size=8\n",
        "                                )\n",
        "\n",
        "trainer = Trainer(\n",
        "    model=model,\n",
        "    args=training_args,\n",
        "    train_dataset=train_hg,\n",
        "    eval_dataset=valid_hg,\n",
        "    tokenizer=tokenizer,\n",
        "    compute_metrics=compute_metrics\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 28,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "CUDA available:  True\n",
            "CUDA version:  12.1\n"
          ]
        }
      ],
      "source": [
        "print(\"CUDA available: \", torch.cuda.is_available())\n",
        "print(\"CUDA version: \", torch.version.cuda)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 29,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 141
        },
        "id": "3s6lVFz_rLwO",
        "outputId": "ee64e8e9-9c8c-42a8-c355-f51410cc33df"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "  0%|          | 0/2500 [00:00<?, ?it/s]c:\\Users\\andre\\1JUPYTER\\dt_labs\\.venv\\Lib\\site-packages\\transformers\\models\\bert\\modeling_bert.py:435: UserWarning: 1Torch was not compiled with flash attention. (Triggered internally at ..\\aten\\src\\ATen\\native\\transformers\\cuda\\sdp_utils.cpp:263.)\n",
            "  attn_output = torch.nn.functional.scaled_dot_product_attention(\n",
            " 20%|██        | 500/2500 [05:35<22:22,  1.49it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{'loss': 0.4994, 'grad_norm': 12.613661766052246, 'learning_rate': 4e-05, 'epoch': 0.2}\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            " 40%|████      | 1000/2500 [11:13<16:46,  1.49it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{'loss': 0.3898, 'grad_norm': 4.661791801452637, 'learning_rate': 3e-05, 'epoch': 0.4}\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            " 60%|██████    | 1500/2500 [16:47<11:02,  1.51it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{'loss': 0.3516, 'grad_norm': 1.5203113555908203, 'learning_rate': 2e-05, 'epoch': 0.6}\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            " 80%|████████  | 2000/2500 [22:25<05:33,  1.50it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{'loss': 0.3121, 'grad_norm': 8.331348419189453, 'learning_rate': 1e-05, 'epoch': 0.8}\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 2500/2500 [28:04<00:00,  1.50it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{'loss': 0.2882, 'grad_norm': 6.287994861602783, 'learning_rate': 0.0, 'epoch': 1.0}\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "                                                   \n",
            "100%|██████████| 2500/2500 [30:45<00:00,  1.35it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{'eval_loss': 0.283893883228302, 'eval_accuracy': 0.883, 'eval_f1': 0.8829425082505502, 'eval_runtime': 159.717, 'eval_samples_per_second': 62.611, 'eval_steps_per_second': 7.826, 'epoch': 1.0}\n",
            "{'train_runtime': 1845.2907, 'train_samples_per_second': 21.677, 'train_steps_per_second': 1.355, 'train_loss': 0.3682089477539062, 'epoch': 1.0}\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "TrainOutput(global_step=2500, training_loss=0.3682089477539062, metrics={'train_runtime': 1845.2907, 'train_samples_per_second': 21.677, 'train_steps_per_second': 1.355, 'total_flos': 4111110240000000.0, 'train_loss': 0.3682089477539062, 'epoch': 1.0})"
            ]
          },
          "execution_count": 29,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "trainer.train()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Salvando o modelo"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 38,
      "metadata": {
        "id": "8eO6WDiOBAhg"
      },
      "outputs": [],
      "source": [
        "torch.save(model.state_dict(), 'model.pth')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FtVZztSa40b3"
      },
      "source": [
        "## Teste de predições individuais"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 34,
      "metadata": {
        "id": "lOHVSyfJJ8zK"
      },
      "outputs": [],
      "source": [
        "from transformers import AutoTokenizer\n",
        "\n",
        "new_tokenizer = AutoTokenizer.from_pretrained(pre_trained_base)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 35,
      "metadata": {
        "id": "t-T7hDZ2J1Qk"
      },
      "outputs": [],
      "source": [
        "def get_prediction(text):\n",
        "    encoding = new_tokenizer(text, return_tensors=\"pt\", padding=\"max_length\", truncation=True, max_length=MAX_LEN)\n",
        "    encoding = {k: v.to(trainer.model.device) for k,v in encoding.items()}\n",
        "\n",
        "    outputs = model(**encoding)\n",
        "\n",
        "    logits = outputs.logits\n",
        "\n",
        "    sigmoid = torch.nn.Sigmoid()\n",
        "    probs = sigmoid(logits.squeeze().cpu())\n",
        "    probs = probs.detach().numpy()\n",
        "    label = np.argmax(probs, axis=-1)\n",
        "\n",
        "    return label"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 36,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "y4dxQ4oYJ5C1",
        "outputId": "d0d77c2d-aff6-412b-e22a-0b721f5b097e"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "0"
            ]
          },
          "execution_count": 36,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "get_prediction(\"This movie is horrible!\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 37,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JXAyOu_6AqoO",
        "outputId": "ffcd019e-4c0c-45eb-f538-d2860c53a0e0"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "1"
            ]
          },
          "execution_count": 37,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "get_prediction(\"This movie is awesome!\")"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "provenance": []
    },
    "gpuClass": "standard",
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.11"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}