text_loaders_splitters.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ae222ed8",
   "metadata": {},
   "source": [
    "## Document Loaders In LangChain"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e65dada0",
   "metadata": {},
   "source": [
    "#### TextLoader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "470ed0a2",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(page_content=\"The stock of NVIDIA Corp (NASDAQ:NVDA) experienced a daily loss of -3.56% and a 3-month gain of 32.35%. With an Earnings Per Share (EPS) (EPS) of $1.92, the question arises: is the stock significantly overvalued? This article aims to provide a detailed valuation analysis of NVIDIA, offering insights into its financial strength, profitability, growth, and more. We invite you to delve into this comprehensive analysis.\\n\\nCompany Overview\\nWarning! GuruFocus has detected 10 Warning Signs with NVDA. Click here to check it out.\\n\\nNVDA 30-Year Financial Data\\n\\nThe intrinsic value of NVDA\\n\\n\\nNVIDIA Corp (NASDAQ:NVDA) is a leading designer of discrete graphics processing units that enhance the experience on computing platforms. The firm's chips are widely used in various end markets, including PC gaming and data centers. In recent years, NVIDIA has broadened its focus from traditional PC graphics applications such as gaming to more complex and favorable opportunities, including artificial intelligence and autonomous driving, leveraging the high-performance capabilities of its products.\\n\\nCurrently, NVIDIA's stock price stands at $418.01, significantly higher than the GF Value of $310.28, indicating the stock might be overvalued. With a market cap of $1 trillion, the valuation seems steep. The following analysis aims to delve deeper into the company's value.\\n\\nIs NVIDIA's Stock Significantly Overvalued? A Comprehensive Valuation Analysis\\nIs NVIDIA's Stock Significantly Overvalued? A Comprehensive Valuation Analysis\\nUnderstanding the GF Value\\nThe GF Value is a unique measure of the intrinsic value of a stock, calculated based on historical trading multiples, a GuruFocus adjustment factor, and future business performance estimates. If the stock price is significantly above the GF Value Line, it is overvalued, and its future return is likely to be poor. Conversely, if it is significantly below the GF Value Line, its future return will likely be higher.\\n\\nAccording to GuruFocus Value calculation, NVIDIA (NASDAQ:NVDA) appears to be significantly overvalued. The stock's current price of $418.01 per share and the market cap of $1 trillion further strengthen this assumption.\\n\\nGiven that NVIDIA is significantly overvalued, the long-term return of its stock is likely to be much lower than its future business growth.\\n\\nIs NVIDIA's Stock Significantly Overvalued? A Comprehensive Valuation Analysis\\nIs NVIDIA's Stock Significantly Overvalued? A Comprehensive Valuation Analysis\\nLink: These companies may deliver higher future returns at reduced risk.\\n\\nFinancial Strength of NVIDIA\\nExamining the financial strength of a company is crucial before investing in its stock. Companies with poor financial strength pose a higher risk of permanent loss. NVIDIA's cash-to-debt ratio of 1.27 is worse than 58.04% of companies in the Semiconductors industry. However, NVIDIA's overall financial strength is 8 out of 10, indicating a strong financial position.\\n\\nIs NVIDIA's Stock Significantly Overvalued? A Comprehensive Valuation Analysis\\nIs NVIDIA's Stock Significantly Overvalued? A Comprehensive Valuation Analysis\\nProfitability and Growth\\nConsistent profitability over the long term reduces the risk for investors. NVIDIA, with its profitability ranking of 10 out of 10, has been profitable for the past 10 years. The company's operating margin of 17.37% ranks better than 76.5% of companies in the Semiconductors industry.\\n\\nHowever, growth is a crucial factor in a company's valuation. NVIDIA's growth ranks worse than 52.99% of companies in the Semiconductors industry, with its 3-year average revenue growth rate better than 87.88% of companies in the industry.\\n\\nROIC vs WACC\\nComparing a company's return on invested capital (ROIC) to its weighted average cost of capital (WACC) is an effective way to evaluate its profitability. Over the past 12 months, NVIDIA's ROIC was 20.32 while its WACC was 16.74, suggesting that the company is creating value for its shareholders.\\n\\nIs NVIDIA's Stock Significantly Overvalued? A Comprehensive Valuation Analysis\\nIs NVIDIA's Stock Significantly Overvalued? A Comprehensive Valuation Analysis\\nConclusion\\nIn conclusion, NVIDIA (NASDAQ:NVDA) appears to be significantly overvalued. Despite its strong financial condition and profitability, its growth ranks lower than 52.99% of companies in the Semiconductors industry. To learn more about NVIDIA stock, you can check out its 30-Year Financials here.\\n\\nTo find out the high quality companies that may deliver above-average returns, please check out GuruFocus High Quality Low Capex Screener.\\n\\nThis article first appeared on GuruFocus.\", metadata={'source': 'nvda_news_1.txt'})]"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from langchain.document_loaders import TextLoader\n",
    "\n",
    "loader = TextLoader(\"nvda_news_1.txt\")\n",
    "loader.load()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "ca5844b4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "langchain.document_loaders.text.TextLoader"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(loader)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "bb95e130",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'nvda_news_1.txt'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loader.file_path"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4da25ff7",
   "metadata": {},
   "source": [
    "#### CSVLoader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ad67ba6e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.document_loaders.csv_loader import CSVLoader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "220b3fcb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(page_content='movie_id: 101\\ntitle: K.G.F: Chapter 2\\nindustry: Bollywood\\nrelease_year: 2022\\nimdb_rating: 8.4\\nstudio: Hombale Films\\nlanguage_id: 3\\nbudget: 1\\nrevenue: 12.5\\nunit: Billions\\ncurrency: INR', metadata={'source': 'movies.csv', 'row': 0}),\n",
       " Document(page_content='movie_id: 102\\ntitle: Doctor Strange in the Multiverse of Madness\\nindustry: Hollywood\\nrelease_year: 2022\\nimdb_rating: 7\\nstudio: Marvel Studios\\nlanguage_id: 5\\nbudget: 200\\nrevenue: 954.8\\nunit: Millions\\ncurrency: USD', metadata={'source': 'movies.csv', 'row': 1}),\n",
       " Document(page_content='movie_id: 103\\ntitle: Thor: The Dark World\\nindustry: Hollywood\\nrelease_year: 2013\\nimdb_rating: 6.8\\nstudio: Marvel Studios\\nlanguage_id: 5\\nbudget: 165\\nrevenue: 644.8\\nunit: Millions\\ncurrency: USD', metadata={'source': 'movies.csv', 'row': 2}),\n",
       " Document(page_content='movie_id: 104\\ntitle: Thor: Ragnarok\\nindustry: Hollywood\\nrelease_year: 2017\\nimdb_rating: 7.9\\nstudio: Marvel Studios\\nlanguage_id: 5\\nbudget: 180\\nrevenue: 854\\nunit: Millions\\ncurrency: USD', metadata={'source': 'movies.csv', 'row': 3}),\n",
       " Document(page_content='movie_id: 105\\ntitle: Thor: Love and Thunder\\nindustry: Hollywood\\nrelease_year: 2022\\nimdb_rating: 6.8\\nstudio: Marvel Studios\\nlanguage_id: 5\\nbudget: 250\\nrevenue: 670\\nunit: Millions\\ncurrency: USD', metadata={'source': 'movies.csv', 'row': 4}),\n",
       " Document(page_content='movie_id: 106\\ntitle: Sholay\\nindustry: Bollywood\\nrelease_year: 1975\\nimdb_rating: 8.1\\nstudio: United Producers\\nlanguage_id: 1\\nbudget: Not Available\\nrevenue: Not Available\\nunit: Not Available\\ncurrency: Not Available', metadata={'source': 'movies.csv', 'row': 5}),\n",
       " Document(page_content='movie_id: 107\\ntitle: Dilwale Dulhania Le Jayenge\\nindustry: Bollywood\\nrelease_year: 1995\\nimdb_rating: 8\\nstudio: Yash Raj Films\\nlanguage_id: 1\\nbudget: 400\\nrevenue: 2000\\nunit: Millions\\ncurrency: INR', metadata={'source': 'movies.csv', 'row': 6}),\n",
       " Document(page_content='movie_id: 108\\ntitle: 3 Idiots\\nindustry: Bollywood\\nrelease_year: 2009\\nimdb_rating: 8.4\\nstudio: Vinod Chopra Films\\nlanguage_id: 1\\nbudget: 550\\nrevenue: 4000\\nunit: Millions\\ncurrency: INR', metadata={'source': 'movies.csv', 'row': 7}),\n",
       " Document(page_content='movie_id: 109\\ntitle: Kabhi Khushi Kabhie Gham\\nindustry: Bollywood\\nrelease_year: 2001\\nimdb_rating: 7.4\\nstudio: Dharma Productions\\nlanguage_id: 1\\nbudget: 390\\nrevenue: 1360\\nunit: Millions\\ncurrency: INR', metadata={'source': 'movies.csv', 'row': 8})]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loader = CSVLoader(file_path=\"movies.csv\")\n",
    "data = loader.load()\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "4e1b3b9a",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Document(page_content='movie_id: 101\\ntitle: K.G.F: Chapter 2\\nindustry: Bollywood\\nrelease_year: 2022\\nimdb_rating: 8.4\\nstudio: Hombale Films\\nlanguage_id: 3\\nbudget: 1\\nrevenue: 12.5\\nunit: Billions\\ncurrency: INR', metadata={'source': 'movies.csv', 'row': 0})"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "e356cd2b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(page_content='movie_id: 101\\ntitle: K.G.F: Chapter 2\\nindustry: Bollywood\\nrelease_year: 2022\\nimdb_rating: 8.4\\nstudio: Hombale Films\\nlanguage_id: 3\\nbudget: 1\\nrevenue: 12.5\\nunit: Billions\\ncurrency: INR', metadata={'source': 'K.G.F: Chapter 2', 'row': 0}),\n",
       " Document(page_content='movie_id: 102\\ntitle: Doctor Strange in the Multiverse of Madness\\nindustry: Hollywood\\nrelease_year: 2022\\nimdb_rating: 7\\nstudio: Marvel Studios\\nlanguage_id: 5\\nbudget: 200\\nrevenue: 954.8\\nunit: Millions\\ncurrency: USD', metadata={'source': 'Doctor Strange in the Multiverse of Madness', 'row': 1}),\n",
       " Document(page_content='movie_id: 103\\ntitle: Thor: The Dark World\\nindustry: Hollywood\\nrelease_year: 2013\\nimdb_rating: 6.8\\nstudio: Marvel Studios\\nlanguage_id: 5\\nbudget: 165\\nrevenue: 644.8\\nunit: Millions\\ncurrency: USD', metadata={'source': 'Thor: The Dark World', 'row': 2}),\n",
       " Document(page_content='movie_id: 104\\ntitle: Thor: Ragnarok\\nindustry: Hollywood\\nrelease_year: 2017\\nimdb_rating: 7.9\\nstudio: Marvel Studios\\nlanguage_id: 5\\nbudget: 180\\nrevenue: 854\\nunit: Millions\\ncurrency: USD', metadata={'source': 'Thor: Ragnarok', 'row': 3}),\n",
       " Document(page_content='movie_id: 105\\ntitle: Thor: Love and Thunder\\nindustry: Hollywood\\nrelease_year: 2022\\nimdb_rating: 6.8\\nstudio: Marvel Studios\\nlanguage_id: 5\\nbudget: 250\\nrevenue: 670\\nunit: Millions\\ncurrency: USD', metadata={'source': 'Thor: Love and Thunder', 'row': 4}),\n",
       " Document(page_content='movie_id: 106\\ntitle: Sholay\\nindustry: Bollywood\\nrelease_year: 1975\\nimdb_rating: 8.1\\nstudio: United Producers\\nlanguage_id: 1\\nbudget: Not Available\\nrevenue: Not Available\\nunit: Not Available\\ncurrency: Not Available', metadata={'source': 'Sholay', 'row': 5}),\n",
       " Document(page_content='movie_id: 107\\ntitle: Dilwale Dulhania Le Jayenge\\nindustry: Bollywood\\nrelease_year: 1995\\nimdb_rating: 8\\nstudio: Yash Raj Films\\nlanguage_id: 1\\nbudget: 400\\nrevenue: 2000\\nunit: Millions\\ncurrency: INR', metadata={'source': 'Dilwale Dulhania Le Jayenge', 'row': 6}),\n",
       " Document(page_content='movie_id: 108\\ntitle: 3 Idiots\\nindustry: Bollywood\\nrelease_year: 2009\\nimdb_rating: 8.4\\nstudio: Vinod Chopra Films\\nlanguage_id: 1\\nbudget: 550\\nrevenue: 4000\\nunit: Millions\\ncurrency: INR', metadata={'source': '3 Idiots', 'row': 7}),\n",
       " Document(page_content='movie_id: 109\\ntitle: Kabhi Khushi Kabhie Gham\\nindustry: Bollywood\\nrelease_year: 2001\\nimdb_rating: 7.4\\nstudio: Dharma Productions\\nlanguage_id: 1\\nbudget: 390\\nrevenue: 1360\\nunit: Millions\\ncurrency: INR', metadata={'source': 'Kabhi Khushi Kabhie Gham', 'row': 8})]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loader = CSVLoader(file_path=\"movies.csv\", source_column=\"title\")\n",
    "data = loader.load()\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "42d13fd6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'movie_id: 101\\ntitle: K.G.F: Chapter 2\\nindustry: Bollywood\\nrelease_year: 2022\\nimdb_rating: 8.4\\nstudio: Hombale Films\\nlanguage_id: 3\\nbudget: 1\\nrevenue: 12.5\\nunit: Billions\\ncurrency: INR'"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[0].page_content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "0d8913df",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'source': 'K.G.F: Chapter 2', 'row': 0}"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[0].metadata"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0163924",
   "metadata": {},
   "source": [
    "#### UnstructuredURLLoader"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48644959",
   "metadata": {},
   "source": [
    "UnstructuredURLLoader of Langchain internally uses unstructured python library to load the content from url's\n",
    "\n",
    "https://unstructured-io.github.io/unstructured/introduction.html\n",
    "\n",
    "https://pypi.org/project/unstructured/#description"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "2e1fbf77",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: unstructured in c:\\users\\dhava\\appdata\\local\\programs\\python\\python310\\lib\\site-packages (0.9.2)\n",
      "Requirement already satisfied: libmagic in c:\\users\\dhava\\appdata\\local\\programs\\python\\python310\\lib\\site-packages (1.0)\n",
      "Requirement already satisfied: python-magic in c:\\users\\dhava\\appdata\\local\\programs\\python\\python310\\lib\\site-packages (0.4.27)\n",
      "Requirement already satisfied: python-magic-bin in c:\\users\\dhava\\appdata\\local\\programs\\python\\python310\\lib\\site-packages (0.4.14)\n",
      "Requirement already satisfied: tabulate in c:\\users\\dhava\\appdata\\local\\programs\\python\\python310\\lib\\site-packages (from unstructured) (0.9.0)\n",
      "Requirement already satisfied: requests in c:\\users\\dhava\\appdata\\local\\programs\\python\\python310\\lib\\site-packages (from unstructured) (2.31.0)\n",
      "Requirement already satisfied: nltk in c:\\users\\dhava\\appdata\\local\\programs\\python\\python310\\lib\\site-packages (from unstructured) (3.8.1)\n",
      "Requirement already satisfied: chardet in c:\\users\\dhava\\appdata\\local\\programs\\python\\python310\\lib\\site-packages (from unstructured) (5.2.0)\n",
      "Requirement already satisfied: filetype in c:\\users\\dhava\\appdata\\local\\programs\\python\\python310\\lib\\site-packages (from unstructured) (1.2.0)\n",
      "Requirement already satisfied: lxml in c:\\users\\dhava\\appdata\\local\\programs\\python\\python310\\lib\\site-packages (from unstructured) (4.9.3)\n",
      "Requirement already satisfied: click in c:\\users\\dhava\\appdata\\local\\programs\\python\\python310\\lib\\site-packages (from nltk->unstructured) (8.1.3)\n",
      "Requirement already satisfied: tqdm in c:\\users\\dhava\\appdata\\local\\programs\\python\\python310\\lib\\site-packages (from nltk->unstructured) (4.65.0)\n",
      "Requirement already satisfied: regex>=2021.8.3 in c:\\users\\dhava\\appdata\\local\\programs\\python\\python310\\lib\\site-packages (from nltk->unstructured) (2023.8.8)\n",
      "Requirement already satisfied: joblib in c:\\users\\dhava\\appdata\\local\\programs\\python\\python310\\lib\\site-packages (from nltk->unstructured) (1.3.2)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in c:\\users\\dhava\\appdata\\local\\programs\\python\\python310\\lib\\site-packages (from requests->unstructured) (2.0.3)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in c:\\users\\dhava\\appdata\\local\\programs\\python\\python310\\lib\\site-packages (from requests->unstructured) (2023.5.7)\n",
      "Requirement already satisfied: idna<4,>=2.5 in c:\\users\\dhava\\appdata\\local\\programs\\python\\python310\\lib\\site-packages (from requests->unstructured) (3.4)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in c:\\users\\dhava\\appdata\\local\\programs\\python\\python310\\lib\\site-packages (from requests->unstructured) (3.1.0)\n",
      "Requirement already satisfied: colorama in c:\\users\\dhava\\appdata\\local\\programs\\python\\python310\\lib\\site-packages (from click->nltk->unstructured) (0.4.6)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "[notice] A new release of pip is available: 23.0.1 -> 23.2.1\n",
      "[notice] To update, run: python.exe -m pip install --upgrade pip\n"
     ]
    }
   ],
   "source": [
    "#installing necessary libraries, libmagic is used for file type detection\n",
    "!pip3 install unstructured libmagic python-magic python-magic-bin"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "9bfca934",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.document_loaders import UnstructuredURLLoader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "93ce75de",
   "metadata": {},
   "outputs": [],
   "source": [
    "loader = UnstructuredURLLoader(\n",
    "    urls = [\n",
    "        \"https://www.moneycontrol.com/news/business/banks/hdfc-bank-re-appoints-sanmoy-chakrabarti-as-chief-risk-officer-11259771.html\",\n",
    "        \"https://www.moneycontrol.com/news/business/markets/market-corrects-post-rbi-ups-inflation-forecast-icrr-bet-on-these-top-10-rate-sensitive-stocks-ideas-11142611.html\"\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "be914bfa",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = loader.load()\n",
    "len(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "52838711",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'English\\n\\nHindi\\n\\nGujarati\\n\\nSpecials\\n\\nTrending Stocks\\n\\nIRFC\\xa0INE053F01010, IRFC, 543257\\n\\nTata Power\\xa0INE'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[0].page_content[0:100]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "8b587350",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'source': 'https://www.moneycontrol.com/news/business/banks/hdfc-bank-re-appoints-sanmoy-chakrabarti-as-chief-risk-officer-11259771.html'}"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[0].metadata"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df45adf4",
   "metadata": {},
   "source": [
    "## Text Splitters"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ce2e736",
   "metadata": {},
   "source": [
    "Why do we need text splitters in first place?\n",
    "\n",
    "LLM's have token limits. Hence we need to split the text which can be large into small chunks so that each chunk size is under the token limit. There are various text splitter classes in langchain that allows us to do this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "id": "90431c5b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Taking some random text from wikipedia\n",
    "\n",
    "text = \"\"\"Interstellar is a 2014 epic science fiction film co-written, directed, and produced by Christopher Nolan. \n",
    "It stars Matthew McConaughey, Anne Hathaway, Jessica Chastain, Bill Irwin, Ellen Burstyn, Matt Damon, and Michael Caine. \n",
    "Set in a dystopian future where humanity is embroiled in a catastrophic blight and famine, the film follows a group of astronauts who travel through a wormhole near Saturn in search of a new home for humankind.\n",
    "\n",
    "Brothers Christopher and Jonathan Nolan wrote the screenplay, which had its origins in a script Jonathan developed in 2007 and was originally set to be directed by Steven Spielberg. \n",
    "Kip Thorne, a Caltech theoretical physicist and 2017 Nobel laureate in Physics,[4] was an executive producer, acted as a scientific consultant, and wrote a tie-in book, The Science of Interstellar. \n",
    "Cinematographer Hoyte van Hoytema shot it on 35 mm movie film in the Panavision anamorphic format and IMAX 70 mm. Principal photography began in late 2013 and took place in Alberta, Iceland, and Los Angeles. \n",
    "Interstellar uses extensive practical and miniature effects, and the company Double Negative created additional digital effects.\n",
    "\n",
    "Interstellar premiered in Los Angeles on October 26, 2014. In the United States, it was first released on film stock, expanding to venues using digital projectors. The film received generally positive reviews from critics and grossed over $677 million worldwide ($715 million after subsequent re-releases), making it the tenth-highest-grossing film of 2014. \n",
    "It has been praised by astronomers for its scientific accuracy and portrayal of theoretical astrophysics.[5][6][7] Interstellar was nominated for five awards at the 87th Academy Awards, winning Best Visual Effects, and received numerous other accolades.\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7af9a95b",
   "metadata": {},
   "source": [
    "#### Manual approach of splitting the text into chunks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "id": "a51fc70e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Interstellar is a 2014 epic science fiction film co-written, directed, and produced by Christopher N'"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Say LLM token limit is 100, in that case we can do simple thing such as this\n",
    "\n",
    "text[0:100]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "id": "2adae99a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "264"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Well but we want complete words and want to do this for entire text, may be we can use Python's split funciton\n",
    "\n",
    "words = text.split(\" \")\n",
    "len(words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "id": "56ec5613",
   "metadata": {},
   "outputs": [],
   "source": [
    "chunks = []\n",
    "\n",
    "s = \"\"\n",
    "for word in words:\n",
    "    s += word + \" \"\n",
    "    if len(s)>200:\n",
    "        chunks.append(s)\n",
    "        s = \"\"\n",
    "        \n",
    "chunks.append(s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "id": "95d902bd",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Interstellar is a 2014 epic science fiction film co-written, directed, and produced by Christopher Nolan. \\nIt stars Matthew McConaughey, Anne Hathaway, Jessica Chastain, Bill Irwin, Ellen Burstyn, Matt ',\n",
       " 'Damon, and Michael Caine. \\nSet in a dystopian future where humanity is embroiled in a catastrophic blight and famine, the film follows a group of astronauts who travel through a wormhole near Saturn in ']"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chunks[:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff06ebc4",
   "metadata": {},
   "source": [
    "**Splitting data into chunks can be done in native python but it is a tidious process. Also if necessary, you may need to experiment with various delimiters in an iterative manner to ensure that each chunk does not exceed the token length limit of the respective LLM.**\n",
    "\n",
    "**Langchain provides a better way through text splitter classes.**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f64b2909",
   "metadata": {},
   "source": [
    "#### Using Text Splitter Classes from Langchain\n",
    "\n",
    "#### CharacterTextSplitter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "id": "9505bedd",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.text_splitter import CharacterTextSplitter\n",
    "\n",
    "splitter = CharacterTextSplitter(\n",
    "    separator = \"\\n\",\n",
    "    chunk_size=200,\n",
    "    chunk_overlap=0\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "id": "4d86bc63",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Created a chunk of size 210, which is longer than the specified 200\n",
      "Created a chunk of size 208, which is longer than the specified 200\n",
      "Created a chunk of size 358, which is longer than the specified 200\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "9"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chunks = splitter.split_text(text)\n",
    "len(chunks)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "id": "e027b9de",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "105\n",
      "120\n",
      "210\n",
      "181\n",
      "197\n",
      "207\n",
      "128\n",
      "357\n",
      "253\n"
     ]
    }
   ],
   "source": [
    "for chunk in chunks:\n",
    "    print(len(chunk))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dcc73da3",
   "metadata": {},
   "source": [
    "As you can see, all though we gave 200 as a chunk size since the split was based on \\n, it ended up creating chunks that are bigger than size 200. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f86f3a10",
   "metadata": {},
   "source": [
    "Another class from Langchain can be used to recursively split the text based on a list of separators. This class is RecursiveTextSplitter. Let's see how it works"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00a61cf1",
   "metadata": {},
   "source": [
    "#### RecursiveTextSplitter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "id": "dacf5e8b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Interstellar is a 2014 epic science fiction film co-written, directed, and produced by Christopher Nolan. \\nIt stars Matthew McConaughey, Anne Hathaway, Jessica Chastain, Bill Irwin, Ellen Burstyn, Matt Damon, and Michael Caine. \\nSet in a dystopian future where humanity is embroiled in a catastrophic blight and famine, the film follows a group of astronauts who travel through a wormhole near Saturn in search of a new home for humankind.\\n\\nBrothers Christopher and Jonathan Nolan wrote the screenplay, which had its origins in a script Jonathan developed in 2007 and was originally set to be directed by Steven Spielberg. \\nKip Thorne, a Caltech theoretical physicist and 2017 Nobel laureate in Physics,[4] was an executive producer, acted as a scientific consultant, and wrote a tie-in book, The Science of Interstellar. \\nCinematographer Hoyte van Hoytema shot it on 35 mm movie film in the Panavision anamorphic format and IMAX 70 mm. Principal photography began in late 2013 and took place in Alberta, Iceland, and Los Angeles. \\nInterstellar uses extensive practical and miniature effects, and the company Double Negative created additional digital effects.\\n\\nInterstellar premiered in Los Angeles on October 26, 2014. In the United States, it was first released on film stock, expanding to venues using digital projectors. The film received generally positive reviews from critics and grossed over $677 million worldwide ($715 million after subsequent re-releases), making it the tenth-highest-grossing film of 2014. \\nIt has been praised by astronomers for its scientific accuracy and portrayal of theoretical astrophysics.[5][6][7] Interstellar was nominated for five awards at the 87th Academy Awards, winning Best Visual Effects, and received numerous other accolades.'"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "id": "848eae34",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
    "\n",
    "r_splitter = RecursiveCharacterTextSplitter(\n",
    "    separators = [\"\\n\\n\", \"\\n\", \" \"],  # List of separators based on requirement (defaults to [\"\\n\\n\", \"\\n\", \" \"])\n",
    "    chunk_size = 200,  # size of each chunk created\n",
    "    chunk_overlap  = 0,  # size of  overlap between chunks in order to maintain the context\n",
    "    length_function = len  # Function to calculate size, currently we are using \"len\" which denotes length of string however you can pass any token counter)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "id": "1151c51d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "105\n",
      "120\n",
      "199\n",
      "10\n",
      "181\n",
      "197\n",
      "198\n",
      "8\n",
      "128\n",
      "191\n",
      "165\n",
      "198\n",
      "54\n"
     ]
    }
   ],
   "source": [
    "chunks = r_splitter.split_text(text)\n",
    "\n",
    "for chunk in chunks:\n",
    "    print(len(chunk))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32135f4d",
   "metadata": {},
   "source": [
    "**Let's understand how exactly it formed these chunks**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "id": "57ef6974",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Interstellar is a 2014 epic science fiction film co-written, directed, and produced by Christopher Nolan. \\nIt stars Matthew McConaughey, Anne Hathaway, Jessica Chastain, Bill Irwin, Ellen Burstyn, Matt Damon, and Michael Caine. \\nSet in a dystopian future where humanity is embroiled in a catastrophic blight and famine, the film follows a group of astronauts who travel through a wormhole near Saturn in search of a new home for humankind.'"
      ]
     },
     "execution_count": 109,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "first_split = text.split(\"\\n\\n\")[0]\n",
    "first_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "id": "2bc7719f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "439"
      ]
     },
     "execution_count": 110,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(first_split)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1018d24",
   "metadata": {},
   "source": [
    "Recursive text splitter uses a list of separators, i.e.  separators = [\"\\n\\n\", \"\\n\", \".\"]\n",
    "\n",
    "So now it will first split using \\n\\n and then if the resulting chunk size is greater than the chunk_size parameter which is 200\n",
    "in our case, then it will use the next separator which is \\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "id": "739cef71",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Interstellar is a 2014 epic science fiction film co-written, directed, and produced by Christopher Nolan. ',\n",
       " 'It stars Matthew McConaughey, Anne Hathaway, Jessica Chastain, Bill Irwin, Ellen Burstyn, Matt Damon, and Michael Caine. ',\n",
       " 'Set in a dystopian future where humanity is embroiled in a catastrophic blight and famine, the film follows a group of astronauts who travel through a wormhole near Saturn in search of a new home for humankind.']"
      ]
     },
     "execution_count": 119,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "second_split = first_split.split(\"\\n\")\n",
    "second_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "id": "903f5921",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "106\n",
      "121\n",
      "210\n"
     ]
    }
   ],
   "source": [
    "for split in second_split:\n",
    "    print(len(split))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed1e69a7",
   "metadata": {},
   "source": [
    "Third split exceeds chunk size 200. Now it will further try to split that using the third separator which is ' ' (space)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "id": "69f4da9d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Set in a dystopian future where humanity is embroiled in a catastrophic blight and famine, the film follows a group of astronauts who travel through a wormhole near Saturn in search of a new home for humankind.'"
      ]
     },
     "execution_count": 115,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "second_split[2]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7fda659a",
   "metadata": {},
   "source": [
    "When you split this using space (i.e. second_split[2].split(\" \")), it will separate out each word and then it will merge those \n",
    "chunks such that their size is close to 200"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f538e50",
   "metadata": {},
   "source": [
    "<img src=\"chunk_size.jpg\"/>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}