{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a05c860c",
   "metadata": {},
   "source": [
    "# How to split text by tokens \n",
    "\n",
    "Language models have a token limit. You should not exceed the token limit. When you split your text into chunks it is therefore a good idea to count the number of tokens. There are many tokenizers. When you count tokens in your text you should use the same tokenizer as used in the language model. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7683b36a",
   "metadata": {},
   "source": [
    "## tiktoken\n",
    "\n",
    ":::{.callout-note}\n",
    "[tiktoken](https://github.com/openai/tiktoken) is a fast `BPE` tokenizer created by `OpenAI`.\n",
    ":::\n",
    "\n",
    "\n",
    "We can use `tiktoken` to estimate tokens used. It will probably be more accurate for the OpenAI models.\n",
    "\n",
    "1. How the text is split: by character passed in.\n",
    "2. How the chunk size is measured: by `tiktoken` tokenizer.\n",
    "\n",
    "[CharacterTextSplitter](https://api.python.langchain.com/en/latest/character/langchain_text_splitters.character.CharacterTextSplitter.html), [RecursiveCharacterTextSplitter](https://api.python.langchain.com/en/latest/character/langchain_text_splitters.character.RecursiveCharacterTextSplitter.html), and [TokenTextSplitter](https://api.python.langchain.com/en/latest/base/langchain_text_splitters.base.TokenTextSplitter.html) can be used with `tiktoken` directly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6c4ef83e-f43a-4658-ad1a-3952e0a5bbe7",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install --upgrade --quiet langchain-text-splitters tiktoken"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "1ad2d0f2",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_text_splitters import CharacterTextSplitter\n",
    "\n",
    "# This is a long document we can split up.\n",
    "with open(\"state_of_the_union.txt\") as f:\n",
    "    state_of_the_union = f.read()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3ba1d8a",
   "metadata": {},
   "source": [
    "To split with a [CharacterTextSplitter](https://api.python.langchain.com/en/latest/character/langchain_text_splitters.character.CharacterTextSplitter.html) and then merge chunks with `tiktoken`, use its `.from_tiktoken_encoder()` method. Note that splits from this method can be larger than the chunk size measured by the `tiktoken` tokenizer.\n",
    "\n",
    "The `.from_tiktoken_encoder()` method takes either `encoding_name` as an argument (e.g. `cl100k_base`), or the `model_name` (e.g. `gpt-4`). All additional arguments like `chunk_size`, `chunk_overlap`, and `separators` are used to instantiate `CharacterTextSplitter`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "825f7c0a",
   "metadata": {},
   "outputs": [],
   "source": [
    "text_splitter = CharacterTextSplitter.from_tiktoken_encoder(\n",
    "    encoding_name=\"cl100k_base\", chunk_size=100, chunk_overlap=0\n",
    ")\n",
    "texts = text_splitter.split_text(state_of_the_union)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "ae35d165",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans.  \n",
      "\n",
      "Last year COVID-19 kept us apart. This year we are finally together again. \n",
      "\n",
      "Tonight, we meet as Democrats Republicans and Independents. But most importantly as Americans. \n",
      "\n",
      "With a duty to one another to the American people to the Constitution.\n"
     ]
    }
   ],
   "source": [
    "print(texts[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de5b6a6e",
   "metadata": {},
   "source": [
    "To implement a hard constraint on the chunk size, we can use `RecursiveCharacterTextSplitter.from_tiktoken_encoder`, where each split will be recursively split if it has a larger size:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "0262a991",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_text_splitters import RecursiveCharacterTextSplitter\n",
    "\n",
    "text_splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(\n",
    "    model_name=\"gpt-4\",\n",
    "    chunk_size=100,\n",
    "    chunk_overlap=0,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04457e3a",
   "metadata": {},
   "source": [
    "We can also load a `TokenTextSplitter` splitter, which works with `tiktoken` directly and will ensure each split is smaller than chunk size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "4454c70e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Madam Speaker, Madam Vice President, our\n"
     ]
    }
   ],
   "source": [
    "from langchain_text_splitters import TokenTextSplitter\n",
    "\n",
    "text_splitter = TokenTextSplitter(chunk_size=10, chunk_overlap=0)\n",
    "\n",
    "texts = text_splitter.split_text(state_of_the_union)\n",
    "print(texts[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3bc155d0",
   "metadata": {},
   "source": [
    "Some written languages (e.g. Chinese and Japanese) have characters which encode to 2 or more tokens. Using the `TokenTextSplitter` directly can split the tokens for a character between two chunks causing malformed Unicode characters. Use `RecursiveCharacterTextSplitter.from_tiktoken_encoder` or `CharacterTextSplitter.from_tiktoken_encoder` to ensure chunks contain valid Unicode strings."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55f95f06",
   "metadata": {},
   "source": [
    "## spaCy\n",
    "\n",
    ":::{.callout-note}\n",
    "[spaCy](https://spacy.io/) is an open-source software library for advanced natural language processing, written in the programming languages Python and Cython.\n",
    ":::\n",
    "\n",
    "LangChain implements splitters based on the [spaCy tokenizer](https://spacy.io/api/tokenizer).\n",
    "\n",
    "1. How the text is split: by `spaCy` tokenizer.\n",
    "2. How the chunk size is measured: by number of characters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d0b9242f-690c-4819-b35a-bb68187281ed",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install --upgrade --quiet  spacy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "f1de7767",
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is a long document we can split up.\n",
    "with open(\"state_of_the_union.txt\") as f:\n",
    "    state_of_the_union = f.read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "cef2b29e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Madam Speaker, Madam Vice President, our First Lady and Second Gentleman.\n",
      "\n",
      "Members of Congress and the Cabinet.\n",
      "\n",
      "Justices of the Supreme Court.\n",
      "\n",
      "My fellow Americans.  \n",
      "\n",
      "\n",
      "\n",
      "Last year COVID-19 kept us apart.\n",
      "\n",
      "This year we are finally together again. \n",
      "\n",
      "\n",
      "\n",
      "Tonight, we meet as Democrats Republicans and Independents.\n",
      "\n",
      "But most importantly as Americans. \n",
      "\n",
      "\n",
      "\n",
      "With a duty to one another to the American people to the Constitution. \n",
      "\n",
      "\n",
      "\n",
      "And with an unwavering resolve that freedom will always triumph over tyranny. \n",
      "\n",
      "\n",
      "\n",
      "Six days ago, Russia’s Vladimir Putin sought to shake the foundations of the free world thinking he could make it bend to his menacing ways.\n",
      "\n",
      "But he badly miscalculated. \n",
      "\n",
      "\n",
      "\n",
      "He thought he could roll into Ukraine and the world would roll over.\n",
      "\n",
      "Instead he met a wall of strength he never imagined. \n",
      "\n",
      "\n",
      "\n",
      "He met the Ukrainian people. \n",
      "\n",
      "\n",
      "\n",
      "From President Zelenskyy to every Ukrainian, their fearlessness, their courage, their determination, inspires the world.\n"
     ]
    }
   ],
   "source": [
    "from langchain_text_splitters import SpacyTextSplitter\n",
    "\n",
    "text_splitter = SpacyTextSplitter(chunk_size=1000)\n",
    "\n",
    "texts = text_splitter.split_text(state_of_the_union)\n",
    "print(texts[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73dbcdb9",
   "metadata": {},
   "source": [
    "## SentenceTransformers\n",
    "\n",
    "The [SentenceTransformersTokenTextSplitter](https://api.python.langchain.com/en/latest/sentence_transformers/langchain_text_splitters.sentence_transformers.SentenceTransformersTokenTextSplitter.html) is a specialized text splitter for use with the sentence-transformer models. The default behaviour is to split the text into chunks that fit the token window of the sentence transformer model that you would like to use.\n",
    "\n",
    "To split text and constrain token counts according to the sentence-transformers tokenizer, instantiate a `SentenceTransformersTokenTextSplitter`. You can optionally specify:\n",
    "\n",
    "- `chunk_overlap`: integer count of token overlap;\n",
    "- `model_name`: sentence-transformer model name, defaulting to `\"sentence-transformers/all-mpnet-base-v2\"`;\n",
    "- `tokens_per_chunk`: desired token count per chunk."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "9dd5419e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2\n"
     ]
    }
   ],
   "source": [
    "from langchain_text_splitters import SentenceTransformersTokenTextSplitter\n",
    "\n",
    "splitter = SentenceTransformersTokenTextSplitter(chunk_overlap=0)\n",
    "text = \"Lorem \"\n",
    "\n",
    "count_start_and_stop_tokens = 2\n",
    "text_token_count = splitter.count_tokens(text=text) - count_start_and_stop_tokens\n",
    "print(text_token_count)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "d7ad2213",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tokens in text to split: 514\n"
     ]
    }
   ],
   "source": [
    "token_multiplier = splitter.maximum_tokens_per_chunk // text_token_count + 1\n",
    "\n",
    "# `text_to_split` does not fit in a single chunk\n",
    "text_to_split = text * token_multiplier\n",
    "\n",
    "print(f\"tokens in text to split: {splitter.count_tokens(text=text_to_split)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "818aea04",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "lorem\n"
     ]
    }
   ],
   "source": [
    "text_chunks = splitter.split_text(text=text_to_split)\n",
    "\n",
    "print(text_chunks[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea2973ac",
   "metadata": {},
   "source": [
    "## NLTK\n",
    "\n",
    ":::{.callout-note}\n",
    "[The Natural Language Toolkit](https://en.wikipedia.org/wiki/Natural_Language_Toolkit), or more commonly [NLTK](https://www.nltk.org/), is a suite of libraries and programs for symbolic and statistical natural language processing (NLP) for English written in the Python programming language.\n",
    ":::\n",
    "\n",
    "\n",
    "Rather than just splitting on \"\\n\\n\", we can use `NLTK` to split based on [NLTK tokenizers](https://www.nltk.org/api/nltk.tokenize.html).\n",
    "\n",
    "1. How the text is split: by `NLTK` tokenizer.\n",
    "2. How the chunk size is measured: by number of characters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6af9886-7d53-4aab-84f6-303c4cce7882",
   "metadata": {},
   "outputs": [],
   "source": [
    "# pip install nltk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "aed17ddf",
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is a long document we can split up.\n",
    "with open(\"state_of_the_union.txt\") as f:\n",
    "    state_of_the_union = f.read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "20fa9c23",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_text_splitters import NLTKTextSplitter\n",
    "\n",
    "text_splitter = NLTKTextSplitter(chunk_size=1000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "5ea10835",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Madam Speaker, Madam Vice President, our First Lady and Second Gentleman.\n",
      "\n",
      "Members of Congress and the Cabinet.\n",
      "\n",
      "Justices of the Supreme Court.\n",
      "\n",
      "My fellow Americans.\n",
      "\n",
      "Last year COVID-19 kept us apart.\n",
      "\n",
      "This year we are finally together again.\n",
      "\n",
      "Tonight, we meet as Democrats Republicans and Independents.\n",
      "\n",
      "But most importantly as Americans.\n",
      "\n",
      "With a duty to one another to the American people to the Constitution.\n",
      "\n",
      "And with an unwavering resolve that freedom will always triumph over tyranny.\n",
      "\n",
      "Six days ago, Russia’s Vladimir Putin sought to shake the foundations of the free world thinking he could make it bend to his menacing ways.\n",
      "\n",
      "But he badly miscalculated.\n",
      "\n",
      "He thought he could roll into Ukraine and the world would roll over.\n",
      "\n",
      "Instead he met a wall of strength he never imagined.\n",
      "\n",
      "He met the Ukrainian people.\n",
      "\n",
      "From President Zelenskyy to every Ukrainian, their fearlessness, their courage, their determination, inspires the world.\n",
      "\n",
      "Groups of citizens blocking tanks with their bodies.\n"
     ]
    }
   ],
   "source": [
    "texts = text_splitter.split_text(state_of_the_union)\n",
    "print(texts[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98a3f975",
   "metadata": {},
   "source": [
    "## KoNLPY\n",
    "\n",
    ":::{.callout-note}\n",
    "[KoNLPy: Korean NLP in Python](https://konlpy.org/en/latest/) is is a Python package for natural language processing (NLP) of the Korean language.\n",
    ":::\n",
    "\n",
    "Token splitting involves the segmentation of text into smaller, more manageable units called tokens. These tokens are often words, phrases, symbols, or other meaningful elements crucial for further processing and analysis. In languages like English, token splitting typically involves separating words by spaces and punctuation marks. The effectiveness of token splitting largely depends on the tokenizer's understanding of the language structure, ensuring the generation of meaningful tokens. Since tokenizers designed for the English language are not equipped to understand the unique semantic structures of other languages, such as Korean, they cannot be effectively used for Korean language processing.\n",
    "\n",
    "### Token splitting for Korean with KoNLPy's Kkma Analyzer\n",
    "In case of Korean text, KoNLPY includes at morphological analyzer called `Kkma` (Korean Knowledge Morpheme Analyzer). `Kkma` provides detailed morphological analysis of Korean text. It breaks down sentences into words and words into their respective morphemes, identifying parts of speech for each token. It can segment a block of text into individual sentences, which is particularly useful for processing long texts.\n",
    "\n",
    "### Usage Considerations\n",
    "While `Kkma` is renowned for its detailed analysis, it is important to note that this precision may impact processing speed. Thus, `Kkma` is best suited for applications where analytical depth is prioritized over rapid text processing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "88ec8f2f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# pip install konlpy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "ddfba6cf",
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is a long Korean document that we want to split up into its component sentences.\n",
    "with open(\"./your_korean_doc.txt\") as f:\n",
    "    korean_document = f.read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "225dfc5c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_text_splitters import KonlpyTextSplitter\n",
    "\n",
    "text_splitter = KonlpyTextSplitter()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "cf156711",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "춘향전 옛날에 남원에 이 도령이라는 벼슬아치 아들이 있었다.\n",
      "\n",
      "그의 외모는 빛나는 달처럼 잘생겼고, 그의 학식과 기예는 남보다 뛰어났다.\n",
      "\n",
      "한편, 이 마을에는 춘향이라는 절세 가인이 살고 있었다.\n",
      "\n",
      "춘 향의 아름다움은 꽃과 같아 마을 사람들 로부터 많은 사랑을 받았다.\n",
      "\n",
      "어느 봄날, 도령은 친구들과 놀러 나갔다가 춘 향을 만 나 첫 눈에 반하고 말았다.\n",
      "\n",
      "두 사람은 서로 사랑하게 되었고, 이내 비밀스러운 사랑의 맹세를 나누었다.\n",
      "\n",
      "하지만 좋은 날들은 오래가지 않았다.\n",
      "\n",
      "도령의 아버지가 다른 곳으로 전근을 가게 되어 도령도 떠나 야만 했다.\n",
      "\n",
      "이별의 아픔 속에서도, 두 사람은 재회를 기약하며 서로를 믿고 기다리기로 했다.\n",
      "\n",
      "그러나 새로 부임한 관아의 사또가 춘 향의 아름다움에 욕심을 내 어 그녀에게 강요를 시작했다.\n",
      "\n",
      "춘 향 은 도령에 대한 자신의 사랑을 지키기 위해, 사또의 요구를 단호히 거절했다.\n",
      "\n",
      "이에 분노한 사또는 춘 향을 감옥에 가두고 혹독한 형벌을 내렸다.\n",
      "\n",
      "이야기는 이 도령이 고위 관직에 오른 후, 춘 향을 구해 내는 것으로 끝난다.\n",
      "\n",
      "두 사람은 오랜 시련 끝에 다시 만나게 되고, 그들의 사랑은 온 세상에 전해 지며 후세에까지 이어진다.\n",
      "\n",
      "- 춘향전 (The Tale of Chunhyang)\n"
     ]
    }
   ],
   "source": [
    "texts = text_splitter.split_text(korean_document)\n",
    "# The sentences are split with \"\\n\\n\" characters.\n",
    "print(texts[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13dc0983",
   "metadata": {},
   "source": [
    "## Hugging Face tokenizer\n",
    "\n",
    "[Hugging Face](https://huggingface.co/docs/tokenizers/index) has many tokenizers.\n",
    "\n",
    "We use Hugging Face tokenizer, the [GPT2TokenizerFast](https://huggingface.co/Ransaka/gpt2-tokenizer-fast) to count the text length in tokens.\n",
    "\n",
    "1. How the text is split: by character passed in.\n",
    "2. How the chunk size is measured: by number of tokens calculated by the `Hugging Face` tokenizer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "a8ce51d5",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import GPT2TokenizerFast\n",
    "\n",
    "tokenizer = GPT2TokenizerFast.from_pretrained(\"gpt2\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "388369ed",
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is a long document we can split up.\n",
    "with open(\"state_of_the_union.txt\") as f:\n",
    "    state_of_the_union = f.read()\n",
    "from langchain_text_splitters import CharacterTextSplitter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "ca5e72c0",
   "metadata": {},
   "outputs": [],
   "source": [
    "text_splitter = CharacterTextSplitter.from_huggingface_tokenizer(\n",
    "    tokenizer, chunk_size=100, chunk_overlap=0\n",
    ")\n",
    "texts = text_splitter.split_text(state_of_the_union)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "37cdfbeb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans.  \n",
      "\n",
      "Last year COVID-19 kept us apart. This year we are finally together again. \n",
      "\n",
      "Tonight, we meet as Democrats Republicans and Independents. But most importantly as Americans. \n",
      "\n",
      "With a duty to one another to the American people to the Constitution.\n"
     ]
    }
   ],
   "source": [
    "print(texts[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a43b0fa6",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  },
  "vscode": {
   "interpreter": {
    "hash": "aee8b7b246df8f9039afb4144a1f6fd8d2ca17a180786b69acc140d282b71a49"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}