commit from `quarto publish`

src/.quarto/idx/index.qmd.json

@@ -1 +1 @@
-{"title":"Optimizing LLM Performance Using Triton","markdown":{"yaml":{"title":"Optimizing LLM Performance Using Triton","format":{"revealjs":{"theme":"dark","transition":"slide","slide-number":true}},"author":"Matej Sirovatka","date":"today"},"headingText":"`whoami`","containsRefs":false,"markdown":"\n\n\n- My name is Matej\n- I'm a Master's student at the Brno University of Technology\n- I currently make GPUs go `brrrrrr` at Hugging Face 🤗\n\n## `What is Triton?`\n\n- NVIDIA's open-source programming language for GPU kernels\n- Designed for AI/ML workloads\n- Simplifies GPU programming compared to CUDA\n\n![](media/optim_scale.png){.center fig-align=\"center\"}\n\n## `Why Optimize with Triton?`\n\n- Simple yet effective\n- Less headache than CUDA\n- GPUs go `brrrrrrr` 🚀\n- Feel cool when your kernel is faster than PyTorch 😎\n\n## `Example Problem: KL Divergence`\n\n- commonly used in LLMs for knowledge distillation\n- for probability distributions $P$ and $Q$, the Kullback-Leibler divergence is defined as:\n\n$$\nD_{KL}(P \\| Q) = \\sum_{i} P_i \\log\\left(\\frac{P_i}{Q_i}\\right)\n$$\n\n\n```python\nimport torch\nfrom torch.nn.functional import kl_div\n\ndef kl_div_torch(p: torch.Tensor, q: torch.Tensor) -> torch.Tensor:\n    return kl_div(p, q, reduction='none')\n```\n\n## `How about Triton?`\n\n```python\nimport triton.language as tl\n\n@triton.jit\ndef kl_div_triton(\n    p_ptr, q_ptr, output_ptr, n_elements, BLOCK_SIZE: tl.constexpr\n):\n    pid = tl.program_id(0)\n    block_start = pid * BLOCK_SIZE\n    offsets = block_start + tl.arange(0, BLOCK_SIZE)\n    mask = offsets < n_elements\n    \n    p = tl.load(p_ptr + offsets, mask=mask)\n    q = tl.load(q_ptr + offsets, mask=mask)\n    \n    output = p * (tl.log(p) - tl.log(q))\n    \n    tl.store(output_ptr + offsets, output, mask=mask)\n```\n\n## `How to integrate with PyTorch?`\n\n- Triton works with pointers\n- How to use our custom kernel with PyTorch autograd?\n\n```python\nimport torch\n\nclass VectorAdd(torch.autograd.Function):\n    @staticmethod\n    def forward(ctx, p, q):\n        ctx.save_for_backward(q)\n        output = torch.empty_like(p)\n        grid = (len(p) + 512 - 1) // 512\n        kl_div_triton[grid](p, q, output, len(p), BLOCK_SIZE=512)\n        return output\n\n    @staticmethod\n    def backward(ctx, grad_output):\n        q = ctx.saved_tensors[0]\n        # Calculate gradients (another triton kernel)\n        return ...\n```\n\n## `Some benchmarks`\n\n- A KL Divergence kernel that is currently used in [Liger Kernel](https://github.com/linkedin/liger-kernel) written by @me\n\n:::: {.columns}\n\n::: {.column width=\"50%\"}\n\n![](media/kl_mem.png){.center fig-align=\"center\"}\n\n:::\n\n::: {.column width=\"50%\"}\n\n![](media/kl_speed.png){.center fig-align=\"center\"}\n\n:::\n\n::::\n\n## `Do I have to write everything?`\n\n- TLDR: No\n- Many cool projects already using Triton\n- Better Integration with PyTorch and even Hugging Face 🤗\n- Liger Kernel, Unsloth AI, etc.\n\n:::: {.columns}\n\n::: {.column width=\"50%\"}\n\n![](media/unsloth.png){.center fig-align=\"center\"}\n\n:::\n\n::: {.column width=\"50%\"}\n\n![](media/liger.png){.center fig-align=\"center\"}\n\n:::\n\n::::\n\n\n## `So how can I use this in my LLM? 🚀`\n\n- Liger Kernel is a great example, providing examples of how to integrate with Hugging Face 🤗 Trainer\n\n```diff\n- from transformers import AutoModelForCausalLM\n+ from liger_kernel.transformers import AutoLigerKernelForCausalLM\n\nmodel_path = \"meta-llama/Meta-Llama-3-8B-Instruct\"\n\n- model = AutoModelForCausalLM.from_pretrained(model_path)\n+ model = AutoLigerKernelForCausalLM.from_pretrained(model_path)\n\n# training/inference logic...\n```\n## `Key Optimization Techniques adapted by Liger Kernel`\n\n- Kernel Fusion\n- Domain-specific optimizations\n- Memory Access Patterns\n- Preemptive memory freeing\n   \n\n## `Aaand some more benchmarks 🚀`\n\n\n:::: {.columns}\n\n::: {.column width=\"50%\"}\n\n![](media/PMA.png){fig-align=\"center\"}\n\n:::\n\n::: {.column width=\"50%\"}\n\n![](media/PMR.png){fig-align=\"center\"}\n\n:::\n\n::::\n\n## `Last benchmark I promise...`\n\n\n![](media/TPS.png){height=\"50%\" width=\"50%\" }\n\n::: {.incremental}\n*Attention is all you need, so I thank you for yours!* 🤗\n:::\n\n","srcMarkdownNoYaml":"\n\n## `whoami`\n\n- My name is Matej\n- I'm a Master's student at the Brno University of Technology\n- I currently make GPUs go `brrrrrr` at Hugging Face 🤗\n\n## `What is Triton?`\n\n- NVIDIA's open-source programming language for GPU kernels\n- Designed for AI/ML workloads\n- Simplifies GPU programming compared to CUDA\n\n![](media/optim_scale.png){.center fig-align=\"center\"}\n\n## `Why Optimize with Triton?`\n\n- Simple yet effective\n- Less headache than CUDA\n- GPUs go `brrrrrrr` 🚀\n- Feel cool when your kernel is faster than PyTorch 😎\n\n## `Example Problem: KL Divergence`\n\n- commonly used in LLMs for knowledge distillation\n- for probability distributions $P$ and $Q$, the Kullback-Leibler divergence is defined as:\n\n$$\nD_{KL}(P \\| Q) = \\sum_{i} P_i \\log\\left(\\frac{P_i}{Q_i}\\right)\n$$\n\n\n```python\nimport torch\nfrom torch.nn.functional import kl_div\n\ndef kl_div_torch(p: torch.Tensor, q: torch.Tensor) -> torch.Tensor:\n    return kl_div(p, q, reduction='none')\n```\n\n## `How about Triton?`\n\n```python\nimport triton.language as tl\n\n@triton.jit\ndef kl_div_triton(\n    p_ptr, q_ptr, output_ptr, n_elements, BLOCK_SIZE: tl.constexpr\n):\n    pid = tl.program_id(0)\n    block_start = pid * BLOCK_SIZE\n    offsets = block_start + tl.arange(0, BLOCK_SIZE)\n    mask = offsets < n_elements\n    \n    p = tl.load(p_ptr + offsets, mask=mask)\n    q = tl.load(q_ptr + offsets, mask=mask)\n    \n    output = p * (tl.log(p) - tl.log(q))\n    \n    tl.store(output_ptr + offsets, output, mask=mask)\n```\n\n## `How to integrate with PyTorch?`\n\n- Triton works with pointers\n- How to use our custom kernel with PyTorch autograd?\n\n```python\nimport torch\n\nclass VectorAdd(torch.autograd.Function):\n    @staticmethod\n    def forward(ctx, p, q):\n        ctx.save_for_backward(q)\n        output = torch.empty_like(p)\n        grid = (len(p) + 512 - 1) // 512\n        kl_div_triton[grid](p, q, output, len(p), BLOCK_SIZE=512)\n        return output\n\n    @staticmethod\n    def backward(ctx, grad_output):\n        q = ctx.saved_tensors[0]\n        # Calculate gradients (another triton kernel)\n        return ...\n```\n\n## `Some benchmarks`\n\n- A KL Divergence kernel that is currently used in [Liger Kernel](https://github.com/linkedin/liger-kernel) written by @me\n\n:::: {.columns}\n\n::: {.column width=\"50%\"}\n\n![](media/kl_mem.png){.center fig-align=\"center\"}\n\n:::\n\n::: {.column width=\"50%\"}\n\n![](media/kl_speed.png){.center fig-align=\"center\"}\n\n:::\n\n::::\n\n## `Do I have to write everything?`\n\n- TLDR: No\n- Many cool projects already using Triton\n- Better Integration with PyTorch and even Hugging Face 🤗\n- Liger Kernel, Unsloth AI, etc.\n\n:::: {.columns}\n\n::: {.column width=\"50%\"}\n\n![](media/unsloth.png){.center fig-align=\"center\"}\n\n:::\n\n::: {.column width=\"50%\"}\n\n![](media/liger.png){.center fig-align=\"center\"}\n\n:::\n\n::::\n\n\n## `So how can I use this in my LLM? 🚀`\n\n- Liger Kernel is a great example, providing examples of how to integrate with Hugging Face 🤗 Trainer\n\n```diff\n- from transformers import AutoModelForCausalLM\n+ from liger_kernel.transformers import AutoLigerKernelForCausalLM\n\nmodel_path = \"meta-llama/Meta-Llama-3-8B-Instruct\"\n\n- model = AutoModelForCausalLM.from_pretrained(model_path)\n+ model = AutoLigerKernelForCausalLM.from_pretrained(model_path)\n\n# training/inference logic...\n```\n## `Key Optimization Techniques adapted by Liger Kernel`\n\n- Kernel Fusion\n- Domain-specific optimizations\n- Memory Access Patterns\n- Preemptive memory freeing\n   \n\n## `Aaand some more benchmarks 🚀`\n\n\n:::: {.columns}\n\n::: {.column width=\"50%\"}\n\n![](media/PMA.png){fig-align=\"center\"}\n\n:::\n\n::: {.column width=\"50%\"}\n\n![](media/PMR.png){fig-align=\"center\"}\n\n:::\n\n::::\n\n## `Last benchmark I promise...`\n\n\n![](media/TPS.png){height=\"50%\" width=\"50%\" }\n\n::: {.incremental}\n*Attention is all you need, so I thank you for yours!* 🤗\n:::\n\n"},"formats":{"revealjs":{"identifier":{"display-name":"RevealJS","target-format":"revealjs","base-format":"revealjs"},"execute":{"fig-width":10,"fig-height":5,"fig-format":"retina","fig-dpi":96,"df-print":"default","error":false,"eval":true,"cache":null,"freeze":false,"echo":false,"output":true,"warning":false,"include":true,"keep-md":false,"keep-ipynb":false,"ipynb":null,"enabled":null,"daemon":null,"daemon-restart":false,"debug":false,"ipynb-filters":[],"ipynb-shell-interactivity":null,"plotly-connected":true,"engine":"markdown"},"render":{"keep-tex":false,"keep-typ":false,"keep-source":false,"keep-hidden":false,"prefer-html":false,"output-divs":true,"output-ext":"html","fig-align":"default","fig-pos":null,"fig-env":null,"code-fold":"none","code-overflow":"scroll","code-link":false,"code-line-numbers":true,"code-tools":false,"tbl-colwidths":"auto","merge-includes":true,"inline-includes":false,"preserve-yaml":false,"latex-auto-mk":true,"latex-auto-install":true,"latex-clean":true,"latex-min-runs":1,"latex-max-runs":10,"latex-makeindex":"makeindex","latex-makeindex-opts":[],"latex-tlmgr-opts":[],"latex-input-paths":[],"latex-output-dir":null,"link-external-icon":false,"link-external-newwindow":false,"self-contained-math":false,"format-resources":[]},"pandoc":{"standalone":true,"wrap":"none","default-image-extension":"png","html-math-method":{"method":"mathjax","url":"https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.9/MathJax.js?config=TeX-AMS_HTML-full"},"slide-level":2,"to":"revealjs","output-file":"index.html"},"language":{"toc-title-document":"Table of contents","toc-title-website":"On this page","related-formats-title":"Other Formats","related-notebooks-title":"Notebooks","source-notebooks-prefix":"Source","other-links-title":"Other Links","code-links-title":"Code Links","launch-dev-container-title":"Launch Dev Container","launch-binder-title":"Launch Binder","article-notebook-label":"Article Notebook","notebook-preview-download":"Download Notebook","notebook-preview-download-src":"Download Source","notebook-preview-back":"Back to Article","manuscript-meca-bundle":"MECA Bundle","section-title-abstract":"Abstract","section-title-appendices":"Appendices","section-title-footnotes":"Footnotes","section-title-references":"References","section-title-reuse":"Reuse","section-title-copyright":"Copyright","section-title-citation":"Citation","appendix-attribution-cite-as":"For attribution, please cite this work as:","appendix-attribution-bibtex":"BibTeX citation:","appendix-view-license":"View License","title-block-author-single":"Author","title-block-author-plural":"Authors","title-block-affiliation-single":"Affiliation","title-block-affiliation-plural":"Affiliations","title-block-published":"Published","title-block-modified":"Modified","title-block-keywords":"Keywords","callout-tip-title":"Tip","callout-note-title":"Note","callout-warning-title":"Warning","callout-important-title":"Important","callout-caution-title":"Caution","code-summary":"Code","code-tools-menu-caption":"Code","code-tools-show-all-code":"Show All Code","code-tools-hide-all-code":"Hide All Code","code-tools-view-source":"View Source","code-tools-source-code":"Source Code","tools-share":"Share","tools-download":"Download","code-line":"Line","code-lines":"Lines","copy-button-tooltip":"Copy to Clipboard","copy-button-tooltip-success":"Copied!","repo-action-links-edit":"Edit this page","repo-action-links-source":"View source","repo-action-links-issue":"Report an issue","back-to-top":"Back to top","search-no-results-text":"No results","search-matching-documents-text":"matching documents","search-copy-link-title":"Copy link to search","search-hide-matches-text":"Hide additional matches","search-more-match-text":"more match in this document","search-more-matches-text":"more matches in this document","search-clear-button-title":"Clear","search-text-placeholder":"","search-detached-cancel-button-title":"Cancel","search-submit-button-title":"Submit","search-label":"Search","toggle-section":"Toggle section","toggle-sidebar":"Toggle sidebar navigation","toggle-dark-mode":"Toggle dark mode","toggle-reader-mode":"Toggle reader mode","toggle-navigation":"Toggle navigation","crossref-fig-title":"Figure","crossref-tbl-title":"Table","crossref-lst-title":"Listing","crossref-thm-title":"Theorem","crossref-lem-title":"Lemma","crossref-cor-title":"Corollary","crossref-prp-title":"Proposition","crossref-cnj-title":"Conjecture","crossref-def-title":"Definition","crossref-exm-title":"Example","crossref-exr-title":"Exercise","crossref-ch-prefix":"Chapter","crossref-apx-prefix":"Appendix","crossref-sec-prefix":"Section","crossref-eq-prefix":"Equation","crossref-lof-title":"List of Figures","crossref-lot-title":"List of Tables","crossref-lol-title":"List of Listings","environment-proof-title":"Proof","environment-remark-title":"Remark","environment-solution-title":"Solution","listing-page-order-by":"Order By","listing-page-order-by-default":"Default","listing-page-order-by-date-asc":"Oldest","listing-page-order-by-date-desc":"Newest","listing-page-order-by-number-desc":"High to Low","listing-page-order-by-number-asc":"Low to High","listing-page-field-date":"Date","listing-page-field-title":"Title","listing-page-field-description":"Description","listing-page-field-author":"Author","listing-page-field-filename":"File Name","listing-page-field-filemodified":"Modified","listing-page-field-subtitle":"Subtitle","listing-page-field-readingtime":"Reading Time","listing-page-field-wordcount":"Word Count","listing-page-field-categories":"Categories","listing-page-minutes-compact":"{0} min","listing-page-category-all":"All","listing-page-no-matches":"No matching items","listing-page-words":"{0} words","listing-page-filter":"Filter","draft":"Draft"},"metadata":{"lang":"en","fig-responsive":false,"quarto-version":"1.6.40","auto-stretch":true,"title":"Optimizing LLM Performance Using Triton","author":"Matej Sirovatka","date":"today","theme":"dark","transition":"slide","slideNumber":true}}},"projectFormats":["html"]}
+{"title":"Optimizing LLM Performance Using Triton","markdown":{"yaml":{"title":"Optimizing LLM Performance Using Triton","format":{"revealjs":{"theme":"dark","transition":"slide","slide-number":true}},"author":"Matej Sirovatka","date":"2025-02-22"},"headingText":"`whoami`","containsRefs":false,"markdown":"\n\n\n- My name is Matej\n- I'm a Master's student at the Brno University of Technology\n- I'm currently working on distributed training at Hugging Face 🤗\n\n## `What is Triton?`\n\n- NVIDIA's open-source programming language for GPU kernels\n- Designed for AI/ML workloads\n- Simplifies GPU programming compared to CUDA\n\n![](media/optim_scale.png){.center fig-align=\"center\"}\n\n## `Why Optimize with Triton?`\n\n- Simple yet effective\n- Less headache than CUDA\n- GPUs go `brrrrrrr` 🚀\n- Feel cool when your kernel is faster than PyTorch 😎\n\n## `Example Problem: KL Divergence`\n\n- commonly used in LLMs for knowledge distillation\n- for probability distributions $P$ and $Q$, the Kullback-Leibler divergence is defined as:\n\n$$\nD_{KL}(P \\| Q) = \\sum_{i} P_i \\log\\left(\\frac{P_i}{Q_i}\\right)\n$$\n\n\n```python\nimport torch\nfrom torch.nn.functional import kl_div\n\ndef kl_div_torch(p: torch.Tensor, q: torch.Tensor) -> torch.Tensor:\n    return kl_div(p, q)\n```\n\n## `How about Triton?`\n\n```python\nimport triton\nimport triton.language as tl\n\n@triton.jit\ndef kl_div_triton(\n    p_ptr, q_ptr, output_ptr, n_elements, BLOCK_SIZE: tl.constexpr\n):\n    pid = tl.program_id(0)\n    block_start = pid * BLOCK_SIZE\n    offsets = block_start + tl.arange(0, BLOCK_SIZE)\n    mask = offsets < n_elements\n    \n    p = tl.load(p_ptr + offsets, mask=mask)\n    q = tl.load(q_ptr + offsets, mask=mask)\n    \n    output = p * (tl.log(p) - tl.log(q))\n    tl.store(output_ptr + offsets, output, mask=mask)\n```\n\n## `How to integrate with PyTorch?`\n\n- How to use our custom kernel with PyTorch autograd?\n\n```python\nimport torch\n\nclass VectorAdd(torch.autograd.Function):\n    @staticmethod\n    def forward(ctx, p, q):\n        ctx.save_for_backward(q)\n        output = torch.empty_like(p)\n        grid = (len(p) + 512 - 1) // 512\n        kl_div_triton[grid](p, q, output, len(p), BLOCK_SIZE=512)\n        return output\n\n    @staticmethod\n    def backward(ctx, grad_output):\n        q = ctx.saved_tensors[0]\n        # Calculate gradients (another triton kernel)\n        return ...\n```\n\n## `Some benchmarks`\n\n- A KL Divergence kernel that is currently used in [Liger Kernel](https://github.com/linkedin/liger-kernel) written by @me\n\n:::: {.columns}\n\n::: {.column width=\"50%\"}\n\n![](media/kl_mem.png){.center fig-align=\"center\"}\n\n:::\n\n::: {.column width=\"50%\"}\n\n![](media/kl_speed.png){.center fig-align=\"center\"}\n\n:::\n\n::::\n\n## `Do I have to write everything?`\n\n- TLDR: No\n- Many cool projects already using Triton\n- Better Integration with PyTorch and even Hugging Face 🤗\n- Liger Kernel, Unsloth AI, etc.\n\n:::: {.columns}\n\n::: {.column width=\"50%\"}\n\n![](media/unsloth.png){.center fig-align=\"center\"}\n\n:::\n\n::: {.column width=\"50%\"}\n\n![](media/liger.png){.center fig-align=\"center\"}\n\n:::\n\n::::\n\n\n## `So how can I use this in my LLM? 🚀`\n\n- Liger Kernel is a great example, providing examples of how to integrate with Hugging Face 🤗 Trainer\n\n```diff\n- from transformers import AutoModelForCausalLM\n+ from liger_kernel.transformers import AutoLigerKernelForCausalLM\n\nmodel_path = \"meta-llama/Meta-Llama-3-8B-Instruct\"\n\n- model = AutoModelForCausalLM.from_pretrained(model_path)\n+ model = AutoLigerKernelForCausalLM.from_pretrained(model_path)\n\n# training/inference logic...\n```\n## `Key Optimization Techniques adapted by Liger Kernel`\n\n- Kernel Fusion\n- Domain-specific optimizations\n- Memory Access Patterns\n- Preemptive memory freeing\n   \n\n## `Aaand some more benchmarks 🚀`\n\n- Saving memory is key to run bigger batch size on smaller GPUs\n\n:::: {.columns}\n\n::: {.column width=\"50%\"}\n\n![](media/PMA.png){fig-align=\"center\"}\n\n:::\n\n::: {.column width=\"50%\"}\n\n![](media/PMR.png){fig-align=\"center\"}\n\n:::\n\n::::\n\n## `Last benchmark I promise...`\n\n- But is it faster? Yes, it is!\n\n![](media/TPS.png){fig-align=\"center\" height=50% width=50%}\n\n:::: {.columns}\n\n::: {.column width=\"60%\"}\n\n*Attention is all you need, so I thank you for yours!* 🤗\n\n:::\n\n::: {.column width=\"40%\"}\n\n![](media/qr.png){height=25% width=25% fig-align=\"center\"}\n\n:::\n\n::::\n\n","srcMarkdownNoYaml":"\n\n## `whoami`\n\n- My name is Matej\n- I'm a Master's student at the Brno University of Technology\n- I'm currently working on distributed training at Hugging Face 🤗\n\n## `What is Triton?`\n\n- NVIDIA's open-source programming language for GPU kernels\n- Designed for AI/ML workloads\n- Simplifies GPU programming compared to CUDA\n\n![](media/optim_scale.png){.center fig-align=\"center\"}\n\n## `Why Optimize with Triton?`\n\n- Simple yet effective\n- Less headache than CUDA\n- GPUs go `brrrrrrr` 🚀\n- Feel cool when your kernel is faster than PyTorch 😎\n\n## `Example Problem: KL Divergence`\n\n- commonly used in LLMs for knowledge distillation\n- for probability distributions $P$ and $Q$, the Kullback-Leibler divergence is defined as:\n\n$$\nD_{KL}(P \\| Q) = \\sum_{i} P_i \\log\\left(\\frac{P_i}{Q_i}\\right)\n$$\n\n\n```python\nimport torch\nfrom torch.nn.functional import kl_div\n\ndef kl_div_torch(p: torch.Tensor, q: torch.Tensor) -> torch.Tensor:\n    return kl_div(p, q)\n```\n\n## `How about Triton?`\n\n```python\nimport triton\nimport triton.language as tl\n\n@triton.jit\ndef kl_div_triton(\n    p_ptr, q_ptr, output_ptr, n_elements, BLOCK_SIZE: tl.constexpr\n):\n    pid = tl.program_id(0)\n    block_start = pid * BLOCK_SIZE\n    offsets = block_start + tl.arange(0, BLOCK_SIZE)\n    mask = offsets < n_elements\n    \n    p = tl.load(p_ptr + offsets, mask=mask)\n    q = tl.load(q_ptr + offsets, mask=mask)\n    \n    output = p * (tl.log(p) - tl.log(q))\n    tl.store(output_ptr + offsets, output, mask=mask)\n```\n\n## `How to integrate with PyTorch?`\n\n- How to use our custom kernel with PyTorch autograd?\n\n```python\nimport torch\n\nclass VectorAdd(torch.autograd.Function):\n    @staticmethod\n    def forward(ctx, p, q):\n        ctx.save_for_backward(q)\n        output = torch.empty_like(p)\n        grid = (len(p) + 512 - 1) // 512\n        kl_div_triton[grid](p, q, output, len(p), BLOCK_SIZE=512)\n        return output\n\n    @staticmethod\n    def backward(ctx, grad_output):\n        q = ctx.saved_tensors[0]\n        # Calculate gradients (another triton kernel)\n        return ...\n```\n\n## `Some benchmarks`\n\n- A KL Divergence kernel that is currently used in [Liger Kernel](https://github.com/linkedin/liger-kernel) written by @me\n\n:::: {.columns}\n\n::: {.column width=\"50%\"}\n\n![](media/kl_mem.png){.center fig-align=\"center\"}\n\n:::\n\n::: {.column width=\"50%\"}\n\n![](media/kl_speed.png){.center fig-align=\"center\"}\n\n:::\n\n::::\n\n## `Do I have to write everything?`\n\n- TLDR: No\n- Many cool projects already using Triton\n- Better Integration with PyTorch and even Hugging Face 🤗\n- Liger Kernel, Unsloth AI, etc.\n\n:::: {.columns}\n\n::: {.column width=\"50%\"}\n\n![](media/unsloth.png){.center fig-align=\"center\"}\n\n:::\n\n::: {.column width=\"50%\"}\n\n![](media/liger.png){.center fig-align=\"center\"}\n\n:::\n\n::::\n\n\n## `So how can I use this in my LLM? 🚀`\n\n- Liger Kernel is a great example, providing examples of how to integrate with Hugging Face 🤗 Trainer\n\n```diff\n- from transformers import AutoModelForCausalLM\n+ from liger_kernel.transformers import AutoLigerKernelForCausalLM\n\nmodel_path = \"meta-llama/Meta-Llama-3-8B-Instruct\"\n\n- model = AutoModelForCausalLM.from_pretrained(model_path)\n+ model = AutoLigerKernelForCausalLM.from_pretrained(model_path)\n\n# training/inference logic...\n```\n## `Key Optimization Techniques adapted by Liger Kernel`\n\n- Kernel Fusion\n- Domain-specific optimizations\n- Memory Access Patterns\n- Preemptive memory freeing\n   \n\n## `Aaand some more benchmarks 🚀`\n\n- Saving memory is key to run bigger batch size on smaller GPUs\n\n:::: {.columns}\n\n::: {.column width=\"50%\"}\n\n![](media/PMA.png){fig-align=\"center\"}\n\n:::\n\n::: {.column width=\"50%\"}\n\n![](media/PMR.png){fig-align=\"center\"}\n\n:::\n\n::::\n\n## `Last benchmark I promise...`\n\n- But is it faster? Yes, it is!\n\n![](media/TPS.png){fig-align=\"center\" height=50% width=50%}\n\n:::: {.columns}\n\n::: {.column width=\"60%\"}\n\n*Attention is all you need, so I thank you for yours!* 🤗\n\n:::\n\n::: {.column width=\"40%\"}\n\n![](media/qr.png){height=25% width=25% fig-align=\"center\"}\n\n:::\n\n::::\n\n"},"formats":{"revealjs":{"identifier":{"display-name":"RevealJS","target-format":"revealjs","base-format":"revealjs"},"execute":{"fig-width":10,"fig-height":5,"fig-format":"retina","fig-dpi":96,"df-print":"default","error":false,"eval":true,"cache":null,"freeze":false,"echo":false,"output":true,"warning":false,"include":true,"keep-md":false,"keep-ipynb":false,"ipynb":null,"enabled":null,"daemon":null,"daemon-restart":false,"debug":false,"ipynb-filters":[],"ipynb-shell-interactivity":null,"plotly-connected":true,"engine":"markdown"},"render":{"keep-tex":false,"keep-typ":false,"keep-source":false,"keep-hidden":false,"prefer-html":false,"output-divs":true,"output-ext":"html","fig-align":"default","fig-pos":null,"fig-env":null,"code-fold":"none","code-overflow":"scroll","code-link":false,"code-line-numbers":true,"code-tools":false,"tbl-colwidths":"auto","merge-includes":true,"inline-includes":false,"preserve-yaml":false,"latex-auto-mk":true,"latex-auto-install":true,"latex-clean":true,"latex-min-runs":1,"latex-max-runs":10,"latex-makeindex":"makeindex","latex-makeindex-opts":[],"latex-tlmgr-opts":[],"latex-input-paths":[],"latex-output-dir":null,"link-external-icon":false,"link-external-newwindow":false,"self-contained-math":false,"format-resources":[]},"pandoc":{"standalone":true,"wrap":"none","default-image-extension":"png","html-math-method":{"method":"mathjax","url":"https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.9/MathJax.js?config=TeX-AMS_HTML-full"},"slide-level":2,"to":"revealjs","output-file":"index.html"},"language":{"toc-title-document":"Table of contents","toc-title-website":"On this page","related-formats-title":"Other Formats","related-notebooks-title":"Notebooks","source-notebooks-prefix":"Source","other-links-title":"Other Links","code-links-title":"Code Links","launch-dev-container-title":"Launch Dev Container","launch-binder-title":"Launch Binder","article-notebook-label":"Article Notebook","notebook-preview-download":"Download Notebook","notebook-preview-download-src":"Download Source","notebook-preview-back":"Back to Article","manuscript-meca-bundle":"MECA Bundle","section-title-abstract":"Abstract","section-title-appendices":"Appendices","section-title-footnotes":"Footnotes","section-title-references":"References","section-title-reuse":"Reuse","section-title-copyright":"Copyright","section-title-citation":"Citation","appendix-attribution-cite-as":"For attribution, please cite this work as:","appendix-attribution-bibtex":"BibTeX citation:","appendix-view-license":"View License","title-block-author-single":"Author","title-block-author-plural":"Authors","title-block-affiliation-single":"Affiliation","title-block-affiliation-plural":"Affiliations","title-block-published":"Published","title-block-modified":"Modified","title-block-keywords":"Keywords","callout-tip-title":"Tip","callout-note-title":"Note","callout-warning-title":"Warning","callout-important-title":"Important","callout-caution-title":"Caution","code-summary":"Code","code-tools-menu-caption":"Code","code-tools-show-all-code":"Show All Code","code-tools-hide-all-code":"Hide All Code","code-tools-view-source":"View Source","code-tools-source-code":"Source Code","tools-share":"Share","tools-download":"Download","code-line":"Line","code-lines":"Lines","copy-button-tooltip":"Copy to Clipboard","copy-button-tooltip-success":"Copied!","repo-action-links-edit":"Edit this page","repo-action-links-source":"View source","repo-action-links-issue":"Report an issue","back-to-top":"Back to top","search-no-results-text":"No results","search-matching-documents-text":"matching documents","search-copy-link-title":"Copy link to search","search-hide-matches-text":"Hide additional matches","search-more-match-text":"more match in this document","search-more-matches-text":"more matches in this document","search-clear-button-title":"Clear","search-text-placeholder":"","search-detached-cancel-button-title":"Cancel","search-submit-button-title":"Submit","search-label":"Search","toggle-section":"Toggle section","toggle-sidebar":"Toggle sidebar navigation","toggle-dark-mode":"Toggle dark mode","toggle-reader-mode":"Toggle reader mode","toggle-navigation":"Toggle navigation","crossref-fig-title":"Figure","crossref-tbl-title":"Table","crossref-lst-title":"Listing","crossref-thm-title":"Theorem","crossref-lem-title":"Lemma","crossref-cor-title":"Corollary","crossref-prp-title":"Proposition","crossref-cnj-title":"Conjecture","crossref-def-title":"Definition","crossref-exm-title":"Example","crossref-exr-title":"Exercise","crossref-ch-prefix":"Chapter","crossref-apx-prefix":"Appendix","crossref-sec-prefix":"Section","crossref-eq-prefix":"Equation","crossref-lof-title":"List of Figures","crossref-lot-title":"List of Tables","crossref-lol-title":"List of Listings","environment-proof-title":"Proof","environment-remark-title":"Remark","environment-solution-title":"Solution","listing-page-order-by":"Order By","listing-page-order-by-default":"Default","listing-page-order-by-date-asc":"Oldest","listing-page-order-by-date-desc":"Newest","listing-page-order-by-number-desc":"High to Low","listing-page-order-by-number-asc":"Low to High","listing-page-field-date":"Date","listing-page-field-title":"Title","listing-page-field-description":"Description","listing-page-field-author":"Author","listing-page-field-filename":"File Name","listing-page-field-filemodified":"Modified","listing-page-field-subtitle":"Subtitle","listing-page-field-readingtime":"Reading Time","listing-page-field-wordcount":"Word Count","listing-page-field-categories":"Categories","listing-page-minutes-compact":"{0} min","listing-page-category-all":"All","listing-page-no-matches":"No matching items","listing-page-words":"{0} words","listing-page-filter":"Filter","draft":"Draft"},"metadata":{"lang":"en","fig-responsive":false,"quarto-version":"1.6.40","auto-stretch":true,"title":"Optimizing LLM Performance Using Triton","author":"Matej Sirovatka","date":"2025-02-22","theme":"dark","transition":"slide","slideNumber":true}}},"projectFormats":["html"]}

src/.quarto/idx/notebooks/advanced_rag.qmd.json

@@ -1 +0,0 @@
-{"title":"Advanced RAG","markdown":{"yaml":{"title":"Advanced RAG","jupyter":"python3","eval":false,"code-annotations":"hover"},"headingText":"Load your knowledge base","containsRefs":false,"markdown":"\n\nThis notebook demonstrates how you can build an advanced RAG (Retrieval Augmented Generation) for answering a user's question about a specific knowledge base (here, the HuggingFace documentation), using LangChain.\n\nFor an introduction to RAG, you can check [this other cookbook](rag_zephyr_langchain.qmd)!\n\nRAG systems are complex, with many moving parts: here a RAG diagram, where we noted in blue all possibilities for system enhancement:\n\n<img src=\"https://huggingface.co/datasets/huggingface/cookbook-images/resolve/main/RAG_workflow.png\" height=\"700\">\n\n::: callout-note\n💡 As you can see, there are many steps to tune in this architecture: tuning the system properly will yield significant performance gains.\n:::\n\nIn this notebook, we will take a look into many of these blue notes to see how to tune your RAG system and get the best performance.\n\n__Let's dig into the model building!__ First, we install the required model dependancies.\n\n```{python}\n!pip install -q torch transformers transformers accelerate bitsandbytes langchain sentence-transformers faiss-gpu openpyxl pacmap\n```\n\n```{python}\n%reload_ext dotenv\n%dotenv\n```\n\n```{python}\nfrom tqdm.notebook import tqdm\nimport pandas as pd\nfrom typing import Optional, List, Tuple\nfrom datasets import Dataset\nimport matplotlib.pyplot as plt\n\npd.set_option(\n    \"display.max_colwidth\", None                    # <1>\n) \n```\n1. This will be helpful when visualizing retriever outputs\n\n\n```{python}\nimport datasets\n\nds = datasets.load_dataset(\"m-ric/huggingface_doc\", split=\"train\")\n```\n\n```{python}\nfrom langchain.docstore.document import Document as LangchainDocument\n\nRAW_KNOWLEDGE_BASE = [\n    LangchainDocument(page_content=doc[\"text\"], metadata={\"source\": doc[\"source\"]})\n    for doc in tqdm(ds)\n]\n```\n\n# 1. Retriever - embeddings 🗂️\nThe __retriever acts like an internal search engine__: given the user query, it returns a few relevant snippets from your knowledge base.\n\nThese snippets will then be fed to the Reader Model to help it generate its answer.\n\nSo __our objective here is, given a user question, to find the most snippets from our knowledge base to answer that question.__\n\nThis is a wide objective, it leaves open some questions. How many snippets should we retrieve? This parameter will be named `top_k`.\n\nHow long should these snippets be? This is called the `chunk size`. There's no one-size-fits-all answers, but here are a few elements:\n- 🔀 Your `chunk size` is allowed to vary from one snippet to the other.\n- Since there will always be some noise in your retrieval, increasing the `top_k` increases the chance to get relevant elements in your retrieved snippets. 🎯 Shooting more arrows increases your probability to hit your target.\n- Meanwhile, the summed length of your retrieved documents should not be too high: for instance, for most current models 16k tokens will probably drown your Reader model in information due to [Lost-in-the-middle phenomenon](https://huggingface.co/papers/2307.03172). 🎯 Give your reader model only the most relevant insights, not a huge pile of books!\n\n::: callout-note\nIn this notebook, we use Langchain library since __it offers a huge variety of options for vector databases and allows us to keep document metadata throughout the processing__.\n:::\n\n### 1.1 Split the documents into chunks\n\n- In this part, __we split the documents from our knowledge base into smaller chunks__ which will be the snippets on which the reader LLM will base its answer.\n- The goal is to prepare a collection of **semantically relevant snippets**. So their size should be adapted to precise ideas: too small will truncate ideas, too large will dilute them.\n\n::: callout-tip\n💡 Many options exist for text splitting: splitting on words, on sentence boundaries, recursive chunking that processes documents in a tree-like way to preserve structure information... To learn more about chunking, I recommend you read [this great notebook](https://github.com/FullStackRetrieval-com/RetrievalTutorials/blob/main/5_Levels_Of_Text_Splitting.ipynb) by Greg Kamradt.\n:::\n\n\n- **Recursive chunking** breaks down the text into smaller parts step by step using a given list of separators sorted from the most important to the least important separator. If the first split doesn't give the right size or shape chunks, the method repeats itself on the new chunks using a different separator. For instance with the list of separators `[\"\\n\\n\", \"\\n\", \".\", \"\"]`:\n    - The method will first break down the document wherever there is a double line break `\"\\n\\n\"`.\n    - Resulting documents will be split again on simple line breaks `\"\\n\"`, then on sentence ends `\".\"`.\n    - And finally, if some chunks are still too big, they will be split whenever they overflow the maximum size.\n\n- With this method, the global structure is well preserved, at the expense of getting slight variations in chunk size.\n\n> [This space](https://huggingface.co/spaces/A-Roucher/chunk_visualizer) lets you visualize how different splitting options affect the chunks you get.\n\n🔬 Let's experiment a bit with chunk sizes, beginning with an arbitrary size, and see how splits work. We use Langchain's implementation of recursive chunking with `RecursiveCharacterTextSplitter`.\n- Parameter `chunk_size` controls the length of individual chunks: this length is counted by default as the number of characters in the chunk.\n- Parameter `chunk_overlap` lets adjacent chunks get a bit of overlap on each other. This reduces the probability that an idea could be cut in half by the split between two adjacent chunks. We ~arbitrarily set this to 1/10th of the chunk size, you could try different values!\n\n```{python}\nfrom langchain.text_splitter import RecursiveCharacterTextSplitter\n\n# We use a hierarchical list of separators specifically tailored for splitting Markdown documents\n# This list is taken from LangChain's MarkdownTextSplitter class.\nMARKDOWN_SEPARATORS = [\n    \"\\n#{1,6} \",\n    \"```\\n\",\n    \"\\n\\\\*\\\\*\\\\*+\\n\",\n    \"\\n---+\\n\",\n    \"\\n___+\\n\",\n    \"\\n\\n\",\n    \"\\n\",\n    \" \",\n    \"\",\n]\n\ntext_splitter = RecursiveCharacterTextSplitter(\n    chunk_size=1000,            # <1>\n    chunk_overlap=100,          # <2>\n    add_start_index=True,       # <3>\n    strip_whitespace=True,      # <4>\n    separators=MARKDOWN_SEPARATORS,\n)\n\ndocs_processed = []\nfor doc in RAW_KNOWLEDGE_BASE:\n    docs_processed += text_splitter.split_documents([doc])\n```\n1. The maximum number of characters in a chunk: we selected this value arbitrally\n2. The number of characters to overlap between chunks\n3. If `True`, includes chunk's start index in metadata\n4. If `True`, strips whitespace from the start and end of every document\n\n\nWe also have to keep in mind that when embedding documents, we will use an embedding model that has accepts a certain maximum sequence length `max_seq_length`.\n\nSo we should make sure that our chunk sizes are below this limit, because any longer chunk will be truncated before processing, thus losing relevancy.\n\n```{python}\n#| colab: {referenced_widgets: [ae043feeb0914c879e2a9008b413d952]}\nfrom sentence_transformers import SentenceTransformer\n\n# To get the value of the max sequence_length, we will query the underlying `SentenceTransformer` object used in the RecursiveCharacterTextSplitter.\nprint(\n    f\"Model's maximum sequence length: {SentenceTransformer('thenlper/gte-small').max_seq_length}\"\n)\n\nfrom transformers import AutoTokenizer\n\ntokenizer = AutoTokenizer.from_pretrained(\"thenlper/gte-small\")\nlengths = [len(tokenizer.encode(doc.page_content)) for doc in tqdm(docs_processed)]\n\n# Plot the distrubution of document lengths, counted as the number of tokens\nfig = pd.Series(lengths).hist()\nplt.title(\"Distribution of document lengths in the knowledge base (in count of tokens)\")\nplt.show()\n```\n\n👀 As you can see, __the chunk lengths are not aligned with our limit of 512 tokens__, and some documents are above the limit, thus some part of them will be lost in truncation!\n - So we should change the `RecursiveCharacterTextSplitter` class to count length in number of tokens instead of number of characters.\n - Then we can choose a specific chunk size, here we would choose a lower threshold than 512:\n    - smaller documents could allow the split to focus more on specific ideas.\n    - But too small chunks would split sentences in half, thus losing meaning again: the proper tuning is a matter of balance.\n\n```{python}\n#| colab: {referenced_widgets: [f900cf4ab3a94f45bfa7298f433566ed]}\nfrom langchain.text_splitter import RecursiveCharacterTextSplitter\nfrom transformers import AutoTokenizer\n\nEMBEDDING_MODEL_NAME = \"thenlper/gte-small\"\n\n\ndef split_documents(\n    chunk_size: int,\n    knowledge_base: List[LangchainDocument],\n    tokenizer_name: Optional[str] = EMBEDDING_MODEL_NAME,\n) -> List[LangchainDocument]:\n    \"\"\"\n    Split documents into chunks of maximum size `chunk_size` tokens and return a list of documents.\n    \"\"\"\n    text_splitter = RecursiveCharacterTextSplitter.from_huggingface_tokenizer(\n        AutoTokenizer.from_pretrained(tokenizer_name),\n        chunk_size=chunk_size,\n        chunk_overlap=int(chunk_size / 10),\n        add_start_index=True,\n        strip_whitespace=True,\n        separators=MARKDOWN_SEPARATORS,\n    )\n\n    docs_processed = []\n    for doc in knowledge_base:\n        docs_processed += text_splitter.split_documents([doc])\n\n    # Remove duplicates\n    unique_texts = {}\n    docs_processed_unique = []\n    for doc in docs_processed:\n        if doc.page_content not in unique_texts:\n            unique_texts[doc.page_content] = True\n            docs_processed_unique.append(doc)\n\n    return docs_processed_unique\n\n\ndocs_processed = split_documents(\n    512,  # We choose a chunk size adapted to our model\n    RAW_KNOWLEDGE_BASE,\n    tokenizer_name=EMBEDDING_MODEL_NAME,\n)\n\n# Let's visualize the chunk sizes we would have in tokens from a common model\nfrom transformers import AutoTokenizer\n\ntokenizer = AutoTokenizer.from_pretrained(EMBEDDING_MODEL_NAME)\nlengths = [len(tokenizer.encode(doc.page_content)) for doc in tqdm(docs_processed)]\nfig = pd.Series(lengths).hist()\nplt.title(\"Distribution of document lengths in the knowledge base (in count of tokens)\")\nplt.show()\n```\n\n➡️ Now the chunk length distribution looks better!\n\n### 1.2 Building the vector database\n\nWe want to compute the embeddings for all the chunks of our knowledge base: to learn more on sentence embeddings, we recommend reading [this guide](https://osanseviero.github.io/hackerllama/blog/posts/sentence_embeddings/).\n\n#### How does retrieval work ?\n\nOnce the chunks are all embedded, we store them into a vector database. When the user types in a query, it gets embedded by the same model previously used, and a similarity search returns the closest documents from the vector database.\n\nThe technical challenge is thus, given a query vector, to quickly find the nearest neighbours of this vector in the vector database. To do this, we need to choose two things: a distance, and a search algorithm to find the nearest neighbors quickly within a database of thousands of records.\n\n##### Nearest Neighbor search algorithm\n\nThere are plentiful choices for the nearest neighbor search algorithm: we go with Facebook's [FAISS](https://github.com/facebookresearch/faiss), since FAISS is performant enough for most use cases, and it is well known thus widely implemented.\n\n##### Distances\n\nRegarding distances, you can find a good guide [here](https://osanseviero.github.io/hackerllama/blog/posts/sentence_embeddings/#distance-between-embeddings). In short:\n\n- **Cosine similarity** computes similarity between two vectors as the cosinus of their relative angle: it allows us to compare vector directions are regardless of their magnitude. Using it requires to normalize all vectors, to rescale them into unit norm.\n- **Dot product** takes into account magnitude, with the sometimes undesirable effect that increasing a vector's length will make it more similar to all others.\n- **Euclidean distance** is the distance between the ends of vectors.\n\nYou can try [this small exercise](https://developers.google.com/machine-learning/clustering/similarity/check-your-understanding) to check your understanding of these concepts. But once vectors are normalized, [the choice of a specific distance does not matter much](https://platform.openai.com/docs/guides/embeddings/which-distance-function-should-i-use).\n\nOur particular model works well with cosine similarity, so choose this distance, and we set it up both in the Embedding model, and in the `distance_strategy` argument of our FAISS index. With cosine similarity, we have to normalize our embeddings.\n\n::: {.callout-warning}\n🚨👇 The cell below takes a few minutes to run on A10G!\n:::\n\n```{python}\nfrom langchain.vectorstores import FAISS\nfrom langchain_community.embeddings import HuggingFaceEmbeddings\nfrom langchain_community.vectorstores.utils import DistanceStrategy\n\nembedding_model = HuggingFaceEmbeddings(\n    model_name=EMBEDDING_MODEL_NAME,\n    multi_process=True,\n    model_kwargs={\"device\": \"cuda\"},\n    encode_kwargs={\"normalize_embeddings\": True},  # set True for cosine similarity\n)\n\nKNOWLEDGE_VECTOR_DATABASE = FAISS.from_documents(\n    docs_processed, embedding_model, distance_strategy=DistanceStrategy.COSINE\n)\n```\n\n👀 To visualize the search for the closest documents, let's project our embeddings from 384 dimensions down to 2 dimensions using PaCMAP.\n\n::: {.callout-note}\n💡 We chose PaCMAP rather than other techniques such as t-SNE or UMAP, since [it is efficient (preserves local and global structure), robust to initialization parameters and fast](https://www.nature.com/articles/s42003-022-03628-x#Abs1).\n:::\n\n\n```{python}\n# embed a user query in the same space\nuser_query = \"How to create a pipeline object?\"\nquery_vector = embedding_model.embed_query(user_query)\n```\n\n```{python}\nimport pacmap\nimport numpy as np\nimport plotly.express as px\n\nembedding_projector = pacmap.PaCMAP(\n    n_components=2, n_neighbors=None, MN_ratio=0.5, FP_ratio=2.0, random_state=1\n)\n\nembeddings_2d = [\n    list(KNOWLEDGE_VECTOR_DATABASE.index.reconstruct_n(idx, 1)[0])\n    for idx in range(len(docs_processed))\n] + [query_vector]\n\n# fit the data (The index of transformed data corresponds to the index of the original data)\ndocuments_projected = embedding_projector.fit_transform(np.array(embeddings_2d), init=\"pca\")\n```\n\n```{python}\ndf = pd.DataFrame.from_dict(\n    [\n        {\n            \"x\": documents_projected[i, 0],\n            \"y\": documents_projected[i, 1],\n            \"source\": docs_processed[i].metadata[\"source\"].split(\"/\")[1],\n            \"extract\": docs_processed[i].page_content[:100] + \"...\",\n            \"symbol\": \"circle\",\n            \"size_col\": 4,\n        }\n        for i in range(len(docs_processed))\n    ]\n    + [\n        {\n            \"x\": documents_projected[-1, 0],\n            \"y\": documents_projected[-1, 1],\n            \"source\": \"User query\",\n            \"extract\": user_query,\n            \"size_col\": 100,\n            \"symbol\": \"star\",\n        }\n    ]\n)\n\n# visualize the embedding\nfig = px.scatter(\n    df,\n    x=\"x\",\n    y=\"y\",\n    color=\"source\",\n    hover_data=\"extract\",\n    size=\"size_col\",\n    symbol=\"symbol\",\n    color_discrete_map={\"User query\": \"black\"},\n    width=1000,\n    height=700,\n)\nfig.update_traces(\n    marker=dict(opacity=1, line=dict(width=0, color=\"DarkSlateGrey\")), selector=dict(mode=\"markers\")\n)\nfig.update_layout(\n    legend_title_text=\"<b>Chunk source</b>\",\n    title=\"<b>2D Projection of Chunk Embeddings via PaCMAP</b>\",\n)\nfig.show()\n```\n\n<img src=\"https://huggingface.co/datasets/huggingface/cookbook-images/resolve/main/PaCMAP_embeddings.png\" height=\"700\">\n\n\n➡️ On the graph above, you can see a spatial representation of the kowledge base documents. As the vector embeddings represent the document's meaning, their closeness in meaning should be reflected in their embedding's closeness.\n\nThe user query's embedding is also shown : we want to find the `k` document that have the closest meaning, thus we pick the `k` closest vectors.\n\nIn the LangChain vector database implementation, this search operation is performed by the method `vector_database.similarity_search(query)`.\n\nHere is the result:\n\n```{python}\nprint(f\"\\nStarting retrieval for {user_query=}...\")\nretrieved_docs = KNOWLEDGE_VECTOR_DATABASE.similarity_search(query=user_query, k=5)\nprint(\"\\n==================================Top document==================================\")\nprint(retrieved_docs[0].page_content)\nprint(\"==================================Metadata==================================\")\nprint(retrieved_docs[0].metadata)\n```\n\n# 2. Reader - LLM 💬\n\nIn this part, the __LLM Reader reads the retrieved context to formulate its answer.__\n\nThere are actually substeps that can all be tuned:\n1. The content of the retrieved documents is aggregated together into the \"context\", with many processing options like _prompt compression_.\n2. The context and the user query are aggregated into a prompt then given to the LLM to generate its answer.\n\n### 2.1. Reader model\n\nThe choice of a reader model is important on a few aspects:\n- the reader model's `max_seq_length` must accomodate our prompt, which includes the context output by the retriever call: the context consists in 5 documents of 512 tokens each, so we aim for a context length of 4k tokens at least.\n- the reader model\n\nFor this example, we chose [`HuggingFaceH4/zephyr-7b-beta`](https://huggingface.co/HuggingFaceH4/zephyr-7b-beta), a small but powerful model.\n\n::: callout-note\nWith many models being released every week, you may want to substitute this model to the latest and greatest. The best way to keep track of open source LLMs is to check the [Open-source LLM leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard).\n:::\n\nTo make inference faster, we will load the quantized version of the model:\n\n```{python}\n#| colab: {referenced_widgets: [db31fd28d3604e78aead26af87b0384f]}\nfrom transformers import pipeline\nimport torch\nfrom transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig\n\nREADER_MODEL_NAME = \"HuggingFaceH4/zephyr-7b-beta\"\n\nbnb_config = BitsAndBytesConfig(\n    load_in_4bit=True,\n    bnb_4bit_use_double_quant=True,\n    bnb_4bit_quant_type=\"nf4\",\n    bnb_4bit_compute_dtype=torch.bfloat16,\n)\nmodel = AutoModelForCausalLM.from_pretrained(READER_MODEL_NAME, quantization_config=bnb_config)\ntokenizer = AutoTokenizer.from_pretrained(READER_MODEL_NAME)\n\nREADER_LLM = pipeline(\n    model=model,\n    tokenizer=tokenizer,\n    task=\"text-generation\",\n    do_sample=True,\n    temperature=0.2,\n    repetition_penalty=1.1,\n    return_full_text=False,\n    max_new_tokens=500,\n)\n```\n\n```{python}\nREADER_LLM(\"What is 4+4? Answer:\")\n```\n\n### 2.2. Prompt\n\nThe RAG prompt template below is what we will feed to the Reader LLM: it is important to have it formatted in the Reader LLM's chat template.\n\nWe give it our context and the user's question.\n\n```{python}\nprompt_in_chat_format = [\n    {\n        \"role\": \"system\",\n        \"content\": \"\"\"Using the information contained in the context,\ngive a comprehensive answer to the question.\nRespond only to the question asked, response should be concise and relevant to the question.\nProvide the number of the source document when relevant.\nIf the answer cannot be deduced from the context, do not give an answer.\"\"\",\n    },\n    {\n        \"role\": \"user\",\n        \"content\": \"\"\"Context:\n{context}\n---\nNow here is the question you need to answer.\n\nQuestion: {question}\"\"\",\n    },\n]\nRAG_PROMPT_TEMPLATE = tokenizer.apply_chat_template(\n    prompt_in_chat_format, tokenize=False, add_generation_prompt=True\n)\nprint(RAG_PROMPT_TEMPLATE)\n```\n\nLet's test our Reader on our previously retrieved documents!\n\n```{python}\nretrieved_docs_text = [\n    doc.page_content for doc in retrieved_docs\n]  # we only need the text of the documents\ncontext = \"\\nExtracted documents:\\n\"\ncontext += \"\".join([f\"Document {str(i)}:::\\n\" + doc for i, doc in enumerate(retrieved_docs_text)])\n\nfinal_prompt = RAG_PROMPT_TEMPLATE.format(\n    question=\"How to create a pipeline object?\", context=context\n)\n\n# Redact an answer\nanswer = READER_LLM(final_prompt)[0][\"generated_text\"]\nprint(answer)\n```\n\n### 2.3. Reranking\n\nA good option for RAG is to retrieve more documents than you want in the end, then rerank the results with a more powerful retrieval model before keeping only the `top_k`.\n\nFor this, [Colbertv2](https://arxiv.org/abs/2112.01488) is a great choice: instead of a bi-encoder like our classical embedding models, it is a cross-encoder that computes more fine-grained interactions between the query tokens and each document's tokens.\n\nIt is easily usable thanks to [the RAGatouille library](https://github.com/bclavie/RAGatouille).\n\n```{python}\nfrom ragatouille import RAGPretrainedModel\n\nRERANKER = RAGPretrainedModel.from_pretrained(\"colbert-ir/colbertv2.0\")\n```\n\n# 3. Assembling it all!\n\n```{python}\nfrom transformers import Pipeline\n\n\ndef answer_with_rag(\n    question: str,\n    llm: Pipeline,\n    knowledge_index: FAISS,\n    reranker: Optional[RAGPretrainedModel] = None,\n    num_retrieved_docs: int = 30,\n    num_docs_final: int = 5,\n) -> Tuple[str, List[LangchainDocument]]:\n    # Gather documents with retriever\n    print(\"=> Retrieving documents...\")\n    relevant_docs = knowledge_index.similarity_search(query=question, k=num_retrieved_docs)\n    relevant_docs = [doc.page_content for doc in relevant_docs]  # keep only the text\n\n    # Optionally rerank results\n    if reranker:\n        print(\"=> Reranking documents...\")\n        relevant_docs = reranker.rerank(question, relevant_docs, k=num_docs_final)\n        relevant_docs = [doc[\"content\"] for doc in relevant_docs]\n\n    relevant_docs = relevant_docs[:num_docs_final]\n\n    # Build the final prompt\n    context = \"\\nExtracted documents:\\n\"\n    context += \"\".join([f\"Document {str(i)}:::\\n\" + doc for i, doc in enumerate(relevant_docs)])\n\n    final_prompt = RAG_PROMPT_TEMPLATE.format(question=question, context=context)\n\n    # Redact an answer\n    print(\"=> Generating answer...\")\n    answer = llm(final_prompt)[0][\"generated_text\"]\n\n    return answer, relevant_docs\n```\n\nLet's see how our RAG pipeline answers a user query.\n\n```{python}\nquestion = \"how to create a pipeline object?\"\n\nanswer, relevant_docs = answer_with_rag(\n    question, READER_LLM, KNOWLEDGE_VECTOR_DATABASE, reranker=RERANKER\n)\n```\n\n```{python}\nprint(\"==================================Answer==================================\")\nprint(f\"{answer}\")\nprint(\"==================================Source docs==================================\")\nfor i, doc in enumerate(relevant_docs):\n    print(f\"Document {i}------------------------------------------------------------\")\n    print(doc)\n```\n\n✅ We now have a fully functional, performant RAG sytem. That's it for today! Congratulations for making it to the end 🥳\n\n\n# To go further 🗺️\n\nThis is not the end of the journey! You can try many steps to improve your RAG system. We recommend doing so in an iterative way: bring small changes to the system and see what improves performance.\n\n### Setting up an evaluation pipeline\n\n- 💬 \"You cannot improve the model performance that you do not measure\", said Gandhi... or at least Llama2 told me he said it. Anyway, you should absolutely start by measuring performance: this means building a small evaluation dataset, then monitor the performance of your RAG system on this evaluation dataset.\n\n### Improving the retriever\n\n🛠️ __You can use these options to tune the results:__\n\n- Tune the chunking method:\n    - Size of the chunks\n    - Method: split on different separators, use [semantic chunking](https://python.langchain.com/docs/modules/data_connection/document_transformers/semantic-chunker)...\n- Change the embedding model\n\n👷‍♀️ __More could be considered:__\n- Try another chunking method, like semantic chunking\n- Change the index used (here, FAISS)\n- Query expansion: reformulate the user query in slightly different ways to retrieve more documents.\n\n### Improving the reader\n\n🛠️ __Here you can try the following options to improve results:__\n- Tune the prompt\n- Switch reranking on/off\n- Choose a more powerful reader model\n\n💡 __Many options could be considered here to further improve the results:__\n- Compress the retrieved context to keep only the most relevant parts to answer the query.\n- Extend the RAG system to make it more user-friendly:\n    - cite source\n    - make conversational\n\n","srcMarkdownNoYaml":"\n\nThis notebook demonstrates how you can build an advanced RAG (Retrieval Augmented Generation) for answering a user's question about a specific knowledge base (here, the HuggingFace documentation), using LangChain.\n\nFor an introduction to RAG, you can check [this other cookbook](rag_zephyr_langchain.qmd)!\n\nRAG systems are complex, with many moving parts: here a RAG diagram, where we noted in blue all possibilities for system enhancement:\n\n<img src=\"https://huggingface.co/datasets/huggingface/cookbook-images/resolve/main/RAG_workflow.png\" height=\"700\">\n\n::: callout-note\n💡 As you can see, there are many steps to tune in this architecture: tuning the system properly will yield significant performance gains.\n:::\n\nIn this notebook, we will take a look into many of these blue notes to see how to tune your RAG system and get the best performance.\n\n__Let's dig into the model building!__ First, we install the required model dependancies.\n\n```{python}\n!pip install -q torch transformers transformers accelerate bitsandbytes langchain sentence-transformers faiss-gpu openpyxl pacmap\n```\n\n```{python}\n%reload_ext dotenv\n%dotenv\n```\n\n```{python}\nfrom tqdm.notebook import tqdm\nimport pandas as pd\nfrom typing import Optional, List, Tuple\nfrom datasets import Dataset\nimport matplotlib.pyplot as plt\n\npd.set_option(\n    \"display.max_colwidth\", None                    # <1>\n) \n```\n1. This will be helpful when visualizing retriever outputs\n\n### Load your knowledge base\n\n```{python}\nimport datasets\n\nds = datasets.load_dataset(\"m-ric/huggingface_doc\", split=\"train\")\n```\n\n```{python}\nfrom langchain.docstore.document import Document as LangchainDocument\n\nRAW_KNOWLEDGE_BASE = [\n    LangchainDocument(page_content=doc[\"text\"], metadata={\"source\": doc[\"source\"]})\n    for doc in tqdm(ds)\n]\n```\n\n# 1. Retriever - embeddings 🗂️\nThe __retriever acts like an internal search engine__: given the user query, it returns a few relevant snippets from your knowledge base.\n\nThese snippets will then be fed to the Reader Model to help it generate its answer.\n\nSo __our objective here is, given a user question, to find the most snippets from our knowledge base to answer that question.__\n\nThis is a wide objective, it leaves open some questions. How many snippets should we retrieve? This parameter will be named `top_k`.\n\nHow long should these snippets be? This is called the `chunk size`. There's no one-size-fits-all answers, but here are a few elements:\n- 🔀 Your `chunk size` is allowed to vary from one snippet to the other.\n- Since there will always be some noise in your retrieval, increasing the `top_k` increases the chance to get relevant elements in your retrieved snippets. 🎯 Shooting more arrows increases your probability to hit your target.\n- Meanwhile, the summed length of your retrieved documents should not be too high: for instance, for most current models 16k tokens will probably drown your Reader model in information due to [Lost-in-the-middle phenomenon](https://huggingface.co/papers/2307.03172). 🎯 Give your reader model only the most relevant insights, not a huge pile of books!\n\n::: callout-note\nIn this notebook, we use Langchain library since __it offers a huge variety of options for vector databases and allows us to keep document metadata throughout the processing__.\n:::\n\n### 1.1 Split the documents into chunks\n\n- In this part, __we split the documents from our knowledge base into smaller chunks__ which will be the snippets on which the reader LLM will base its answer.\n- The goal is to prepare a collection of **semantically relevant snippets**. So their size should be adapted to precise ideas: too small will truncate ideas, too large will dilute them.\n\n::: callout-tip\n💡 Many options exist for text splitting: splitting on words, on sentence boundaries, recursive chunking that processes documents in a tree-like way to preserve structure information... To learn more about chunking, I recommend you read [this great notebook](https://github.com/FullStackRetrieval-com/RetrievalTutorials/blob/main/5_Levels_Of_Text_Splitting.ipynb) by Greg Kamradt.\n:::\n\n\n- **Recursive chunking** breaks down the text into smaller parts step by step using a given list of separators sorted from the most important to the least important separator. If the first split doesn't give the right size or shape chunks, the method repeats itself on the new chunks using a different separator. For instance with the list of separators `[\"\\n\\n\", \"\\n\", \".\", \"\"]`:\n    - The method will first break down the document wherever there is a double line break `\"\\n\\n\"`.\n    - Resulting documents will be split again on simple line breaks `\"\\n\"`, then on sentence ends `\".\"`.\n    - And finally, if some chunks are still too big, they will be split whenever they overflow the maximum size.\n\n- With this method, the global structure is well preserved, at the expense of getting slight variations in chunk size.\n\n> [This space](https://huggingface.co/spaces/A-Roucher/chunk_visualizer) lets you visualize how different splitting options affect the chunks you get.\n\n🔬 Let's experiment a bit with chunk sizes, beginning with an arbitrary size, and see how splits work. We use Langchain's implementation of recursive chunking with `RecursiveCharacterTextSplitter`.\n- Parameter `chunk_size` controls the length of individual chunks: this length is counted by default as the number of characters in the chunk.\n- Parameter `chunk_overlap` lets adjacent chunks get a bit of overlap on each other. This reduces the probability that an idea could be cut in half by the split between two adjacent chunks. We ~arbitrarily set this to 1/10th of the chunk size, you could try different values!\n\n```{python}\nfrom langchain.text_splitter import RecursiveCharacterTextSplitter\n\n# We use a hierarchical list of separators specifically tailored for splitting Markdown documents\n# This list is taken from LangChain's MarkdownTextSplitter class.\nMARKDOWN_SEPARATORS = [\n    \"\\n#{1,6} \",\n    \"```\\n\",\n    \"\\n\\\\*\\\\*\\\\*+\\n\",\n    \"\\n---+\\n\",\n    \"\\n___+\\n\",\n    \"\\n\\n\",\n    \"\\n\",\n    \" \",\n    \"\",\n]\n\ntext_splitter = RecursiveCharacterTextSplitter(\n    chunk_size=1000,            # <1>\n    chunk_overlap=100,          # <2>\n    add_start_index=True,       # <3>\n    strip_whitespace=True,      # <4>\n    separators=MARKDOWN_SEPARATORS,\n)\n\ndocs_processed = []\nfor doc in RAW_KNOWLEDGE_BASE:\n    docs_processed += text_splitter.split_documents([doc])\n```\n1. The maximum number of characters in a chunk: we selected this value arbitrally\n2. The number of characters to overlap between chunks\n3. If `True`, includes chunk's start index in metadata\n4. If `True`, strips whitespace from the start and end of every document\n\n\nWe also have to keep in mind that when embedding documents, we will use an embedding model that has accepts a certain maximum sequence length `max_seq_length`.\n\nSo we should make sure that our chunk sizes are below this limit, because any longer chunk will be truncated before processing, thus losing relevancy.\n\n```{python}\n#| colab: {referenced_widgets: [ae043feeb0914c879e2a9008b413d952]}\nfrom sentence_transformers import SentenceTransformer\n\n# To get the value of the max sequence_length, we will query the underlying `SentenceTransformer` object used in the RecursiveCharacterTextSplitter.\nprint(\n    f\"Model's maximum sequence length: {SentenceTransformer('thenlper/gte-small').max_seq_length}\"\n)\n\nfrom transformers import AutoTokenizer\n\ntokenizer = AutoTokenizer.from_pretrained(\"thenlper/gte-small\")\nlengths = [len(tokenizer.encode(doc.page_content)) for doc in tqdm(docs_processed)]\n\n# Plot the distrubution of document lengths, counted as the number of tokens\nfig = pd.Series(lengths).hist()\nplt.title(\"Distribution of document lengths in the knowledge base (in count of tokens)\")\nplt.show()\n```\n\n👀 As you can see, __the chunk lengths are not aligned with our limit of 512 tokens__, and some documents are above the limit, thus some part of them will be lost in truncation!\n - So we should change the `RecursiveCharacterTextSplitter` class to count length in number of tokens instead of number of characters.\n - Then we can choose a specific chunk size, here we would choose a lower threshold than 512:\n    - smaller documents could allow the split to focus more on specific ideas.\n    - But too small chunks would split sentences in half, thus losing meaning again: the proper tuning is a matter of balance.\n\n```{python}\n#| colab: {referenced_widgets: [f900cf4ab3a94f45bfa7298f433566ed]}\nfrom langchain.text_splitter import RecursiveCharacterTextSplitter\nfrom transformers import AutoTokenizer\n\nEMBEDDING_MODEL_NAME = \"thenlper/gte-small\"\n\n\ndef split_documents(\n    chunk_size: int,\n    knowledge_base: List[LangchainDocument],\n    tokenizer_name: Optional[str] = EMBEDDING_MODEL_NAME,\n) -> List[LangchainDocument]:\n    \"\"\"\n    Split documents into chunks of maximum size `chunk_size` tokens and return a list of documents.\n    \"\"\"\n    text_splitter = RecursiveCharacterTextSplitter.from_huggingface_tokenizer(\n        AutoTokenizer.from_pretrained(tokenizer_name),\n        chunk_size=chunk_size,\n        chunk_overlap=int(chunk_size / 10),\n        add_start_index=True,\n        strip_whitespace=True,\n        separators=MARKDOWN_SEPARATORS,\n    )\n\n    docs_processed = []\n    for doc in knowledge_base:\n        docs_processed += text_splitter.split_documents([doc])\n\n    # Remove duplicates\n    unique_texts = {}\n    docs_processed_unique = []\n    for doc in docs_processed:\n        if doc.page_content not in unique_texts:\n            unique_texts[doc.page_content] = True\n            docs_processed_unique.append(doc)\n\n    return docs_processed_unique\n\n\ndocs_processed = split_documents(\n    512,  # We choose a chunk size adapted to our model\n    RAW_KNOWLEDGE_BASE,\n    tokenizer_name=EMBEDDING_MODEL_NAME,\n)\n\n# Let's visualize the chunk sizes we would have in tokens from a common model\nfrom transformers import AutoTokenizer\n\ntokenizer = AutoTokenizer.from_pretrained(EMBEDDING_MODEL_NAME)\nlengths = [len(tokenizer.encode(doc.page_content)) for doc in tqdm(docs_processed)]\nfig = pd.Series(lengths).hist()\nplt.title(\"Distribution of document lengths in the knowledge base (in count of tokens)\")\nplt.show()\n```\n\n➡️ Now the chunk length distribution looks better!\n\n### 1.2 Building the vector database\n\nWe want to compute the embeddings for all the chunks of our knowledge base: to learn more on sentence embeddings, we recommend reading [this guide](https://osanseviero.github.io/hackerllama/blog/posts/sentence_embeddings/).\n\n#### How does retrieval work ?\n\nOnce the chunks are all embedded, we store them into a vector database. When the user types in a query, it gets embedded by the same model previously used, and a similarity search returns the closest documents from the vector database.\n\nThe technical challenge is thus, given a query vector, to quickly find the nearest neighbours of this vector in the vector database. To do this, we need to choose two things: a distance, and a search algorithm to find the nearest neighbors quickly within a database of thousands of records.\n\n##### Nearest Neighbor search algorithm\n\nThere are plentiful choices for the nearest neighbor search algorithm: we go with Facebook's [FAISS](https://github.com/facebookresearch/faiss), since FAISS is performant enough for most use cases, and it is well known thus widely implemented.\n\n##### Distances\n\nRegarding distances, you can find a good guide [here](https://osanseviero.github.io/hackerllama/blog/posts/sentence_embeddings/#distance-between-embeddings). In short:\n\n- **Cosine similarity** computes similarity between two vectors as the cosinus of their relative angle: it allows us to compare vector directions are regardless of their magnitude. Using it requires to normalize all vectors, to rescale them into unit norm.\n- **Dot product** takes into account magnitude, with the sometimes undesirable effect that increasing a vector's length will make it more similar to all others.\n- **Euclidean distance** is the distance between the ends of vectors.\n\nYou can try [this small exercise](https://developers.google.com/machine-learning/clustering/similarity/check-your-understanding) to check your understanding of these concepts. But once vectors are normalized, [the choice of a specific distance does not matter much](https://platform.openai.com/docs/guides/embeddings/which-distance-function-should-i-use).\n\nOur particular model works well with cosine similarity, so choose this distance, and we set it up both in the Embedding model, and in the `distance_strategy` argument of our FAISS index. With cosine similarity, we have to normalize our embeddings.\n\n::: {.callout-warning}\n🚨👇 The cell below takes a few minutes to run on A10G!\n:::\n\n```{python}\nfrom langchain.vectorstores import FAISS\nfrom langchain_community.embeddings import HuggingFaceEmbeddings\nfrom langchain_community.vectorstores.utils import DistanceStrategy\n\nembedding_model = HuggingFaceEmbeddings(\n    model_name=EMBEDDING_MODEL_NAME,\n    multi_process=True,\n    model_kwargs={\"device\": \"cuda\"},\n    encode_kwargs={\"normalize_embeddings\": True},  # set True for cosine similarity\n)\n\nKNOWLEDGE_VECTOR_DATABASE = FAISS.from_documents(\n    docs_processed, embedding_model, distance_strategy=DistanceStrategy.COSINE\n)\n```\n\n👀 To visualize the search for the closest documents, let's project our embeddings from 384 dimensions down to 2 dimensions using PaCMAP.\n\n::: {.callout-note}\n💡 We chose PaCMAP rather than other techniques such as t-SNE or UMAP, since [it is efficient (preserves local and global structure), robust to initialization parameters and fast](https://www.nature.com/articles/s42003-022-03628-x#Abs1).\n:::\n\n\n```{python}\n# embed a user query in the same space\nuser_query = \"How to create a pipeline object?\"\nquery_vector = embedding_model.embed_query(user_query)\n```\n\n```{python}\nimport pacmap\nimport numpy as np\nimport plotly.express as px\n\nembedding_projector = pacmap.PaCMAP(\n    n_components=2, n_neighbors=None, MN_ratio=0.5, FP_ratio=2.0, random_state=1\n)\n\nembeddings_2d = [\n    list(KNOWLEDGE_VECTOR_DATABASE.index.reconstruct_n(idx, 1)[0])\n    for idx in range(len(docs_processed))\n] + [query_vector]\n\n# fit the data (The index of transformed data corresponds to the index of the original data)\ndocuments_projected = embedding_projector.fit_transform(np.array(embeddings_2d), init=\"pca\")\n```\n\n```{python}\ndf = pd.DataFrame.from_dict(\n    [\n        {\n            \"x\": documents_projected[i, 0],\n            \"y\": documents_projected[i, 1],\n            \"source\": docs_processed[i].metadata[\"source\"].split(\"/\")[1],\n            \"extract\": docs_processed[i].page_content[:100] + \"...\",\n            \"symbol\": \"circle\",\n            \"size_col\": 4,\n        }\n        for i in range(len(docs_processed))\n    ]\n    + [\n        {\n            \"x\": documents_projected[-1, 0],\n            \"y\": documents_projected[-1, 1],\n            \"source\": \"User query\",\n            \"extract\": user_query,\n            \"size_col\": 100,\n            \"symbol\": \"star\",\n        }\n    ]\n)\n\n# visualize the embedding\nfig = px.scatter(\n    df,\n    x=\"x\",\n    y=\"y\",\n    color=\"source\",\n    hover_data=\"extract\",\n    size=\"size_col\",\n    symbol=\"symbol\",\n    color_discrete_map={\"User query\": \"black\"},\n    width=1000,\n    height=700,\n)\nfig.update_traces(\n    marker=dict(opacity=1, line=dict(width=0, color=\"DarkSlateGrey\")), selector=dict(mode=\"markers\")\n)\nfig.update_layout(\n    legend_title_text=\"<b>Chunk source</b>\",\n    title=\"<b>2D Projection of Chunk Embeddings via PaCMAP</b>\",\n)\nfig.show()\n```\n\n<img src=\"https://huggingface.co/datasets/huggingface/cookbook-images/resolve/main/PaCMAP_embeddings.png\" height=\"700\">\n\n\n➡️ On the graph above, you can see a spatial representation of the kowledge base documents. As the vector embeddings represent the document's meaning, their closeness in meaning should be reflected in their embedding's closeness.\n\nThe user query's embedding is also shown : we want to find the `k` document that have the closest meaning, thus we pick the `k` closest vectors.\n\nIn the LangChain vector database implementation, this search operation is performed by the method `vector_database.similarity_search(query)`.\n\nHere is the result:\n\n```{python}\nprint(f\"\\nStarting retrieval for {user_query=}...\")\nretrieved_docs = KNOWLEDGE_VECTOR_DATABASE.similarity_search(query=user_query, k=5)\nprint(\"\\n==================================Top document==================================\")\nprint(retrieved_docs[0].page_content)\nprint(\"==================================Metadata==================================\")\nprint(retrieved_docs[0].metadata)\n```\n\n# 2. Reader - LLM 💬\n\nIn this part, the __LLM Reader reads the retrieved context to formulate its answer.__\n\nThere are actually substeps that can all be tuned:\n1. The content of the retrieved documents is aggregated together into the \"context\", with many processing options like _prompt compression_.\n2. The context and the user query are aggregated into a prompt then given to the LLM to generate its answer.\n\n### 2.1. Reader model\n\nThe choice of a reader model is important on a few aspects:\n- the reader model's `max_seq_length` must accomodate our prompt, which includes the context output by the retriever call: the context consists in 5 documents of 512 tokens each, so we aim for a context length of 4k tokens at least.\n- the reader model\n\nFor this example, we chose [`HuggingFaceH4/zephyr-7b-beta`](https://huggingface.co/HuggingFaceH4/zephyr-7b-beta), a small but powerful model.\n\n::: callout-note\nWith many models being released every week, you may want to substitute this model to the latest and greatest. The best way to keep track of open source LLMs is to check the [Open-source LLM leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard).\n:::\n\nTo make inference faster, we will load the quantized version of the model:\n\n```{python}\n#| colab: {referenced_widgets: [db31fd28d3604e78aead26af87b0384f]}\nfrom transformers import pipeline\nimport torch\nfrom transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig\n\nREADER_MODEL_NAME = \"HuggingFaceH4/zephyr-7b-beta\"\n\nbnb_config = BitsAndBytesConfig(\n    load_in_4bit=True,\n    bnb_4bit_use_double_quant=True,\n    bnb_4bit_quant_type=\"nf4\",\n    bnb_4bit_compute_dtype=torch.bfloat16,\n)\nmodel = AutoModelForCausalLM.from_pretrained(READER_MODEL_NAME, quantization_config=bnb_config)\ntokenizer = AutoTokenizer.from_pretrained(READER_MODEL_NAME)\n\nREADER_LLM = pipeline(\n    model=model,\n    tokenizer=tokenizer,\n    task=\"text-generation\",\n    do_sample=True,\n    temperature=0.2,\n    repetition_penalty=1.1,\n    return_full_text=False,\n    max_new_tokens=500,\n)\n```\n\n```{python}\nREADER_LLM(\"What is 4+4? Answer:\")\n```\n\n### 2.2. Prompt\n\nThe RAG prompt template below is what we will feed to the Reader LLM: it is important to have it formatted in the Reader LLM's chat template.\n\nWe give it our context and the user's question.\n\n```{python}\nprompt_in_chat_format = [\n    {\n        \"role\": \"system\",\n        \"content\": \"\"\"Using the information contained in the context,\ngive a comprehensive answer to the question.\nRespond only to the question asked, response should be concise and relevant to the question.\nProvide the number of the source document when relevant.\nIf the answer cannot be deduced from the context, do not give an answer.\"\"\",\n    },\n    {\n        \"role\": \"user\",\n        \"content\": \"\"\"Context:\n{context}\n---\nNow here is the question you need to answer.\n\nQuestion: {question}\"\"\",\n    },\n]\nRAG_PROMPT_TEMPLATE = tokenizer.apply_chat_template(\n    prompt_in_chat_format, tokenize=False, add_generation_prompt=True\n)\nprint(RAG_PROMPT_TEMPLATE)\n```\n\nLet's test our Reader on our previously retrieved documents!\n\n```{python}\nretrieved_docs_text = [\n    doc.page_content for doc in retrieved_docs\n]  # we only need the text of the documents\ncontext = \"\\nExtracted documents:\\n\"\ncontext += \"\".join([f\"Document {str(i)}:::\\n\" + doc for i, doc in enumerate(retrieved_docs_text)])\n\nfinal_prompt = RAG_PROMPT_TEMPLATE.format(\n    question=\"How to create a pipeline object?\", context=context\n)\n\n# Redact an answer\nanswer = READER_LLM(final_prompt)[0][\"generated_text\"]\nprint(answer)\n```\n\n### 2.3. Reranking\n\nA good option for RAG is to retrieve more documents than you want in the end, then rerank the results with a more powerful retrieval model before keeping only the `top_k`.\n\nFor this, [Colbertv2](https://arxiv.org/abs/2112.01488) is a great choice: instead of a bi-encoder like our classical embedding models, it is a cross-encoder that computes more fine-grained interactions between the query tokens and each document's tokens.\n\nIt is easily usable thanks to [the RAGatouille library](https://github.com/bclavie/RAGatouille).\n\n```{python}\nfrom ragatouille import RAGPretrainedModel\n\nRERANKER = RAGPretrainedModel.from_pretrained(\"colbert-ir/colbertv2.0\")\n```\n\n# 3. Assembling it all!\n\n```{python}\nfrom transformers import Pipeline\n\n\ndef answer_with_rag(\n    question: str,\n    llm: Pipeline,\n    knowledge_index: FAISS,\n    reranker: Optional[RAGPretrainedModel] = None,\n    num_retrieved_docs: int = 30,\n    num_docs_final: int = 5,\n) -> Tuple[str, List[LangchainDocument]]:\n    # Gather documents with retriever\n    print(\"=> Retrieving documents...\")\n    relevant_docs = knowledge_index.similarity_search(query=question, k=num_retrieved_docs)\n    relevant_docs = [doc.page_content for doc in relevant_docs]  # keep only the text\n\n    # Optionally rerank results\n    if reranker:\n        print(\"=> Reranking documents...\")\n        relevant_docs = reranker.rerank(question, relevant_docs, k=num_docs_final)\n        relevant_docs = [doc[\"content\"] for doc in relevant_docs]\n\n    relevant_docs = relevant_docs[:num_docs_final]\n\n    # Build the final prompt\n    context = \"\\nExtracted documents:\\n\"\n    context += \"\".join([f\"Document {str(i)}:::\\n\" + doc for i, doc in enumerate(relevant_docs)])\n\n    final_prompt = RAG_PROMPT_TEMPLATE.format(question=question, context=context)\n\n    # Redact an answer\n    print(\"=> Generating answer...\")\n    answer = llm(final_prompt)[0][\"generated_text\"]\n\n    return answer, relevant_docs\n```\n\nLet's see how our RAG pipeline answers a user query.\n\n```{python}\nquestion = \"how to create a pipeline object?\"\n\nanswer, relevant_docs = answer_with_rag(\n    question, READER_LLM, KNOWLEDGE_VECTOR_DATABASE, reranker=RERANKER\n)\n```\n\n```{python}\nprint(\"==================================Answer==================================\")\nprint(f\"{answer}\")\nprint(\"==================================Source docs==================================\")\nfor i, doc in enumerate(relevant_docs):\n    print(f\"Document {i}------------------------------------------------------------\")\n    print(doc)\n```\n\n✅ We now have a fully functional, performant RAG sytem. That's it for today! Congratulations for making it to the end 🥳\n\n\n# To go further 🗺️\n\nThis is not the end of the journey! You can try many steps to improve your RAG system. We recommend doing so in an iterative way: bring small changes to the system and see what improves performance.\n\n### Setting up an evaluation pipeline\n\n- 💬 \"You cannot improve the model performance that you do not measure\", said Gandhi... or at least Llama2 told me he said it. Anyway, you should absolutely start by measuring performance: this means building a small evaluation dataset, then monitor the performance of your RAG system on this evaluation dataset.\n\n### Improving the retriever\n\n🛠️ __You can use these options to tune the results:__\n\n- Tune the chunking method:\n    - Size of the chunks\n    - Method: split on different separators, use [semantic chunking](https://python.langchain.com/docs/modules/data_connection/document_transformers/semantic-chunker)...\n- Change the embedding model\n\n👷‍♀️ __More could be considered:__\n- Try another chunking method, like semantic chunking\n- Change the index used (here, FAISS)\n- Query expansion: reformulate the user query in slightly different ways to retrieve more documents.\n\n### Improving the reader\n\n🛠️ __Here you can try the following options to improve results:__\n- Tune the prompt\n- Switch reranking on/off\n- Choose a more powerful reader model\n\n💡 __Many options could be considered here to further improve the results:__\n- Compress the retrieved context to keep only the most relevant parts to answer the query.\n- Extend the RAG system to make it more user-friendly:\n    - cite source\n    - make conversational\n\n"},"formats":{"html":{"identifier":{"display-name":"HTML","target-format":"html","base-format":"html"},"execute":{"fig-width":7,"fig-height":5,"fig-format":"retina","fig-dpi":96,"df-print":"default","error":false,"eval":false,"cache":null,"freeze":false,"echo":true,"output":true,"warning":true,"include":true,"keep-md":false,"keep-ipynb":false,"ipynb":null,"enabled":null,"daemon":null,"daemon-restart":false,"debug":false,"ipynb-filters":[],"ipynb-shell-interactivity":null,"plotly-connected":true,"engine":"jupyter"},"render":{"keep-tex":false,"keep-typ":false,"keep-source":false,"keep-hidden":false,"prefer-html":false,"output-divs":true,"output-ext":"html","fig-align":"default","fig-pos":null,"fig-env":null,"code-fold":"none","code-overflow":"scroll","code-link":false,"code-line-numbers":false,"code-tools":false,"tbl-colwidths":"auto","merge-includes":true,"inline-includes":false,"preserve-yaml":false,"latex-auto-mk":true,"latex-auto-install":true,"latex-clean":true,"latex-min-runs":1,"latex-max-runs":10,"latex-makeindex":"makeindex","latex-makeindex-opts":[],"latex-tlmgr-opts":[],"latex-input-paths":[],"latex-output-dir":null,"link-external-icon":false,"link-external-newwindow":false,"self-contained-math":false,"format-resources":[],"notebook-links":true},"pandoc":{"standalone":true,"wrap":"none","default-image-extension":"png","to":"html","css":["../styles.css"],"toc":true,"output-file":"advanced_rag.html"},"language":{"toc-title-document":"Table of contents","toc-title-website":"On this page","related-formats-title":"Other Formats","related-notebooks-title":"Notebooks","source-notebooks-prefix":"Source","other-links-title":"Other Links","code-links-title":"Code Links","launch-dev-container-title":"Launch Dev Container","launch-binder-title":"Launch Binder","article-notebook-label":"Article Notebook","notebook-preview-download":"Download Notebook","notebook-preview-download-src":"Download Source","notebook-preview-back":"Back to Article","manuscript-meca-bundle":"MECA Bundle","section-title-abstract":"Abstract","section-title-appendices":"Appendices","section-title-footnotes":"Footnotes","section-title-references":"References","section-title-reuse":"Reuse","section-title-copyright":"Copyright","section-title-citation":"Citation","appendix-attribution-cite-as":"For attribution, please cite this work as:","appendix-attribution-bibtex":"BibTeX citation:","appendix-view-license":"View License","title-block-author-single":"Author","title-block-author-plural":"Authors","title-block-affiliation-single":"Affiliation","title-block-affiliation-plural":"Affiliations","title-block-published":"Published","title-block-modified":"Modified","title-block-keywords":"Keywords","callout-tip-title":"Tip","callout-note-title":"Note","callout-warning-title":"Warning","callout-important-title":"Important","callout-caution-title":"Caution","code-summary":"Code","code-tools-menu-caption":"Code","code-tools-show-all-code":"Show All Code","code-tools-hide-all-code":"Hide All Code","code-tools-view-source":"View Source","code-tools-source-code":"Source Code","tools-share":"Share","tools-download":"Download","code-line":"Line","code-lines":"Lines","copy-button-tooltip":"Copy to Clipboard","copy-button-tooltip-success":"Copied!","repo-action-links-edit":"Edit this page","repo-action-links-source":"View source","repo-action-links-issue":"Report an issue","back-to-top":"Back to top","search-no-results-text":"No results","search-matching-documents-text":"matching documents","search-copy-link-title":"Copy link to search","search-hide-matches-text":"Hide additional matches","search-more-match-text":"more match in this document","search-more-matches-text":"more matches in this document","search-clear-button-title":"Clear","search-text-placeholder":"","search-detached-cancel-button-title":"Cancel","search-submit-button-title":"Submit","search-label":"Search","toggle-section":"Toggle section","toggle-sidebar":"Toggle sidebar navigation","toggle-dark-mode":"Toggle dark mode","toggle-reader-mode":"Toggle reader mode","toggle-navigation":"Toggle navigation","crossref-fig-title":"Figure","crossref-tbl-title":"Table","crossref-lst-title":"Listing","crossref-thm-title":"Theorem","crossref-lem-title":"Lemma","crossref-cor-title":"Corollary","crossref-prp-title":"Proposition","crossref-cnj-title":"Conjecture","crossref-def-title":"Definition","crossref-exm-title":"Example","crossref-exr-title":"Exercise","crossref-ch-prefix":"Chapter","crossref-apx-prefix":"Appendix","crossref-sec-prefix":"Section","crossref-eq-prefix":"Equation","crossref-lof-title":"List of Figures","crossref-lot-title":"List of Tables","crossref-lol-title":"List of Listings","environment-proof-title":"Proof","environment-remark-title":"Remark","environment-solution-title":"Solution","listing-page-order-by":"Order By","listing-page-order-by-default":"Default","listing-page-order-by-date-asc":"Oldest","listing-page-order-by-date-desc":"Newest","listing-page-order-by-number-desc":"High to Low","listing-page-order-by-number-asc":"Low to High","listing-page-field-date":"Date","listing-page-field-title":"Title","listing-page-field-description":"Description","listing-page-field-author":"Author","listing-page-field-filename":"File Name","listing-page-field-filemodified":"Modified","listing-page-field-subtitle":"Subtitle","listing-page-field-readingtime":"Reading Time","listing-page-field-wordcount":"Word Count","listing-page-field-categories":"Categories","listing-page-minutes-compact":"{0} min","listing-page-category-all":"All","listing-page-no-matches":"No matching items","listing-page-words":"{0} words","listing-page-filter":"Filter","draft":"Draft"},"metadata":{"lang":"en","fig-responsive":true,"quarto-version":"1.6.40","theme":"cosmo","title":"Advanced RAG","jupyter":"python3","code-annotations":"hover"},"extensions":{"book":{"multiFile":true}}}},"projectFormats":["html"]}

src/.quarto/idx/notebooks/automatic_embedding.ipynb.json

@@ -1 +0,0 @@
-{"title":"Inference Endpoints","markdown":{"yaml":{"title":"Inference Endpoints"},"headingText":"How to use Inference Endpoints to Embed Documents","containsRefs":false,"markdown":"\n\n\n_Authored by: [Derek Thomas](https://huggingface.co/derek-thomas)_\n\n## Goal\nI have a dataset I want to embed for semantic search (or QA, or RAG), I want the easiest way to do embed this and put it in a new dataset.\n\n## Approach\nI'm using a dataset from my favorite subreddit [r/bestofredditorupdates](https://www.reddit.com/r/bestofredditorupdates/). Because it has long entries, I will use the new [jinaai/jina-embeddings-v2-base-en](https://huggingface.co/jinaai/jina-embeddings-v2-base-en) since it has an 8k context length. I will deploy this using [Inference Endpoint](https://huggingface.co/inference-endpoints) to save time and money. To follow this tutorial, you will need to **have already added a payment method**. If you haven't, you can add one here in [billing](https://huggingface.co/docs/hub/billing#billing). To make it even easier, I'll make this fully API based.\n\nTo make this MUCH faster I will use the [Text Embeddings Inference](https://github.com/huggingface/text-embeddings-inference) image. This has many benefits like:\n- No model graph compilation step\n- Small docker images and fast boot times. Get ready for true serverless!\n- Token based dynamic batching\n- Optimized transformers code for inference using Flash Attention, Candle and cuBLASLt\n- Safetensors weight loading\n- Production ready (distributed tracing with Open Telemetry, Prometheus metrics)\n\n![img](https://media.githubusercontent.com/media/huggingface/text-embeddings-inference/main/assets/bs1-tp.png)\n\n## Requirements\n\n## Imports\n\n## Config\n`DATASET_IN` is where your text data is\n`DATASET_OUT` is where your embeddings will be stored\n\nNote I used 5 for the `MAX_WORKERS` since `jina-embeddings-v2` are quite memory hungry. \n\nHugging Face offers a number of GPUs that you can choose from a number of GPUs that you can choose in Inference Endpoints. Here they are in table form:\n\n| GPU                 | instanceType   | instanceSize | vRAM  |\n|---------------------|----------------|--------------|-------|\n| 1x Nvidia Tesla T4 | g4dn.xlarge | small | 16GB |\n| 4x Nvidia Tesla T4 | g4dn.12xlarge | large | 64GB |\n| 1x Nvidia A10G | g5.2xlarge | medium | 24GB |\n| 4x Nvidia A10G | g5.12xlarge | xxlarge | 96GB |\n| 1x Nvidia A100* | p4de | xlarge | 80GB |\n| 2x Nvidia A100* | p4de | 2xlarge | 160GB |\n\n\\*Note that for A100s you might get a note to email us to get access.\n\nSome users might have payment registered in an organization. This allows you to connect to an organization (that you are a member of) with a payment method.\n\nLeave it blank is you want to use your username.\n\n## Get Dataset\n\n# Inference Endpoints\n## Create Inference Endpoint\nWe are going to use the [API](https://huggingface.co/docs/inference-endpoints/api_reference) to create an [Inference Endpoint](https://huggingface.co/inference-endpoints). This should provide a few main benefits:\n- It's convenient (No clicking)\n- It's repeatable (We have the code to run it easily)\n- It's cheaper (No time spent waiting for it to load, and automatically shut it down)\n\n\n\nThere are a few design choices here:\n- As discussed before we are using `jinaai/jina-embeddings-v2-base-en` as our model. \n    - For reproducibility we are pinning it to a specific revision.\n- If you are interested in more models, check out the supported list [here](https://huggingface.co/docs/text-embeddings-inference/supported_models). \n    - Note that most embedding models are based on the BERT architecture.\n- `MAX_BATCH_TOKENS` is chosen based on our number of workers and the context window of our embedding model.\n- `type=\"protected\"` utilized the security from Inference Endpoints detailed here.\n- I'm using **1x Nvidia A10** since `jina-embeddings-v2` is memory hungry (remember the 8k context length). \n- You should consider further tuning `MAX_BATCH_TOKENS` and `MAX_CONCURRENT_REQUESTS` if you have high workloads\n\n\n## Wait until it's running\n\nWhen we use `endpoint.client.post` we get a bytes string back. This is a little tedious because we need to convert this to an `np.array`, but it's just a couple quick lines in python.\n\nYou may have inputs that exceed the context. In such scenarios, it's up to you to handle them. In my case, I'd like to truncate rather than have an error. Let's test that it works.\n\n# Get Embeddings\n\nHere I send a document, update it with the embedding, and return it. This happens in parallel with `MAX_WORKERS`.\n\n## Pause Inference Endpoint\nNow that we have finished, let's pause the endpoint so we don't incur any extra charges, this will also allow us to analyze the cost.\n\n# Push updated dataset to Hub\nWe now have our documents updated with the embeddings we wanted. First we need to convert it back to a `Dataset` format. I find it easiest to go from list of dicts -> `pd.DataFrame` -> `Dataset`\n\nI'm uploading it to the user's account by default (as opposed to uploading to an organization) but feel free to push to wherever you want by setting the user in the `repo_id` or in the config by setting `DATASET_OUT`\n\n# Analyze Usage\n1. Go to your `dashboard_url` printed below\n1. Click on the Usage & Cost tab\n1. See how much you have spent\n\nWe can see that it only took `$0.04` to pay for this!\n\n\n\n# Delete Endpoint\nNow that we are done, we don't need our endpoint anymore. We can delete our endpoint programmatically. \n\n![Cost](https://huggingface.co/datasets/huggingface/cookbook-images/resolve/main/automatic_embedding_tei_inference_endpoints.png)\n","srcMarkdownNoYaml":"\n\n# How to use Inference Endpoints to Embed Documents\n\n_Authored by: [Derek Thomas](https://huggingface.co/derek-thomas)_\n\n## Goal\nI have a dataset I want to embed for semantic search (or QA, or RAG), I want the easiest way to do embed this and put it in a new dataset.\n\n## Approach\nI'm using a dataset from my favorite subreddit [r/bestofredditorupdates](https://www.reddit.com/r/bestofredditorupdates/). Because it has long entries, I will use the new [jinaai/jina-embeddings-v2-base-en](https://huggingface.co/jinaai/jina-embeddings-v2-base-en) since it has an 8k context length. I will deploy this using [Inference Endpoint](https://huggingface.co/inference-endpoints) to save time and money. To follow this tutorial, you will need to **have already added a payment method**. If you haven't, you can add one here in [billing](https://huggingface.co/docs/hub/billing#billing). To make it even easier, I'll make this fully API based.\n\nTo make this MUCH faster I will use the [Text Embeddings Inference](https://github.com/huggingface/text-embeddings-inference) image. This has many benefits like:\n- No model graph compilation step\n- Small docker images and fast boot times. Get ready for true serverless!\n- Token based dynamic batching\n- Optimized transformers code for inference using Flash Attention, Candle and cuBLASLt\n- Safetensors weight loading\n- Production ready (distributed tracing with Open Telemetry, Prometheus metrics)\n\n![img](https://media.githubusercontent.com/media/huggingface/text-embeddings-inference/main/assets/bs1-tp.png)\n\n## Requirements\n\n## Imports\n\n## Config\n`DATASET_IN` is where your text data is\n`DATASET_OUT` is where your embeddings will be stored\n\nNote I used 5 for the `MAX_WORKERS` since `jina-embeddings-v2` are quite memory hungry. \n\nHugging Face offers a number of GPUs that you can choose from a number of GPUs that you can choose in Inference Endpoints. Here they are in table form:\n\n| GPU                 | instanceType   | instanceSize | vRAM  |\n|---------------------|----------------|--------------|-------|\n| 1x Nvidia Tesla T4 | g4dn.xlarge | small | 16GB |\n| 4x Nvidia Tesla T4 | g4dn.12xlarge | large | 64GB |\n| 1x Nvidia A10G | g5.2xlarge | medium | 24GB |\n| 4x Nvidia A10G | g5.12xlarge | xxlarge | 96GB |\n| 1x Nvidia A100* | p4de | xlarge | 80GB |\n| 2x Nvidia A100* | p4de | 2xlarge | 160GB |\n\n\\*Note that for A100s you might get a note to email us to get access.\n\nSome users might have payment registered in an organization. This allows you to connect to an organization (that you are a member of) with a payment method.\n\nLeave it blank is you want to use your username.\n\n## Get Dataset\n\n# Inference Endpoints\n## Create Inference Endpoint\nWe are going to use the [API](https://huggingface.co/docs/inference-endpoints/api_reference) to create an [Inference Endpoint](https://huggingface.co/inference-endpoints). This should provide a few main benefits:\n- It's convenient (No clicking)\n- It's repeatable (We have the code to run it easily)\n- It's cheaper (No time spent waiting for it to load, and automatically shut it down)\n\n\n\nThere are a few design choices here:\n- As discussed before we are using `jinaai/jina-embeddings-v2-base-en` as our model. \n    - For reproducibility we are pinning it to a specific revision.\n- If you are interested in more models, check out the supported list [here](https://huggingface.co/docs/text-embeddings-inference/supported_models). \n    - Note that most embedding models are based on the BERT architecture.\n- `MAX_BATCH_TOKENS` is chosen based on our number of workers and the context window of our embedding model.\n- `type=\"protected\"` utilized the security from Inference Endpoints detailed here.\n- I'm using **1x Nvidia A10** since `jina-embeddings-v2` is memory hungry (remember the 8k context length). \n- You should consider further tuning `MAX_BATCH_TOKENS` and `MAX_CONCURRENT_REQUESTS` if you have high workloads\n\n\n## Wait until it's running\n\nWhen we use `endpoint.client.post` we get a bytes string back. This is a little tedious because we need to convert this to an `np.array`, but it's just a couple quick lines in python.\n\nYou may have inputs that exceed the context. In such scenarios, it's up to you to handle them. In my case, I'd like to truncate rather than have an error. Let's test that it works.\n\n# Get Embeddings\n\nHere I send a document, update it with the embedding, and return it. This happens in parallel with `MAX_WORKERS`.\n\n## Pause Inference Endpoint\nNow that we have finished, let's pause the endpoint so we don't incur any extra charges, this will also allow us to analyze the cost.\n\n# Push updated dataset to Hub\nWe now have our documents updated with the embeddings we wanted. First we need to convert it back to a `Dataset` format. I find it easiest to go from list of dicts -> `pd.DataFrame` -> `Dataset`\n\nI'm uploading it to the user's account by default (as opposed to uploading to an organization) but feel free to push to wherever you want by setting the user in the `repo_id` or in the config by setting `DATASET_OUT`\n\n# Analyze Usage\n1. Go to your `dashboard_url` printed below\n1. Click on the Usage & Cost tab\n1. See how much you have spent\n\nWe can see that it only took `$0.04` to pay for this!\n\n\n\n# Delete Endpoint\nNow that we are done, we don't need our endpoint anymore. We can delete our endpoint programmatically. \n\n![Cost](https://huggingface.co/datasets/huggingface/cookbook-images/resolve/main/automatic_embedding_tei_inference_endpoints.png)\n"},"formats":{"html":{"identifier":{"display-name":"HTML","target-format":"html","base-format":"html"},"execute":{"fig-width":7,"fig-height":5,"fig-format":"retina","fig-dpi":96,"df-print":"default","error":false,"eval":true,"cache":null,"freeze":false,"echo":true,"output":true,"warning":true,"include":true,"keep-md":false,"keep-ipynb":false,"ipynb":null,"enabled":false,"daemon":null,"daemon-restart":false,"debug":false,"ipynb-filters":[],"ipynb-shell-interactivity":null,"plotly-connected":true,"engine":"jupyter"},"render":{"keep-tex":false,"keep-typ":false,"keep-source":false,"keep-hidden":false,"prefer-html":false,"output-divs":true,"output-ext":"html","fig-align":"default","fig-pos":null,"fig-env":null,"code-fold":"none","code-overflow":"scroll","code-link":false,"code-line-numbers":false,"code-tools":false,"tbl-colwidths":"auto","merge-includes":true,"inline-includes":false,"preserve-yaml":false,"latex-auto-mk":true,"latex-auto-install":true,"latex-clean":true,"latex-min-runs":1,"latex-max-runs":10,"latex-makeindex":"makeindex","latex-makeindex-opts":[],"latex-tlmgr-opts":[],"latex-input-paths":[],"latex-output-dir":null,"link-external-icon":false,"link-external-newwindow":false,"self-contained-math":false,"format-resources":[],"notebook-links":true},"pandoc":{"standalone":true,"wrap":"none","default-image-extension":"png","to":"html","css":["../styles.css"],"toc":true,"output-file":"automatic_embedding.html"},"language":{"toc-title-document":"Table of contents","toc-title-website":"On this page","related-formats-title":"Other Formats","related-notebooks-title":"Notebooks","source-notebooks-prefix":"Source","other-links-title":"Other Links","code-links-title":"Code Links","launch-dev-container-title":"Launch Dev Container","launch-binder-title":"Launch Binder","article-notebook-label":"Article Notebook","notebook-preview-download":"Download Notebook","notebook-preview-download-src":"Download Source","notebook-preview-back":"Back to Article","manuscript-meca-bundle":"MECA Bundle","section-title-abstract":"Abstract","section-title-appendices":"Appendices","section-title-footnotes":"Footnotes","section-title-references":"References","section-title-reuse":"Reuse","section-title-copyright":"Copyright","section-title-citation":"Citation","appendix-attribution-cite-as":"For attribution, please cite this work as:","appendix-attribution-bibtex":"BibTeX citation:","appendix-view-license":"View License","title-block-author-single":"Author","title-block-author-plural":"Authors","title-block-affiliation-single":"Affiliation","title-block-affiliation-plural":"Affiliations","title-block-published":"Published","title-block-modified":"Modified","title-block-keywords":"Keywords","callout-tip-title":"Tip","callout-note-title":"Note","callout-warning-title":"Warning","callout-important-title":"Important","callout-caution-title":"Caution","code-summary":"Code","code-tools-menu-caption":"Code","code-tools-show-all-code":"Show All Code","code-tools-hide-all-code":"Hide All Code","code-tools-view-source":"View Source","code-tools-source-code":"Source Code","tools-share":"Share","tools-download":"Download","code-line":"Line","code-lines":"Lines","copy-button-tooltip":"Copy to Clipboard","copy-button-tooltip-success":"Copied!","repo-action-links-edit":"Edit this page","repo-action-links-source":"View source","repo-action-links-issue":"Report an issue","back-to-top":"Back to top","search-no-results-text":"No results","search-matching-documents-text":"matching documents","search-copy-link-title":"Copy link to search","search-hide-matches-text":"Hide additional matches","search-more-match-text":"more match in this document","search-more-matches-text":"more matches in this document","search-clear-button-title":"Clear","search-text-placeholder":"","search-detached-cancel-button-title":"Cancel","search-submit-button-title":"Submit","search-label":"Search","toggle-section":"Toggle section","toggle-sidebar":"Toggle sidebar navigation","toggle-dark-mode":"Toggle dark mode","toggle-reader-mode":"Toggle reader mode","toggle-navigation":"Toggle navigation","crossref-fig-title":"Figure","crossref-tbl-title":"Table","crossref-lst-title":"Listing","crossref-thm-title":"Theorem","crossref-lem-title":"Lemma","crossref-cor-title":"Corollary","crossref-prp-title":"Proposition","crossref-cnj-title":"Conjecture","crossref-def-title":"Definition","crossref-exm-title":"Example","crossref-exr-title":"Exercise","crossref-ch-prefix":"Chapter","crossref-apx-prefix":"Appendix","crossref-sec-prefix":"Section","crossref-eq-prefix":"Equation","crossref-lof-title":"List of Figures","crossref-lot-title":"List of Tables","crossref-lol-title":"List of Listings","environment-proof-title":"Proof","environment-remark-title":"Remark","environment-solution-title":"Solution","listing-page-order-by":"Order By","listing-page-order-by-default":"Default","listing-page-order-by-date-asc":"Oldest","listing-page-order-by-date-desc":"Newest","listing-page-order-by-number-desc":"High to Low","listing-page-order-by-number-asc":"Low to High","listing-page-field-date":"Date","listing-page-field-title":"Title","listing-page-field-description":"Description","listing-page-field-author":"Author","listing-page-field-filename":"File Name","listing-page-field-filemodified":"Modified","listing-page-field-subtitle":"Subtitle","listing-page-field-readingtime":"Reading Time","listing-page-field-wordcount":"Word Count","listing-page-field-categories":"Categories","listing-page-minutes-compact":"{0} min","listing-page-category-all":"All","listing-page-no-matches":"No matching items","listing-page-words":"{0} words","listing-page-filter":"Filter","draft":"Draft"},"metadata":{"lang":"en","fig-responsive":true,"quarto-version":"1.6.40","theme":"cosmo","title":"Inference Endpoints"},"extensions":{"book":{"multiFile":true}}}},"projectFormats":["html"]}

src/.quarto/idx/notebooks/faiss.ipynb.json

@@ -1 +0,0 @@
-{"title":"Similarity Search","markdown":{"yaml":{"title":"Similarity Search"},"headingText":"Embedding multimodal data for similarity search using 🤗 transformers, 🤗 datasets and FAISS","containsRefs":false,"markdown":"\n\n\n_Authored by: [Merve Noyan](https://huggingface.co/merve)_\n\nEmbeddings are semantically meaningful compressions of information. They can be used to do similarity search, zero-shot classification or simply train a new model. Use cases for similarity search include searching for similar products in e-commerce, content search in social media and more.\nThis notebook walks you through using 🤗transformers, 🤗datasets and FAISS to create and index embeddings from a feature extraction model to later use them for similarity search.\nLet's install necessary libraries.\n\nFor this tutorial, we will use [CLIP model](https://huggingface.co/openai/clip-vit-base-patch16) to extract the features. CLIP is a revolutionary model that introduced joint training of a text encoder and an image encoder to connect two modalities.\n\nLoad the dataset. To keep this notebook light, we will use a small captioning dataset, [jmhessel/newyorker_caption_contest](https://huggingface.co/datasets/jmhessel/newyorker_caption_contest).\n\nSee an example.\n\nWe don't have to write any function to embed examples or create an index. 🤗 datasets library's FAISS integration abstracts these processes. We can simply use `map` method of the dataset to create a new column with the embeddings for each example like below. Let's create one for text features on the prompt column.\n\nWe can do the same and get the image embeddings.\n\n## Querying the data with text prompts\n\nWe can now query the dataset with text or image to get similar items from it.\n\n## Querying the data with image prompts\n\nImage similarity inference is similar, where you just call `get_image_features`.\n\nSearch for the similar image.\n\nDisplay the most similar image to the beaver image.\n\n## Saving, pushing and loading the embeddings\nWe can save the dataset with embeddings with `save_faiss_index`.\n\n\nIt's a good practice to store the embeddings in a dataset repository, so we will create one and push our embeddings there to pull later.\nWe will login to Hugging Face Hub, create a dataset repository there and push our indexes there and load using `snapshot_download`.\n\n  We can load the embeddings to the dataset with no embeddings using `load_faiss_index`.\n","srcMarkdownNoYaml":"\n\n# Embedding multimodal data for similarity search using 🤗 transformers, 🤗 datasets and FAISS\n\n_Authored by: [Merve Noyan](https://huggingface.co/merve)_\n\nEmbeddings are semantically meaningful compressions of information. They can be used to do similarity search, zero-shot classification or simply train a new model. Use cases for similarity search include searching for similar products in e-commerce, content search in social media and more.\nThis notebook walks you through using 🤗transformers, 🤗datasets and FAISS to create and index embeddings from a feature extraction model to later use them for similarity search.\nLet's install necessary libraries.\n\nFor this tutorial, we will use [CLIP model](https://huggingface.co/openai/clip-vit-base-patch16) to extract the features. CLIP is a revolutionary model that introduced joint training of a text encoder and an image encoder to connect two modalities.\n\nLoad the dataset. To keep this notebook light, we will use a small captioning dataset, [jmhessel/newyorker_caption_contest](https://huggingface.co/datasets/jmhessel/newyorker_caption_contest).\n\nSee an example.\n\nWe don't have to write any function to embed examples or create an index. 🤗 datasets library's FAISS integration abstracts these processes. We can simply use `map` method of the dataset to create a new column with the embeddings for each example like below. Let's create one for text features on the prompt column.\n\nWe can do the same and get the image embeddings.\n\n## Querying the data with text prompts\n\nWe can now query the dataset with text or image to get similar items from it.\n\n## Querying the data with image prompts\n\nImage similarity inference is similar, where you just call `get_image_features`.\n\nSearch for the similar image.\n\nDisplay the most similar image to the beaver image.\n\n## Saving, pushing and loading the embeddings\nWe can save the dataset with embeddings with `save_faiss_index`.\n\n\nIt's a good practice to store the embeddings in a dataset repository, so we will create one and push our embeddings there to pull later.\nWe will login to Hugging Face Hub, create a dataset repository there and push our indexes there and load using `snapshot_download`.\n\n  We can load the embeddings to the dataset with no embeddings using `load_faiss_index`.\n"},"formats":{"html":{"identifier":{"display-name":"HTML","target-format":"html","base-format":"html"},"execute":{"fig-width":7,"fig-height":5,"fig-format":"retina","fig-dpi":96,"df-print":"default","error":false,"eval":true,"cache":null,"freeze":false,"echo":true,"output":true,"warning":true,"include":true,"keep-md":false,"keep-ipynb":false,"ipynb":null,"enabled":false,"daemon":null,"daemon-restart":false,"debug":false,"ipynb-filters":[],"ipynb-shell-interactivity":null,"plotly-connected":true,"engine":"jupyter"},"render":{"keep-tex":false,"keep-typ":false,"keep-source":false,"keep-hidden":false,"prefer-html":false,"output-divs":true,"output-ext":"html","fig-align":"default","fig-pos":null,"fig-env":null,"code-fold":"none","code-overflow":"scroll","code-link":false,"code-line-numbers":false,"code-tools":false,"tbl-colwidths":"auto","merge-includes":true,"inline-includes":false,"preserve-yaml":false,"latex-auto-mk":true,"latex-auto-install":true,"latex-clean":true,"latex-min-runs":1,"latex-max-runs":10,"latex-makeindex":"makeindex","latex-makeindex-opts":[],"latex-tlmgr-opts":[],"latex-input-paths":[],"latex-output-dir":null,"link-external-icon":false,"link-external-newwindow":false,"self-contained-math":false,"format-resources":[],"notebook-links":true},"pandoc":{"standalone":true,"wrap":"none","default-image-extension":"png","to":"html","css":["../styles.css"],"toc":true,"output-file":"faiss.html"},"language":{"toc-title-document":"Table of contents","toc-title-website":"On this page","related-formats-title":"Other Formats","related-notebooks-title":"Notebooks","source-notebooks-prefix":"Source","other-links-title":"Other Links","code-links-title":"Code Links","launch-dev-container-title":"Launch Dev Container","launch-binder-title":"Launch Binder","article-notebook-label":"Article Notebook","notebook-preview-download":"Download Notebook","notebook-preview-download-src":"Download Source","notebook-preview-back":"Back to Article","manuscript-meca-bundle":"MECA Bundle","section-title-abstract":"Abstract","section-title-appendices":"Appendices","section-title-footnotes":"Footnotes","section-title-references":"References","section-title-reuse":"Reuse","section-title-copyright":"Copyright","section-title-citation":"Citation","appendix-attribution-cite-as":"For attribution, please cite this work as:","appendix-attribution-bibtex":"BibTeX citation:","appendix-view-license":"View License","title-block-author-single":"Author","title-block-author-plural":"Authors","title-block-affiliation-single":"Affiliation","title-block-affiliation-plural":"Affiliations","title-block-published":"Published","title-block-modified":"Modified","title-block-keywords":"Keywords","callout-tip-title":"Tip","callout-note-title":"Note","callout-warning-title":"Warning","callout-important-title":"Important","callout-caution-title":"Caution","code-summary":"Code","code-tools-menu-caption":"Code","code-tools-show-all-code":"Show All Code","code-tools-hide-all-code":"Hide All Code","code-tools-view-source":"View Source","code-tools-source-code":"Source Code","tools-share":"Share","tools-download":"Download","code-line":"Line","code-lines":"Lines","copy-button-tooltip":"Copy to Clipboard","copy-button-tooltip-success":"Copied!","repo-action-links-edit":"Edit this page","repo-action-links-source":"View source","repo-action-links-issue":"Report an issue","back-to-top":"Back to top","search-no-results-text":"No results","search-matching-documents-text":"matching documents","search-copy-link-title":"Copy link to search","search-hide-matches-text":"Hide additional matches","search-more-match-text":"more match in this document","search-more-matches-text":"more matches in this document","search-clear-button-title":"Clear","search-text-placeholder":"","search-detached-cancel-button-title":"Cancel","search-submit-button-title":"Submit","search-label":"Search","toggle-section":"Toggle section","toggle-sidebar":"Toggle sidebar navigation","toggle-dark-mode":"Toggle dark mode","toggle-reader-mode":"Toggle reader mode","toggle-navigation":"Toggle navigation","crossref-fig-title":"Figure","crossref-tbl-title":"Table","crossref-lst-title":"Listing","crossref-thm-title":"Theorem","crossref-lem-title":"Lemma","crossref-cor-title":"Corollary","crossref-prp-title":"Proposition","crossref-cnj-title":"Conjecture","crossref-def-title":"Definition","crossref-exm-title":"Example","crossref-exr-title":"Exercise","crossref-ch-prefix":"Chapter","crossref-apx-prefix":"Appendix","crossref-sec-prefix":"Section","crossref-eq-prefix":"Equation","crossref-lof-title":"List of Figures","crossref-lot-title":"List of Tables","crossref-lol-title":"List of Listings","environment-proof-title":"Proof","environment-remark-title":"Remark","environment-solution-title":"Solution","listing-page-order-by":"Order By","listing-page-order-by-default":"Default","listing-page-order-by-date-asc":"Oldest","listing-page-order-by-date-desc":"Newest","listing-page-order-by-number-desc":"High to Low","listing-page-order-by-number-asc":"Low to High","listing-page-field-date":"Date","listing-page-field-title":"Title","listing-page-field-description":"Description","listing-page-field-author":"Author","listing-page-field-filename":"File Name","listing-page-field-filemodified":"Modified","listing-page-field-subtitle":"Subtitle","listing-page-field-readingtime":"Reading Time","listing-page-field-wordcount":"Word Count","listing-page-field-categories":"Categories","listing-page-minutes-compact":"{0} min","listing-page-category-all":"All","listing-page-no-matches":"No matching items","listing-page-words":"{0} words","listing-page-filter":"Filter","draft":"Draft"},"metadata":{"lang":"en","fig-responsive":true,"quarto-version":"1.6.40","theme":"cosmo","title":"Similarity Search"},"extensions":{"book":{"multiFile":true}}}},"projectFormats":["html"]}

src/.quarto/idx/notebooks/rag_evaluation.qmd.json

@@ -1 +0,0 @@
-{"title":"RAG Evaluation","markdown":{"yaml":{"title":"RAG Evaluation","jupyter":"python3","eval":false},"headingText":"Load your knowledge base","containsRefs":false,"markdown":"\n\n```{python}\n!pip install -q torch transformers transformers langchain sentence-transformers faiss-gpu openpyxl openai\n```\n\n```{python}\n%reload_ext autoreload\n%autoreload 2\n%reload_ext dotenv\n%dotenv\n```\n\n```{python}\nfrom tqdm.notebook import tqdm\nimport pandas as pd\nfrom typing import Optional, List, Tuple\nfrom langchain_core.language_models import BaseChatModel\nimport json\nimport datasets\n\npd.set_option(\"display.max_colwidth\", None)\n```\n\n\n```{python}\nds = datasets.load_dataset(\"m-ric/huggingface_doc\", split=\"train\")\n```\n\n# 1. Build a synthetic dataset for evaluation\nWe first build a synthetic dataset of questions and associated contexts. The method is to get elements from our knowledge base, and ask an LLM to generate questions based on these documents.\n\nThen we setup other LLM agents to act as quality filters for the generated QA couples: each of them will act as the filter for a specific flaw.\n\n### 1.1. Prepare source documents\n\n```{python}\nfrom langchain.text_splitter import RecursiveCharacterTextSplitter\nfrom langchain.docstore.document import Document as LangchainDocument\n\nlangchain_docs = [\n    LangchainDocument(page_content=doc[\"text\"], metadata={\"source\": doc[\"source\"]})\n    for doc in tqdm(ds)\n]\n\n\ntext_splitter = RecursiveCharacterTextSplitter(\n    chunk_size=2000,\n    chunk_overlap=200,\n    add_start_index=True,\n    separators=[\"\\n\\n\", \"\\n\", \".\", \" \", \"\"],\n)\n\ndocs_processed = []\nfor doc in langchain_docs:\n    docs_processed += text_splitter.split_documents([doc])\n```\n\n### 1.2. Setup agents for question generation\n\nWe use [Mixtral](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1) for QA couple generation because it it has excellent performance in leaderboards such as [Chatbot Arena](https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard).\n\n```{python}\nfrom langchain_community.llms import HuggingFaceHub\n\nrepo_id = \"mistralai/Mixtral-8x7B-Instruct-v0.1\"\n\nllm = HuggingFaceHub(\n    repo_id=repo_id,\n    task=\"text-generation\",\n    model_kwargs={\n        \"max_new_tokens\": 512,\n        \"top_k\": 30,\n        \"temperature\": 0.1,\n        \"repetition_penalty\": 1.03,\n    },\n)\n```\n\n```{python}\nfrom langchain_community.chat_models import ChatHuggingFace\n\nchat_model = ChatHuggingFace(llm=llm)\n```\n\n```{python}\nfrom langchain.prompts import ChatPromptTemplate\n\nQA_generation_prompt = \"\"\"\nYour task is to write a factoid question and an answer given a context.\nYour factoid question should be answerable with a specific, concise piece of factual information from the context.\nYour factoid question should be formulated in the same style as questions users could ask in a search engine.\nThis means that your factoid question MUST NOT mention something like \"according to the passage\" or \"context\".\n\nProvide your answer as follows:\n\nOutput:::\nFactoid question: (your factoid question)\nAnswer: (your answer to the factoid question)\n\nNow here is the context.\n\nContext: {context}\\n\nOutput:::\"\"\"\n\nQA_generation_prompt = ChatPromptTemplate.from_template(QA_generation_prompt)\nQA_generation_agent = QA_generation_prompt | chat_model\n```\n\nNow let's generate our QA couples.\nFor this example, we generate only 10 QA couples and will load the rest from the Hub.\n\nBut for your specific knowledge base, given that you want to get at least ~100 test samples, and accounting for the fact that we will filter out around half of these with our critique agents later on, you should generate much more, in the >200 samples.\n\n```{python}\nimport random\n\nN_GENERATIONS = (\n    10  # We intentionally generate only 10 QA couples here for cost and time considerations\n)\n\nprint(f\"Generating {N_GENERATIONS} QA couples...\")\noutputs = []\nfor context in tqdm(random.sample(langchain_docs, N_GENERATIONS)):\n    # Generate QA couple\n    output_QA_couple = QA_generation_agent.invoke({\"context\": context.page_content}).content\n    try:\n        question = output_QA_couple.split(\"Factoid question: \")[1].split(\"Answer: \")[0]\n        answer = output_QA_couple.split(\"Answer: \")[1]\n        outputs.append(\n            {\n                \"context\": context.page_content,\n                \"question\": question,\n                \"answer\": answer,\n                \"source_doc\": context.metadata[\"source\"],\n            }\n        )\n    except:\n        continue\n```\n\n```{python}\ndisplay(pd.DataFrame(outputs).head(1))\n```\n\n### 1.3. Setup critique agents\n\nThe questions generated by the previous agent can have many flaws: we should do a quality check before validating these questions.\n\nWe thus build critique agents that will rate each question on several criteria, given in [this paper](https://huggingface.co/papers/2312.10003):\n- **Groundedness:** can the question be answered from the given context?\n- **Relevance:** is the question relevant to users? For instance, `\"What is the date when transformers 4.29.1 was released?\"` is not relevant for ML practicioners.\n\nOne last failure case we've noticed is when a function is tailored for the particular setting where the question was generated, but undecipherable by itself, like `\"What is the name of the function used in this guide?\"`.\nWe also build a critique agent for this criteria:\n- **Stand-alone**: is the question understandable free of any context, for someone with domain knowledge/Internet access? The opposite of this would be `What is the function used in this article?` for a question generated from a specific blog article.\n\nWe systematically score functions with all these agents, and whenever the score is too low for any one of the agents, we eliminate the question from our eval dataset.\n\n💡 ___When asking the agents to output a score, we first ask them to produce its rationale. This will help us verify scores, but most importantly, asking it to first output rationale gives the model more tokens to think and elaborate an answer before summarizing it into a single score token.___\n\nWe now build and run these critique agents.\n\n```{python}\nquestion_groundedness_critique_prompt = \"\"\"\nYou will be given a context and a question.\nYour task is to provide a 'total rating' scoring how well one can answer the given question unambiguously with the given context.\nGive your answer on a scale of 1 to 5, where 1 means that the question is not answerable at all given the context, and 5 means that the question is clearly and unambiguously answerable with the context.\n\nProvide your answer as follows:\n\nAnswer:::\nEvaluation: (your rationale for the rating)\nTotal rating: (your rating)\n\nNow here are the question and context.\n\nQuestion: {question}\\n\nContext: {context}\\n\nAnswer::: \"\"\"\n\nquestion_relevance_critique_prompt = \"\"\"\nYou will be given a question.\nYour task is to provide a 'total rating' representing how useful this question can be to machine learning developers building NLP applications with the Hugging Face ecosystem.\nGive your answer on a scale of 1 to 5, where 1 means that the question is not useful at all, and 5 means that the question is extremely useful.\n\nProvide your answer as follows:\n\nAnswer:::\nEvaluation: (your rationale for the rating)\nTotal rating: (your rating)\n\nNow here is the question.\n\nQuestion: {question}\\n\nAnswer::: \"\"\"\n\nquestion_standalone_critique_prompt = \"\"\"\nYou will be given a question.\nYour task is to provide a 'total rating' representing how context-independant this question is.\nGive your answer on a scale of 1 to 5, where 1 means that the question only makes sense in a specific context, and 5 means that the question makes sense by itself.\nFor instance, if the question refers to a particular setting, like 'in the context' or 'in the document', the rating must be 1.\nThe questions can contain obscure technical nouns or acronyms like Gradio, Hub, Hugging Face or Space and still be a 5: it must simply be clear to an operator with access to documentation what the question is about.\n\nProvide your answer as follows:\n\nAnswer:::\nEvaluation: (your rationale for the rating)\nTotal rating: (your rating)\n\nNow here is the question.\n\nQuestion: {question}\\n\nAnswer::: \"\"\"\n\nquestion_groundedness_critique_prompt = ChatPromptTemplate.from_template(\n    question_groundedness_critique_prompt\n)\nquestion_groundedness_critique_agent = question_groundedness_critique_prompt | chat_model\n\nquestion_relevance_critique_prompt = ChatPromptTemplate.from_template(\n    question_relevance_critique_prompt\n)\nquestion_relevance_critique_agent = question_relevance_critique_prompt | chat_model\n\nquestion_standalone_critique_prompt = ChatPromptTemplate.from_template(\n    question_standalone_critique_prompt\n)\nquestion_standalone_critique_agent = question_standalone_critique_prompt | chat_model\n```\n\n```{python}\nprint(\"Generating critique for each QA couple...\")\nfor output in tqdm(outputs):\n    # Critique the generated QA couple\n    question_groundedness_evaluation = question_groundedness_critique_agent.invoke(\n        {\"context\": output[\"context\"], \"question\": output[\"question\"]}\n    ).content\n    question_relevance_evaluation = question_relevance_critique_agent.invoke(\n        {\"question\": output[\"question\"]}\n    ).content\n    question_standalone_evaluation = question_standalone_critique_agent.invoke(\n        {\"question\": output[\"question\"]}\n    ).content\n\n    try:\n        groundedness_score = int(question_groundedness_evaluation.split(\"Total rating: \")[1][0])\n        groundedness_eval = question_groundedness_evaluation.split(\"Total rating: \")[0].split(\n            \"Evaluation: \"\n        )[1]\n        relevance_score = int(question_relevance_evaluation.split(\"Total rating: \")[1][0])\n        relevance_eval = question_relevance_evaluation.split(\"Total rating: \")[0].split(\n            \"Evaluation: \"\n        )[1]\n        standalone_score = int(question_standalone_evaluation.split(\"Total rating: \")[1][0])\n        standalone_eval = question_standalone_evaluation.split(\"Total rating: \")[0].split(\n            \"Evaluation: \"\n        )[1]\n        output.update(\n            {\n                \"groundedness_score\": groundedness_score,\n                \"groundedness_eval\": groundedness_eval,\n                \"relevance_score\": relevance_score,\n                \"relevance_eval\": relevance_eval,\n                \"standalone_score\": standalone_score,\n                \"standalone_eval\": standalone_eval,\n            }\n        )\n    except:\n        continue\n```\n\nNow let us filter out bad questions based on our critique agent scores:\n\n```{python}\nimport pandas as pd\n\npd.set_option(\"display.max_colwidth\", None)\n\ngenerated_questions = pd.DataFrame.from_dict(outputs)\n\nprint(\"Evaluation dataset before filtering:\")\ndisplay(\n    generated_questions[\n        [\"question\", \"answer\", \"groundedness_score\", \"relevance_score\", \"standalone_score\"]\n    ]\n)\ngenerated_questions = generated_questions.loc[\n    (generated_questions[\"groundedness_score\"] >= 4)\n    & (generated_questions[\"relevance_score\"] >= 4)\n    & (generated_questions[\"standalone_score\"] >= 4)\n]\nprint(\"============================================\")\nprint(\"Final evaluation dataset:\")\ndisplay(\n    generated_questions[\n        [\"question\", \"answer\", \"groundedness_score\", \"relevance_score\", \"standalone_score\"]\n    ]\n)\n\neval_dataset = datasets.Dataset.from_pandas(\n    generated_questions, split=\"train\", preserve_index=False\n)\n```\n\nNow our synthetic evaluation dataset is complete! We can evaluate different RAG systems on this evaluation dataset.\n\nWe have generated only a few QA couples here to reduce time and cost. But let's kick start the next part by loading a pre-generated dataset:\n\n```{python}\neval_dataset = datasets.load_dataset(\"m-ric/huggingface_doc_qa_eval\", split=\"train\")\n```\n\n# 2. Build our RAG System\n\n### 2.1. Preprocessing documents to build our vector database\n\n- In this part, __we split the documents from our knowledge base into smaller chunks__: these will be the snippets that are picked by the Retriever, to then be ingested by the Reader LLM as supporting elements for its answer.\n- The goal is to build semantically relevant snippets: not too small to be sufficient for supporting an answer, and not too large too avoid diluting individual ideas.\n\nMany options exist for text splitting:\n- split every `n` words / characters, but this has the risk of cutting in half paragraphs or even sentences\n- split after `n` words / character, but only on sentence boundaries\n- **recursive split** tries to preserve even more of the document structure, by processing it tree-like way, splitting first on the largest units (chapters) then recursively splitting on smaller units (paragraphs, sentences).\n\nTo learn more about chunking, I recommend you read [this great notebook](https://github.com/FullStackRetrieval-com/RetrievalTutorials/blob/main/5_Levels_Of_Text_Splitting.ipynb) by Greg Kamradt.\n\n[This space](https://huggingface.co/spaces/m-ric/chunk_visualizer) lets you visualize how different splitting options affect the chunks you get.\n\n> In the following, we use Langchain's `RecursiveCharacterTextSplitter`.\n\n💡 _To measure chunk length in our Text Splitter, our length function will not be the count of characters, but the count of tokens in the tokenized text: indeed, for subsequent embedder that processes token, measuring length in tokens is more relevant and empirically performs better._\n\n```{python}\nfrom langchain.docstore.document import Document as LangchainDocument\n\nRAW_KNOWLEDGE_BASE = [\n    LangchainDocument(page_content=doc[\"text\"], metadata={\"source\": doc[\"source\"]})\n    for doc in tqdm(ds)\n]\n```\n\n```{python}\nfrom langchain.text_splitter import RecursiveCharacterTextSplitter\nfrom transformers import AutoTokenizer\n\n\ndef split_documents(\n    chunk_size: int,\n    knowledge_base: List[LangchainDocument],\n    tokenizer_name: str,\n) -> List[LangchainDocument]:\n    \"\"\"\n    Split documents into chunks of size `chunk_size` characters and return a list of documents.\n    \"\"\"\n    text_splitter = RecursiveCharacterTextSplitter.from_huggingface_tokenizer(\n        AutoTokenizer.from_pretrained(tokenizer_name),\n        chunk_size=chunk_size,\n        chunk_overlap=int(chunk_size / 10),\n        add_start_index=True,\n        strip_whitespace=True,\n        separators=[\"\\n\\n\", \"\\n\", \".\", \" \", \"\"],\n    )\n\n    docs_processed = []\n    for doc in knowledge_base:\n        docs_processed += text_splitter.split_documents([doc])\n\n    # Remove duplicates\n    unique_texts = {}\n    docs_processed_unique = []\n    for doc in docs_processed:\n        if doc.page_content not in unique_texts:\n            unique_texts[doc.page_content] = True\n            docs_processed_unique.append(doc)\n\n    return docs_processed_unique\n```\n\n### 2.2. Retriever - embeddings 🗂️\nThe __retriever acts like an internal search engine__: given the user query, it returns the most relevant documents from your knowledge base.\n\n> For the knowledge base, we use Langchain vector databases since __it offers a convenient [FAISS](https://github.com/facebookresearch/faiss) index and allows us to keep document metadata throughout the processing__.\n\n🛠️ __Options included:__\n\n- Tune the chunking method:\n    - Size of the chunks\n    - Method: split on different separators, use [semantic chunking](https://python.langchain.com/docs/modules/data_connection/document_transformers/semantic-chunker)...\n- Change the embedding model\n\n```{python}\nfrom langchain.vectorstores import FAISS\nfrom langchain_community.embeddings import HuggingFaceEmbeddings\nfrom langchain_community.vectorstores.utils import DistanceStrategy\nimport os\n\n\ndef load_embeddings(\n    langchain_docs: List[LangchainDocument],\n    chunk_size: int,\n    embedding_model_name: Optional[str] = \"thenlper/gte-small\",\n) -> FAISS:\n    \"\"\"\n    Creates a FAISS index from the given embedding model and documents. Loads the index directly if it already exists.\n\n    Args:\n        langchain_docs: list of documents\n        chunk_size: size of the chunks to split the documents into\n        embedding_model_name: name of the embedding model to use\n\n    Returns:\n        FAISS index\n    \"\"\"\n    # load embedding_model\n    embedding_model = HuggingFaceEmbeddings(\n        model_name=embedding_model_name,\n        multi_process=True,\n        model_kwargs={\"device\": \"cuda\"},\n        encode_kwargs={\"normalize_embeddings\": True},  # set True to compute cosine similarity\n    )\n\n    # Check if embeddings already exist on disk\n    index_name = f\"index_chunk:{chunk_size}_embeddings:{embedding_model_name.replace('/', '~')}\"\n    index_folder_path = f\"./data/indexes/{index_name}/\"\n    if os.path.isdir(index_folder_path):\n        return FAISS.load_local(\n            index_folder_path,\n            embedding_model,\n            distance_strategy=DistanceStrategy.COSINE,\n        )\n\n    else:\n        print(\"Index not found, generating it...\")\n        docs_processed = split_documents(\n            chunk_size,\n            langchain_docs,\n            embedding_model_name,\n        )\n        knowledge_index = FAISS.from_documents(\n            docs_processed, embedding_model, distance_strategy=DistanceStrategy.COSINE\n        )\n        knowledge_index.save_local(index_folder_path)\n        return knowledge_index\n```\n\n### 2.3. Reader - LLM 💬\n\nIn this part, the __LLM Reader reads the retrieved documents to formulate its answer.__\n\n🛠️ Here we tried the following options to improve results:\n- Switch reranking on/off\n- Change the reader model\n\n```{python}\nRAG_PROMPT_TEMPLATE = \"\"\"\n<|system|>\nUsing the information contained in the context,\ngive a comprehensive answer to the question.\nRespond only to the question asked, response should be concise and relevant to the question.\nProvide the number of the source document when relevant.\nIf the answer cannot be deduced from the context, do not give an answer.</s>\n<|user|>\nContext:\n{context}\n---\nNow here is the question you need to answer.\n\nQuestion: {question}\n</s>\n<|assistant|>\n\"\"\"\n```\n\n```{python}\nfrom langchain_community.llms import HuggingFaceHub\n\nrepo_id = \"HuggingFaceH4/zephyr-7b-beta\"\nREADER_MODEL_NAME = \"zephyr-7b-beta\"\n\nREADER_LLM = HuggingFaceHub(\n    repo_id=repo_id,\n    task=\"text-generation\",\n    model_kwargs={\n        \"max_new_tokens\": 512,\n        \"top_k\": 30,\n        \"temperature\": 0.1,\n        \"repetition_penalty\": 1.03,\n    },\n)\n```\n\n```{python}\nfrom ragatouille import RAGPretrainedModel\nfrom langchain_core.vectorstores import VectorStore\nfrom langchain_core.language_models.llms import LLM\n\n\ndef answer_with_rag(\n    question: str,\n    llm: LLM,\n    knowledge_index: VectorStore,\n    reranker: Optional[RAGPretrainedModel] = None,\n    num_retrieved_docs: int = 30,\n    num_docs_final: int = 7,\n) -> Tuple[str, List[LangchainDocument]]:\n    \"\"\"Answer a question using RAG with the given knowledge index.\"\"\"\n    # Gather documents with retriever\n    relevant_docs = knowledge_index.similarity_search(query=question, k=num_retrieved_docs)\n    relevant_docs = [doc.page_content for doc in relevant_docs]  # keep only the text\n\n    # Optionally rerank results\n    if reranker:\n        relevant_docs = reranker.rerank(question, relevant_docs, k=num_docs_final)\n        relevant_docs = [doc[\"content\"] for doc in relevant_docs]\n\n    relevant_docs = relevant_docs[:num_docs_final]\n\n    # Build the final prompt\n    context = \"\\nExtracted documents:\\n\"\n    context += \"\".join([f\"Document {str(i)}:::\\n\" + doc for i, doc in enumerate(relevant_docs)])\n\n    final_prompt = RAG_PROMPT_TEMPLATE.format(question=question, context=context)\n\n    # Redact an answer\n    answer = llm(final_prompt)\n\n    return answer, relevant_docs\n```\n\n# 3. Benchmarking the RAG system\n\nThe RAG system and the evaluation datasets are now ready. The last step is to judge the RAG system's output on this evlauation dataset.\n\nTo this end, __we setup a judge agent__. ⚖️🤖\n\nOut of [the different RAG evaluation metrics](https://docs.ragas.io/en/latest/concepts/metrics/index.html), we choose to focus only on faithfulness since it the best end-to-end metric of our system's performance.\n\n> We use GPT4 as a judge for its empirically good performance, but you could try with other models such as [kaist-ai/prometheus-13b-v1.0](https://huggingface.co/kaist-ai/prometheus-13b-v1.0) or [BAAI/JudgeLM-33B-v1.0](https://huggingface.co/BAAI/JudgeLM-33B-v1.0).\n\n💡 _In the evaluation prompt, we give a detailed description each metric on the scale 1-5, as is done in [Prometheus's prompt template](https://huggingface.co/kaist-ai/prometheus-13b-v1.0): this helps the model ground its metric precisely. If instead you give the judge LLM a vague scale to work with, the outputs will not be consistent enough between different examples._\n\n💡 _Again, prompting the LLM to output rationale before giving its final score gives it more tokens to help it formalize and elaborate a judgement._\n\n```{python}\ndef run_rag_tests(\n    eval_dataset: datasets.Dataset,\n    llm: BaseChatModel,\n    knowledge_index: VectorStore,\n    output_file: str,\n    reranker: Optional[RAGPretrainedModel] = None,\n    verbose: Optional[bool] = True,\n    test_settings: Optional[str] = None,  # To document the test settings used\n):\n    \"\"\"Runs RAG tests on the given dataset and saves the results to the given output file.\"\"\"\n    try:  # load previous generations if they exist\n        with open(output_file, \"r\") as f:\n            outputs = json.load(f)\n    except:\n        outputs = []\n\n    for example in tqdm(eval_dataset):\n        question = example[\"question\"]\n        if question in [output[\"question\"] for output in outputs]:\n            continue\n\n        answer, relevant_docs = answer_with_rag(question, llm, knowledge_index, reranker=reranker)\n        if verbose:\n            print(\"=======================================================\")\n            print(f\"Question: {question}\")\n            print(f\"Answer: {answer}\")\n            print(f'True answer: {example[\"answer\"]}')\n        result = {\n            \"question\": question,\n            \"true_answer\": example[\"answer\"],\n            \"source_doc\": example[\"source_doc\"],\n            \"generated_answer\": answer,\n            \"retrieved_docs\": [doc for doc in relevant_docs],\n        }\n        if test_settings:\n            result[\"test_settings\"] = test_settings\n        outputs.append(result)\n\n        with open(output_file, \"w\") as f:\n            json.dump(outputs, f)\n```\n\n```{python}\nEVALUATION_PROMPT = \"\"\"###Task Description:\nAn instruction (might include an Input inside it), a response to evaluate, a reference answer that gets a score of 5, and a score rubric representing a evaluation criteria are given.\n1. Write a detailed feedback that assess the quality of the response strictly based on the given score rubric, not evaluating in general.\n2. After writing a feedback, write a score that is an integer between 1 and 5. You should refer to the score rubric.\n3. The output format should look as follows: \\\"Feedback: {{write a feedback for criteria}} [RESULT] {{an integer number between 1 and 5}}\\\"\n4. Please do not generate any other opening, closing, and explanations. Be sure to include [RESULT] in your output.\n\n###The instruction to evaluate:\n{instruction}\n\n###Response to evaluate:\n{response}\n\n###Reference Answer (Score 5):\n{reference_answer}\n\n###Score Rubrics:\n[Is the response correct, accurate, and factual based on the reference answer?]\nScore 1: The response is completely incorrect, inaccurate, and/or not factual.\nScore 2: The response is mostly incorrect, inaccurate, and/or not factual.\nScore 3: The response is somewhat correct, accurate, and/or factual.\nScore 4: The response is mostly correct, accurate, and factual.\nScore 5: The response is completely correct, accurate, and factual.\n\n###Feedback:\"\"\"\n\nfrom langchain.prompts.chat import (\n    ChatPromptTemplate,\n    HumanMessagePromptTemplate,\n)\nfrom langchain.schema import SystemMessage\n\n\nevaluation_prompt_template = ChatPromptTemplate.from_messages(\n    [\n        SystemMessage(content=\"You are a fair evaluator language model.\"),\n        HumanMessagePromptTemplate.from_template(EVALUATION_PROMPT),\n    ]\n)\n```\n\n```{python}\nfrom langchain.chat_models import ChatOpenAI\n\neval_chat_model = ChatOpenAI(model=\"gpt-4-1106-preview\", temperature=0)\nevaluator_name = \"GPT4\"\n\n\ndef evaluate_answers(\n    answer_path: str,\n    eval_chat_model: BaseChatModel,\n    evaluator_name: str,\n    evaluation_prompt_template: ChatPromptTemplate,\n) -> None:\n    \"\"\"Evaluates generated answers. Modifies the given answer file in place for better checkpointing.\"\"\"\n    answers = []\n    if os.path.isfile(answer_path):  # load previous generations if they exist\n        answers = json.load(open(answer_path, \"r\"))\n\n    for experiment in tqdm(answers):\n        if f\"eval_score_{evaluator_name}\" in experiment:\n            continue\n\n        eval_prompt = evaluation_prompt_template.format_messages(\n            instruction=experiment[\"question\"],\n            response=experiment[\"generated_answer\"],\n            reference_answer=experiment[\"true_answer\"],\n        )\n        eval_result = eval_chat_model.invoke(eval_prompt)\n        feedback, score = [item.strip() for item in eval_result.content.split(\"[RESULT]\")]\n        experiment[f\"eval_score_{evaluator_name}\"] = score\n        experiment[f\"eval_feedback_{evaluator_name}\"] = feedback\n\n        with open(answer_path, \"w\") as f:\n            json.dump(answers, f)\n```\n\n🚀 Let's run the tests and evaluate answers!👇\n\n```{python}\nif not os.path.exists(\"./output\"):\n    os.mkdir(\"./output\")\n\nfor chunk_size in [200]:  # Add other chunk sizes (in tokens) as needed\n    for embeddings in [\"thenlper/gte-small\"]:  # Add other embeddings as needed\n        for rerank in [True, False]:\n            settings_name = f\"chunk:{chunk_size}_embeddings:{embeddings.replace('/', '~')}_rerank:{rerank}_reader-model:{READER_MODEL_NAME}\"\n            output_file_name = f\"./output/rag_{settings_name}.json\"\n\n            print(f\"Running evaluation for {settings_name}:\")\n\n            print(\"Loading knowledge base embeddings...\")\n            knowledge_index = load_embeddings(\n                RAW_KNOWLEDGE_BASE,\n                chunk_size=chunk_size,\n                embedding_model_name=embeddings,\n            )\n\n            print(\"Running RAG...\")\n            reranker = (\n                RAGPretrainedModel.from_pretrained(\"colbert-ir/colbertv2.0\") if rerank else None\n            )\n            run_rag_tests(\n                eval_dataset=eval_dataset,\n                llm=READER_LLM,\n                knowledge_index=knowledge_index,\n                output_file=output_file_name,\n                reranker=reranker,\n                verbose=False,\n                test_settings=settings_name,\n            )\n\n            print(\"Running evaluation...\")\n            evaluate_answers(\n                output_file_name,\n                eval_chat_model,\n                evaluator_name,\n                evaluation_prompt_template,\n            )\n```\n\n### Inspect results\n\n```{python}\nimport glob\n\noutputs = []\nfor file in glob.glob(\"./output/*.json\"):\n    output = pd.DataFrame(json.load(open(file, \"r\")))\n    output[\"settings\"] = file\n    outputs.append(output)\nresult = pd.concat(outputs)\n```\n\n```{python}\nresult[\"eval_score_GPT4\"] = result[\"eval_score_GPT4\"].apply(\n    lambda x: int(x) if isinstance(x, str) else 1\n)\nresult[\"eval_score_GPT4\"] = (result[\"eval_score_GPT4\"] - 1) / 4\n```\n\n```{python}\naverage_scores = result.groupby(\"settings\")[\"eval_score_GPT4\"].mean()\naverage_scores.sort_values()\n```\n\n## Example results\n\nLet us load the results that I obtained by tweaking the different options available in this notebook.\nFor more detail on why these options could work on not, see the notebook on [advanced_RAG](advanced_rag).\n\nAs you can see in the graph below, some tweaks do not bring any improvement, some give huge performance boosts.\n\n➡️ ___There is no single good recipe: you should try several different directions when tuning your RAG systems.___\n\n```{python}\nimport plotly.express as px\n\nscores = datasets.load_dataset(\"m-ric/rag_scores_cookbook\", split=\"train\")\nscores = pd.Series(scores[\"score\"], index=scores[\"settings\"])\n```\n\n```{python}\nfig = px.bar(\n    scores,\n    color=scores,\n    labels={\n        \"value\": \"Accuracy\",\n        \"settings\": \"Configuration\",\n    },\n    color_continuous_scale=\"bluered\",\n)\nfig.update_layout(w\n    width=1000,\n    height=600,\n    barmode=\"group\",\n    yaxis_range=[0, 100],\n    title=\"<b>Accuracy of different RAG configurations</b>\",\n    xaxis_title=\"RAG settings\",\n    font=dict(size=15),\n)\nfig.layout.yaxis.ticksuffix = \"%\"\nfig.update_coloraxes(showscale=False)\nfig.update_traces(texttemplate=\"%{y:.1f}\", textposition=\"outside\")\nfig.show()\n```\n\n<img src=\"https://huggingface.co/datasets/huggingface/cookbook-images/resolve/main/RAG_settings_accuracy.png\" height=\"500\" width=\"800\">\n\nAs you can see, these had varying impact on performance. In particular, tuning the chunk size is both easy and very impactful.\n\nBut this is our case: your results could be very different: now that you have a robust evaluation pipeline, you can set on to explore other options! 🗺️\n\n","srcMarkdownNoYaml":"\n\n```{python}\n!pip install -q torch transformers transformers langchain sentence-transformers faiss-gpu openpyxl openai\n```\n\n```{python}\n%reload_ext autoreload\n%autoreload 2\n%reload_ext dotenv\n%dotenv\n```\n\n```{python}\nfrom tqdm.notebook import tqdm\nimport pandas as pd\nfrom typing import Optional, List, Tuple\nfrom langchain_core.language_models import BaseChatModel\nimport json\nimport datasets\n\npd.set_option(\"display.max_colwidth\", None)\n```\n\n### Load your knowledge base\n\n```{python}\nds = datasets.load_dataset(\"m-ric/huggingface_doc\", split=\"train\")\n```\n\n# 1. Build a synthetic dataset for evaluation\nWe first build a synthetic dataset of questions and associated contexts. The method is to get elements from our knowledge base, and ask an LLM to generate questions based on these documents.\n\nThen we setup other LLM agents to act as quality filters for the generated QA couples: each of them will act as the filter for a specific flaw.\n\n### 1.1. Prepare source documents\n\n```{python}\nfrom langchain.text_splitter import RecursiveCharacterTextSplitter\nfrom langchain.docstore.document import Document as LangchainDocument\n\nlangchain_docs = [\n    LangchainDocument(page_content=doc[\"text\"], metadata={\"source\": doc[\"source\"]})\n    for doc in tqdm(ds)\n]\n\n\ntext_splitter = RecursiveCharacterTextSplitter(\n    chunk_size=2000,\n    chunk_overlap=200,\n    add_start_index=True,\n    separators=[\"\\n\\n\", \"\\n\", \".\", \" \", \"\"],\n)\n\ndocs_processed = []\nfor doc in langchain_docs:\n    docs_processed += text_splitter.split_documents([doc])\n```\n\n### 1.2. Setup agents for question generation\n\nWe use [Mixtral](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1) for QA couple generation because it it has excellent performance in leaderboards such as [Chatbot Arena](https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard).\n\n```{python}\nfrom langchain_community.llms import HuggingFaceHub\n\nrepo_id = \"mistralai/Mixtral-8x7B-Instruct-v0.1\"\n\nllm = HuggingFaceHub(\n    repo_id=repo_id,\n    task=\"text-generation\",\n    model_kwargs={\n        \"max_new_tokens\": 512,\n        \"top_k\": 30,\n        \"temperature\": 0.1,\n        \"repetition_penalty\": 1.03,\n    },\n)\n```\n\n```{python}\nfrom langchain_community.chat_models import ChatHuggingFace\n\nchat_model = ChatHuggingFace(llm=llm)\n```\n\n```{python}\nfrom langchain.prompts import ChatPromptTemplate\n\nQA_generation_prompt = \"\"\"\nYour task is to write a factoid question and an answer given a context.\nYour factoid question should be answerable with a specific, concise piece of factual information from the context.\nYour factoid question should be formulated in the same style as questions users could ask in a search engine.\nThis means that your factoid question MUST NOT mention something like \"according to the passage\" or \"context\".\n\nProvide your answer as follows:\n\nOutput:::\nFactoid question: (your factoid question)\nAnswer: (your answer to the factoid question)\n\nNow here is the context.\n\nContext: {context}\\n\nOutput:::\"\"\"\n\nQA_generation_prompt = ChatPromptTemplate.from_template(QA_generation_prompt)\nQA_generation_agent = QA_generation_prompt | chat_model\n```\n\nNow let's generate our QA couples.\nFor this example, we generate only 10 QA couples and will load the rest from the Hub.\n\nBut for your specific knowledge base, given that you want to get at least ~100 test samples, and accounting for the fact that we will filter out around half of these with our critique agents later on, you should generate much more, in the >200 samples.\n\n```{python}\nimport random\n\nN_GENERATIONS = (\n    10  # We intentionally generate only 10 QA couples here for cost and time considerations\n)\n\nprint(f\"Generating {N_GENERATIONS} QA couples...\")\noutputs = []\nfor context in tqdm(random.sample(langchain_docs, N_GENERATIONS)):\n    # Generate QA couple\n    output_QA_couple = QA_generation_agent.invoke({\"context\": context.page_content}).content\n    try:\n        question = output_QA_couple.split(\"Factoid question: \")[1].split(\"Answer: \")[0]\n        answer = output_QA_couple.split(\"Answer: \")[1]\n        outputs.append(\n            {\n                \"context\": context.page_content,\n                \"question\": question,\n                \"answer\": answer,\n                \"source_doc\": context.metadata[\"source\"],\n            }\n        )\n    except:\n        continue\n```\n\n```{python}\ndisplay(pd.DataFrame(outputs).head(1))\n```\n\n### 1.3. Setup critique agents\n\nThe questions generated by the previous agent can have many flaws: we should do a quality check before validating these questions.\n\nWe thus build critique agents that will rate each question on several criteria, given in [this paper](https://huggingface.co/papers/2312.10003):\n- **Groundedness:** can the question be answered from the given context?\n- **Relevance:** is the question relevant to users? For instance, `\"What is the date when transformers 4.29.1 was released?\"` is not relevant for ML practicioners.\n\nOne last failure case we've noticed is when a function is tailored for the particular setting where the question was generated, but undecipherable by itself, like `\"What is the name of the function used in this guide?\"`.\nWe also build a critique agent for this criteria:\n- **Stand-alone**: is the question understandable free of any context, for someone with domain knowledge/Internet access? The opposite of this would be `What is the function used in this article?` for a question generated from a specific blog article.\n\nWe systematically score functions with all these agents, and whenever the score is too low for any one of the agents, we eliminate the question from our eval dataset.\n\n💡 ___When asking the agents to output a score, we first ask them to produce its rationale. This will help us verify scores, but most importantly, asking it to first output rationale gives the model more tokens to think and elaborate an answer before summarizing it into a single score token.___\n\nWe now build and run these critique agents.\n\n```{python}\nquestion_groundedness_critique_prompt = \"\"\"\nYou will be given a context and a question.\nYour task is to provide a 'total rating' scoring how well one can answer the given question unambiguously with the given context.\nGive your answer on a scale of 1 to 5, where 1 means that the question is not answerable at all given the context, and 5 means that the question is clearly and unambiguously answerable with the context.\n\nProvide your answer as follows:\n\nAnswer:::\nEvaluation: (your rationale for the rating)\nTotal rating: (your rating)\n\nNow here are the question and context.\n\nQuestion: {question}\\n\nContext: {context}\\n\nAnswer::: \"\"\"\n\nquestion_relevance_critique_prompt = \"\"\"\nYou will be given a question.\nYour task is to provide a 'total rating' representing how useful this question can be to machine learning developers building NLP applications with the Hugging Face ecosystem.\nGive your answer on a scale of 1 to 5, where 1 means that the question is not useful at all, and 5 means that the question is extremely useful.\n\nProvide your answer as follows:\n\nAnswer:::\nEvaluation: (your rationale for the rating)\nTotal rating: (your rating)\n\nNow here is the question.\n\nQuestion: {question}\\n\nAnswer::: \"\"\"\n\nquestion_standalone_critique_prompt = \"\"\"\nYou will be given a question.\nYour task is to provide a 'total rating' representing how context-independant this question is.\nGive your answer on a scale of 1 to 5, where 1 means that the question only makes sense in a specific context, and 5 means that the question makes sense by itself.\nFor instance, if the question refers to a particular setting, like 'in the context' or 'in the document', the rating must be 1.\nThe questions can contain obscure technical nouns or acronyms like Gradio, Hub, Hugging Face or Space and still be a 5: it must simply be clear to an operator with access to documentation what the question is about.\n\nProvide your answer as follows:\n\nAnswer:::\nEvaluation: (your rationale for the rating)\nTotal rating: (your rating)\n\nNow here is the question.\n\nQuestion: {question}\\n\nAnswer::: \"\"\"\n\nquestion_groundedness_critique_prompt = ChatPromptTemplate.from_template(\n    question_groundedness_critique_prompt\n)\nquestion_groundedness_critique_agent = question_groundedness_critique_prompt | chat_model\n\nquestion_relevance_critique_prompt = ChatPromptTemplate.from_template(\n    question_relevance_critique_prompt\n)\nquestion_relevance_critique_agent = question_relevance_critique_prompt | chat_model\n\nquestion_standalone_critique_prompt = ChatPromptTemplate.from_template(\n    question_standalone_critique_prompt\n)\nquestion_standalone_critique_agent = question_standalone_critique_prompt | chat_model\n```\n\n```{python}\nprint(\"Generating critique for each QA couple...\")\nfor output in tqdm(outputs):\n    # Critique the generated QA couple\n    question_groundedness_evaluation = question_groundedness_critique_agent.invoke(\n        {\"context\": output[\"context\"], \"question\": output[\"question\"]}\n    ).content\n    question_relevance_evaluation = question_relevance_critique_agent.invoke(\n        {\"question\": output[\"question\"]}\n    ).content\n    question_standalone_evaluation = question_standalone_critique_agent.invoke(\n        {\"question\": output[\"question\"]}\n    ).content\n\n    try:\n        groundedness_score = int(question_groundedness_evaluation.split(\"Total rating: \")[1][0])\n        groundedness_eval = question_groundedness_evaluation.split(\"Total rating: \")[0].split(\n            \"Evaluation: \"\n        )[1]\n        relevance_score = int(question_relevance_evaluation.split(\"Total rating: \")[1][0])\n        relevance_eval = question_relevance_evaluation.split(\"Total rating: \")[0].split(\n            \"Evaluation: \"\n        )[1]\n        standalone_score = int(question_standalone_evaluation.split(\"Total rating: \")[1][0])\n        standalone_eval = question_standalone_evaluation.split(\"Total rating: \")[0].split(\n            \"Evaluation: \"\n        )[1]\n        output.update(\n            {\n                \"groundedness_score\": groundedness_score,\n                \"groundedness_eval\": groundedness_eval,\n                \"relevance_score\": relevance_score,\n                \"relevance_eval\": relevance_eval,\n                \"standalone_score\": standalone_score,\n                \"standalone_eval\": standalone_eval,\n            }\n        )\n    except:\n        continue\n```\n\nNow let us filter out bad questions based on our critique agent scores:\n\n```{python}\nimport pandas as pd\n\npd.set_option(\"display.max_colwidth\", None)\n\ngenerated_questions = pd.DataFrame.from_dict(outputs)\n\nprint(\"Evaluation dataset before filtering:\")\ndisplay(\n    generated_questions[\n        [\"question\", \"answer\", \"groundedness_score\", \"relevance_score\", \"standalone_score\"]\n    ]\n)\ngenerated_questions = generated_questions.loc[\n    (generated_questions[\"groundedness_score\"] >= 4)\n    & (generated_questions[\"relevance_score\"] >= 4)\n    & (generated_questions[\"standalone_score\"] >= 4)\n]\nprint(\"============================================\")\nprint(\"Final evaluation dataset:\")\ndisplay(\n    generated_questions[\n        [\"question\", \"answer\", \"groundedness_score\", \"relevance_score\", \"standalone_score\"]\n    ]\n)\n\neval_dataset = datasets.Dataset.from_pandas(\n    generated_questions, split=\"train\", preserve_index=False\n)\n```\n\nNow our synthetic evaluation dataset is complete! We can evaluate different RAG systems on this evaluation dataset.\n\nWe have generated only a few QA couples here to reduce time and cost. But let's kick start the next part by loading a pre-generated dataset:\n\n```{python}\neval_dataset = datasets.load_dataset(\"m-ric/huggingface_doc_qa_eval\", split=\"train\")\n```\n\n# 2. Build our RAG System\n\n### 2.1. Preprocessing documents to build our vector database\n\n- In this part, __we split the documents from our knowledge base into smaller chunks__: these will be the snippets that are picked by the Retriever, to then be ingested by the Reader LLM as supporting elements for its answer.\n- The goal is to build semantically relevant snippets: not too small to be sufficient for supporting an answer, and not too large too avoid diluting individual ideas.\n\nMany options exist for text splitting:\n- split every `n` words / characters, but this has the risk of cutting in half paragraphs or even sentences\n- split after `n` words / character, but only on sentence boundaries\n- **recursive split** tries to preserve even more of the document structure, by processing it tree-like way, splitting first on the largest units (chapters) then recursively splitting on smaller units (paragraphs, sentences).\n\nTo learn more about chunking, I recommend you read [this great notebook](https://github.com/FullStackRetrieval-com/RetrievalTutorials/blob/main/5_Levels_Of_Text_Splitting.ipynb) by Greg Kamradt.\n\n[This space](https://huggingface.co/spaces/m-ric/chunk_visualizer) lets you visualize how different splitting options affect the chunks you get.\n\n> In the following, we use Langchain's `RecursiveCharacterTextSplitter`.\n\n💡 _To measure chunk length in our Text Splitter, our length function will not be the count of characters, but the count of tokens in the tokenized text: indeed, for subsequent embedder that processes token, measuring length in tokens is more relevant and empirically performs better._\n\n```{python}\nfrom langchain.docstore.document import Document as LangchainDocument\n\nRAW_KNOWLEDGE_BASE = [\n    LangchainDocument(page_content=doc[\"text\"], metadata={\"source\": doc[\"source\"]})\n    for doc in tqdm(ds)\n]\n```\n\n```{python}\nfrom langchain.text_splitter import RecursiveCharacterTextSplitter\nfrom transformers import AutoTokenizer\n\n\ndef split_documents(\n    chunk_size: int,\n    knowledge_base: List[LangchainDocument],\n    tokenizer_name: str,\n) -> List[LangchainDocument]:\n    \"\"\"\n    Split documents into chunks of size `chunk_size` characters and return a list of documents.\n    \"\"\"\n    text_splitter = RecursiveCharacterTextSplitter.from_huggingface_tokenizer(\n        AutoTokenizer.from_pretrained(tokenizer_name),\n        chunk_size=chunk_size,\n        chunk_overlap=int(chunk_size / 10),\n        add_start_index=True,\n        strip_whitespace=True,\n        separators=[\"\\n\\n\", \"\\n\", \".\", \" \", \"\"],\n    )\n\n    docs_processed = []\n    for doc in knowledge_base:\n        docs_processed += text_splitter.split_documents([doc])\n\n    # Remove duplicates\n    unique_texts = {}\n    docs_processed_unique = []\n    for doc in docs_processed:\n        if doc.page_content not in unique_texts:\n            unique_texts[doc.page_content] = True\n            docs_processed_unique.append(doc)\n\n    return docs_processed_unique\n```\n\n### 2.2. Retriever - embeddings 🗂️\nThe __retriever acts like an internal search engine__: given the user query, it returns the most relevant documents from your knowledge base.\n\n> For the knowledge base, we use Langchain vector databases since __it offers a convenient [FAISS](https://github.com/facebookresearch/faiss) index and allows us to keep document metadata throughout the processing__.\n\n🛠️ __Options included:__\n\n- Tune the chunking method:\n    - Size of the chunks\n    - Method: split on different separators, use [semantic chunking](https://python.langchain.com/docs/modules/data_connection/document_transformers/semantic-chunker)...\n- Change the embedding model\n\n```{python}\nfrom langchain.vectorstores import FAISS\nfrom langchain_community.embeddings import HuggingFaceEmbeddings\nfrom langchain_community.vectorstores.utils import DistanceStrategy\nimport os\n\n\ndef load_embeddings(\n    langchain_docs: List[LangchainDocument],\n    chunk_size: int,\n    embedding_model_name: Optional[str] = \"thenlper/gte-small\",\n) -> FAISS:\n    \"\"\"\n    Creates a FAISS index from the given embedding model and documents. Loads the index directly if it already exists.\n\n    Args:\n        langchain_docs: list of documents\n        chunk_size: size of the chunks to split the documents into\n        embedding_model_name: name of the embedding model to use\n\n    Returns:\n        FAISS index\n    \"\"\"\n    # load embedding_model\n    embedding_model = HuggingFaceEmbeddings(\n        model_name=embedding_model_name,\n        multi_process=True,\n        model_kwargs={\"device\": \"cuda\"},\n        encode_kwargs={\"normalize_embeddings\": True},  # set True to compute cosine similarity\n    )\n\n    # Check if embeddings already exist on disk\n    index_name = f\"index_chunk:{chunk_size}_embeddings:{embedding_model_name.replace('/', '~')}\"\n    index_folder_path = f\"./data/indexes/{index_name}/\"\n    if os.path.isdir(index_folder_path):\n        return FAISS.load_local(\n            index_folder_path,\n            embedding_model,\n            distance_strategy=DistanceStrategy.COSINE,\n        )\n\n    else:\n        print(\"Index not found, generating it...\")\n        docs_processed = split_documents(\n            chunk_size,\n            langchain_docs,\n            embedding_model_name,\n        )\n        knowledge_index = FAISS.from_documents(\n            docs_processed, embedding_model, distance_strategy=DistanceStrategy.COSINE\n        )\n        knowledge_index.save_local(index_folder_path)\n        return knowledge_index\n```\n\n### 2.3. Reader - LLM 💬\n\nIn this part, the __LLM Reader reads the retrieved documents to formulate its answer.__\n\n🛠️ Here we tried the following options to improve results:\n- Switch reranking on/off\n- Change the reader model\n\n```{python}\nRAG_PROMPT_TEMPLATE = \"\"\"\n<|system|>\nUsing the information contained in the context,\ngive a comprehensive answer to the question.\nRespond only to the question asked, response should be concise and relevant to the question.\nProvide the number of the source document when relevant.\nIf the answer cannot be deduced from the context, do not give an answer.</s>\n<|user|>\nContext:\n{context}\n---\nNow here is the question you need to answer.\n\nQuestion: {question}\n</s>\n<|assistant|>\n\"\"\"\n```\n\n```{python}\nfrom langchain_community.llms import HuggingFaceHub\n\nrepo_id = \"HuggingFaceH4/zephyr-7b-beta\"\nREADER_MODEL_NAME = \"zephyr-7b-beta\"\n\nREADER_LLM = HuggingFaceHub(\n    repo_id=repo_id,\n    task=\"text-generation\",\n    model_kwargs={\n        \"max_new_tokens\": 512,\n        \"top_k\": 30,\n        \"temperature\": 0.1,\n        \"repetition_penalty\": 1.03,\n    },\n)\n```\n\n```{python}\nfrom ragatouille import RAGPretrainedModel\nfrom langchain_core.vectorstores import VectorStore\nfrom langchain_core.language_models.llms import LLM\n\n\ndef answer_with_rag(\n    question: str,\n    llm: LLM,\n    knowledge_index: VectorStore,\n    reranker: Optional[RAGPretrainedModel] = None,\n    num_retrieved_docs: int = 30,\n    num_docs_final: int = 7,\n) -> Tuple[str, List[LangchainDocument]]:\n    \"\"\"Answer a question using RAG with the given knowledge index.\"\"\"\n    # Gather documents with retriever\n    relevant_docs = knowledge_index.similarity_search(query=question, k=num_retrieved_docs)\n    relevant_docs = [doc.page_content for doc in relevant_docs]  # keep only the text\n\n    # Optionally rerank results\n    if reranker:\n        relevant_docs = reranker.rerank(question, relevant_docs, k=num_docs_final)\n        relevant_docs = [doc[\"content\"] for doc in relevant_docs]\n\n    relevant_docs = relevant_docs[:num_docs_final]\n\n    # Build the final prompt\n    context = \"\\nExtracted documents:\\n\"\n    context += \"\".join([f\"Document {str(i)}:::\\n\" + doc for i, doc in enumerate(relevant_docs)])\n\n    final_prompt = RAG_PROMPT_TEMPLATE.format(question=question, context=context)\n\n    # Redact an answer\n    answer = llm(final_prompt)\n\n    return answer, relevant_docs\n```\n\n# 3. Benchmarking the RAG system\n\nThe RAG system and the evaluation datasets are now ready. The last step is to judge the RAG system's output on this evlauation dataset.\n\nTo this end, __we setup a judge agent__. ⚖️🤖\n\nOut of [the different RAG evaluation metrics](https://docs.ragas.io/en/latest/concepts/metrics/index.html), we choose to focus only on faithfulness since it the best end-to-end metric of our system's performance.\n\n> We use GPT4 as a judge for its empirically good performance, but you could try with other models such as [kaist-ai/prometheus-13b-v1.0](https://huggingface.co/kaist-ai/prometheus-13b-v1.0) or [BAAI/JudgeLM-33B-v1.0](https://huggingface.co/BAAI/JudgeLM-33B-v1.0).\n\n💡 _In the evaluation prompt, we give a detailed description each metric on the scale 1-5, as is done in [Prometheus's prompt template](https://huggingface.co/kaist-ai/prometheus-13b-v1.0): this helps the model ground its metric precisely. If instead you give the judge LLM a vague scale to work with, the outputs will not be consistent enough between different examples._\n\n💡 _Again, prompting the LLM to output rationale before giving its final score gives it more tokens to help it formalize and elaborate a judgement._\n\n```{python}\ndef run_rag_tests(\n    eval_dataset: datasets.Dataset,\n    llm: BaseChatModel,\n    knowledge_index: VectorStore,\n    output_file: str,\n    reranker: Optional[RAGPretrainedModel] = None,\n    verbose: Optional[bool] = True,\n    test_settings: Optional[str] = None,  # To document the test settings used\n):\n    \"\"\"Runs RAG tests on the given dataset and saves the results to the given output file.\"\"\"\n    try:  # load previous generations if they exist\n        with open(output_file, \"r\") as f:\n            outputs = json.load(f)\n    except:\n        outputs = []\n\n    for example in tqdm(eval_dataset):\n        question = example[\"question\"]\n        if question in [output[\"question\"] for output in outputs]:\n            continue\n\n        answer, relevant_docs = answer_with_rag(question, llm, knowledge_index, reranker=reranker)\n        if verbose:\n            print(\"=======================================================\")\n            print(f\"Question: {question}\")\n            print(f\"Answer: {answer}\")\n            print(f'True answer: {example[\"answer\"]}')\n        result = {\n            \"question\": question,\n            \"true_answer\": example[\"answer\"],\n            \"source_doc\": example[\"source_doc\"],\n            \"generated_answer\": answer,\n            \"retrieved_docs\": [doc for doc in relevant_docs],\n        }\n        if test_settings:\n            result[\"test_settings\"] = test_settings\n        outputs.append(result)\n\n        with open(output_file, \"w\") as f:\n            json.dump(outputs, f)\n```\n\n```{python}\nEVALUATION_PROMPT = \"\"\"###Task Description:\nAn instruction (might include an Input inside it), a response to evaluate, a reference answer that gets a score of 5, and a score rubric representing a evaluation criteria are given.\n1. Write a detailed feedback that assess the quality of the response strictly based on the given score rubric, not evaluating in general.\n2. After writing a feedback, write a score that is an integer between 1 and 5. You should refer to the score rubric.\n3. The output format should look as follows: \\\"Feedback: {{write a feedback for criteria}} [RESULT] {{an integer number between 1 and 5}}\\\"\n4. Please do not generate any other opening, closing, and explanations. Be sure to include [RESULT] in your output.\n\n###The instruction to evaluate:\n{instruction}\n\n###Response to evaluate:\n{response}\n\n###Reference Answer (Score 5):\n{reference_answer}\n\n###Score Rubrics:\n[Is the response correct, accurate, and factual based on the reference answer?]\nScore 1: The response is completely incorrect, inaccurate, and/or not factual.\nScore 2: The response is mostly incorrect, inaccurate, and/or not factual.\nScore 3: The response is somewhat correct, accurate, and/or factual.\nScore 4: The response is mostly correct, accurate, and factual.\nScore 5: The response is completely correct, accurate, and factual.\n\n###Feedback:\"\"\"\n\nfrom langchain.prompts.chat import (\n    ChatPromptTemplate,\n    HumanMessagePromptTemplate,\n)\nfrom langchain.schema import SystemMessage\n\n\nevaluation_prompt_template = ChatPromptTemplate.from_messages(\n    [\n        SystemMessage(content=\"You are a fair evaluator language model.\"),\n        HumanMessagePromptTemplate.from_template(EVALUATION_PROMPT),\n    ]\n)\n```\n\n```{python}\nfrom langchain.chat_models import ChatOpenAI\n\neval_chat_model = ChatOpenAI(model=\"gpt-4-1106-preview\", temperature=0)\nevaluator_name = \"GPT4\"\n\n\ndef evaluate_answers(\n    answer_path: str,\n    eval_chat_model: BaseChatModel,\n    evaluator_name: str,\n    evaluation_prompt_template: ChatPromptTemplate,\n) -> None:\n    \"\"\"Evaluates generated answers. Modifies the given answer file in place for better checkpointing.\"\"\"\n    answers = []\n    if os.path.isfile(answer_path):  # load previous generations if they exist\n        answers = json.load(open(answer_path, \"r\"))\n\n    for experiment in tqdm(answers):\n        if f\"eval_score_{evaluator_name}\" in experiment:\n            continue\n\n        eval_prompt = evaluation_prompt_template.format_messages(\n            instruction=experiment[\"question\"],\n            response=experiment[\"generated_answer\"],\n            reference_answer=experiment[\"true_answer\"],\n        )\n        eval_result = eval_chat_model.invoke(eval_prompt)\n        feedback, score = [item.strip() for item in eval_result.content.split(\"[RESULT]\")]\n        experiment[f\"eval_score_{evaluator_name}\"] = score\n        experiment[f\"eval_feedback_{evaluator_name}\"] = feedback\n\n        with open(answer_path, \"w\") as f:\n            json.dump(answers, f)\n```\n\n🚀 Let's run the tests and evaluate answers!👇\n\n```{python}\nif not os.path.exists(\"./output\"):\n    os.mkdir(\"./output\")\n\nfor chunk_size in [200]:  # Add other chunk sizes (in tokens) as needed\n    for embeddings in [\"thenlper/gte-small\"]:  # Add other embeddings as needed\n        for rerank in [True, False]:\n            settings_name = f\"chunk:{chunk_size}_embeddings:{embeddings.replace('/', '~')}_rerank:{rerank}_reader-model:{READER_MODEL_NAME}\"\n            output_file_name = f\"./output/rag_{settings_name}.json\"\n\n            print(f\"Running evaluation for {settings_name}:\")\n\n            print(\"Loading knowledge base embeddings...\")\n            knowledge_index = load_embeddings(\n                RAW_KNOWLEDGE_BASE,\n                chunk_size=chunk_size,\n                embedding_model_name=embeddings,\n            )\n\n            print(\"Running RAG...\")\n            reranker = (\n                RAGPretrainedModel.from_pretrained(\"colbert-ir/colbertv2.0\") if rerank else None\n            )\n            run_rag_tests(\n                eval_dataset=eval_dataset,\n                llm=READER_LLM,\n                knowledge_index=knowledge_index,\n                output_file=output_file_name,\n                reranker=reranker,\n                verbose=False,\n                test_settings=settings_name,\n            )\n\n            print(\"Running evaluation...\")\n            evaluate_answers(\n                output_file_name,\n                eval_chat_model,\n                evaluator_name,\n                evaluation_prompt_template,\n            )\n```\n\n### Inspect results\n\n```{python}\nimport glob\n\noutputs = []\nfor file in glob.glob(\"./output/*.json\"):\n    output = pd.DataFrame(json.load(open(file, \"r\")))\n    output[\"settings\"] = file\n    outputs.append(output)\nresult = pd.concat(outputs)\n```\n\n```{python}\nresult[\"eval_score_GPT4\"] = result[\"eval_score_GPT4\"].apply(\n    lambda x: int(x) if isinstance(x, str) else 1\n)\nresult[\"eval_score_GPT4\"] = (result[\"eval_score_GPT4\"] - 1) / 4\n```\n\n```{python}\naverage_scores = result.groupby(\"settings\")[\"eval_score_GPT4\"].mean()\naverage_scores.sort_values()\n```\n\n## Example results\n\nLet us load the results that I obtained by tweaking the different options available in this notebook.\nFor more detail on why these options could work on not, see the notebook on [advanced_RAG](advanced_rag).\n\nAs you can see in the graph below, some tweaks do not bring any improvement, some give huge performance boosts.\n\n➡️ ___There is no single good recipe: you should try several different directions when tuning your RAG systems.___\n\n```{python}\nimport plotly.express as px\n\nscores = datasets.load_dataset(\"m-ric/rag_scores_cookbook\", split=\"train\")\nscores = pd.Series(scores[\"score\"], index=scores[\"settings\"])\n```\n\n```{python}\nfig = px.bar(\n    scores,\n    color=scores,\n    labels={\n        \"value\": \"Accuracy\",\n        \"settings\": \"Configuration\",\n    },\n    color_continuous_scale=\"bluered\",\n)\nfig.update_layout(w\n    width=1000,\n    height=600,\n    barmode=\"group\",\n    yaxis_range=[0, 100],\n    title=\"<b>Accuracy of different RAG configurations</b>\",\n    xaxis_title=\"RAG settings\",\n    font=dict(size=15),\n)\nfig.layout.yaxis.ticksuffix = \"%\"\nfig.update_coloraxes(showscale=False)\nfig.update_traces(texttemplate=\"%{y:.1f}\", textposition=\"outside\")\nfig.show()\n```\n\n<img src=\"https://huggingface.co/datasets/huggingface/cookbook-images/resolve/main/RAG_settings_accuracy.png\" height=\"500\" width=\"800\">\n\nAs you can see, these had varying impact on performance. In particular, tuning the chunk size is both easy and very impactful.\n\nBut this is our case: your results could be very different: now that you have a robust evaluation pipeline, you can set on to explore other options! 🗺️\n\n"},"formats":{"html":{"identifier":{"display-name":"HTML","target-format":"html","base-format":"html"},"execute":{"fig-width":7,"fig-height":5,"fig-format":"retina","fig-dpi":96,"df-print":"default","error":false,"eval":false,"cache":null,"freeze":false,"echo":true,"output":true,"warning":true,"include":true,"keep-md":false,"keep-ipynb":false,"ipynb":null,"enabled":null,"daemon":null,"daemon-restart":false,"debug":false,"ipynb-filters":[],"ipynb-shell-interactivity":null,"plotly-connected":true,"engine":"jupyter"},"render":{"keep-tex":false,"keep-typ":false,"keep-source":false,"keep-hidden":false,"prefer-html":false,"output-divs":true,"output-ext":"html","fig-align":"default","fig-pos":null,"fig-env":null,"code-fold":"none","code-overflow":"scroll","code-link":false,"code-line-numbers":false,"code-tools":false,"tbl-colwidths":"auto","merge-includes":true,"inline-includes":false,"preserve-yaml":false,"latex-auto-mk":true,"latex-auto-install":true,"latex-clean":true,"latex-min-runs":1,"latex-max-runs":10,"latex-makeindex":"makeindex","latex-makeindex-opts":[],"latex-tlmgr-opts":[],"latex-input-paths":[],"latex-output-dir":null,"link-external-icon":false,"link-external-newwindow":false,"self-contained-math":false,"format-resources":[],"notebook-links":true},"pandoc":{"standalone":true,"wrap":"none","default-image-extension":"png","to":"html","css":["../styles.css"],"toc":true,"output-file":"rag_evaluation.html"},"language":{"toc-title-document":"Table of contents","toc-title-website":"On this page","related-formats-title":"Other Formats","related-notebooks-title":"Notebooks","source-notebooks-prefix":"Source","other-links-title":"Other Links","code-links-title":"Code Links","launch-dev-container-title":"Launch Dev Container","launch-binder-title":"Launch Binder","article-notebook-label":"Article Notebook","notebook-preview-download":"Download Notebook","notebook-preview-download-src":"Download Source","notebook-preview-back":"Back to Article","manuscript-meca-bundle":"MECA Bundle","section-title-abstract":"Abstract","section-title-appendices":"Appendices","section-title-footnotes":"Footnotes","section-title-references":"References","section-title-reuse":"Reuse","section-title-copyright":"Copyright","section-title-citation":"Citation","appendix-attribution-cite-as":"For attribution, please cite this work as:","appendix-attribution-bibtex":"BibTeX citation:","appendix-view-license":"View License","title-block-author-single":"Author","title-block-author-plural":"Authors","title-block-affiliation-single":"Affiliation","title-block-affiliation-plural":"Affiliations","title-block-published":"Published","title-block-modified":"Modified","title-block-keywords":"Keywords","callout-tip-title":"Tip","callout-note-title":"Note","callout-warning-title":"Warning","callout-important-title":"Important","callout-caution-title":"Caution","code-summary":"Code","code-tools-menu-caption":"Code","code-tools-show-all-code":"Show All Code","code-tools-hide-all-code":"Hide All Code","code-tools-view-source":"View Source","code-tools-source-code":"Source Code","tools-share":"Share","tools-download":"Download","code-line":"Line","code-lines":"Lines","copy-button-tooltip":"Copy to Clipboard","copy-button-tooltip-success":"Copied!","repo-action-links-edit":"Edit this page","repo-action-links-source":"View source","repo-action-links-issue":"Report an issue","back-to-top":"Back to top","search-no-results-text":"No results","search-matching-documents-text":"matching documents","search-copy-link-title":"Copy link to search","search-hide-matches-text":"Hide additional matches","search-more-match-text":"more match in this document","search-more-matches-text":"more matches in this document","search-clear-button-title":"Clear","search-text-placeholder":"","search-detached-cancel-button-title":"Cancel","search-submit-button-title":"Submit","search-label":"Search","toggle-section":"Toggle section","toggle-sidebar":"Toggle sidebar navigation","toggle-dark-mode":"Toggle dark mode","toggle-reader-mode":"Toggle reader mode","toggle-navigation":"Toggle navigation","crossref-fig-title":"Figure","crossref-tbl-title":"Table","crossref-lst-title":"Listing","crossref-thm-title":"Theorem","crossref-lem-title":"Lemma","crossref-cor-title":"Corollary","crossref-prp-title":"Proposition","crossref-cnj-title":"Conjecture","crossref-def-title":"Definition","crossref-exm-title":"Example","crossref-exr-title":"Exercise","crossref-ch-prefix":"Chapter","crossref-apx-prefix":"Appendix","crossref-sec-prefix":"Section","crossref-eq-prefix":"Equation","crossref-lof-title":"List of Figures","crossref-lot-title":"List of Tables","crossref-lol-title":"List of Listings","environment-proof-title":"Proof","environment-remark-title":"Remark","environment-solution-title":"Solution","listing-page-order-by":"Order By","listing-page-order-by-default":"Default","listing-page-order-by-date-asc":"Oldest","listing-page-order-by-date-desc":"Newest","listing-page-order-by-number-desc":"High to Low","listing-page-order-by-number-asc":"Low to High","listing-page-field-date":"Date","listing-page-field-title":"Title","listing-page-field-description":"Description","listing-page-field-author":"Author","listing-page-field-filename":"File Name","listing-page-field-filemodified":"Modified","listing-page-field-subtitle":"Subtitle","listing-page-field-readingtime":"Reading Time","listing-page-field-wordcount":"Word Count","listing-page-field-categories":"Categories","listing-page-minutes-compact":"{0} min","listing-page-category-all":"All","listing-page-no-matches":"No matching items","listing-page-words":"{0} words","listing-page-filter":"Filter","draft":"Draft"},"metadata":{"lang":"en","fig-responsive":true,"quarto-version":"1.6.40","theme":"cosmo","title":"RAG Evaluation","jupyter":"python3"},"extensions":{"book":{"multiFile":true}}}},"projectFormats":["html"]}

src/.quarto/idx/notebooks/rag_zephyr_langchain.qmd.json

@@ -1 +0,0 @@
-{"title":"Simple RAG","markdown":{"yaml":{"title":"Simple RAG","jupyter":"python3","eval":false,"code-annotations":"hover"},"headingText":"Prepare the data","containsRefs":false,"markdown":"\n\n```{python}\n!pip install -q torch transformers accelerate bitsandbytes transformers sentence-transformers faiss-gpu\n```\n\n```{python}\n!pip install -q langchain\n```\n\n::: callout-note\nIf running in Google Colab, you may need to run this cell to make sure you're using UTF-8 locale to install LangChain\n```{python}\nimport locale\nlocale.getpreferredencoding = lambda: \"UTF-8\"\n```\n:::\n\n\n\nIn this example, we'll load all of the issues (both open and closed) from [PEFT library's repo](https://github.com/huggingface/peft).\n\nFirst, you need to acquire a [GitHub personal access token](https://github.com/settings/tokens?type=beta) to access the GitHub API.\n\n```{python}\nfrom getpass import getpass\n\nACCESS_TOKEN = getpass(\"YOUR_GITHUB_PERSONAL_TOKEN\")  # <1>\n```\n1. You can also use an environment variable to store your token.\n\nNext, we'll load all of the issues in the [huggingface/peft](https://github.com/huggingface/peft) repo:\n- By default, pull requests are considered issues as well, here we chose to exclude them from data with by setting `include_prs=False`\n- Setting `state = \"all\"` means we will load both open and closed issues.\n\n```{python}\nfrom langchain.document_loaders import GitHubIssuesLoader\n\nloader = GitHubIssuesLoader(\n    repo=\"huggingface/peft\",\n    access_token=ACCESS_TOKEN,\n    include_prs=False,\n    state=\"all\"\n)\n\ndocs = loader.load()\n```\n\nThe content of individual GitHub issues may be longer than what an embedding model can take as input. If we want to embed all of the available content, we need to chunk the documents into appropriately sized pieces.\n\nThe most common and straightforward approach to chunking is to define a fixed size of chunks and whether there should be any overlap between them. Keeping some overlap between chunks allows us to preserve some semantic context between the chunks.\n\nOther approaches are typically more involved and take into account the documents' structure and context. For example, one may want to split a document based on sentences or paragraphs, or create chunks based on the\n\nThe fixed-size chunking, however, works well for most common cases, so that is what we'll do here.\n\n```{python}\nfrom langchain.text_splitter import CharacterTextSplitter\n\nsplitter = CharacterTextSplitter(chunk_size=512, chunk_overlap=30)\n\nchunked_docs = splitter.split_documents(docs)\n```\n\n## Create the embeddings + retriever\n\nNow that the docs are all of the appropriate size, we can create a database with their embeddings.\n\nTo create document chunk embeddings we'll use the `HuggingFaceEmbeddings` and the [`BAAI/bge-base-en-v1.5`](https://huggingface.co/BAAI/bge-base-en-v1.5) embeddings model. To create the vector database, we'll use `FAISS`, a library developed by Facebook AI. This library offers efficient similarity search and clustering of dense vectors, which is what we need here. FAISS is currently one of the most used libraries for NN search in massive datasets. \n\n::: callout-tip\nThere are many other embeddings models available on the Hub, and you can keep an eye on the best performing ones by checking the [Massive Text Embedding Benchmark (MTEB) Leaderboard](https://huggingface.co/spaces/mteb/leaderboard).\n:::\n\nWe'll access both the embeddings model and FAISS via LangChain API.\n\n```{python}\nfrom langchain.vectorstores import FAISS\nfrom langchain.embeddings import HuggingFaceEmbeddings\n\ndb = FAISS.from_documents(chunked_docs,\n                          HuggingFaceEmbeddings(model_name='BAAI/bge-base-en-v1.5'))\n```\n\nWe need a way to return(retrieve) the documents given an unstructured query. For that, we'll use the `as_retriever` method using the `db` as a backbone:\n- `search_type=\"similarity\"` means we want to perform similarity search between the query and documents\n- `search_kwargs={'k': 4}` instructs the retriever to return top 4 results.\n\n```{python}\nretriever = db.as_retriever(\n    search_type=\"similarity\",            # <1>\n    search_kwargs={'k': 4}               # <1>\n)\n```\n1. The ideal search type is context dependent, and you should experiment to find the best one for your data.\n\nThe vector database and retriever are now set up, next we need to set up the next piece of the chain - the model.\n\n## Load quantized model\n\nFor this example, we chose [`HuggingFaceH4/zephyr-7b-beta`](https://huggingface.co/HuggingFaceH4/zephyr-7b-beta), a small but powerful model.\nTo make inference faster, we will load the quantized version of the model:\n\n:::::: {.callout-tip}\nWith many models being released every week, you may want to substitute this model to the latest and greatest. The best way to keep track of open source LLMs is to check the [Open-source LLM leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard).\n:::\n\n```{python}\nimport torch\nfrom transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig\n\nmodel_name = 'HuggingFaceH4/zephyr-7b-beta'\n\nbnb_config = BitsAndBytesConfig(\n    load_in_4bit=True,\n    bnb_4bit_use_double_quant=True,\n    bnb_4bit_quant_type=\"nf4\",\n    bnb_4bit_compute_dtype=torch.bfloat16\n)\n\nmodel = AutoModelForCausalLM.from_pretrained(model_name, quantization_config=bnb_config)\ntokenizer = AutoTokenizer.from_pretrained(model_name)\n```\n\n## Setup the LLM chain\n\nFinally, we have all the pieces we need to set up the LLM chain.\n\nFirst, create a text_generation pipeline using the loaded model and its tokenizer.\n\nNext, create a prompt template - this should follow the format of the model, so if you substitute the model checkpoint, make sure to use the appropriate formatting.\n\n```{python}\nfrom langchain.llms import HuggingFacePipeline\nfrom langchain.prompts import PromptTemplate\nfrom transformers import pipeline\nfrom langchain_core.output_parsers import StrOutputParser\n\ntext_generation_pipeline = pipeline(\n    model=model,                # <1> \n    tokenizer=tokenizer,        # <2> \n    task=\"text-generation\",     # <3> \n    temperature=0.2,            # <4> \n    do_sample=True,             # <5> \n    repetition_penalty=1.1,     # <6> \n    return_full_text=True,      # <7> \n    max_new_tokens=400,         # <8> \n)\n\nllm = HuggingFacePipeline(pipeline=text_generation_pipeline)\n\nprompt_template = \"\"\"\n<|system|>\nAnswer the question based on your knowledge. Use the following context to help:\n\n{context}\n\n</s>\n<|user|>\n{question}\n</s>\n<|assistant|>\n\n \"\"\"\n\nprompt = PromptTemplate(\n    input_variables=[\"context\", \"question\"],\n    template=prompt_template,\n)\n\nllm_chain = prompt | llm | StrOutputParser()\n```\n\n1. The pre-trained model for text generation.\n2. Tokenizer to preprocess input text and postprocess generated output.\n3. Specifies the task as text generation.\n4. Controls the randomness in the output generation. Lower values make the output more deterministic.\n5. Enables sampling to introduce randomness in the output generation.\n6. Penalizes repetition in the output to encourage diversity.\n7. Returns the full generated text including the input prompt.\n8. Limits the maximum number of new tokens generated.\n\nNote: _You can also use `tokenizer.apply_chat_template` to convert a list of messages (as dicts: `{'role': 'user', 'content': '(...)'}`) into a string with the appropriate chat format._\n\n\nFinally, we need to combine the `llm_chain` with the retriever to create a RAG chain. We pass the original question through to the final generation step, as well as the retrieved context docs:\n\n```{python}\nfrom langchain_core.runnables import RunnablePassthrough\n\nretriever = db.as_retriever()\n\nrag_chain = (\n {\"context\": retriever, \"question\": RunnablePassthrough()}\n    | llm_chain\n)\n```\n\n## Compare the results\n\nLet's see the difference RAG makes in generating answers to the library-specific questions.\n\n```{python}\nquestion = \"How do you combine multiple adapters?\"\n```\n\nFirst, let's see what kind of answer we can get with just the model itself, no context added:\n\n```{python}\n#| colab: {base_uri: 'https://localhost:8080/', height: 125}\nllm_chain.invoke({\"context\":\"\", \"question\": question})\n```\n\nAs you can see, the model interpreted the question as one about physical computer adapters, while in the context of PEFT, \"adapters\" refer to LoRA adapters.\nLet's see if adding context from GitHub issues helps the model give a more relevant answer:\n\n```{python}\n#| colab: {base_uri: 'https://localhost:8080/', height: 125}\nrag_chain.invoke(question)\n```\n\nAs we can see, the added context, really helps the exact same model, provide a much more relevant and informed answer to the library-specific question.\n\nNotably, combining multiple adapters for inference has been added to the library, and one can find this information in the documentation, so for the next iteration of this RAG it may be worth including documentation embeddings.\n\n","srcMarkdownNoYaml":"\n\n```{python}\n!pip install -q torch transformers accelerate bitsandbytes transformers sentence-transformers faiss-gpu\n```\n\n```{python}\n!pip install -q langchain\n```\n\n::: callout-note\nIf running in Google Colab, you may need to run this cell to make sure you're using UTF-8 locale to install LangChain\n```{python}\nimport locale\nlocale.getpreferredencoding = lambda: \"UTF-8\"\n```\n:::\n\n\n## Prepare the data\n\nIn this example, we'll load all of the issues (both open and closed) from [PEFT library's repo](https://github.com/huggingface/peft).\n\nFirst, you need to acquire a [GitHub personal access token](https://github.com/settings/tokens?type=beta) to access the GitHub API.\n\n```{python}\nfrom getpass import getpass\n\nACCESS_TOKEN = getpass(\"YOUR_GITHUB_PERSONAL_TOKEN\")  # <1>\n```\n1. You can also use an environment variable to store your token.\n\nNext, we'll load all of the issues in the [huggingface/peft](https://github.com/huggingface/peft) repo:\n- By default, pull requests are considered issues as well, here we chose to exclude them from data with by setting `include_prs=False`\n- Setting `state = \"all\"` means we will load both open and closed issues.\n\n```{python}\nfrom langchain.document_loaders import GitHubIssuesLoader\n\nloader = GitHubIssuesLoader(\n    repo=\"huggingface/peft\",\n    access_token=ACCESS_TOKEN,\n    include_prs=False,\n    state=\"all\"\n)\n\ndocs = loader.load()\n```\n\nThe content of individual GitHub issues may be longer than what an embedding model can take as input. If we want to embed all of the available content, we need to chunk the documents into appropriately sized pieces.\n\nThe most common and straightforward approach to chunking is to define a fixed size of chunks and whether there should be any overlap between them. Keeping some overlap between chunks allows us to preserve some semantic context between the chunks.\n\nOther approaches are typically more involved and take into account the documents' structure and context. For example, one may want to split a document based on sentences or paragraphs, or create chunks based on the\n\nThe fixed-size chunking, however, works well for most common cases, so that is what we'll do here.\n\n```{python}\nfrom langchain.text_splitter import CharacterTextSplitter\n\nsplitter = CharacterTextSplitter(chunk_size=512, chunk_overlap=30)\n\nchunked_docs = splitter.split_documents(docs)\n```\n\n## Create the embeddings + retriever\n\nNow that the docs are all of the appropriate size, we can create a database with their embeddings.\n\nTo create document chunk embeddings we'll use the `HuggingFaceEmbeddings` and the [`BAAI/bge-base-en-v1.5`](https://huggingface.co/BAAI/bge-base-en-v1.5) embeddings model. To create the vector database, we'll use `FAISS`, a library developed by Facebook AI. This library offers efficient similarity search and clustering of dense vectors, which is what we need here. FAISS is currently one of the most used libraries for NN search in massive datasets. \n\n::: callout-tip\nThere are many other embeddings models available on the Hub, and you can keep an eye on the best performing ones by checking the [Massive Text Embedding Benchmark (MTEB) Leaderboard](https://huggingface.co/spaces/mteb/leaderboard).\n:::\n\nWe'll access both the embeddings model and FAISS via LangChain API.\n\n```{python}\nfrom langchain.vectorstores import FAISS\nfrom langchain.embeddings import HuggingFaceEmbeddings\n\ndb = FAISS.from_documents(chunked_docs,\n                          HuggingFaceEmbeddings(model_name='BAAI/bge-base-en-v1.5'))\n```\n\nWe need a way to return(retrieve) the documents given an unstructured query. For that, we'll use the `as_retriever` method using the `db` as a backbone:\n- `search_type=\"similarity\"` means we want to perform similarity search between the query and documents\n- `search_kwargs={'k': 4}` instructs the retriever to return top 4 results.\n\n```{python}\nretriever = db.as_retriever(\n    search_type=\"similarity\",            # <1>\n    search_kwargs={'k': 4}               # <1>\n)\n```\n1. The ideal search type is context dependent, and you should experiment to find the best one for your data.\n\nThe vector database and retriever are now set up, next we need to set up the next piece of the chain - the model.\n\n## Load quantized model\n\nFor this example, we chose [`HuggingFaceH4/zephyr-7b-beta`](https://huggingface.co/HuggingFaceH4/zephyr-7b-beta), a small but powerful model.\nTo make inference faster, we will load the quantized version of the model:\n\n:::::: {.callout-tip}\nWith many models being released every week, you may want to substitute this model to the latest and greatest. The best way to keep track of open source LLMs is to check the [Open-source LLM leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard).\n:::\n\n```{python}\nimport torch\nfrom transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig\n\nmodel_name = 'HuggingFaceH4/zephyr-7b-beta'\n\nbnb_config = BitsAndBytesConfig(\n    load_in_4bit=True,\n    bnb_4bit_use_double_quant=True,\n    bnb_4bit_quant_type=\"nf4\",\n    bnb_4bit_compute_dtype=torch.bfloat16\n)\n\nmodel = AutoModelForCausalLM.from_pretrained(model_name, quantization_config=bnb_config)\ntokenizer = AutoTokenizer.from_pretrained(model_name)\n```\n\n## Setup the LLM chain\n\nFinally, we have all the pieces we need to set up the LLM chain.\n\nFirst, create a text_generation pipeline using the loaded model and its tokenizer.\n\nNext, create a prompt template - this should follow the format of the model, so if you substitute the model checkpoint, make sure to use the appropriate formatting.\n\n```{python}\nfrom langchain.llms import HuggingFacePipeline\nfrom langchain.prompts import PromptTemplate\nfrom transformers import pipeline\nfrom langchain_core.output_parsers import StrOutputParser\n\ntext_generation_pipeline = pipeline(\n    model=model,                # <1> \n    tokenizer=tokenizer,        # <2> \n    task=\"text-generation\",     # <3> \n    temperature=0.2,            # <4> \n    do_sample=True,             # <5> \n    repetition_penalty=1.1,     # <6> \n    return_full_text=True,      # <7> \n    max_new_tokens=400,         # <8> \n)\n\nllm = HuggingFacePipeline(pipeline=text_generation_pipeline)\n\nprompt_template = \"\"\"\n<|system|>\nAnswer the question based on your knowledge. Use the following context to help:\n\n{context}\n\n</s>\n<|user|>\n{question}\n</s>\n<|assistant|>\n\n \"\"\"\n\nprompt = PromptTemplate(\n    input_variables=[\"context\", \"question\"],\n    template=prompt_template,\n)\n\nllm_chain = prompt | llm | StrOutputParser()\n```\n\n1. The pre-trained model for text generation.\n2. Tokenizer to preprocess input text and postprocess generated output.\n3. Specifies the task as text generation.\n4. Controls the randomness in the output generation. Lower values make the output more deterministic.\n5. Enables sampling to introduce randomness in the output generation.\n6. Penalizes repetition in the output to encourage diversity.\n7. Returns the full generated text including the input prompt.\n8. Limits the maximum number of new tokens generated.\n\nNote: _You can also use `tokenizer.apply_chat_template` to convert a list of messages (as dicts: `{'role': 'user', 'content': '(...)'}`) into a string with the appropriate chat format._\n\n\nFinally, we need to combine the `llm_chain` with the retriever to create a RAG chain. We pass the original question through to the final generation step, as well as the retrieved context docs:\n\n```{python}\nfrom langchain_core.runnables import RunnablePassthrough\n\nretriever = db.as_retriever()\n\nrag_chain = (\n {\"context\": retriever, \"question\": RunnablePassthrough()}\n    | llm_chain\n)\n```\n\n## Compare the results\n\nLet's see the difference RAG makes in generating answers to the library-specific questions.\n\n```{python}\nquestion = \"How do you combine multiple adapters?\"\n```\n\nFirst, let's see what kind of answer we can get with just the model itself, no context added:\n\n```{python}\n#| colab: {base_uri: 'https://localhost:8080/', height: 125}\nllm_chain.invoke({\"context\":\"\", \"question\": question})\n```\n\nAs you can see, the model interpreted the question as one about physical computer adapters, while in the context of PEFT, \"adapters\" refer to LoRA adapters.\nLet's see if adding context from GitHub issues helps the model give a more relevant answer:\n\n```{python}\n#| colab: {base_uri: 'https://localhost:8080/', height: 125}\nrag_chain.invoke(question)\n```\n\nAs we can see, the added context, really helps the exact same model, provide a much more relevant and informed answer to the library-specific question.\n\nNotably, combining multiple adapters for inference has been added to the library, and one can find this information in the documentation, so for the next iteration of this RAG it may be worth including documentation embeddings.\n\n"},"formats":{"html":{"identifier":{"display-name":"HTML","target-format":"html","base-format":"html"},"execute":{"fig-width":7,"fig-height":5,"fig-format":"retina","fig-dpi":96,"df-print":"default","error":false,"eval":false,"cache":null,"freeze":false,"echo":true,"output":true,"warning":true,"include":true,"keep-md":false,"keep-ipynb":false,"ipynb":null,"enabled":null,"daemon":null,"daemon-restart":false,"debug":false,"ipynb-filters":[],"ipynb-shell-interactivity":null,"plotly-connected":true,"engine":"jupyter"},"render":{"keep-tex":false,"keep-typ":false,"keep-source":false,"keep-hidden":false,"prefer-html":false,"output-divs":true,"output-ext":"html","fig-align":"default","fig-pos":null,"fig-env":null,"code-fold":"none","code-overflow":"scroll","code-link":false,"code-line-numbers":false,"code-tools":false,"tbl-colwidths":"auto","merge-includes":true,"inline-includes":false,"preserve-yaml":false,"latex-auto-mk":true,"latex-auto-install":true,"latex-clean":true,"latex-min-runs":1,"latex-max-runs":10,"latex-makeindex":"makeindex","latex-makeindex-opts":[],"latex-tlmgr-opts":[],"latex-input-paths":[],"latex-output-dir":null,"link-external-icon":false,"link-external-newwindow":false,"self-contained-math":false,"format-resources":[],"notebook-links":true},"pandoc":{"standalone":true,"wrap":"none","default-image-extension":"png","to":"html","css":["../styles.css"],"toc":true,"output-file":"rag_zephyr_langchain.html"},"language":{"toc-title-document":"Table of contents","toc-title-website":"On this page","related-formats-title":"Other Formats","related-notebooks-title":"Notebooks","source-notebooks-prefix":"Source","other-links-title":"Other Links","code-links-title":"Code Links","launch-dev-container-title":"Launch Dev Container","launch-binder-title":"Launch Binder","article-notebook-label":"Article Notebook","notebook-preview-download":"Download Notebook","notebook-preview-download-src":"Download Source","notebook-preview-back":"Back to Article","manuscript-meca-bundle":"MECA Bundle","section-title-abstract":"Abstract","section-title-appendices":"Appendices","section-title-footnotes":"Footnotes","section-title-references":"References","section-title-reuse":"Reuse","section-title-copyright":"Copyright","section-title-citation":"Citation","appendix-attribution-cite-as":"For attribution, please cite this work as:","appendix-attribution-bibtex":"BibTeX citation:","appendix-view-license":"View License","title-block-author-single":"Author","title-block-author-plural":"Authors","title-block-affiliation-single":"Affiliation","title-block-affiliation-plural":"Affiliations","title-block-published":"Published","title-block-modified":"Modified","title-block-keywords":"Keywords","callout-tip-title":"Tip","callout-note-title":"Note","callout-warning-title":"Warning","callout-important-title":"Important","callout-caution-title":"Caution","code-summary":"Code","code-tools-menu-caption":"Code","code-tools-show-all-code":"Show All Code","code-tools-hide-all-code":"Hide All Code","code-tools-view-source":"View Source","code-tools-source-code":"Source Code","tools-share":"Share","tools-download":"Download","code-line":"Line","code-lines":"Lines","copy-button-tooltip":"Copy to Clipboard","copy-button-tooltip-success":"Copied!","repo-action-links-edit":"Edit this page","repo-action-links-source":"View source","repo-action-links-issue":"Report an issue","back-to-top":"Back to top","search-no-results-text":"No results","search-matching-documents-text":"matching documents","search-copy-link-title":"Copy link to search","search-hide-matches-text":"Hide additional matches","search-more-match-text":"more match in this document","search-more-matches-text":"more matches in this document","search-clear-button-title":"Clear","search-text-placeholder":"","search-detached-cancel-button-title":"Cancel","search-submit-button-title":"Submit","search-label":"Search","toggle-section":"Toggle section","toggle-sidebar":"Toggle sidebar navigation","toggle-dark-mode":"Toggle dark mode","toggle-reader-mode":"Toggle reader mode","toggle-navigation":"Toggle navigation","crossref-fig-title":"Figure","crossref-tbl-title":"Table","crossref-lst-title":"Listing","crossref-thm-title":"Theorem","crossref-lem-title":"Lemma","crossref-cor-title":"Corollary","crossref-prp-title":"Proposition","crossref-cnj-title":"Conjecture","crossref-def-title":"Definition","crossref-exm-title":"Example","crossref-exr-title":"Exercise","crossref-ch-prefix":"Chapter","crossref-apx-prefix":"Appendix","crossref-sec-prefix":"Section","crossref-eq-prefix":"Equation","crossref-lof-title":"List of Figures","crossref-lot-title":"List of Tables","crossref-lol-title":"List of Listings","environment-proof-title":"Proof","environment-remark-title":"Remark","environment-solution-title":"Solution","listing-page-order-by":"Order By","listing-page-order-by-default":"Default","listing-page-order-by-date-asc":"Oldest","listing-page-order-by-date-desc":"Newest","listing-page-order-by-number-desc":"High to Low","listing-page-order-by-number-asc":"Low to High","listing-page-field-date":"Date","listing-page-field-title":"Title","listing-page-field-description":"Description","listing-page-field-author":"Author","listing-page-field-filename":"File Name","listing-page-field-filemodified":"Modified","listing-page-field-subtitle":"Subtitle","listing-page-field-readingtime":"Reading Time","listing-page-field-wordcount":"Word Count","listing-page-field-categories":"Categories","listing-page-minutes-compact":"{0} min","listing-page-category-all":"All","listing-page-no-matches":"No matching items","listing-page-words":"{0} words","listing-page-filter":"Filter","draft":"Draft"},"metadata":{"lang":"en","fig-responsive":true,"quarto-version":"1.6.40","theme":"cosmo","title":"Simple RAG","jupyter":"python3","code-annotations":"hover"},"extensions":{"book":{"multiFile":true}}}},"projectFormats":["html"]}

src/.quarto/idx/notebooks/single_gpu.ipynb.json

@@ -1 +0,0 @@
-{"title":"Single GPU Fine-tuning","markdown":{"yaml":{"title":"Single GPU Fine-tuning"},"headingText":"Fine-tuning a Code LLM on Custom Code on a single GPU","containsRefs":false,"markdown":"\n\n\n_Authored by: [Maria Khalusova](https://github.com/MKhalusova)_\n\nPublicly available code LLMs such as Codex, StarCoder, and Code Llama are great at generating code that adheres to general programming principles and syntax, but they may not align with an organization's internal conventions, or be aware of proprietary libraries.\n\nIn this notebook, we'll see show how you can fine-tune a code LLM on private code bases to enhance its contextual awareness and improve a model's usefulness to your organization's needs. Since the code LLMs are quite large, fine-tuning them in a traditional manner can be resource-draining. Worry not! We will show how you can optimize fine-tuning to fit on a single GPU.\n\n\n## Dataset\n\nFor this example, we picked the top 10 Hugging Face public repositories on GitHub. We have excluded non-code files from the data, such as images, audio files, presentations, and so on. For Jupyter notebooks, we've kept only cells containing code. The resulting code is stored as a dataset that you can find on the Hugging Face Hub under [`smangrul/hf-stack-v1`](https://huggingface.co/datasets/smangrul/hf-stack-v1). It contains repo id, file path, and file content.\n\n\n## Model\n\nWe'll finetune [`bigcode/starcoderbase-1b`](https://huggingface.co/bigcode/starcoderbase-1b), which is a 1B parameter model trained on 80+ programming languages. This is a gated model, so if you plan to run this notebook with this exact model, you'll need to gain access to it on the model's page. Log in to your Hugging Face account to do so:\n\nTo get started, let's install all the necessary libraries. As you can see, in addition to `transformers` and `datasets`, we'll be using `peft`, `bitsandbytes`, and `flash-attn` to optimize the training.\n\nBy employing parameter-efficient training techniques, we can run this notebook on a single A100 High-RAM GPU.\n\nLet's define some variables now. Feel free to play with these.\n\n## Prepare the data\n\nBegin by loading the data. As the dataset is likely to be quite large, make sure to enable the streaming mode. Streaming allows us to load the data progressively as we iterate over the dataset instead of downloading the whole dataset at once.\n\nWe'll reserve the first 4000 examples as the validation set, and everything else will be the training data.\n\nAt this step, the dataset still contains raw data with code of arbitraty length. For training, we need inputs of fixed length. Let's create an Iterable dataset that would return constant-length chunks of tokens from a stream of text files.\n\nFirst, let's estimate the average number of characters per token in the dataset, which will help us later estimate the number of tokens in the text buffer later. By default, we'll only take 400 examples (`nb_examples`) from the dataset. Using only a subset of the entire dataset will reduce computational cost while still providing a reasonable estimate of the overall character-to-token ratio.\n\nThe character-to-token ratio can also be used as an indicator of the quality of text tokenization. For instance, a character-to-token ratio of 1.0 would mean that each character is represented with a token, which is not very meaningful. This would indicate poor tokenization. In standard English text, one token is typically equivalent to approximately four characters, meaning the character-to-token ratio is around 4.0. We can expect a lower ratio in the code dataset, but generally speaking, a number between 2.0 and 3.5 can be considered good enough.\n\n**Optional FIM transformations**\n\n\nAutoregressive language models typically generate sequences from left to right. By applying the FIM transformations, the model can also learn to infill text.  Check out [\"Efficient Training of Language Models to Fill in the Middle\" paper](https://arxiv.org/pdf/2207.14255.pdf) to learn more about the technique.\nWe'll define the FIM transformations here and will use them when creating the Iterable Dataset. However, if you want to omit transformations, feel free to set `fim_rate` to 0.\n\nLet's define the `ConstantLengthDataset`, an Iterable dataset that will return constant-length chunks of tokens. To do so, we'll read a buffer of text from the original dataset until we hit the size limits and then apply tokenizer to convert the raw text into tokenized inputs. Optionally, we'll perform FIM transformations on some sequences (the proportion of sequences affected is controlled by `fim_rate`).\n\nOnce defined, we can create instances of the `ConstantLengthDataset` from both training and validation data.\n\n## Prepare the model\n\nNow that the data is prepared, it's time to load the model! We're going to load the quantized version of the model.\n\nThis will allow us to reduce memory usage, as quantization represents data with fewer bits. We'll use the `bitsandbytes` library to quantize the model, as it has a nice integration with `transformers`. All we need to do is define a `bitsandbytes` config, and then use it when loading the model.\n\nThere are different variants of 4bit quantization, but generally, we recommend using NF4 quantization for better performance (`bnb_4bit_quant_type=\"nf4\"`).\n\nThe `bnb_4bit_use_double_quant` option adds a second quantization after the first one to save an additional 0.4 bits per parameter.\n\nTo learn more about quantization, check out the [\"Making LLMs even more accessible with bitsandbytes, 4-bit quantization and QLoRA\" blog post](https://huggingface.co/blog/4bit-transformers-bitsandbytes).\n\nOnce defined, pass the config to the `from_pretrained` method to load the quantized version of the model.\n\nWhen using a quantized model for training, you need to call the `prepare_model_for_kbit_training()` function to preprocess the quantized model for training.\n\nNow that the quantized model is ready, we can set up a LoRA configuration. LoRA makes fine-tuning more efficient by drastically reducing the number of trainable parameters.\n\nTo train a model using LoRA technique, we need to wrap the base model as a `PeftModel`. This involves definign LoRA configuration with `LoraConfig`, and wrapping the original model with `get_peft_model()` using the `LoraConfig`.\n\nTo learn more about LoRA and its parameters, refer to [PEFT documentation](https://huggingface.co/docs/peft/conceptual_guides/lora).\n\nAs you can see, by applying LoRA technique we will now need to train less than 1% of the parameters.\n\n## Train the model\n\nNow that we have prepared the data, and optimized the model, we are ready to bring everything together to start the training.\n\nTo instantiate a `Trainer`, you need to define the training configuration. The most important is the `TrainingArguments`, which is a class that contains all the attributes to configure the training.\n\nThese are similar to any other kind of model training you may run, so we won't go into detail here.\n\nAs a final step, instantiate the `Trainer` and call the `train` method.   \n\nFinally, you can push the fine-tuned model to your Hub repository to share with your team.\n\n## Inference\n\nOnce the model is uploaded to Hub, we can use it for inference. To do so we first initialize the original base model and its tokenizer. Next, we need to merge the fine-duned weights with the base model.\n\nNow we can use the merged model for inference. For convenience, we'll define a `get_code_completion` - feel free to experiment with text generation parameters!\n\nNow all we need to do to get code completion is call the `get_code_complete` function and pass the first few lines that we want to be completed as a prefix, and an empty string as a suffix.\n\nAs someone who has just used the PEFT library earlier in this notebook, you can see that the generated result for creating a `LoraConfig` is rather good!\n\nIf you go back to the cell where we instantiate the model for inference, and comment out the lines where we merge the fine-tuned weights, you can see what the original model would've generated for the exact same prefix:\n\nWhile it is Python syntax, you can see that the original model has no understanding of what a `LoraConfig` should be doing.\n\nTo learn how this kind of fine-tuning compares to full fine-tuning, and how to use a model like this as your copilot in VS Code via Inference Endpoints, or locally, check out the [\"Personal Copilot: Train Your Own Coding Assistant\" blog post](https://huggingface.co/blog/personal-copilot). This notebook complements the original blog post.\n\n","srcMarkdownNoYaml":"\n\n# Fine-tuning a Code LLM on Custom Code on a single GPU\n\n_Authored by: [Maria Khalusova](https://github.com/MKhalusova)_\n\nPublicly available code LLMs such as Codex, StarCoder, and Code Llama are great at generating code that adheres to general programming principles and syntax, but they may not align with an organization's internal conventions, or be aware of proprietary libraries.\n\nIn this notebook, we'll see show how you can fine-tune a code LLM on private code bases to enhance its contextual awareness and improve a model's usefulness to your organization's needs. Since the code LLMs are quite large, fine-tuning them in a traditional manner can be resource-draining. Worry not! We will show how you can optimize fine-tuning to fit on a single GPU.\n\n\n## Dataset\n\nFor this example, we picked the top 10 Hugging Face public repositories on GitHub. We have excluded non-code files from the data, such as images, audio files, presentations, and so on. For Jupyter notebooks, we've kept only cells containing code. The resulting code is stored as a dataset that you can find on the Hugging Face Hub under [`smangrul/hf-stack-v1`](https://huggingface.co/datasets/smangrul/hf-stack-v1). It contains repo id, file path, and file content.\n\n\n## Model\n\nWe'll finetune [`bigcode/starcoderbase-1b`](https://huggingface.co/bigcode/starcoderbase-1b), which is a 1B parameter model trained on 80+ programming languages. This is a gated model, so if you plan to run this notebook with this exact model, you'll need to gain access to it on the model's page. Log in to your Hugging Face account to do so:\n\nTo get started, let's install all the necessary libraries. As you can see, in addition to `transformers` and `datasets`, we'll be using `peft`, `bitsandbytes`, and `flash-attn` to optimize the training.\n\nBy employing parameter-efficient training techniques, we can run this notebook on a single A100 High-RAM GPU.\n\nLet's define some variables now. Feel free to play with these.\n\n## Prepare the data\n\nBegin by loading the data. As the dataset is likely to be quite large, make sure to enable the streaming mode. Streaming allows us to load the data progressively as we iterate over the dataset instead of downloading the whole dataset at once.\n\nWe'll reserve the first 4000 examples as the validation set, and everything else will be the training data.\n\nAt this step, the dataset still contains raw data with code of arbitraty length. For training, we need inputs of fixed length. Let's create an Iterable dataset that would return constant-length chunks of tokens from a stream of text files.\n\nFirst, let's estimate the average number of characters per token in the dataset, which will help us later estimate the number of tokens in the text buffer later. By default, we'll only take 400 examples (`nb_examples`) from the dataset. Using only a subset of the entire dataset will reduce computational cost while still providing a reasonable estimate of the overall character-to-token ratio.\n\nThe character-to-token ratio can also be used as an indicator of the quality of text tokenization. For instance, a character-to-token ratio of 1.0 would mean that each character is represented with a token, which is not very meaningful. This would indicate poor tokenization. In standard English text, one token is typically equivalent to approximately four characters, meaning the character-to-token ratio is around 4.0. We can expect a lower ratio in the code dataset, but generally speaking, a number between 2.0 and 3.5 can be considered good enough.\n\n**Optional FIM transformations**\n\n\nAutoregressive language models typically generate sequences from left to right. By applying the FIM transformations, the model can also learn to infill text.  Check out [\"Efficient Training of Language Models to Fill in the Middle\" paper](https://arxiv.org/pdf/2207.14255.pdf) to learn more about the technique.\nWe'll define the FIM transformations here and will use them when creating the Iterable Dataset. However, if you want to omit transformations, feel free to set `fim_rate` to 0.\n\nLet's define the `ConstantLengthDataset`, an Iterable dataset that will return constant-length chunks of tokens. To do so, we'll read a buffer of text from the original dataset until we hit the size limits and then apply tokenizer to convert the raw text into tokenized inputs. Optionally, we'll perform FIM transformations on some sequences (the proportion of sequences affected is controlled by `fim_rate`).\n\nOnce defined, we can create instances of the `ConstantLengthDataset` from both training and validation data.\n\n## Prepare the model\n\nNow that the data is prepared, it's time to load the model! We're going to load the quantized version of the model.\n\nThis will allow us to reduce memory usage, as quantization represents data with fewer bits. We'll use the `bitsandbytes` library to quantize the model, as it has a nice integration with `transformers`. All we need to do is define a `bitsandbytes` config, and then use it when loading the model.\n\nThere are different variants of 4bit quantization, but generally, we recommend using NF4 quantization for better performance (`bnb_4bit_quant_type=\"nf4\"`).\n\nThe `bnb_4bit_use_double_quant` option adds a second quantization after the first one to save an additional 0.4 bits per parameter.\n\nTo learn more about quantization, check out the [\"Making LLMs even more accessible with bitsandbytes, 4-bit quantization and QLoRA\" blog post](https://huggingface.co/blog/4bit-transformers-bitsandbytes).\n\nOnce defined, pass the config to the `from_pretrained` method to load the quantized version of the model.\n\nWhen using a quantized model for training, you need to call the `prepare_model_for_kbit_training()` function to preprocess the quantized model for training.\n\nNow that the quantized model is ready, we can set up a LoRA configuration. LoRA makes fine-tuning more efficient by drastically reducing the number of trainable parameters.\n\nTo train a model using LoRA technique, we need to wrap the base model as a `PeftModel`. This involves definign LoRA configuration with `LoraConfig`, and wrapping the original model with `get_peft_model()` using the `LoraConfig`.\n\nTo learn more about LoRA and its parameters, refer to [PEFT documentation](https://huggingface.co/docs/peft/conceptual_guides/lora).\n\nAs you can see, by applying LoRA technique we will now need to train less than 1% of the parameters.\n\n## Train the model\n\nNow that we have prepared the data, and optimized the model, we are ready to bring everything together to start the training.\n\nTo instantiate a `Trainer`, you need to define the training configuration. The most important is the `TrainingArguments`, which is a class that contains all the attributes to configure the training.\n\nThese are similar to any other kind of model training you may run, so we won't go into detail here.\n\nAs a final step, instantiate the `Trainer` and call the `train` method.   \n\nFinally, you can push the fine-tuned model to your Hub repository to share with your team.\n\n## Inference\n\nOnce the model is uploaded to Hub, we can use it for inference. To do so we first initialize the original base model and its tokenizer. Next, we need to merge the fine-duned weights with the base model.\n\nNow we can use the merged model for inference. For convenience, we'll define a `get_code_completion` - feel free to experiment with text generation parameters!\n\nNow all we need to do to get code completion is call the `get_code_complete` function and pass the first few lines that we want to be completed as a prefix, and an empty string as a suffix.\n\nAs someone who has just used the PEFT library earlier in this notebook, you can see that the generated result for creating a `LoraConfig` is rather good!\n\nIf you go back to the cell where we instantiate the model for inference, and comment out the lines where we merge the fine-tuned weights, you can see what the original model would've generated for the exact same prefix:\n\nWhile it is Python syntax, you can see that the original model has no understanding of what a `LoraConfig` should be doing.\n\nTo learn how this kind of fine-tuning compares to full fine-tuning, and how to use a model like this as your copilot in VS Code via Inference Endpoints, or locally, check out the [\"Personal Copilot: Train Your Own Coding Assistant\" blog post](https://huggingface.co/blog/personal-copilot). This notebook complements the original blog post.\n\n"},"formats":{"html":{"identifier":{"display-name":"HTML","target-format":"html","base-format":"html"},"execute":{"fig-width":7,"fig-height":5,"fig-format":"retina","fig-dpi":96,"df-print":"default","error":false,"eval":true,"cache":null,"freeze":false,"echo":true,"output":true,"warning":true,"include":true,"keep-md":false,"keep-ipynb":false,"ipynb":null,"enabled":false,"daemon":null,"daemon-restart":false,"debug":false,"ipynb-filters":[],"ipynb-shell-interactivity":null,"plotly-connected":true,"engine":"jupyter"},"render":{"keep-tex":false,"keep-typ":false,"keep-source":false,"keep-hidden":false,"prefer-html":false,"output-divs":true,"output-ext":"html","fig-align":"default","fig-pos":null,"fig-env":null,"code-fold":"none","code-overflow":"scroll","code-link":false,"code-line-numbers":false,"code-tools":false,"tbl-colwidths":"auto","merge-includes":true,"inline-includes":false,"preserve-yaml":false,"latex-auto-mk":true,"latex-auto-install":true,"latex-clean":true,"latex-min-runs":1,"latex-max-runs":10,"latex-makeindex":"makeindex","latex-makeindex-opts":[],"latex-tlmgr-opts":[],"latex-input-paths":[],"latex-output-dir":null,"link-external-icon":false,"link-external-newwindow":false,"self-contained-math":false,"format-resources":[],"notebook-links":true},"pandoc":{"standalone":true,"wrap":"none","default-image-extension":"png","to":"html","css":["../styles.css"],"toc":true,"output-file":"single_gpu.html"},"language":{"toc-title-document":"Table of contents","toc-title-website":"On this page","related-formats-title":"Other Formats","related-notebooks-title":"Notebooks","source-notebooks-prefix":"Source","other-links-title":"Other Links","code-links-title":"Code Links","launch-dev-container-title":"Launch Dev Container","launch-binder-title":"Launch Binder","article-notebook-label":"Article Notebook","notebook-preview-download":"Download Notebook","notebook-preview-download-src":"Download Source","notebook-preview-back":"Back to Article","manuscript-meca-bundle":"MECA Bundle","section-title-abstract":"Abstract","section-title-appendices":"Appendices","section-title-footnotes":"Footnotes","section-title-references":"References","section-title-reuse":"Reuse","section-title-copyright":"Copyright","section-title-citation":"Citation","appendix-attribution-cite-as":"For attribution, please cite this work as:","appendix-attribution-bibtex":"BibTeX citation:","appendix-view-license":"View License","title-block-author-single":"Author","title-block-author-plural":"Authors","title-block-affiliation-single":"Affiliation","title-block-affiliation-plural":"Affiliations","title-block-published":"Published","title-block-modified":"Modified","title-block-keywords":"Keywords","callout-tip-title":"Tip","callout-note-title":"Note","callout-warning-title":"Warning","callout-important-title":"Important","callout-caution-title":"Caution","code-summary":"Code","code-tools-menu-caption":"Code","code-tools-show-all-code":"Show All Code","code-tools-hide-all-code":"Hide All Code","code-tools-view-source":"View Source","code-tools-source-code":"Source Code","tools-share":"Share","tools-download":"Download","code-line":"Line","code-lines":"Lines","copy-button-tooltip":"Copy to Clipboard","copy-button-tooltip-success":"Copied!","repo-action-links-edit":"Edit this page","repo-action-links-source":"View source","repo-action-links-issue":"Report an issue","back-to-top":"Back to top","search-no-results-text":"No results","search-matching-documents-text":"matching documents","search-copy-link-title":"Copy link to search","search-hide-matches-text":"Hide additional matches","search-more-match-text":"more match in this document","search-more-matches-text":"more matches in this document","search-clear-button-title":"Clear","search-text-placeholder":"","search-detached-cancel-button-title":"Cancel","search-submit-button-title":"Submit","search-label":"Search","toggle-section":"Toggle section","toggle-sidebar":"Toggle sidebar navigation","toggle-dark-mode":"Toggle dark mode","toggle-reader-mode":"Toggle reader mode","toggle-navigation":"Toggle navigation","crossref-fig-title":"Figure","crossref-tbl-title":"Table","crossref-lst-title":"Listing","crossref-thm-title":"Theorem","crossref-lem-title":"Lemma","crossref-cor-title":"Corollary","crossref-prp-title":"Proposition","crossref-cnj-title":"Conjecture","crossref-def-title":"Definition","crossref-exm-title":"Example","crossref-exr-title":"Exercise","crossref-ch-prefix":"Chapter","crossref-apx-prefix":"Appendix","crossref-sec-prefix":"Section","crossref-eq-prefix":"Equation","crossref-lof-title":"List of Figures","crossref-lot-title":"List of Tables","crossref-lol-title":"List of Listings","environment-proof-title":"Proof","environment-remark-title":"Remark","environment-solution-title":"Solution","listing-page-order-by":"Order By","listing-page-order-by-default":"Default","listing-page-order-by-date-asc":"Oldest","listing-page-order-by-date-desc":"Newest","listing-page-order-by-number-desc":"High to Low","listing-page-order-by-number-asc":"Low to High","listing-page-field-date":"Date","listing-page-field-title":"Title","listing-page-field-description":"Description","listing-page-field-author":"Author","listing-page-field-filename":"File Name","listing-page-field-filemodified":"Modified","listing-page-field-subtitle":"Subtitle","listing-page-field-readingtime":"Reading Time","listing-page-field-wordcount":"Word Count","listing-page-field-categories":"Categories","listing-page-minutes-compact":"{0} min","listing-page-category-all":"All","listing-page-no-matches":"No matching items","listing-page-words":"{0} words","listing-page-filter":"Filter","draft":"Draft"},"metadata":{"lang":"en","fig-responsive":true,"quarto-version":"1.6.40","theme":"cosmo","title":"Single GPU Fine-tuning"},"extensions":{"book":{"multiFile":true}}}},"projectFormats":["html"]}

src/.quarto/idx/presentation.qmd.json

@@ -1 +0,0 @@
-{"title":"Optimizing LLM Performance Using Triton","markdown":{"yaml":{"title":"Optimizing LLM Performance Using Triton","format":{"revealjs":{"theme":"dark","transition":"slide","slide-number":true}},"author":"Matej Sirovatka","date":"today"},"headingText":"`whoami`","containsRefs":false,"markdown":"\n\n\n- My name is Matej\n- I'm a Master's student at the Brno University of Technology\n- I currently make GPUs go `brrrrrr` at Hugging Face 🤗\n\n## `What is Triton?`\n\n- NVIDIA's open-source programming language for GPU kernels\n- Designed for AI/ML workloads\n- Simplifies GPU programming compared to CUDA\n\n![](media/optim_scale.png){.center fig-align=\"center\"}\n\n## `Why Optimize with Triton?`\n\n- Simple yet effective\n- Less headache than CUDA\n- GPUs go `brrrrrrr` 🚀\n- Feel cool when your kernel is faster than PyTorch 😎\n\n## `Example Problem: KL Divergence`\n\n- commonly used in LLMs for knowledge distillation\n- for probability distributions $P$ and $Q$, the Kullback-Leibler divergence is defined as:\n\n$$\nD_{KL}(P \\| Q) = \\sum_{i} P_i \\log\\left(\\frac{P_i}{Q_i}\\right)\n$$\n\n\n```python\nimport torch\nfrom torch.nn.functional import kl_div\n\ndef kl_div_torch(p: torch.Tensor, q: torch.Tensor) -> torch.Tensor:\n    return kl_div(p, q, reduction='none')\n```\n\n## `How about Triton?`\n\n```python\nimport triton.language as tl\n\n@triton.jit\ndef kl_div_triton(\n    p_ptr, q_ptr, output_ptr, n_elements, BLOCK_SIZE: tl.constexpr\n):\n    pid = tl.program_id(0)\n    block_start = pid * BLOCK_SIZE\n    offsets = block_start + tl.arange(0, BLOCK_SIZE)\n    mask = offsets < n_elements\n    \n    p = tl.load(p_ptr + offsets, mask=mask)\n    q = tl.load(q_ptr + offsets, mask=mask)\n    \n    output = p * (tl.log(p) - tl.log(q))\n    \n    tl.store(output_ptr + offsets, output, mask=mask)\n```\n\n## `How to integrate with PyTorch?`\n\n- Triton works with pointers\n- How to use our custom kernel with PyTorch autograd?\n\n```python\nimport torch\n\nclass VectorAdd(torch.autograd.Function):\n    @staticmethod\n    def forward(ctx, p, q):\n        ctx.save_for_backward(q)\n        output = torch.empty_like(p)\n        grid = (len(p) + 512 - 1) // 512\n        kl_div_triton[grid](p, q, output, len(p), BLOCK_SIZE=512)\n        return output\n\n    @staticmethod\n    def backward(ctx, grad_output):\n        q = ctx.saved_tensors[0]\n        # Calculate gradients (another triton kernel)\n        return ...\n```\n\n## `Some benchmarks`\n\n- A KL Divergence kernel that is currently used in [Liger Kernel](https://github.com/linkedin/liger-kernel) written by @me\n\n:::: {.columns}\n\n::: {.column width=\"50%\"}\n\n![](media/kl_mem.png){.center fig-align=\"center\"}\n\n:::\n\n::: {.column width=\"50%\"}\n\n![](media/kl_speed.png){.center fig-align=\"center\"}\n\n:::\n\n::::\n\n## `Do I have to write everything?`\n\n- TLDR: No\n- Many cool projects already using Triton\n- Better Integration with PyTorch and even Hugging Face 🤗\n- Liger Kernel, Unsloth AI, etc.\n\n:::: {.columns}\n\n::: {.column width=\"50%\"}\n\n![](media/unsloth.png){.center fig-align=\"center\"}\n\n:::\n\n::: {.column width=\"50%\"}\n\n![](media/liger.png){.center fig-align=\"center\"}\n\n:::\n\n::::\n\n\n## `So how can I use this in my LLM? 🚀`\n\n- Liger Kernel is a great example, providing examples of how to integrate with Hugging Face 🤗 Trainer\n\n```diff\n- from transformers import AutoModelForCausalLM\n+ from liger_kernel.transformers import AutoLigerKernelForCausalLM\n\nmodel_path = \"meta-llama/Meta-Llama-3-8B-Instruct\"\n\n- model = AutoModelForCausalLM.from_pretrained(model_path)\n+ model = AutoLigerKernelForCausalLM.from_pretrained(model_path)\n\n# training/inference logic...\n```\n## `Key Optimization Techniques adapted by Liger Kernel`\n\n- Kernel Fusion\n- Domain-specific optimizations\n- Memory Access Patterns\n- Preemptive memory freeing\n   \n\n## `Aaand some more benchmarks 🚀`\n\n\n:::: {.columns}\n\n::: {.column width=\"50%\"}\n\n![](media/PMA.png){fig-align=\"center\"}\n\n:::\n\n::: {.column width=\"50%\"}\n\n![](media/PMR.png){fig-align=\"center\"}\n\n:::\n\n::::\n\n## `Last benchmark I promise...`\n\n\n![](media/TPS.png){height=\"50%\" width=\"50%\" }\n\n::: {.incremental}\n*Attention is all you need, so I thank you for yours!* 🤗\n:::\n\n","srcMarkdownNoYaml":"\n\n## `whoami`\n\n- My name is Matej\n- I'm a Master's student at the Brno University of Technology\n- I currently make GPUs go `brrrrrr` at Hugging Face 🤗\n\n## `What is Triton?`\n\n- NVIDIA's open-source programming language for GPU kernels\n- Designed for AI/ML workloads\n- Simplifies GPU programming compared to CUDA\n\n![](media/optim_scale.png){.center fig-align=\"center\"}\n\n## `Why Optimize with Triton?`\n\n- Simple yet effective\n- Less headache than CUDA\n- GPUs go `brrrrrrr` 🚀\n- Feel cool when your kernel is faster than PyTorch 😎\n\n## `Example Problem: KL Divergence`\n\n- commonly used in LLMs for knowledge distillation\n- for probability distributions $P$ and $Q$, the Kullback-Leibler divergence is defined as:\n\n$$\nD_{KL}(P \\| Q) = \\sum_{i} P_i \\log\\left(\\frac{P_i}{Q_i}\\right)\n$$\n\n\n```python\nimport torch\nfrom torch.nn.functional import kl_div\n\ndef kl_div_torch(p: torch.Tensor, q: torch.Tensor) -> torch.Tensor:\n    return kl_div(p, q, reduction='none')\n```\n\n## `How about Triton?`\n\n```python\nimport triton.language as tl\n\n@triton.jit\ndef kl_div_triton(\n    p_ptr, q_ptr, output_ptr, n_elements, BLOCK_SIZE: tl.constexpr\n):\n    pid = tl.program_id(0)\n    block_start = pid * BLOCK_SIZE\n    offsets = block_start + tl.arange(0, BLOCK_SIZE)\n    mask = offsets < n_elements\n    \n    p = tl.load(p_ptr + offsets, mask=mask)\n    q = tl.load(q_ptr + offsets, mask=mask)\n    \n    output = p * (tl.log(p) - tl.log(q))\n    \n    tl.store(output_ptr + offsets, output, mask=mask)\n```\n\n## `How to integrate with PyTorch?`\n\n- Triton works with pointers\n- How to use our custom kernel with PyTorch autograd?\n\n```python\nimport torch\n\nclass VectorAdd(torch.autograd.Function):\n    @staticmethod\n    def forward(ctx, p, q):\n        ctx.save_for_backward(q)\n        output = torch.empty_like(p)\n        grid = (len(p) + 512 - 1) // 512\n        kl_div_triton[grid](p, q, output, len(p), BLOCK_SIZE=512)\n        return output\n\n    @staticmethod\n    def backward(ctx, grad_output):\n        q = ctx.saved_tensors[0]\n        # Calculate gradients (another triton kernel)\n        return ...\n```\n\n## `Some benchmarks`\n\n- A KL Divergence kernel that is currently used in [Liger Kernel](https://github.com/linkedin/liger-kernel) written by @me\n\n:::: {.columns}\n\n::: {.column width=\"50%\"}\n\n![](media/kl_mem.png){.center fig-align=\"center\"}\n\n:::\n\n::: {.column width=\"50%\"}\n\n![](media/kl_speed.png){.center fig-align=\"center\"}\n\n:::\n\n::::\n\n## `Do I have to write everything?`\n\n- TLDR: No\n- Many cool projects already using Triton\n- Better Integration with PyTorch and even Hugging Face 🤗\n- Liger Kernel, Unsloth AI, etc.\n\n:::: {.columns}\n\n::: {.column width=\"50%\"}\n\n![](media/unsloth.png){.center fig-align=\"center\"}\n\n:::\n\n::: {.column width=\"50%\"}\n\n![](media/liger.png){.center fig-align=\"center\"}\n\n:::\n\n::::\n\n\n## `So how can I use this in my LLM? 🚀`\n\n- Liger Kernel is a great example, providing examples of how to integrate with Hugging Face 🤗 Trainer\n\n```diff\n- from transformers import AutoModelForCausalLM\n+ from liger_kernel.transformers import AutoLigerKernelForCausalLM\n\nmodel_path = \"meta-llama/Meta-Llama-3-8B-Instruct\"\n\n- model = AutoModelForCausalLM.from_pretrained(model_path)\n+ model = AutoLigerKernelForCausalLM.from_pretrained(model_path)\n\n# training/inference logic...\n```\n## `Key Optimization Techniques adapted by Liger Kernel`\n\n- Kernel Fusion\n- Domain-specific optimizations\n- Memory Access Patterns\n- Preemptive memory freeing\n   \n\n## `Aaand some more benchmarks 🚀`\n\n\n:::: {.columns}\n\n::: {.column width=\"50%\"}\n\n![](media/PMA.png){fig-align=\"center\"}\n\n:::\n\n::: {.column width=\"50%\"}\n\n![](media/PMR.png){fig-align=\"center\"}\n\n:::\n\n::::\n\n## `Last benchmark I promise...`\n\n\n![](media/TPS.png){height=\"50%\" width=\"50%\" }\n\n::: {.incremental}\n*Attention is all you need, so I thank you for yours!* 🤗\n:::\n\n"},"formats":{"revealjs":{"identifier":{"display-name":"RevealJS","target-format":"revealjs","base-format":"revealjs"},"execute":{"fig-width":10,"fig-height":5,"fig-format":"retina","fig-dpi":96,"df-print":"default","error":false,"eval":true,"cache":null,"freeze":false,"echo":false,"output":true,"warning":false,"include":true,"keep-md":false,"keep-ipynb":false,"ipynb":null,"enabled":null,"daemon":null,"daemon-restart":false,"debug":false,"ipynb-filters":[],"ipynb-shell-interactivity":null,"plotly-connected":true,"engine":"markdown"},"render":{"keep-tex":false,"keep-typ":false,"keep-source":false,"keep-hidden":false,"prefer-html":false,"output-divs":true,"output-ext":"html","fig-align":"default","fig-pos":null,"fig-env":null,"code-fold":"none","code-overflow":"scroll","code-link":false,"code-line-numbers":true,"code-tools":false,"tbl-colwidths":"auto","merge-includes":true,"inline-includes":false,"preserve-yaml":false,"latex-auto-mk":true,"latex-auto-install":true,"latex-clean":true,"latex-min-runs":1,"latex-max-runs":10,"latex-makeindex":"makeindex","latex-makeindex-opts":[],"latex-tlmgr-opts":[],"latex-input-paths":[],"latex-output-dir":null,"link-external-icon":false,"link-external-newwindow":false,"self-contained-math":false,"format-resources":[]},"pandoc":{"standalone":true,"wrap":"none","default-image-extension":"png","html-math-method":{"method":"mathjax","url":"https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.9/MathJax.js?config=TeX-AMS_HTML-full"},"slide-level":2,"to":"revealjs","output-file":"presentation.html"},"language":{"toc-title-document":"Table of contents","toc-title-website":"On this page","related-formats-title":"Other Formats","related-notebooks-title":"Notebooks","source-notebooks-prefix":"Source","other-links-title":"Other Links","code-links-title":"Code Links","launch-dev-container-title":"Launch Dev Container","launch-binder-title":"Launch Binder","article-notebook-label":"Article Notebook","notebook-preview-download":"Download Notebook","notebook-preview-download-src":"Download Source","notebook-preview-back":"Back to Article","manuscript-meca-bundle":"MECA Bundle","section-title-abstract":"Abstract","section-title-appendices":"Appendices","section-title-footnotes":"Footnotes","section-title-references":"References","section-title-reuse":"Reuse","section-title-copyright":"Copyright","section-title-citation":"Citation","appendix-attribution-cite-as":"For attribution, please cite this work as:","appendix-attribution-bibtex":"BibTeX citation:","appendix-view-license":"View License","title-block-author-single":"Author","title-block-author-plural":"Authors","title-block-affiliation-single":"Affiliation","title-block-affiliation-plural":"Affiliations","title-block-published":"Published","title-block-modified":"Modified","title-block-keywords":"Keywords","callout-tip-title":"Tip","callout-note-title":"Note","callout-warning-title":"Warning","callout-important-title":"Important","callout-caution-title":"Caution","code-summary":"Code","code-tools-menu-caption":"Code","code-tools-show-all-code":"Show All Code","code-tools-hide-all-code":"Hide All Code","code-tools-view-source":"View Source","code-tools-source-code":"Source Code","tools-share":"Share","tools-download":"Download","code-line":"Line","code-lines":"Lines","copy-button-tooltip":"Copy to Clipboard","copy-button-tooltip-success":"Copied!","repo-action-links-edit":"Edit this page","repo-action-links-source":"View source","repo-action-links-issue":"Report an issue","back-to-top":"Back to top","search-no-results-text":"No results","search-matching-documents-text":"matching documents","search-copy-link-title":"Copy link to search","search-hide-matches-text":"Hide additional matches","search-more-match-text":"more match in this document","search-more-matches-text":"more matches in this document","search-clear-button-title":"Clear","search-text-placeholder":"","search-detached-cancel-button-title":"Cancel","search-submit-button-title":"Submit","search-label":"Search","toggle-section":"Toggle section","toggle-sidebar":"Toggle sidebar navigation","toggle-dark-mode":"Toggle dark mode","toggle-reader-mode":"Toggle reader mode","toggle-navigation":"Toggle navigation","crossref-fig-title":"Figure","crossref-tbl-title":"Table","crossref-lst-title":"Listing","crossref-thm-title":"Theorem","crossref-lem-title":"Lemma","crossref-cor-title":"Corollary","crossref-prp-title":"Proposition","crossref-cnj-title":"Conjecture","crossref-def-title":"Definition","crossref-exm-title":"Example","crossref-exr-title":"Exercise","crossref-ch-prefix":"Chapter","crossref-apx-prefix":"Appendix","crossref-sec-prefix":"Section","crossref-eq-prefix":"Equation","crossref-lof-title":"List of Figures","crossref-lot-title":"List of Tables","crossref-lol-title":"List of Listings","environment-proof-title":"Proof","environment-remark-title":"Remark","environment-solution-title":"Solution","listing-page-order-by":"Order By","listing-page-order-by-default":"Default","listing-page-order-by-date-asc":"Oldest","listing-page-order-by-date-desc":"Newest","listing-page-order-by-number-desc":"High to Low","listing-page-order-by-number-asc":"Low to High","listing-page-field-date":"Date","listing-page-field-title":"Title","listing-page-field-description":"Description","listing-page-field-author":"Author","listing-page-field-filename":"File Name","listing-page-field-filemodified":"Modified","listing-page-field-subtitle":"Subtitle","listing-page-field-readingtime":"Reading Time","listing-page-field-wordcount":"Word Count","listing-page-field-categories":"Categories","listing-page-minutes-compact":"{0} min","listing-page-category-all":"All","listing-page-no-matches":"No matching items","listing-page-words":"{0} words","listing-page-filter":"Filter","draft":"Draft"},"metadata":{"lang":"en","fig-responsive":false,"quarto-version":"1.6.40","auto-stretch":true,"title":"Optimizing LLM Performance Using Triton","author":"Matej Sirovatka","date":"today","theme":"dark","transition":"slide","slideNumber":true}}},"projectFormats":["html"]}

src/_site/index.html

@@ -10,7 +10,7 @@
   <meta name="generator" content="quarto-1.6.40">
 
   <meta name="author" content="Matej Sirovatka">
-  <meta name="dcterms.date" content="2025-02-21">
+  <meta name="dcterms.date" content="2025-02-22">
   <title>Optimizing LLM Performance Using Triton</title>
   <meta name="apple-mobile-web-app-capable" content="yes">
   <meta name="apple-mobile-web-app-status-bar-style" content="black-translucent">
@@ -203,14 +203,14 @@ Matej Sirovatka
 </div>
 </div>
 
-  <p class="date">2025-02-21</p>
+  <p class="date">2025-02-22</p>
 </section>
 <section id="whoami" class="slide level2">
 <h2><code>whoami</code></h2>
 <ul>
 <li>My name is Matej</li>
 <li>I’m a Master’s student at the Brno University of Technology</li>
-<li>I currently make GPUs go <code>brrrrrr</code> at Hugging Face 🤗</li>
+<li>I’m currently working on distributed training at Hugging Face 🤗</li>
 </ul>
 </section>
 <section id="what-is-triton" class="slide level2">
@@ -244,32 +244,31 @@ D_{KL}(P \| Q) = \sum_{i} P_i \log\left(\frac{P_i}{Q_i}\right)
 <span id="cb1-2"><a></a><span class="im">from</span> torch.nn.functional <span class="im">import</span> kl_div</span>
 <span id="cb1-3"><a></a></span>
 <span id="cb1-4"><a></a><span class="kw">def</span> kl_div_torch(p: torch.Tensor, q: torch.Tensor) <span class="op">-&gt;</span> torch.Tensor:</span>
-<span id="cb1-5"><a></a>    <span class="cf">return</span> kl_div(p, q, reduction<span class="op">=</span><span class="st">'none'</span>)</span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
+<span id="cb1-5"><a></a>    <span class="cf">return</span> kl_div(p, q)</span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
 </section>
 <section id="how-about-triton" class="slide level2">
 <h2><code>How about Triton?</code></h2>
-<div class="sourceCode" id="cb2"><pre class="sourceCode numberSource python number-lines code-with-copy"><code class="sourceCode python"><span id="cb2-1"><a></a><span class="im">import</span> triton.language <span class="im">as</span> tl</span>
-<span id="cb2-2"><a></a></span>
-<span id="cb2-3"><a></a><span class="at">@triton.jit</span></span>
-<span id="cb2-4"><a></a><span class="kw">def</span> kl_div_triton(</span>
-<span id="cb2-5"><a></a>    p_ptr, q_ptr, output_ptr, n_elements, BLOCK_SIZE: tl.constexpr</span>
-<span id="cb2-6"><a></a>):</span>
-<span id="cb2-7"><a></a>    pid <span class="op">=</span> tl.program_id(<span class="dv">0</span>)</span>
-<span id="cb2-8"><a></a>    block_start <span class="op">=</span> pid <span class="op">*</span> BLOCK_SIZE</span>
-<span id="cb2-9"><a></a>    offsets <span class="op">=</span> block_start <span class="op">+</span> tl.arange(<span class="dv">0</span>, BLOCK_SIZE)</span>
-<span id="cb2-10"><a></a>    mask <span class="op">=</span> offsets <span class="op">&lt;</span> n_elements</span>
-<span id="cb2-11"><a></a>    </span>
-<span id="cb2-12"><a></a>    p <span class="op">=</span> tl.load(p_ptr <span class="op">+</span> offsets, mask<span class="op">=</span>mask)</span>
-<span id="cb2-13"><a></a>    q <span class="op">=</span> tl.load(q_ptr <span class="op">+</span> offsets, mask<span class="op">=</span>mask)</span>
-<span id="cb2-14"><a></a>    </span>
-<span id="cb2-15"><a></a>    output <span class="op">=</span> p <span class="op">*</span> (tl.log(p) <span class="op">-</span> tl.log(q))</span>
-<span id="cb2-16"><a></a>    </span>
+<div class="sourceCode" id="cb2"><pre class="sourceCode numberSource python number-lines code-with-copy"><code class="sourceCode python"><span id="cb2-1"><a></a><span class="im">import</span> triton</span>
+<span id="cb2-2"><a></a><span class="im">import</span> triton.language <span class="im">as</span> tl</span>
+<span id="cb2-3"><a></a></span>
+<span id="cb2-4"><a></a><span class="at">@triton.jit</span></span>
+<span id="cb2-5"><a></a><span class="kw">def</span> kl_div_triton(</span>
+<span id="cb2-6"><a></a>    p_ptr, q_ptr, output_ptr, n_elements, BLOCK_SIZE: tl.constexpr</span>
+<span id="cb2-7"><a></a>):</span>
+<span id="cb2-8"><a></a>    pid <span class="op">=</span> tl.program_id(<span class="dv">0</span>)</span>
+<span id="cb2-9"><a></a>    block_start <span class="op">=</span> pid <span class="op">*</span> BLOCK_SIZE</span>
+<span id="cb2-10"><a></a>    offsets <span class="op">=</span> block_start <span class="op">+</span> tl.arange(<span class="dv">0</span>, BLOCK_SIZE)</span>
+<span id="cb2-11"><a></a>    mask <span class="op">=</span> offsets <span class="op">&lt;</span> n_elements</span>
+<span id="cb2-12"><a></a>    </span>
+<span id="cb2-13"><a></a>    p <span class="op">=</span> tl.load(p_ptr <span class="op">+</span> offsets, mask<span class="op">=</span>mask)</span>
+<span id="cb2-14"><a></a>    q <span class="op">=</span> tl.load(q_ptr <span class="op">+</span> offsets, mask<span class="op">=</span>mask)</span>
+<span id="cb2-15"><a></a>    </span>
+<span id="cb2-16"><a></a>    output <span class="op">=</span> p <span class="op">*</span> (tl.log(p) <span class="op">-</span> tl.log(q))</span>
 <span id="cb2-17"><a></a>    tl.store(output_ptr <span class="op">+</span> offsets, output, mask<span class="op">=</span>mask)</span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
 </section>
 <section id="how-to-integrate-with-pytorch" class="slide level2">
 <h2><code>How to integrate with PyTorch?</code></h2>
 <ul>
-<li>Triton works with pointers</li>
 <li>How to use our custom kernel with PyTorch autograd?</li>
 </ul>
 <div class="sourceCode" id="cb3"><pre class="sourceCode numberSource python number-lines code-with-copy"><code class="sourceCode python"><span id="cb3-1"><a></a><span class="im">import</span> torch</span>
@@ -358,6 +357,9 @@ D_{KL}(P \| Q) = \sum_{i} P_i \log\left(\frac{P_i}{Q_i}\right)
 </section>
 <section id="aaand-some-more-benchmarks" class="slide level2">
 <h2><code>Aaand some more benchmarks 🚀</code></h2>
+<ul>
+<li>Saving memory is key to run bigger batch size on smaller GPUs</li>
+</ul>
 <div class="columns">
 <div class="column" style="width:50%;">
 <div class="quarto-figure quarto-figure-center">
@@ -375,10 +377,24 @@ D_{KL}(P \| Q) = \sum_{i} P_i \log\left(\frac{P_i}{Q_i}\right)
 </section>
 <section id="last-benchmark-i-promise..." class="slide level2">
 <h2><code>Last benchmark I promise...</code></h2>
-<p><img data-src="media/TPS.png" style="width:50.0%;height:50.0%"></p>
-<div>
+<ul>
+<li>But is it faster? Yes, it is!</li>
+</ul>
+<div class="quarto-figure quarto-figure-center">
+<figure>
+<p><img data-src="media/TPS.png" class="quarto-figure quarto-figure-center" style="width:50.0%;height:50.0%"></p>
+</figure>
+</div>
+<div class="columns">
+<div class="column" style="width:60%;">
 <p><em>Attention is all you need, so I thank you for yours!</em> 🤗</p>
+</div><div class="column" style="width:40%;">
+<div class="quarto-figure quarto-figure-center">
+<figure>
+<p><img data-src="media/qr.png" class="quarto-figure quarto-figure-center" style="width:25.0%;height:25.0%"></p>
+</figure>
 </div>
+</div></div>
 
 
 </section>

src/_site/search.json

@@ -227,7 +227,7 @@
     "href": "index.html#whoami",
     "title": "Optimizing LLM Performance Using Triton",
     "section": "whoami",
-    "text": "whoami\n\nMy name is Matej\nI’m a Master’s student at the Brno University of Technology\nI currently make GPUs go brrrrrr at Hugging Face 🤗",
+    "text": "whoami\n\nMy name is Matej\nI’m a Master’s student at the Brno University of Technology\nI’m currently working on distributed training at Hugging Face 🤗",
     "crumbs": [
       "About",
       "About Quarto"
@@ -260,7 +260,7 @@
     "href": "index.html#example-problem-kl-divergence",
     "title": "Optimizing LLM Performance Using Triton",
     "section": "Example Problem: KL Divergence",
-    "text": "Example Problem: KL Divergence\n\ncommonly used in LLMs for knowledge distillation\nfor probability distributions \\(P\\) and \\(Q\\), the Kullback-Leibler divergence is defined as:\n\n\\[\nD_{KL}(P \\| Q) = \\sum_{i} P_i \\log\\left(\\frac{P_i}{Q_i}\\right)\n\\]\nimport torch\nfrom torch.nn.functional import kl_div\n\ndef kl_div_torch(p: torch.Tensor, q: torch.Tensor) -> torch.Tensor:\n    return kl_div(p, q, reduction='none')",
+    "text": "Example Problem: KL Divergence\n\ncommonly used in LLMs for knowledge distillation\nfor probability distributions \\(P\\) and \\(Q\\), the Kullback-Leibler divergence is defined as:\n\n\\[\nD_{KL}(P \\| Q) = \\sum_{i} P_i \\log\\left(\\frac{P_i}{Q_i}\\right)\n\\]\nimport torch\nfrom torch.nn.functional import kl_div\n\ndef kl_div_torch(p: torch.Tensor, q: torch.Tensor) -> torch.Tensor:\n    return kl_div(p, q)",
     "crumbs": [
       "About",
       "About Quarto"
@@ -271,7 +271,7 @@
     "href": "index.html#how-about-triton",
     "title": "Optimizing LLM Performance Using Triton",
     "section": "How about Triton?",
-    "text": "How about Triton?\nimport triton.language as tl\n\n@triton.jit\ndef kl_div_triton(\n    p_ptr, q_ptr, output_ptr, n_elements, BLOCK_SIZE: tl.constexpr\n):\n    pid = tl.program_id(0)\n    block_start = pid * BLOCK_SIZE\n    offsets = block_start + tl.arange(0, BLOCK_SIZE)\n    mask = offsets < n_elements\n    \n    p = tl.load(p_ptr + offsets, mask=mask)\n    q = tl.load(q_ptr + offsets, mask=mask)\n    \n    output = p * (tl.log(p) - tl.log(q))\n    \n    tl.store(output_ptr + offsets, output, mask=mask)",
+    "text": "How about Triton?\nimport triton\nimport triton.language as tl\n\n@triton.jit\ndef kl_div_triton(\n    p_ptr, q_ptr, output_ptr, n_elements, BLOCK_SIZE: tl.constexpr\n):\n    pid = tl.program_id(0)\n    block_start = pid * BLOCK_SIZE\n    offsets = block_start + tl.arange(0, BLOCK_SIZE)\n    mask = offsets < n_elements\n    \n    p = tl.load(p_ptr + offsets, mask=mask)\n    q = tl.load(q_ptr + offsets, mask=mask)\n    \n    output = p * (tl.log(p) - tl.log(q))\n    tl.store(output_ptr + offsets, output, mask=mask)",
     "crumbs": [
       "About",
       "About Quarto"
@@ -282,7 +282,7 @@
     "href": "index.html#how-to-integrate-with-pytorch",
     "title": "Optimizing LLM Performance Using Triton",
     "section": "How to integrate with PyTorch?",
-    "text": "How to integrate with PyTorch?\n\nTriton works with pointers\nHow to use our custom kernel with PyTorch autograd?\n\nimport torch\n\nclass VectorAdd(torch.autograd.Function):\n    @staticmethod\n    def forward(ctx, p, q):\n        ctx.save_for_backward(q)\n        output = torch.empty_like(p)\n        grid = (len(p) + 512 - 1) // 512\n        kl_div_triton[grid](p, q, output, len(p), BLOCK_SIZE=512)\n        return output\n\n    @staticmethod\n    def backward(ctx, grad_output):\n        q = ctx.saved_tensors[0]\n        # Calculate gradients (another triton kernel)\n        return ...",
+    "text": "How to integrate with PyTorch?\n\nHow to use our custom kernel with PyTorch autograd?\n\nimport torch\n\nclass VectorAdd(torch.autograd.Function):\n    @staticmethod\n    def forward(ctx, p, q):\n        ctx.save_for_backward(q)\n        output = torch.empty_like(p)\n        grid = (len(p) + 512 - 1) // 512\n        kl_div_triton[grid](p, q, output, len(p), BLOCK_SIZE=512)\n        return output\n\n    @staticmethod\n    def backward(ctx, grad_output):\n        q = ctx.saved_tensors[0]\n        # Calculate gradients (another triton kernel)\n        return ...",
     "crumbs": [
       "About",
       "About Quarto"
@@ -337,7 +337,7 @@
     "href": "index.html#aaand-some-more-benchmarks",
     "title": "Optimizing LLM Performance Using Triton",
     "section": "Aaand some more benchmarks 🚀",
-    "text": "Aaand some more benchmarks 🚀",
+    "text": "Aaand some more benchmarks 🚀\n\nSaving memory is key to run bigger batch size on smaller GPUs",
     "crumbs": [
       "About",
       "About Quarto"
@@ -348,7 +348,7 @@
     "href": "index.html#last-benchmark-i-promise...",
     "title": "Optimizing LLM Performance Using Triton",
     "section": "Last benchmark I promise...",
-    "text": "Last benchmark I promise...\n\n\nAttention is all you need, so I thank you for yours! 🤗",
+    "text": "Last benchmark I promise...\n\nBut is it faster? Yes, it is!\n\n\n\n\n\n\n\n\nAttention is all you need, so I thank you for yours! 🤗",
     "crumbs": [
       "About",
       "About Quarto"

src/index.qmd

@@ -6,14 +6,14 @@ format:
     transition: slide
     slide-number: true
 author: "Matej Sirovatka"
-date: today
+date: "2025-02-22"
 ---
 
 ## `whoami`
 
 - My name is Matej
 - I'm a Master's student at the Brno University of Technology
-- I currently make GPUs go `brrrrrr` at Hugging Face 🤗
+- I'm currently working on distributed training at Hugging Face 🤗
 
 ## `What is Triton?`
 
@@ -45,12 +45,13 @@ import torch
 from torch.nn.functional import kl_div
 
 def kl_div_torch(p: torch.Tensor, q: torch.Tensor) -> torch.Tensor:
-    return kl_div(p, q, reduction='none')
+    return kl_div(p, q)
 ```
 
 ## `How about Triton?`
 
 ```python
+import triton
 import triton.language as tl
 
 @triton.jit
@@ -66,13 +67,11 @@ def kl_div_triton(
     q = tl.load(q_ptr + offsets, mask=mask)
     
     output = p * (tl.log(p) - tl.log(q))
-    
     tl.store(output_ptr + offsets, output, mask=mask)
 ```
 
 ## `How to integrate with PyTorch?`
 
-- Triton works with pointers
 - How to use our custom kernel with PyTorch autograd?
 
 ```python
@@ -163,6 +162,7 @@ model_path = "meta-llama/Meta-Llama-3-8B-Instruct"
 
 ## `Aaand some more benchmarks 🚀`
 
+- Saving memory is key to run bigger batch size on smaller GPUs
 
 :::: {.columns}
 
@@ -182,10 +182,23 @@ model_path = "meta-llama/Meta-Llama-3-8B-Instruct"
 
 ## `Last benchmark I promise...`
 
+- But is it faster? Yes, it is!
+
+![](media/TPS.png){fig-align="center" height=50% width=50%}
+
+:::: {.columns}
 
-![](media/TPS.png){height="50%" width="50%" }
+::: {.column width="60%"}
 
-::: {.incremental}
 *Attention is all you need, so I thank you for yours!* 🤗
+
 :::
 
+::: {.column width="40%"}
+
+![](media/qr.png){height=25% width=25% fig-align="center"}
+
+:::
+
+::::
+