Boosting Your RAG App Speed and Cost-Efficiency with LLM-Based Chunking Strategy
Published: 2023-12-11 17:03:12

##  RAG app : from zero to one 

You can now on ChatGPT upload your document and ask question about it.
That’s call a RAG (Retrieval Augmented Generation) app.

It’s currently one of the hottest use case of LLM

The framework Langchain make it pretty is to build. That’s why you’ve seen so much (chat to pdf like)

Basically a RAG app is :

- Splitting a document
- Embedding the chunks into a vector database
- Retrieve information by similarity search using a LLM to answer users input.

From the user perspective, **it give the impression to plug a document to your LLM**

Usually splitting is pretty straightforward. You just set a size in characters for chunks and that’s it.

(Now you can also set a size in tokens with Langchain, but nothing more)

 ## The problem with chunking your document 🥲

The problem is that **your vector database is full of random chunks that don't make sense by itself.**

When the document is small you can add a bit of context in your prompt

But with big documents **you may have either obvious or off-the-mark answers**

Langchain came up with some algorithm to fix that issue like « map re-rank »

![Map re-rank Algorithm from Langchain](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0pu0fl5m9nub3s7a2wc0.png)

The idea is to do many call to the LLM with variations on the input, to find the best chunks on the database.

Which create 2 problems :

1. **Time consuming** 
    
    As user you have to wait until getting a response (which sometimes is missing the point 😞)
    
2. It requires **making a lot of calls to your LLM**. 
    
    This becomes problematic when your LLM is called GPT-4, as the calls can quickly become quite expensive 💸.

## How to speed up and improve cost efficiency 💪

**A better chunking strategy is the key 🔑**

Instead of using an LLM to fixed the issue, why not doing it before the issue happened ?

(As they say, “an ounce of prevention is worth a pound of cure,” right ?)

Let's consider the PDF example:

1. Feed one page at a time to your LLM.
2. Ask it to summarize the useful points.
3. Receive the chunks back.

**Therefore, each chunk will focus on a single relevant idea from the document.**

This what my prompt for this look like :

```
"""You are an assistant tasked with dividing text from {document_source} into distinct, meaningful, and relevant sections.

- **Context** : The transcripts are from {document_context}. We will send you one page at a time from these PDF transcripts. Your task is to return the divided sections of that page
- **Objective** : Mentally summarize the page and then create sections by reformulating the text.

**Criteria for each section** : 
1. Each section will be stored in a vector database and retrieved via similarity search.
2. The section should be coherent and readable as a standalone unit.
3. It should focus on a single piece of information (1 section = 1 useful information).
4. Omit sections that do not provide relevant information.

**Avoid Repetition** : Ensure that you don't repeat or unnecessarily reformulate information. Each section should present unique and valuable content.

**Working Language** : {language}
**Response Format** : You should only respond in the following JSON format:
<json>
[
     {{
            "page_content": "First chunk of text from the page...",
            "metadata": {{
                "page_number": {page_number},
                "chunk_id": 1
            }}
        }},
        {{
            "page_content": "Second chunk of text...",
            "metadata": {{
                "page_number": {page_number},
                "chunk_id": 2
            }}
        }},
        ...
]
</json>
"""
```

For those unfamiliar with Python, `{document_source}` `{document_context}` `{language}` and `{page_number}` are variables. Then I can configure a bit more my LLM 

And that's how the previously mentioned problems are solved: by moving them at the upload ! 

1. It’s still **time consuming but only when you uploa**d
2. You still **make multiple of call to your LLM, but only once**

### When to use this strategy

From a user's perspective, waiting for the upload of their document can be really frustrating.

**However, if your RAG app relies on documents that are not uploaded by users but rather added before by you as a developer, then this is clearly the best strategy.**

### Want more ?

Thanks for reading 🫶

if you want work, or have any question with me feel free to contact me on [Linkedin](https://www.linkedin.com/in/cl%C3%A9ment-rbn-3a3587156/). 

My profile is in french but don’t worry I am fluent in english 😉