semvec/pypi-packages huggingface dataset:
smart-chunker 0.0.4
Smart-Chunker is a semantic chunker to prepare a long document for RAG
License: Apache License Version 2.0
Keywords: smart-chunker, rag, chunker, cross-encoder, encoder, reranker


Smart-Chunker
===============

This smart chunker is a semantic chunker to prepare a
long document for retrieval augmented generation (RAG).

Unlike a usual chunker, it does not split the text into
identical groups of N tokens. Instead, it uses a cross-encoder
to calculate the similarity function between neighboring
sentences and divides the text based on the most significant
boundaries of semantic transitions, i.e. minima in the
above-mentioned similarity function.

The BAAI/bge-reranker-v2-m3, or any other model that supports the
AutoModelForSequenceClassification interface, should be used
as a cross encoder.

The smart chunker supports Russian and English.