Embedding creation
#6
by
Cognitus-Stuti
- opened
How are you creating the embeddings here
The query embeddings are created on the fly here, and the corpus embeddings are created with this script for binary and this script for int8. For these two scripts, the embeddings are already created by calling the encode method of the Sentence Transformer model: https://huggingface.co/mixedbread-ai/mxbai-embed-large-v1#sentence-transformers
- Tom Aarsen
The above script doesn't show how the ivf binary indexes are created which is used when doing approximte search in the demo. Could you please share that also