Update BM25S model

.gitattributes

@@ -33,3 +33,6 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+corpus.jsonl filter=lfs diff=lfs merge=lfs -text
+corpus.mmindex.json filter=lfs diff=lfs merge=lfs -text
+vocab.index.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,3 +1,157 @@
----
-license: mit
----
+---
+language: en
+library_name: bm25s
+tags:
+- bm25
+- bm25s
+- retrieval
+- search
+- lexical
+---
+
+# BM25S Index
+
+This is a BM25S index created with the [`bm25s` library](https://github.com/xhluca/bm25s) (version `0.2.5`), an ultra-fast implementation of BM25. It can be used for lexical retrieval tasks.
+
+BM25S Related Links:
+
+* 🏠[Homepage](https://bm25s.github.io)
+* 💻[GitHub Repository](https://github.com/xhluca/bm25s)
+* 🤗[Blog Post](https://huggingface.co/blog/xhluca/bm25s)
+* 📝[Technical Report](https://arxiv.org/abs/2407.03618)
+
+
+## Installation
+
+You can install the `bm25s` library with `pip`:
+
+```bash
+pip install "bm25s==0.2.5"
+
+# Include extra dependencies like stemmer
+pip install "bm25s[full]==0.2.5"
+
+# For huggingface hub usage
+pip install huggingface_hub
+```
+
+## Loading a `bm25s` index
+
+You can use this index for information retrieval tasks. Here is an example:
+
+```python
+import bm25s
+from bm25s.hf import BM25HF
+
+# Load the index
+retriever = BM25HF.load_from_hub("tien314/bm25s-version2")
+
+# You can retrieve now
+query = "a cat is a feline"
+results = retriever.retrieve(bm25s.tokenize(query), k=3)
+```
+
+## Saving a `bm25s` index
+
+You can save a `bm25s` index to the Hugging Face Hub. Here is an example:
+
+```python
+import bm25s
+from bm25s.hf import BM25HF
+
+corpus = [
+    "a cat is a feline and likes to purr",
+    "a dog is the human's best friend and loves to play",
+    "a bird is a beautiful animal that can fly",
+    "a fish is a creature that lives in water and swims",
+]
+
+retriever = BM25HF(corpus=corpus)
+retriever.index(bm25s.tokenize(corpus))
+
+token = None  # You can get a token from the Hugging Face website
+retriever.save_to_hub("tien314/bm25s-version2", token=token)
+```
+
+## Advanced usage
+
+You can leverage more advanced features of the BM25S library during `load_from_hub`:
+
+```python
+# Load corpus and index in memory-map (mmap=True) to reduce memory
+retriever = BM25HF.load_from_hub("tien314/bm25s-version2", load_corpus=True, mmap=True)
+
+# Load a different branch/revision
+retriever = BM25HF.load_from_hub("tien314/bm25s-version2", revision="main")
+
+# Change directory where the local files should be downloaded
+retriever = BM25HF.load_from_hub("tien314/bm25s-version2", local_dir="/path/to/dir")
+
+# Load private repositories with a token:
+retriever = BM25HF.load_from_hub("tien314/bm25s-version2", token=token)
+```
+
+## Tokenizer
+
+If you have saved a `Tokenizer` object with the index using the following approach:
+
+```python
+from bm25s.hf import TokenizerHF
+
+token = "your_hugging_face_token"
+tokenizer = TokenizerHF(corpus=corpus, stopwords="english")
+tokenizer.save_to_hub("tien314/bm25s-version2", token=token)
+
+# and stopwords too
+tokenizer.save_stopwords_to_hub("tien314/bm25s-version2", token=token)
+```
+
+Then, you can load the tokenizer using the following code:
+
+```python
+from bm25s.hf import TokenizerHF
+
+tokenizer = TokenizerHF(corpus=corpus, stopwords=[])
+tokenizer.load_vocab_from_hub("tien314/bm25s-version2", token=token)
+tokenizer.load_stopwords_from_hub("tien314/bm25s-version2", token=token)
+```
+
+
+## Stats
+
+This dataset was created using the following data:
+
+| Statistic | Value |
+| --- | --- |
+| Number of documents | 9826150 |
+| Number of tokens | 90771053 |
+| Average tokens per document | 9.24 |
+
+## Parameters
+
+The index was created with the following parameters:
+
+| Parameter | Value |
+| --- | --- |
+| k1 | `1.5` |
+| b | `0.75` |
+| delta | `0.5` |
+| method | `lucene` |
+| idf method | `lucene` |
+
+## Citation
+
+To cite `bm25s`, please use the following bibtex:
+
+```
+@misc{lu_2024_bm25s,
+      title={BM25S: Orders of magnitude faster lexical search via eager sparse scoring}, 
+      author={Xing Han Lù},
+      year={2024},
+      eprint={2407.03618},
+      archivePrefix={arXiv},
+      primaryClass={cs.IR},
+      url={https://arxiv.org/abs/2407.03618}, 
+}
+```
+

corpus.jsonl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cc67e52d86fb0491bc0bbe3779c98a4f7ed410969e18ef8f3e6ecb925be2d1f1
+size 904224061

corpus.mmindex.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:70a643508f5acf2f522626054bf56b1c2de09f1667402c9d566fff72ecbe0393
+size 97039423

data.csc.index.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3c9a5b996421a169d4f2c4699a8736411d2aa66d5214b8e0c50dc45c79e208e6
+size 363084340

indices.csc.index.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:77212125429072a15bb878f8f3a85e1711ef394da51aae760e0744e40e6a08b8
+size 363084340

indptr.csc.index.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0244fd6aed85ad7725b845e3c2d14193aa722822fc39091131dd82ba17af10cf
+size 9116092

params.index.json

@@ -0,0 +1,12 @@
+{
+    "k1": 1.5,
+    "b": 0.75,
+    "delta": 0.5,
+    "method": "lucene",
+    "idf_method": "lucene",
+    "dtype": "float32",
+    "int_dtype": "int32",
+    "num_docs": 9826150,
+    "version": "0.2.5",
+    "backend": "numpy"
+}

vocab.index.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:63b81555f7457d5a9310be43a7fce6818caaf2a0b261d0646aee1cdfaeb5e4b5
+size 45333270