xhluca commited on
Commit
54a6f12
1 Parent(s): 2149496

Update BM25S model

Browse files
.gitattributes CHANGED
@@ -33,3 +33,6 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ corpus.jsonl filter=lfs diff=lfs merge=lfs -text
37
+ corpus.mmindex.json filter=lfs diff=lfs merge=lfs -text
38
+ vocab.index.json filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,91 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language: en
3
+ tags:
4
+ - bm25
5
+ - bm25s
6
+ - retrieval
7
+ - search
8
+ - lexical
9
+ ---
10
+
11
+ # BM25S Index
12
+
13
+ This is a BM25S index created with the [`bm25s` library](https://github.com/xhluca/bm25s) (version `0.0.1dev0`), an ultra-fast implementation of BM25. It can be used for lexical retrieval tasks.
14
+
15
+ [BM25S GitHub Repository](https://github.com/xhluca/bm25s)
16
+
17
+ ## Installation
18
+
19
+ You can install the `bm25s` library with `pip`:
20
+
21
+ ```bash
22
+ pip install "bm25s==0.0.1dev0"
23
+
24
+ # Include extra dependencies like stemmer
25
+ pip install "bm25s[full]==0.0.1dev0"
26
+
27
+ # For huggingface hub usage
28
+ pip install huggingface_hub
29
+ ```
30
+
31
+ ## Loading a `bm25s` index
32
+
33
+ You can use this index for information retrieval tasks. Here is an example:
34
+
35
+ ```python
36
+ import bm25s
37
+ from bm25s.hf import BM25HF
38
+
39
+ # Load the index
40
+ retriever = BM25HF.load_from_hub("xhluca/bm25s-nq-index", revision="main")
41
+
42
+ # You can retrieve now
43
+ query = "a cat is a feline"
44
+ results = retriever.retrieve(query, k=3)
45
+ ```
46
+
47
+ ## Saving a `bm25s` index
48
+
49
+ You can save a `bm25s` index to the Hugging Face Hub. Here is an example:
50
+
51
+ ```python
52
+ import bm25s
53
+ from bm25s.hf import BM25HF
54
+
55
+ # Create a BM25 index and add documents
56
+ retriever = BM25HF()
57
+ corpus = [
58
+ "a cat is a feline and likes to purr",
59
+ "a dog is the human's best friend and loves to play",
60
+ "a bird is a beautiful animal that can fly",
61
+ "a fish is a creature that lives in water and swims",
62
+ ]
63
+ corpus_tokens = bm25s.tokenize(corpus)
64
+ retriever.index(corpus_tokens)
65
+
66
+ token = None # You can get a token from the Hugging Face website
67
+ retriever.save_to_hub("xhluca/bm25s-nq-index", token=token)
68
+ ```
69
+
70
+
71
+ ## Stats
72
+
73
+ This dataset was created using the following data:
74
+
75
+ | Statistic | Value |
76
+ | --- | --- |
77
+ | Number of documents | 2681468 |
78
+ | Number of tokens | 116237970 |
79
+ | Average tokens per document | 43.3486321671562 |
80
+
81
+ ## Parameters
82
+
83
+ The index was created with the following parameters:
84
+
85
+ | Parameter | Value |
86
+ | --- | --- |
87
+ | k1 | `1.5` |
88
+ | b | `0.75` |
89
+ | delta | `0.5` |
90
+ | method | `lucene` |
91
+ | idf method | `lucene` |
corpus.jsonl ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:44b59b73c4d65a2ca25aed039c6e80f74f80af1e8345b394af11cf74e78b94a6
3
+ size 1444980459
corpus.mmindex.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:935ad3f79be3e0476835047c3788af0857acc7d794646285cf739134590b3db1
3
+ size 27471651
data.csc.index.npy ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:eb6e57b4f6cba920cbd90fb455adb9119689f397adeade8e4737929e11e41781
3
+ size 464952008
indices.csc.index.npy ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9bf64d392fca3d08b7a0655a88372ee0815bf82d0f10f9337c384aba9cb46199
3
+ size 464952008
indptr.csc.index.npy ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5179cf8c792ab866f48bfb934ff7b09c25833b6e4e3b5f7501e2dd3c5558e712
3
+ size 3572064
params.index.json ADDED
@@ -0,0 +1,11 @@
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "k1": 1.5,
3
+ "b": 0.75,
4
+ "delta": 0.5,
5
+ "method": "lucene",
6
+ "idf_method": "lucene",
7
+ "dtype": "float32",
8
+ "int_dtype": "int32",
9
+ "num_docs": 2681468,
10
+ "version": "0.0.1dev0"
11
+ }
vocab.index.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:138b5bc83f8c203e5cb316f9360caa425da73891f30f12d67d49d4c7e9cb1e9a
3
+ size 17082686