Hugging Face's logo

nlpai-lab
/
KoE5

Feature Extraction
Transformers
Safetensors
Korean
English
xlm-roberta
text-embeddings-inference
Inference Endpoints
Model card Files Community
3

sbert-compat

#3
by taeminlee - opened
base: refs/heads/main
←
from: refs/pr/3
Discussion Files changed
+11
-149
Files changed (5) hide show
  1. 1_Pooling/config.json +0 -10
  2. 2_Normalize/__init__ +0 -0
  3. README.md +11 -110
  4. config_sentence_transformers.json +0 -9
  5. modules.json +0 -20
1_Pooling/config.json DELETED
@@ -1,10 +0,0 @@
1
- {
2
- "word_embedding_dimension": 1024,
3
- "pooling_mode_cls_token": false,
4
- "pooling_mode_mean_tokens": true,
5
- "pooling_mode_max_tokens": false,
6
- "pooling_mode_mean_sqrt_len_tokens": false,
7
- "pooling_mode_weightedmean_tokens": false,
8
- "pooling_mode_lasttoken": false,
9
- "include_prompt": true
10
- }
 
 
 
 
 
 
 
 
 
 
 
2_Normalize/__init__ DELETED
File without changes
README.md CHANGED
@@ -14,18 +14,13 @@ pipeline_tag: feature-extraction
14
  # 🔎 KoE5
15
 
16
  Introducing KoE5, a model with advanced retrieval abilities.
17
- It has shown remarkable performance in Korean text retrieval.
 
18
 
19
- For details, visit the [KURE repository](https://github.com/nlpai-lab/KURE)
20
 
21
  ---
22
 
23
- ## Model Versions
24
- | Model Name | Dimension | Sequence Length | Introduction |
25
- |:----:|:---:|:---:|:---:|
26
- | [KURE-v1](https://huggingface.co/nlpai-lab/KURE-v1) | 1024 | 8192 | Fine-tuned [BAAI/bge-m3](https://huggingface.co/BAAI/bge-m3) with Korean data via [CachedGISTEmbedLoss](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#cachedgistembedloss)
27
- | [KoE5](https://huggingface.co/nlpai-lab/KoE5) | 1024 | 512 | Fine-tuned [intfloat/multilingual-e5-large](https://huggingface.co/intfloat/multilingual-e5-large) with [ko-triplet-v1.0](https://huggingface.co/datasets/nlpai-lab/ko-triplet-v1.0) via [CachedMultipleNegativesRankingLoss](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#cachedmultiplenegativesrankingloss) |
28
-
29
  ### Model Description
30
 
31
  This is the model card of a 🤗 transformers model that has been pushed on the Hub.
@@ -86,105 +81,17 @@ print(similarities)
86
 
87
  ## Evaluation
88
  ### Metrics
89
- - Recall, Precision, NDCG, F1
90
  ### Benchmark Datasets
91
- - [Ko-StrategyQA](https://huggingface.co/datasets/taeminlee/Ko-StrategyQA): 한국어 ODQA multi-hop 검색 데이터셋 (StrategyQA 번역)
92
- - [AutoRAGRetrieval](https://huggingface.co/datasets/yjoonjang/markers_bm): 금융, 공공, 의료, 법률, 커머스 5개 분야에 대해, pdf를 파싱하여 구성한 한국어 문서 검색 데이터셋
93
- - [MIRACLRetrieval]([url](https://huggingface.co/datasets/miracl/miracl)): Wikipedia 기반의 한국어 문서 검색 데이터셋
94
- - [PublicHealthQA]([url](https://huggingface.co/datasets/xhluca/publichealth-qa)): 의료 및 공중보건 도메인에 대한 한국어 문서 검색 데이터셋
95
- - [BelebeleRetrieval]([url](https://huggingface.co/datasets/facebook/belebele)): FLORES-200 기반의 한국어 문서 검색 데이터셋
96
- - [MrTidyRetrieval](https://huggingface.co/datasets/mteb/mrtidy): Wikipedia 기반의 한국어 문서 검색 데이터셋
97
- - [MultiLongDocRetrieval](https://huggingface.co/datasets/Shitao/MLDR): 다양한 도메인의 한국어 장문 검색 데이터셋
98
- - [XPQARetrieval](https://huggingface.co/datasets/jinaai/xpqa): 다양한 도메인의 한국어 문서 검색 데이터셋
99
 
100
  ## Results
101
-
102
- 아래는 모든 모델의, 모든 벤치마크 데이터셋에 대한 평균 결과입니다.
103
- 자세한 결과는 [KURE Github](https://github.com/nlpai-lab/KURE/tree/main/eval/results)에서 확인하실 수 있습니다.
104
- ### Top-k 1
105
- | Model | Average Recall_top1 | Average Precision_top1 | Average NDCG_top1 | Average F1_top1 |
106
- |-----------------------------------------|----------------------|------------------------|-------------------|-----------------|
107
- | **nlpai-lab/KURE-v1** | **0.52640** | **0.60551** | **0.60551** | **0.55784** |
108
- | dragonkue/BGE-m3-ko | 0.52361 | 0.60394 | 0.60394 | 0.55535 |
109
- | BAAI/bge-m3 | 0.51778 | 0.59846 | 0.59846 | 0.54998 |
110
- | Snowflake/snowflake-arctic-embed-l-v2.0 | 0.51246 | 0.59384 | 0.59384 | 0.54489 |
111
- | nlpai-lab/KoE5 | 0.50157 | 0.57790 | 0.57790 | 0.53178 |
112
- | intfloat/multilingual-e5-large | 0.50052 | 0.57727 | 0.57727 | 0.53122 |
113
- | jinaai/jina-embeddings-v3 | 0.48287 | 0.56068 | 0.56068 | 0.51361 |
114
- | BAAI/bge-multilingual-gemma2 | 0.47904 | 0.55472 | 0.55472 | 0.50916 |
115
- | intfloat/multilingual-e5-large-instruct | 0.47842 | 0.55435 | 0.55435 | 0.50826 |
116
- | intfloat/multilingual-e5-base | 0.46950 | 0.54490 | 0.54490 | 0.49947 |
117
- | intfloat/e5-mistral-7b-instruct | 0.46772 | 0.54394 | 0.54394 | 0.49781 |
118
- | Alibaba-NLP/gte-multilingual-base | 0.46469 | 0.53744 | 0.53744 | 0.49353 |
119
- | Alibaba-NLP/gte-Qwen2-7B-instruct | 0.46633 | 0.53625 | 0.53625 | 0.49429 |
120
- | openai/text-embedding-3-large | 0.44884 | 0.51688 | 0.51688 | 0.47572 |
121
- | Salesforce/SFR-Embedding-2_R | 0.43748 | 0.50815 | 0.50815 | 0.46504 |
122
- | upskyy/bge-m3-korean | 0.43125 | 0.50245 | 0.50245 | 0.45945 |
123
- | jhgan/ko-sroberta-multitask | 0.33788 | 0.38497 | 0.38497 | 0.35678 |
124
-
125
- ### Top-k 3
126
- | Model | Average Recall_top1 | Average Precision_top1 | Average NDCG_top1 | Average F1_top1 |
127
- |-----------------------------------------|----------------------|------------------------|-------------------|-----------------|
128
- | **nlpai-lab/KURE-v1** | **0.68678** | **0.28711** | **0.65538** | **0.39835** |
129
- | dragonkue/BGE-m3-ko | 0.67834 | 0.28385 | 0.64950 | 0.39378 |
130
- | BAAI/bge-m3 | 0.67526 | 0.28374 | 0.64556 | 0.39291 |
131
- | Snowflake/snowflake-arctic-embed-l-v2.0 | 0.67128 | 0.28193 | 0.64042 | 0.39072 |
132
- | intfloat/multilingual-e5-large | 0.65807 | 0.27777 | 0.62822 | 0.38423 |
133
- | nlpai-lab/KoE5 | 0.65174 | 0.27329 | 0.62369 | 0.37882 |
134
- | BAAI/bge-multilingual-gemma2 | 0.64415 | 0.27416 | 0.61105 | 0.37782 |
135
- | jinaai/jina-embeddings-v3 | 0.64116 | 0.27165 | 0.60954 | 0.37511 |
136
- | intfloat/multilingual-e5-large-instruct | 0.64353 | 0.27040 | 0.60790 | 0.37453 |
137
- | Alibaba-NLP/gte-multilingual-base | 0.63744 | 0.26404 | 0.59695 | 0.36764 |
138
- | Alibaba-NLP/gte-Qwen2-7B-instruct | 0.63163 | 0.25937 | 0.59237 | 0.36263 |
139
- | intfloat/multilingual-e5-base | 0.62099 | 0.26144 | 0.59179 | 0.36203 |
140
- | intfloat/e5-mistral-7b-instruct | 0.62087 | 0.26144 | 0.58917 | 0.36188 |
141
- | openai/text-embedding-3-large | 0.61035 | 0.25356 | 0.57329 | 0.35270 |
142
- | Salesforce/SFR-Embedding-2_R | 0.60001 | 0.25253 | 0.56346 | 0.34952 |
143
- | upskyy/bge-m3-korean | 0.59215 | 0.25076 | 0.55722 | 0.34623 |
144
- | jhgan/ko-sroberta-multitask | 0.46930 | 0.18994 | 0.43293 | 0.26696 |
145
-
146
- ### Top-k 5
147
- | Model | Average Recall_top1 | Average Precision_top1 | Average NDCG_top1 | Average F1_top1 |
148
- |-----------------------------------------|----------------------|------------------------|-------------------|-----------------|
149
- | **nlpai-lab/KURE-v1** | **0.73851** | **0.19130** | **0.67479** | **0.29903** |
150
- | dragonkue/BGE-m3-ko | 0.72517 | 0.18799 | 0.66692 | 0.29401 |
151
- | BAAI/bge-m3 | 0.72954 | 0.18975 | 0.66615 | 0.29632 |
152
- | Snowflake/snowflake-arctic-embed-l-v2.0 | 0.72962 | 0.18875 | 0.66236 | 0.29542 |
153
- | nlpai-lab/KoE5 | 0.70820 | 0.18287 | 0.64499 | 0.28628 |
154
- | intfloat/multilingual-e5-large | 0.70124 | 0.18316 | 0.64402 | 0.28588 |
155
- | BAAI/bge-multilingual-gemma2 | 0.70258 | 0.18556 | 0.63338 | 0.28851 |
156
- | jinaai/jina-embeddings-v3 | 0.69933 | 0.18256 | 0.63133 | 0.28505 |
157
- | intfloat/multilingual-e5-large-instruct | 0.69018 | 0.17838 | 0.62486 | 0.27933 |
158
- | Alibaba-NLP/gte-multilingual-base | 0.69365 | 0.17789 | 0.61896 | 0.27879 |
159
- | intfloat/multilingual-e5-base | 0.67250 | 0.17406 | 0.61119 | 0.27247 |
160
- | Alibaba-NLP/gte-Qwen2-7B-instruct | 0.67447 | 0.17114 | 0.60952 | 0.26943 |
161
- | intfloat/e5-mistral-7b-instruct | 0.67449 | 0.17484 | 0.60935 | 0.27349 |
162
- | openai/text-embedding-3-large | 0.66365 | 0.17004 | 0.59389 | 0.26677 |
163
- | Salesforce/SFR-Embedding-2_R | 0.65622 | 0.17018 | 0.58494 | 0.26612 |
164
- | upskyy/bge-m3-korean | 0.65477 | 0.17015 | 0.58073 | 0.26589 |
165
- | jhgan/ko-sroberta-multitask | 0.53136 | 0.13264 | 0.45879 | 0.20976 |
166
-
167
- ### Top-k 10
168
- | Model | Average Recall_top1 | Average Precision_top1 | Average NDCG_top1 | Average F1_top1 |
169
- |-----------------------------------------|----------------------|------------------------|-------------------|-----------------|
170
- | **nlpai-lab/KURE-v1** | **0.79682** | **0.10624** | **0.69473** | **0.18524** |
171
- | dragonkue/BGE-m3-ko | 0.78450 | 0.10492 | 0.68748 | 0.18288 |
172
- | BAAI/bge-m3 | 0.79195 | 0.10592 | 0.68723 | 0.18456 |
173
- | Snowflake/snowflake-arctic-embed-l-v2.0 | 0.78669 | 0.10462 | 0.68189 | 0.18260 |
174
- | intfloat/multilingual-e5-large | 0.75902 | 0.10147 | 0.66370 | 0.17693 |
175
- | nlpai-lab/KoE5 | 0.75296 | 0.09937 | 0.66012 | 0.17369 |
176
- | BAAI/bge-multilingual-gemma2 | 0.76153 | 0.10364 | 0.65330 | 0.18003 |
177
- | jinaai/jina-embeddings-v3 | 0.76277 | 0.10240 | 0.65290 | 0.17843 |
178
- | intfloat/multilingual-e5-large-instruct | 0.74851 | 0.09888 | 0.64451 | 0.17283 |
179
- | Alibaba-NLP/gte-multilingual-base | 0.75631 | 0.09938 | 0.64025 | 0.17363 |
180
- | Alibaba-NLP/gte-Qwen2-7B-instruct | 0.74092 | 0.09607 | 0.63258 | 0.16847 |
181
- | intfloat/multilingual-e5-base | 0.73512 | 0.09717 | 0.63216 | 0.16977 |
182
- | intfloat/e5-mistral-7b-instruct | 0.73795 | 0.09777 | 0.63076 | 0.17078 |
183
- | openai/text-embedding-3-large | 0.72946 | 0.09571 | 0.61670 | 0.16739 |
184
- | Salesforce/SFR-Embedding-2_R | 0.71662 | 0.09546 | 0.60589 | 0.16651 |
185
- | upskyy/bge-m3-korean | 0.71895 | 0.09583 | 0.60258 | 0.16712 |
186
- | jhgan/ko-sroberta-multitask | 0.61225 | 0.07826 | 0.48687 | 0.13757 |
187
- <br/>
188
 
189
  ## FAQ
190
 
@@ -203,12 +110,6 @@ Here are some rules of thumb:
203
 
204
  If you find our paper or models helpful, please consider cite as follows:
205
  ```text
206
- @misc{KURE,
207
- publisher = {Youngjoon Jang, Junyoung Son, Taemin Lee},
208
- year = {2024},
209
- url = {https://github.com/nlpai-lab/KURE}
210
- },
211
-
212
  @misc{KoE5,
213
  author = {NLP & AI Lab and Human-Inspired AI research},
214
  title = {KoE5: A New Dataset and Model for Improving Korean Embedding Performance},
 
14
  # 🔎 KoE5
15
 
16
  Introducing KoE5, a model with advanced retrieval abilities.
17
+ It has shown remarkable performance in Korean text retrieval, speficially overwhelming most multilingual embedding models.
18
+ To our knowledge, It is one of the best publicly opened Korean retrieval models.
19
 
20
+ For details, visit the [KoE5 repository](https://github.com/nlpai-lab/KoE5)
21
 
22
  ---
23
 
 
 
 
 
 
 
24
  ### Model Description
25
 
26
  This is the model card of a 🤗 transformers model that has been pushed on the Hub.
 
81
 
82
  ## Evaluation
83
  ### Metrics
84
+ - NDCG@1, F1@1, NDCG@3, F1@3
85
  ### Benchmark Datasets
86
+ - Ko-strategyQA
87
+ - AutoRAG-benchmark
88
+ - PublicHealthQA
 
 
 
 
 
89
 
90
  ## Results
91
+ - By datasets
92
+ <img src="KoE5-results-by-datasets.png" width=100%>
93
+ - Average
94
+ <img src="KoE5-results-avg.png" width=100%>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
95
 
96
  ## FAQ
97
 
 
110
 
111
  If you find our paper or models helpful, please consider cite as follows:
112
  ```text
 
 
 
 
 
 
113
  @misc{KoE5,
114
  author = {NLP & AI Lab and Human-Inspired AI research},
115
  title = {KoE5: A New Dataset and Model for Improving Korean Embedding Performance},
config_sentence_transformers.json DELETED
@@ -1,9 +0,0 @@
1
- {
2
- "__version__": {
3
- "sentence_transformers": "2.6.1",
4
- "transformers": "4.39.3",
5
- "pytorch": "2.2.2"
6
- },
7
- "prompts": {},
8
- "default_prompt_name": null
9
- }
 
 
 
 
 
 
 
 
 
 
modules.json DELETED
@@ -1,20 +0,0 @@
1
- [
2
- {
3
- "idx": 0,
4
- "name": "0",
5
- "path": "",
6
- "type": "sentence_transformers.models.Transformer"
7
- },
8
- {
9
- "idx": 1,
10
- "name": "1",
11
- "path": "1_Pooling",
12
- "type": "sentence_transformers.models.Pooling"
13
- },
14
- {
15
- "idx": 2,
16
- "name": "2",
17
- "path": "2_Normalize",
18
- "type": "sentence_transformers.models.Normalize"
19
- }
20
- ]