Commit
•
aa5681f
1
Parent(s):
79e70c5
Add README.md
Browse files
README.md
ADDED
@@ -0,0 +1,49 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
---
|
2 |
+
license: apache-2.0
|
3 |
+
datasets:
|
4 |
+
- llm-book/aio-retriever
|
5 |
+
language:
|
6 |
+
- ja
|
7 |
+
library_name: transformers
|
8 |
+
pipeline_tag: feature-extraction
|
9 |
+
---
|
10 |
+
|
11 |
+
# bert-base-japanese-v3-unsup-simcse-jawiki
|
12 |
+
|
13 |
+
「[大規模言語モデル入門](https://www.amazon.co.jp/dp/4297136333)」の第8章で紹介している教師なしSimCSEのモデルです。
|
14 |
+
[cl-tohoku/bert-base-japanese-v3](https://huggingface.co/cl-tohoku/bert-base-japanese-v3) を [llm-book/jawiki-sentences](https://huggingface.co/datasets/llm-book/jawiki-sentences) でファインチューニングして構築されています。
|
15 |
+
|
16 |
+
|
17 |
+
## 関連リンク
|
18 |
+
|
19 |
+
* [GitHubリポジトリ](https://github.com/ghmagazine/llm-book)
|
20 |
+
* [Colabノートブック(訓練)](https://colab.research.google.com/github/ghmagazine/llm-book/blob/main/chapter8/8-3-simcse-training.ipynb)
|
21 |
+
* [Colabノートブック(推論)](https://colab.research.google.com/github/ghmagazine/llm-book/blob/main/chapter8/8-4-simcse-faiss.ipynb)
|
22 |
+
* [データセット](https://huggingface.co/datasets/llm-book/jawiki-sentences)
|
23 |
+
* [大規模言語モデル入門(Amazon.co.jp)](https://www.amazon.co.jp/dp/4297136333/)
|
24 |
+
* [大規模言語モデル入門(gihyo.jp)](https://gihyo.jp/book/2023/978-4-297-13633-8)
|
25 |
+
|
26 |
+
|
27 |
+
## 使い方
|
28 |
+
|
29 |
+
```py
|
30 |
+
from torch.nn.functional import cosine_similarity
|
31 |
+
from transformers import pipeline
|
32 |
+
|
33 |
+
sim_enc_pipeline = pipeline(model="llm-book/bert-base-japanese-v3-unsup-simcse-jawiki", task="feature-extraction")
|
34 |
+
|
35 |
+
text = "川べりでサーフボードを持った人たちがいます"
|
36 |
+
sim_text = "サーファーたちが川べりに立っています"
|
37 |
+
|
38 |
+
# text と sim_text のベクトルを獲得
|
39 |
+
text_emb = sim_enc_pipeline(text, return_tensors=True)[0][0]
|
40 |
+
sim_emb = sim_enc_pipeline(sim_text, return_tensors=True)[0][0]
|
41 |
+
# text と sim_text の類似度を計算
|
42 |
+
sim_pair_score = cosine_similarity(text_emb, sim_emb, dim=0)
|
43 |
+
print(sim_pair_score.item()) # -> 0.8568589687347412
|
44 |
+
```
|
45 |
+
|
46 |
+
|
47 |
+
## ライセンス
|
48 |
+
|
49 |
+
[Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0)
|