|
--- |
|
pipeline_tag: sentence-similarity |
|
language: ja |
|
license: cc-by-sa-4.0 |
|
tags: |
|
- transformers |
|
- sentence-similarity |
|
- feature-extraction |
|
- sentence-transformers |
|
inference: false |
|
widget: |
|
- source_sentence: "This widget can't work correctly now." |
|
sentences: |
|
- "Sorry :(" |
|
- "Try this model in your local environment!" |
|
example_title: "notification" |
|
--- |
|
|
|
# Japanese SimCSE (BERT-base) |
|
[日本語のREADME/Japanese README](https://huggingface.co/pkshatech/simcse-ja-bert-base-clcmlp/blob/main/README_JA.md) |
|
|
|
## summary |
|
model name: `pkshatech/simcse-ja-bert-base-clcmlp` |
|
|
|
|
|
This is a Japanese [SimCSE](https://arxiv.org/abs/2104.08821) model. You can easily extract sentence embedding representations from Japanese sentences. This model is based on [`cl-tohoku/bert-base-japanese-v2`](https://huggingface.co/cl-tohoku/bert-base-japanese-v2) and trained on [JSNLI](https://nlp.ist.i.kyoto-u.ac.jp/?%E6%97%A5%E6%9C%AC%E8%AA%9ESNLI%28JSNLI%29%E3%83%87%E3%83%BC%E3%82%BF%E3%82%BB%E3%83%83%E3%83%88) dataset, which is a Japanese natural language inference dataset. |
|
|
|
|
|
## Usage (Sentence-Transformers) |
|
You can use this model easily with [sentence-transformers](https://www.SBERT.net). |
|
|
|
You need [fugashi](https://github.com/polm/fugashi) and [unidic-lite](https://pypi.org/project/unidic-lite/) for tokenization. |
|
|
|
Please install sentence-transformers, fugashi, and unidic-lite with pip as follows: |
|
``` |
|
pip install -U fugashi[unidic-lite] sentence-transformers |
|
``` |
|
|
|
You can load the model and convert sentences to dense vectors as follows: |
|
|
|
```python |
|
from sentence_transformers import SentenceTransformer |
|
sentences = [ |
|
"PKSHA Technologyは機械学習/深層学習技術に関わるアルゴリズムソリューションを展開している。", |
|
"この深層学習モデルはPKSHA Technologyによって学習され、公開された。", |
|
"広目天は、仏教における四天王の一尊であり、サンスクリット語の「種々の眼をした者」を名前の由来とする。", |
|
] |
|
|
|
model = SentenceTransformer('pkshatech/simcse-ja-bert-base-clcmlp') |
|
embeddings = model.encode(sentences) |
|
print(embeddings) |
|
``` |
|
|
|
Since the loss function used during training is cosine similarity, we recommend using cosine similarity for downstream tasks. |
|
|
|
## Model Detail |
|
|
|
### Tokenization |
|
We use the same tokenizer as `tohoku/bert-base-japanese-v2`. Please see the [README of `tohoku/bert-base-japanese-v2`](https://huggingface.co/cl-tohoku/bert-base-japanese-v2) for details. |
|
|
|
### Training |
|
We set `tohoku/bert-base-japanese-v2` as the initial value and trained it on the train set of [JSNLI](https://nlp.ist.i.kyoto-u.ac.jp/?%E6%97%A5%E6%9C%AC%E8%AA%9ESNLI%28JSNLI%29%E3%83%87%E3%83%BC%E3%82%BF%E3%82%BB%E3%83%83%E3%83%88). We trained 20 epochs and published the checkpoint of the model with the highest Spearman's correlation coefficient on the validation set [^1] of the train set of [JSTS](https://github.com/yahoojapan/JGLUE) |
|
|
|
### Training Parameters |
|
|
|
| Parameter | Value | |
|
| --- | --- | |
|
|pooling_strategy | [CLS] -> single fully-connected layer | |
|
| max_seq_length | 128 | |
|
| with hard negative | true | |
|
| temperature of contrastive loss | 0.05 | |
|
| Batch size | 200 | |
|
| Learning rate | 1e-5 | |
|
| Weight decay | 0.01 | |
|
| Max gradient norm | 1.0 | |
|
| Warmup steps | 2012 | |
|
| Scheduler | WarmupLinear | |
|
| Epochs | 20 | |
|
| Evaluation steps | 250 | |
|
|
|
|
|
# Licenses |
|
This models are distributed under the terms of the Creative [Creative Commons Attribution-ShareAlike 4.0](https://creativecommons.org/licenses/by-sa/4.0/). |
|
|
|
|
|
[^1]: When we trained this model, the test data of JGLUE was not released, so we used the dev set of JGLUE as a private evaluation data. Therefore, we selected the checkpoint on the train set of JGLUE insted of its dev set. |
|
|
|
|
