init

1_Pooling/config.json

@@ -0,0 +1,7 @@
+{
+  "word_embedding_dimension": 768,
+  "pooling_mode_cls_token": false,
+  "pooling_mode_mean_tokens": true,
+  "pooling_mode_max_tokens": false,
+  "pooling_mode_mean_sqrt_len_tokens": false
+}

README.md

@@ -1,3 +1,106 @@
 ---
+pipeline_tag: sentence-similarity
+language: ja
 license: apache-2.0
+tags:
+  - transformers
+  - sentence-similarity
+  - feature-extraction
+  - sentence-transformers
+inference: false
+datasets:
+　- mc4
+  - clips/mqa
+  - shunk031/JGLUE
+  - paws-x
+  - hpprc/janli
+  - MoritzLaurer/multilingual-NLI-26lang-2mil7
+  - castorini/mr-tydi
+  - hpprc/jsick
 ---
+
+
+# GLuCoSE (General Luke-based COntrastive Sentence Embedding)-base-Japanese
+
+GLuCoSE (General LUke-based COntrastive Sentence Embedding, "glucose") is a Japanese text embedding model based on [LUKE](https://github.com/studio-ousia/luke). In order to create a general-purpose, user-friendly Japanese text embedding model, GLuCoSE has been trained on a mix of web data and various datasets associated with natural language inference and search. This model is not only suitable for sentence vector similarity tasks but also for semantic search tasks.
+- Maximum token count: 512
+- Output dimension: 768
+- Pooling: mean pooling
+- Supported language: Japanese
+
+## Usage
+You can use this model easily with [sentence-transformers](https://www.SBERT.net).  
+
+First, install sentence-transformers with pip as follows:  
+
+```
+pip install -U sentence-transformers
+```
+
+You can load the model and convert sentences into dense vectors as shown below:  
+
+```python
+from sentence_transformers import SentenceTransformer
+sentences = [
+    "PKSHA Technologyは機械学習/深層学習技術に関わるアルゴリズムソリューションを展開している。",
+    "この深層学習モデルはPKSHA Technologyによって学習され、公開された。",
+    "広目天は、仏教における四天王の一尊であり、サンスクリット語の「種々の眼をした者」を名前の由来とする。",
+]
+
+model = SentenceTransformer('pkshatech/GLuCoSE-base-ja')
+embeddings = model.encode(sentences)
+print(embeddings)
+```
+
+Since the loss function used during training is cosine similarity, we recommend using cosine similarity for downstream tasks.
+
+This text embedding model can also be used in LangChain. Please refer to [this page](https://python.langchain.com/docs/modules/data_connection/text_embedding/integrations/sentence_transformers) for more information.
+
+## Resources Used
+The following resources were used to train this model.
+### Pre-trained model
+- [studio-ousia/luke-japanese-base-lite](https://huggingface.co/studio-ousia/luke-japanese-base-lite)
+
+### Datasets
+- [mC4](https://huggingface.co/datasets/mc4)
+- [MQA](https://huggingface.co/datasets/clips/mqa)
+- [JNLI](https://github.com/yahoojapan/JGLUE)
+- [JSNLI](https://nlp.ist.i.kyoto-u.ac.jp/?%E6%97%A5%E6%9C%AC%E8%AA%9ESNLI%28JSNLI%29%E3%83%87%E3%83%BC%E3%82%BF%E3%82%BB%E3%83%83%E3%83%88)
+- [JaNLI](https://github.com/verypluming/JaNLI)
+- [PAWS-X](https://huggingface.co/datasets/paws-x)
+- [JSeM](https://github.com/DaisukeBekki/JSeM)
+- [MoritzLaurer/multilingual-NLI-26lang-2mil7](https://huggingface.co/datasets/MoritzLaurer/multilingual-NLI-26lang-2mil7)
+    - [MultiNLI](https://huggingface.co/datasets/multi_nli)
+    - [WANLI](https://huggingface.co/datasets/alisawuffles/WANLI)
+    - [FeverNLI](https://github.com/easonnie/combine-FEVER-NSMN/blob/master/other_resources/nli_fever.md)
+    - [LingNLI](https://arxiv.org/pdf/2104.07179.pdf)
+- [JSICK](https://github.com/verypluming/JSICK)
+- [Mr.Tidy](https://huggingface.co/datasets/castorini/mr-tydi)
+- [JSTS](https://github.com/yahoojapan/JGLUE) (used for validation) [^1]
+
+## Benchmarks
+### Semantic Similarity Calculation ([JSTS](https://github.com/yahoojapan/JGLUE) dev set)
+Evaluation by Spearman's correlation coefficient and Pearson's correlation coefficient.
+| Model | Spearman | Pearson |
+| --- | --- | --- |
+| [text-embedding-ada-002](https://platform.openai.com/docs/guides/embeddings) |0.837[^2] | 0.790[^2] |
+| [pkshatech/simcse-ja-bert-base-clcmlp](https://huggingface.co/pkshatech/simcse-ja-bert-base-clcmlp)[^3] | 0.850 | 0.801 |
+| pkshatech/GLuCoSE-base-ja | **0.865** | **0.821** |
+
+### Zero-shot Search ([AIO3](https://sites.google.com/view/project-aio/competition3?authuser=0) dev set)
+Evaluation by top-k retrieval accuracy[^4] (the fraction of questions that have a correct answer in the top-k retrieved documents at least once.)
+| Model | Top-1 | Top-5 | Top-10 | Top-50 |
+| --- | --- | --- | --- | --- |
+| [text-embedding-ada-002](https://platform.openai.com/docs/guides/embeddings) | 33.50 | **57.80** | 65.10 | 76.60 |
+| [pkshatech/simcse-ja-bert-base-clcmlp](https://huggingface.co/pkshatech/simcse-ja-bert-base-clcmlp)[^3] | 30.60 | 54.50 | 62.50 | 76.70 |
+| pkshatech/GLuCoSE-base-ja | **34.20** | 57.50 | **65.20** | **77.60** |
+
+## License
+This model is published under the [Apache License, Version 2.0](https://www.apache.org/licenses/LICENSE-2.0).
+
+[^1]: When we trained this model, the test data of JGLUE was not released, so we used the dev set of JGLUE as a private evaluation data. Therefore, we selected the checkpoint on the train set of JGLUE insted of its dev set.
+[^2]: https://qiita.com/akeyhero/items/ce371bfed64399027c23 
+[^3]: This is the model that PKSHA Technology published before this model.
+[^4]: For more details, please refer to https://arxiv.org/pdf/2004.04906.pdf.
+
+```

README_JA.md

@@ -0,0 +1,95 @@
+---
+pipeline_tag: sentence-similarity
+language: ja
+license: apache-2.0
+tags:
+  - transformers
+  - sentence-similarity
+  - feature-extraction
+  - sentence-transformers
+inference: false
+---
+
+# GLuCoSE (General Luke-based COntrastive Sentence Embedding)-base-Japanese
+
+GLuCoSE (General LUke-based COntrastive Sentence Embedding, "ぐるこーす")は[LUKE](https://github.com/studio-ousia/luke)をベースにした日本語のテキスト埋め込みモデルです。汎用的で気軽に使えるテキスト埋め込みモデルを目指して、Webデータと自然言語推論や検索などの複数のデータセットを組み合わせたデータで学習されています。文ベクトルの類似度タスクだけでなく意味検索タスクにもお使いいただけます。
+- 最大トークン数: 512
+- 出力次元数: 768
+- pooling: mean pooling
+- 対応言語: 日本語
+
+
+## 使い方
+[sentence-transformers](https://www.SBERT.net)を使って、このモデルを簡単に利用することができます。  
+
+下記のように、pipでsentence-transformersをインストールしてください。  
+
+```
+pip install -U sentence-transformers
+```
+
+下記のようにモデルをロードして、文を密なベクトルに変換することができます。  
+
+```python
+from sentence_transformers import SentenceTransformer
+sentences = [
+    "PKSHA Technologyは機械学習/深層学習技術に関わるアルゴリズムソリューションを展開している。",
+    "この深層学習モデルはPKSHA Technologyによって学習され、公開された。",
+    "広目天は、仏教における四天王の一尊であり、サンスクリット語の「種々の眼をした者」を名前の由来とする。",
+]
+
+model = SentenceTransformer('pkshatech/GLuCoSE-base-ja')
+embeddings = model.encode(sentences)
+print(embeddings)
+```
+
+学習時の損失関数にcosine類似度を使っているため、下流のタスクでcosine類似度を類似度計算に使うことをおすすめします。  
+
+LangChainでもこのテキスト埋め込みモデルを利用することができます。[こちらのページ](https://python.langchain.com/docs/modules/data_connection/text_embedding/integrations/sentence_transformers)を参考にしてください。
+
+## 使用したリソース
+このモデルの学習に下記の資源を使用しています。
+### 事前学習モデル
+- [studio-ousia/luke-japanese-base-lite](https://huggingface.co/studio-ousia/luke-japanese-base-lite)
+
+### データセット
+- [mC4](https://huggingface.co/datasets/mc4)
+- [MQA](https://huggingface.co/datasets/clips/mqa)
+- [JNLI](https://github.com/yahoojapan/JGLUE)
+- [JSNLI](https://nlp.ist.i.kyoto-u.ac.jp/?%E6%97%A5%E6%9C%AC%E8%AA%9ESNLI%28JSNLI%29%E3%83%87%E3%83%BC%E3%82%BF%E3%82%BB%E3%83%83%E3%83%88)
+- [JaNLI](https://github.com/verypluming/JaNLI)
+- [PAWS-X](https://huggingface.co/datasets/paws-x)
+- [JSeM](https://github.com/DaisukeBekki/JSeM)
+- [MoritzLaurer/multilingual-NLI-26lang-2mil7](https://huggingface.co/datasets/MoritzLaurer/multilingual-NLI-26lang-2mil7)
+    - [MultiNLI](https://huggingface.co/datasets/multi_nli)
+    - [WANLI](https://huggingface.co/datasets/alisawuffles/WANLI)
+    - [FeverNLI](https://github.com/easonnie/combine-FEVER-NSMN/blob/master/other_resources/nli_fever.md)
+    - [LingNLI](https://arxiv.org/pdf/2104.07179.pdf)
+- [JSICK](https://github.com/verypluming/JSICK)
+- [Mr.Tidy](https://huggingface.co/datasets/castorini/mr-tydi)
+- [JSTS](https://github.com/yahoojapan/JGLUE) (validasion用) [^1]
+
+## ベンチマーク
+### 意味的類似度計算 ([JSTS](https://github.com/yahoojapan/JGLUE) dev set)
+Spearmanの相関係数、Pearsonの相関係数で評価
+| モデル | Spearman |Pearson|
+| --- | --- | --- |
+| [text-embedding-ada-002](https://platform.openai.com/docs/guides/embeddings) | 0.837[^2] | 0.790[^2] |
+| [pkshatech/simcse-ja-bert-base-clcmlp](https://huggingface.co/pkshatech/simcse-ja-bert-base-clcmlp)[^3]　| 0.850 | 0.801 |
+| pkshatech/GLuCoSE-base-ja | **0.865**  | **0.821** | 
+
+### zero-shot 検索([第3回AI王](https://sites.google.com/view/project-aio/competition3?authuser=0) dev set)
+top-k　retrieval accuracy[^4] (検索された上位k個の文書に少なくとも1回は正解が含まれる問題の割合)で評価
+| モデル | Top-1 | Top-5| Top-10| Top-50|
+| --- | --- | --- |---|---|
+| [text-embedding-ada-002](https://platform.openai.com/docs/guides/embeddings) |33.50 | **57.80** | 65.10| 76.60|
+| [pkshatech/simcse-ja-bert-base-clcmlp](https://huggingface.co/pkshatech/simcse-ja-bert-base-clcmlp)[^3]　| 30.60 | 54.50 | 62.50 | 76.70 | 
+| pkshatech/GLuCoSE-base-ja | **34.20** | 57.50 | **65.20** | **77.60** |
+
+## ライセンス
+このモデルは [Apache License, Version 2.0](https://www.apache.org/licenses/LICENSE-2.0)の下で公開されています。
+
+[^1]: モデル学習時には、JGLUEのテストデータが公開されていなかったため、プライベートな��価データとしてJGLUEのdev setを使用していました。そのため、玉突き的にJGLUEのtrain setでcheckpointを選択しています。
+[^2]: https://qiita.com/akeyhero/items/ce371bfed64399027c23 
+[^3]: このモデルの前にPKSHA Technologyが公開したモデルです。  
+[^4]: 詳しくは、https://arxiv.org/pdf/2004.04906.pdf を参照してください。

added_tokens.json

@@ -0,0 +1,4 @@
+{
+  "<ent2>": 32771,
+  "<ent>": 32770
+}

config.json

@@ -0,0 +1,32 @@
+{
+  "_name_or_path": "models/luke-japanese/hf_xlm_roberta",
+  "architectures": [
+    "LukeModel"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "bert_model_name": "models/luke-japanese/hf_xlm_roberta",
+  "bos_token_id": 0,
+  "classifier_dropout": null,
+  "cls_entity_prediction": false,
+  "entity_emb_size": 256,
+  "entity_vocab_size": 4,
+  "eos_token_id": 2,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_norm_eps": 1e-05,
+  "max_position_embeddings": 514,
+  "model_type": "luke",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 1,
+  "position_embedding_type": "absolute",
+  "torch_dtype": "float32",
+  "transformers_version": "4.28.1",
+  "type_vocab_size": 1,
+  "use_cache": true,
+  "use_entity_aware_attention": true,
+  "vocab_size": 32772
+}

config_sentence_transformers.json

@@ -0,0 +1,7 @@
+{
+  "__version__": {
+    "sentence_transformers": "2.2.2",
+    "transformers": "4.28.1",
+    "pytorch": "2.0.0+cu117"
+  }
+}

entity_vocab.json

@@ -0,0 +1,6 @@
+{
+  "[MASK2]": 3,
+  "[MASK]": 0,
+  "[PAD]": 2,
+  "[UNK]": 1
+}

modules.json

@@ -0,0 +1,14 @@
+[
+  {
+    "idx": 0,
+    "name": "0",
+    "path": "",
+    "type": "sentence_transformers.models.Transformer"
+  },
+  {
+    "idx": 1,
+    "name": "1",
+    "path": "1_Pooling",
+    "type": "sentence_transformers.models.Pooling"
+  }
+]

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b3c698698882f31d4ce74494bcc357c7fe0525c4401907520dd529dffc5b61bc
+size 532361569

sentence_bert_config.json

@@ -0,0 +1,4 @@
+{
+  "max_seq_length": 512,
+  "do_lower_case": false
+}

sentencepiece.bpe.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d8b73a5e054936c920cf5b7d1ec21ce9c281977078269963beb821c6c86fbff7
+size 841889

special_tokens_map.json

@@ -0,0 +1,31 @@
+{
+  "additional_special_tokens": [
+    {
+      "content": "<ent>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    },
+    {
+      "content": "<ent2>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    }
+  ],
+  "bos_token": "<s>",
+  "cls_token": "<s>",
+  "eos_token": "</s>",
+  "mask_token": {
+    "content": "<mask>",
+    "lstrip": true,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "<pad>",
+  "sep_token": "</s>",
+  "unk_token": "<unk>"
+}

tokenizer_config.json

@@ -0,0 +1,94 @@
+{
+  "additional_special_tokens": [
+    {
+      "__type": "AddedToken",
+      "content": "<ent>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false
+    },
+    {
+      "__type": "AddedToken",
+      "content": "<ent2>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false
+    },
+    {
+      "__type": "AddedToken",
+      "content": "<ent>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false
+    },
+    {
+      "__type": "AddedToken",
+      "content": "<ent2>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false
+    },
+    {
+      "__type": "AddedToken",
+      "content": "<ent>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false
+    },
+    {
+      "__type": "AddedToken",
+      "content": "<ent2>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false
+    }
+  ],
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "<s>",
+  "entity_mask2_token": "[MASK2]",
+  "entity_mask_token": "[MASK]",
+  "entity_pad_token": "[PAD]",
+  "entity_token_1": {
+    "__type": "AddedToken",
+    "content": "<ent>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "entity_token_2": {
+    "__type": "AddedToken",
+    "content": "<ent2>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "entity_unk_token": "[UNK]",
+  "eos_token": "</s>",
+  "mask_token": {
+    "__type": "AddedToken",
+    "content": "<mask>",
+    "lstrip": true,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "max_entity_length": 32,
+  "max_mention_length": 30,
+  "model_max_length": 512,
+  "pad_token": "<pad>",
+  "sep_token": "</s>",
+  "sp_model_kwargs": {},
+  "task": null,
+  "tokenizer_class": "MLukeTokenizer",
+  "tokenizer_file": "models/luke-japanese/hf_luke_japanese_lite_epoch20/tokenizer.json",
+  "unk_token": "<unk>"
+}