Hugging Face's logo

pkshatech
/
RoSEtta-base-ja

Sentence Similarity
sentence-transformers
Safetensors
Japanese
retrieva-bert
feature-extraction
custom_code
Inference Endpoints
Model card Files Community
RoSEtta-base-ja / README.md
yano0's picture
yano0
Update README.md
eef3369 verified
preview code
|
raw
history blame
5.28 kB
metadata
language:
  - ja
library_name: sentence-transformers
tags:
  - sentence-transformers
  - sentence-similarity
  - feature-extraction
metrics: null
widget: []
pipeline_tag: sentence-similarity
license: apache-2.0
datasets:
  - hpprc/emb
  - hpprc/mqa-ja
  - google-research-datasets/paws-x

Model Details

This is a text embedding model based on RoFormer with a maximum input sequence length of 1024. The model is pre-trained with Wikipedia and cc100 and fine-tuned as a sentence embedding model. Fine-tuning begins with weakly supervised learning using mc4 and MQA. After that, we perform the same 3-stage learning process as GLuCoSE v2.

Model Description

  • Model Type: Sentence Transformer
  • Maximum Sequence Length: 1024 tokens
  • Output Dimensionality: 768 tokens
  • Similarity Function: Cosine Similarity

Model Sources

Full Model Architecture 

SentenceTransformer(
  (0): Transformer({'max_seq_length': 1024, 'do_lower_case': False}) with Transformer model: RetrievaBertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer
import torch.nn.functional as F

# Download from the 🤗 Hub
# The argument "trust_remote_code=True" is required to load the model
model = SentenceTransformer("pkshatech/RoSEtta-base-ja",trust_remote_code=True)

# Don't forget to add the prefix "query: " for query-side or "passage: " for passage-side texts.
sentences = [
    'query: PKSHAはどんな会社ですか？',
    'passage: 研究開発したアルゴリズムを、多くの企業のソフトウエア・オペレーションに導入しています。',
    'query: 日本で一番高い山は？',
    'passage: 富士山（ふじさん）は、標高3776.12 m、日本最高峰（剣ヶ峰）の独立峰で、その優美な風貌は日本国外でも日本の象徴として広く知られている。',
]
embeddings = model.encode(sentences,convert_to_tensor=True)
print(embeddings.shape)
# [4, 768]

# Get the similarity scores for the embeddings
similarities = F.cosine_similarity(embeddings.unsqueeze(0), embeddings.unsqueeze(1), dim=2)
print(similarities)
# tensor([[1.0000, 0.5910, 0.4332, 0.5421],
#         [0.5910, 1.0000, 0.4977, 0.6969],
#         [0.4332, 0.4977, 1.0000, 0.7475],
#         [0.5421, 0.6969, 0.7475, 1.0000]])

Benchmarks

Retieval

Evaluated with MIRACL-ja, JQARA and MLDR-ja.

model size MIRACL
Recall@5		 JQaRA
nDCG@10		 MLDR
nDCG@10
me5-base 0.3B 84.2 47.2 25.4
GLuCoSE 0.1B 53.3 30.8 25.2
RoSEtta 0.2B 79.3 57.7 32.3

JMTEB

Evaluated with JMTEB.

  • The time-consuming datasets ['amazon_review_classification', 'mrtydi', 'jaqket', 'esci'] were excluded, and the evaluation was conducted on the other 12 datasets.
  • The average is a macro-average per task.
model size Class. Ret. STS. Clus. Pair. Avg.
me5-base 0.3B 75.1 80.6 80.5 52.6 62.4 70.2
GLuCoSE 0.1B 82.6 69.8 78.2 51.5 66.2 69.7
RoSEtta 0.2B 79.0 84.3 81.4 53.2 61.7 71.9

Authors

Chihiro Yano, Mocho Go, Hideyuki Tachibana, Hiroto Takegawa, Yotaro Watanabe

License

This model is published under the Apache License, Version 2.0.