comsat-embed-ja-0.3b-preview

comsat-embed-ja-0.3b-preview is an encoder-based embedding model developed by Sionic AI, optimized for Japanese semantic retrieval tasks. Trained on over 1.5M Japanese examples, it encodes queries and documents into vectors so that the most relevant documents can be found by similarity. The model is designed to provide high-quality text representations for real-world information retrieval scenarios, including document search, question answering, knowledge base retrieval, and enterprise semantic search. At only 0.3B parameters, it delivers robust performance across Japanese search environments where accurate semantic matching is essential.

Highlights

• Japanese-specialized — trained on 1.5M+ Japanese examples; achieves state-of-the-art average NDCG@10 (0.7785) on the 11-task JMTEB(v2) retrieval benchmark among the compared models with ≤4B parameters.
• Compact & efficient — 0.3B (310M) parameters, well suited to cost-efficient, low-latency deployment.
• Long context — handles inputs up to 8,192 tokens.
• Asymmetric encoding — queries and documents are encoded with their respective prefixes (検索クエリ: / 検索文書: ).
• Embeddings — 768-dimensional, mean-pooled and L2-normalized, compared with cosine similarity.

Usage

First install the Sentence Transformers library

pip install -U sentence-transformers

Sentence Transformers Usage

⚠️ Encode queries with the query prompt and documents with the document prompt. (Both use their own prefix; skipping the prompt slightly degrades retrieval quality.)

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("sionic-ai/comsat-embed-ja-0.3b-preview")

queries   = ["日本の首都はどこですか？"]
documents = ["日本の首都は東京です。"]

# Prefixes ("検索クエリ: " / "検索文書: ") are applied automatically by prompt_name
q_emb = model.encode(queries,   prompt_name="query",    normalize_embeddings=True)
d_emb = model.encode(documents, prompt_name="document", normalize_embeddings=True)

# Option: sentence-transformers 5.x helper API (equivalent)
# q_emb = model.encode_query(queries)
# d_emb = model.encode_document(documents)

scores = q_emb @ d_emb.T   # cosine similarity
print(scores)

Transformers Usage

import torch
import torch.nn.functional as F
from torch import Tensor
from transformers import AutoTokenizer, AutoModel


def mean_pool(last_hidden_states: Tensor, attention_mask: Tensor) -> Tensor:
    mask = attention_mask.unsqueeze(-1).to(last_hidden_states.dtype)
    return (last_hidden_states * mask).sum(dim=1) / mask.sum(dim=1).clamp(min=1e-9)


# Prepend the prefixes manually when using plain Transformers
queries   = ["検索クエリ: 日本の首都はどこですか？"]
documents = ["検索文書: 日本の首都は東京です。"]
input_texts = queries + documents

tokenizer = AutoTokenizer.from_pretrained("sionic-ai/comsat-embed-ja-0.3b-preview")
model = AutoModel.from_pretrained("sionic-ai/comsat-embed-ja-0.3b-preview")

batch_dict = tokenizer(
    input_texts,
    padding=True,
    truncation=True,
    max_length=8192,
    return_tensors="pt",
)
with torch.no_grad():
    outputs = model(**batch_dict)

embeddings = mean_pool(outputs.last_hidden_state, batch_dict["attention_mask"])
embeddings = F.normalize(embeddings, p=2, dim=1)
scores = embeddings[:1] @ embeddings[1:].T   # cosine similarity
print(scores.tolist())

JMTEB Retrieval Benchmark

• NLPJournalTitleAbsRetrieval.V2: Japanese academic paper retrieval — retrieve the abstract from the paper title.
• NLPJournalTitleIntroRetrieval.V2: Japanese academic paper retrieval — retrieve the introduction from the title.
• NLPJournalAbsIntroRetrieval.V2: Japanese academic paper retrieval — retrieve the introduction from the abstract.
• NLPJournalAbsArticleRetrieval.V2: Japanese academic paper retrieval — retrieve the article body from the abstract.
• MintakaRetrieval: A multilingual open-domain QA retrieval dataset (Japanese subset).
• JaGovFaqsRetrieval: A Japanese government FAQ retrieval dataset.
• JaqketRetrieval: A Japanese open-domain quiz QA retrieval dataset.
• MultiLongDocRetrieval: A long-document retrieval dataset (Japanese subset).
• JaCWIRRetrieval: A Japanese casual web information retrieval dataset.
• MIRACLRetrieval: A Wikipedia-based retrieval dataset (Japanese subset).
• MrTidyRetrieval: A Wikipedia-based Japanese retrieval dataset.

Performance (JMTEB v2 Retrieval, NDCG@10)

Only models with ≤4B parameters are shown. All scores are NDCG@10; for multilingual tasks the Japanese subset is used (Mintaka/MultiLongDoc/MIRACL=ja, MrTidy=japanese).

Model	Avg	NLPJ-TitleAbs	NLPJ-TitleIntro	NLPJ-AbsIntro	NLPJ-AbsArticle	Mintaka	JaGovFaqs	Jaqket	MultiLongDoc	JaCWIR	MIRACL	MrTidy
comsat-embed-ja-0.3b-preview	0.7785	0.9807	0.9772	0.9945	0.9951	0.3686	0.7902	0.7617	0.4689	0.8721	0.7143	0.6402
Qwen/Qwen3-Embedding-4B	0.7779	0.9753	0.9589	0.9881	0.9959	0.5201	0.7179	0.6136	0.5659	0.8560	0.7244	0.6406
codefuse-ai/F2LLM-v2-4B	0.7705	0.9853	0.9803	0.9937	0.9966	0.4767	0.8064	0.6528	0.4701	0.8166	0.6527	0.6442
sbintuitions/sarashina-embedding-v2-1b	0.7659	0.9804	0.9782	0.9954	0.9858	0.4365	0.7561	0.7371	0.4529	0.8552	0.6552	0.5916
cl-nagoya/ruri-v3-130m	0.7641	0.9807	0.9643	0.9894	0.9959	0.3283	0.7729	0.7514	0.4565	0.8349	0.7157	0.6149
cl-nagoya/ruri-v3-310m	0.7630	0.9785	0.9653	0.9908	0.9959	0.3353	0.7726	0.7342	0.4393	0.8405	0.7233	0.6168
cl-nagoya/ruri-v3-70m	0.7473	0.9705	0.9620	0.9862	0.9896	0.2974	0.7461	0.7093	0.4392	0.8201	0.7050	0.5947
nvidia/llama-nemotron-embed-vl-1b-v2	0.7464	0.9765	0.9669	0.9898	0.9966	0.2949	0.7076	0.6495	0.4257	0.8605	0.7143	0.6277
codefuse-ai/F2LLM-v2-1.7B	0.7426	0.9790	0.9699	0.9932	0.9980	0.3584	0.7857	0.6012	0.4597	0.8181	0.6153	0.5897
cl-nagoya/ruri-v3-30m	0.7330	0.9748	0.9540	0.9910	0.9893	0.2836	0.7236	0.6530	0.4626	0.8193	0.6635	0.5481
cl-nagoya/ruri-large-v2	0.7260	0.9750	0.8184	0.9145	0.9083	0.3377	0.7744	0.7336	0.3933	0.8021	0.7136	0.6152
Qwen/Qwen3-VL-Embedding-2B	0.7253	0.9644	0.9454	0.9833	0.9946	0.3026	0.6915	0.5605	0.4638	0.8510	0.6402	0.5815
BAAI/bge-m3	0.7245	0.9592	0.9164	0.9710	0.9528	0.2145	0.7066	0.5122	0.5034	0.8509	0.7285	0.6545
google/embeddinggemma-300m	0.7230	0.9627	0.9231	0.9757	0.9866	0.2683	0.7209	0.6749	0.3852	0.8524	0.6542	0.5490
Snowflake/snowflake-arctic-embed-l-v2.0	0.7111	0.9727	0.9444	0.9873	0.9643	0.2344	0.7203	0.4328	0.4648	0.8549	0.6608	0.5856
sbintuitions/sarashina-embedding-v1-1b	0.7078	0.9688	0.9661	0.9916	0.9920	0.4025	0.7223	0.6412	0.3420	0.8254	0.5124	0.4219
codefuse-ai/F2LLM-v2-0.6B	0.7078	0.9671	0.9590	0.9921	0.9966	0.2734	0.7609	0.4896	0.4185	0.8018	0.5723	0.5548
cl-nagoya/ruri-base-v2	0.7043	0.9658	0.7821	0.8982	0.9045	0.2997	0.7532	0.6947	0.3675	0.8044	0.6840	0.5928

Avg is the mean over the 11 JMTEB(v2) retrieval tasks (higher is better). Reproduction: evaluated with the MTEB/JMTEB retrieval pipeline (NDCG@10, full corpus); the query prompt (検索クエリ: ) is applied to queries and the document prompt (検索文書: ) to documents.

License

• Model weights: cc-by-nc-4.0 (non-commercial use).