tr-en-sci-e5-base

Turkish → English scientific literature retrieval. Type a query in Turkish, find the relevant English paper (title + abstract). A 278M-parameter embedding model fine-tuned from intfloat/multilingual-e5-base on ~380k Turkish–English abstract pairs written by the authors of articles on DergiPark.

Demo: huggingface.co/spaces/musapeker/tr-en-sci-search (runs entirely in the browser).

Türkçe özet: Türkçe sorguyla İngilizce bilimsel makale bulmak için eğitilmiş bir embedding modeli. DergiPark'taki yazarların kendi yazdığı Türkçe–İngilizce başlık, özet ve anahtar kelime çiftleriyle eğitildi. Eğitimde hiç görülmemiş dergilerden oluşan test kümesinde, kendisinin yaklaşık iki katı büyüklükteki bge-m3 ve turkish-e5-large modellerini geçiyor.

Results

Public test set of musapeker/dergipark-tr-en-abstracts: 16,983 English documents (title + abstract) from journals never seen in training, 2,000 Turkish queries per query type, exactly one relevant document per query (the same article).

Model	Title nDCG@10	Title R@1	Title R@10	Title nDCG@10 (no shared words)	Keywords nDCG@10	Keywords R@1	Keywords R@10
tr-en-sci-e5-base (this model, 278M)	0.979	0.960	0.995	0.971	0.927	0.868	0.981
bge-m3 (568M)	0.944	0.908	0.979	0.922	0.834	0.743	0.922
ytu-ce-cosmos/turkish-e5-large (560M)	0.942	0.907	0.974	0.927	0.890	0.826	0.953
multilingual-e5-large-instruct (560M)	0.889	0.840	0.934	0.850	0.789	0.687	0.886
Qwen3-Embedding-0.6B	0.819	0.749	0.888	0.752	0.775	0.680	0.864
multilingual-e5-base (base model, 278M)	0.760	0.671	0.849	0.672	0.668	0.559	0.783
MT (opus-mt-tr-en) + e5-base-v2	0.695	0.649	0.742	0.660	0.845	0.771	0.916
MT (opus-mt-tr-en) + BM25	0.645	0.589	0.703	0.578	0.762	0.674	0.848
BM25 (Turkish query on English docs)	0.292	0.221	0.370	0.095	0.462	0.306	0.641

• Title queries: the Turkish title of the article. Keywords queries: the Turkish author keywords, joined with ; .
• No shared words: the subset of title queries (1,150 of 2,000) that share no content word (≥3 letters, no digits) with their target document, i.e. names, years and international terms cannot help. This is the purely cross-lingual part of the task.
• R@k: share of queries whose relevant document is in the top k.

Usage

The model follows the E5 convention: prefix queries with query: and documents with passage: .

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("musapeker/tr-en-sci-e5-base")

queries = ["query: göz dibi görüntülerinde derin öğrenme ile diyabetik retinopati tespiti"]
docs = [
    "passage: Deep learning based detection of diabetic retinopathy from fundus images using convolutional neural networks.",
    "passage: The effect of climate change on wheat yield in Central Anatolia: a regression analysis.",
]
q_emb = model.encode(queries, normalize_embeddings=True)
d_emb = model.encode(docs, normalize_embeddings=True)
print(model.similarity(q_emb, d_emb))  # first document scores highest

For best results, index documents as passage: <English title>. <English abstract>.

In the browser (transformers.js)

An int8 ONNX export (onnx/model_quantized.onnx, 279 MB) is included. On 1,000 test queries against 1,000 documents, R@1 is 0.990 for the PyTorch model and 0.988 when int8 query embeddings are matched against PyTorch document embeddings.

import { pipeline } from "@huggingface/transformers";

const extractor = await pipeline("feature-extraction", "musapeker/tr-en-sci-e5-base", { dtype: "q8" });
const out = await extractor(["query: tahrir defterleri ve Osmanlı nüfus tarihi"], { pooling: "mean", normalize: true });

Training

Base model	intfloat/multilingual-e5-base (XLM-RoBERTa, 278M)
Training data	379,792 Turkish–English article pairs from 2,222 DergiPark journals (journal-disjoint from dev/test)
Training rows	837,539 (Turkish title → English doc, Turkish keywords → English doc, and Turkish abstract → English doc for 30% of articles)
Loss	CachedMultipleNegativesRankingLoss (GradCache), in-batch negatives
Batch size	256 (GradCache mini-batch 16), no-duplicates sampler so one article never appears twice in a batch
Learning rate / schedule	2e-5, 10% linear warmup, 1 epoch (3,271 steps)
Max sequence length	256 tokens
Precision / hardware	fp16, one RTX 4060 8 GB, 5 h 04 min
Seed	42

Development-set nDCG@10 (10,000 documents, Turkish title queries) rose from 0.812 (base model) to 0.987.

Data and splits

Pairs were harvested from DergiPark's public OAI-PMH interface (843,394 records, October 2026). Both language sides were checked with a language heuristic, mis-tagged sides were swapped, copied titles removed, and pairs filtered by length, length ratio and exact duplicates, leaving 447,244 pairs. Splits are by journal: train 2,222 journals / dev 146 / test 417, stratified by subject area, so test journals and their topics are never seen in training.

Limitations

• Known-item evaluation. Each query has exactly one relevant document: its own article. Real searches can have many relevant papers; scores here measure how well the model links a Turkish description to the right English paper, not full topical relevance.
• Query style. Training and test queries are titles, keyword lists and abstracts, not natural-language questions. Conversational queries may perform worse.
• Domain. Documents are abstracts of Turkish journals across all fields (education, health, engineering, social sciences, humanities). Performance on other English corpora (e.g. PubMed) has not been measured yet.
• Truncation. Trained with a 256-token limit; about 65% of English title+abstract documents are longer and were truncated during training.
• Baselines' training data. Some baselines (e.g. turkish-e5-large, whose training data is not documented) may have seen DergiPark text during their own training.

License and data provenance

The model is released under CC BY-NC 4.0 (non-commercial). It was trained on abstracts from all harvested DergiPark journals, including journals that publish under non-commercial licenses or state no license. The model weights do not redistribute these texts. The companion dataset contains only pairs from journals with a Creative Commons license that allows derivatives (CC BY, BY-SA, BY-NC, BY-NC-SA); every row has a license column.

Citation

A paper is in preparation. Until then, please cite this repository.