NGNN MTEB Local Publish Preview
This repository is a public research preview for review and discussion. It is not official MTEB leaderboard data.
It contains 9 rows across 3 tasks for these public result labels: Full embedding baseline, 256-dim baseline, Sparse NGNN.
Model Summary
This preview reports exploratory MTEB-style results for an NGNN-based embedding compression path. The experiments compare full-size text embeddings, a 256-dimensional baseline, and a sparse NGNN compressed representation across three benchmark tasks.
The base embedding source is OpenAI text-embedding-3-large. The NGNN path is
evaluated as an experimental compression method for reducing embedding
representation size while tracking task quality.
This is not a released standalone encoder and not an official MTEB leaderboard submission.
Public Comparison Rows
The preview intentionally exposes only three public comparison rows:
Full embedding baseline: the uncompressed embedding reference.256-dim baseline: a 256-dimensional PCA/SVD baseline.Sparse NGNN: the restricted public NGNN sparse compressed representation.
The preview intentionally omits additional non-public comparison rows and keeps only the restricted public comparison set.
Intended Use
This preview is intended for research review of embedding compression behavior. It can be used to inspect how a sparse NGNN compressed representation compares with full embeddings and a 256-dimensional baseline on the included tasks.
It should not be used as evidence of official MTEB leaderboard performance, and it should not be treated as a production-ready public encoder.
Evaluation
The included results cover three tasks:
- SciFact
- STSBenchmark
- Banking77Classification.v2
These rows are exploratory task-local evaluations and are not official MTEB
leaderboard submissions. No mteb.evaluate(...) run or ResultCache export was
used to create this preview. All rows set eligible_for_mteb_leaderboard=false.
Restricted Result Table
| model | task | type | split | metric | value | scope | leaderboard eligible |
|---|---|---|---|---|---|---|---|
| Full embedding baseline | Banking77Classification.v2 | Classification | test | accuracy | 0.8176202860858257 | hf_report | False |
| 256-dim baseline | Banking77Classification.v2 | Classification | test | accuracy | 0.8120936280884266 | exploratory | False |
| Sparse NGNN | Banking77Classification.v2 | Classification | test | accuracy | 0.8312743823146944 | exploratory | False |
| Full embedding baseline | STSBenchmark | STS | test | spearman | 0.8357254303666832 | hf_report | False |
| 256-dim baseline | STSBenchmark | STS | test | spearman | 0.8398060719526397 | exploratory | False |
| Sparse NGNN | STSBenchmark | STS | test | spearman | 0.8394133976928108 | exploratory | False |
| Full embedding baseline | SciFact | Retrieval | test | ndcg_at_10 | 0.9363324490406888 | hf_report | False |
| 256-dim baseline | SciFact | Retrieval | test | ndcg_at_10 | 0.9276906542101906 | exploratory | False |
| Sparse NGNN | SciFact | Retrieval | test | ndcg_at_10 | 0.9222986675189114 | exploratory | False |
The same restricted result table is also available as public_results_table.md,
with machine-readable copies in public_results.csv and public_results.json.
Limitations
These results are exploratory and task-local. The NGNN path is not yet a frozen public encoder, and the current preview does not establish leaderboard eligibility or broad generalization beyond the included tasks.