WeMM-Embedding-2B-Quantized (4-Pillar SVD Vocab + PAS-Guarded FP8 + INT4)
Empirical Milestone Results
Live empirical forward-pass measurements against unquantized base BF16 encent/WeMM-Embedding-2B:
| Metric | Previous Milestone (v11) | Current 4-Pillar Milestone (v14) | Status |
|---|---|---|---|
| Mean Cosine Fidelity (2048d) | 96.5900% | 96.7267% | +0.1367% IMPROVED |
| Cosine Degradation (2048d) | 3.4100% | 3.2733% | -0.1367% IMPROVED |
| Min Fidelity | 95.2405% | 95.2773% | IMPROVED |
| Max Fidelity | 98.1155% | 98.1290% | IMPROVED |
| Fidelity Std Dev | 0.9052% | 0.8134% | IMPROVED |
| Checkpoint Size on Disk | 1.437 GB | 1.440 GB (1,475 MB) | Within $\le 1.45$ GB envelope |
Empirical Matryoshka Representation Learning (MRL) Benchmark
| Dimension | Mean Text Fidelity | Degradation vs BF16 2048d | Storage per 1M Vectors |
|---|---|---|---|
| 2048 (Full) | 96.7267% | 3.2733% | 7.81 GB |
| 1024 | 96.9952% | 3.0048% | 3.91 GB |
| 512 | 97.2845% | 2.7155% | 1.95 GB |
| 256 | 97.6994% | 2.3006% | 0.98 GB |
| 128 | 97.9021% | 2.0979% | 0.49 GB |
| 64 | 98.2917% | 1.7083% | 0.24 GB |
4-Pillar Quantization Architecture
- Pillar 1: SVD Rank-32 Core + INT4 Group-64 Residual on 248k Vocabulary:
- Truncated randomized SVD factors the vocabulary embedding manifold into rank-32 FP8 core vectors ({248k \times 32}$) + FP16 projection matrix ({32 \times 2048}$), quantizing only the small residual in symmetric Group-64 INT4.
- Pillar 2: Pre-Attention Spike (PAS) Boundary Protection:
- The GatedDeltaNet recurrent write projections (_proj$, _proj$) at boundary layers {2, 6, 10, 14, 18, 22} preceding full attention are preserved in FP8 E4M3 to prevent cascading memory state divergence.
- Pillar 3: Full Attention Softmax Preservation:
- All 6 full attention layers ({3, 7, 11, 15, 19, 23}) are preserved in FP8 E4M3, preventing quadratic attention distortion.
- Pillar 4: Group-64 Symmetric INT4 MLP Projections:
- SwiGLU feed-forward networks operate in INT4 with per-64-element dynamic scaling.
Quickstart
`python from sentence_transformers import SentenceTransformer
model = SentenceTransformer('ewin-reg/WeMM-Embedding-2B-Quantized', trust_remote_code=True) embeddings = model.encode(['Semantic document retrieval with post-training quantization.']) print('Embedding Shape:', embeddings.shape) `
- Downloads last month
- -
Model tree for ewin-reg/WeMM-Embedding-2B-Quantized
Evaluation results
- Model Size on Disk on Empirical Evaluation Suiteself-reported1.441
- Text Cosine Fidelity on Empirical Evaluation Suiteself-reported96.727
- Text Degradation on Empirical Evaluation Suiteself-reported3.273