semvec/pypi-packages huggingface dataset:
word2vec-matryoshka 0.1.5
Word2Vec with Matryoshka Representation Learning (Rust+PyO3), gensim-style API, numpy interop.
License: MIT
Requires Python: >=3.10

# word2vec‑matryoshka

[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/feisan/word2vec_matryoshka)

Lightweight Word2Vec for Python, backed by Rust (PyO3). It implements a small, practical subset of Word2Vec (Skip‑gram/CBOW with Negative Sampling or Hierarchical Softmax) and adds Matryoshka multi‑level representations: you can train on multiple prefix dimensions and query a prefix level without recomputing vectors.

This project is a learning/practice (“vibe coding”) effort inspired by gensim. It intentionally covers only a tiny part of gensim’s functionality and adds Matryoshka prefix representations. If you need a full‑featured Word2Vec, please try gensim first.

## Install

```bash
pip install word2vec-matryoshka
```

## Quick start

```python
from word2vec_matryoshka import Word2Vec, KeyedVectors, set_seed

set_seed(42)
corpus = [["hello", "world"], ["computer", "science"]]

model = Word2Vec(
    sentences=corpus,
    vector_size=64,
    window=5,
    min_count=1,
    workers=2,
    negative=5,
    sg=1,    # 1=Skip-gram, 0=CBOW
    hs=0,    # 1=Hierarchical Softmax, 0=Negative Sampling
    levels=[16, 32, 64],  # Matryoshka prefix dimensions
)

# Full vector
vec = model.wv["hello"]

# Top-5 similar at a prefix level (e.g., 32 dims)
sims = model.wv.most_similar("hello", topn=5, level=32)

# Save/load vectors (base path creates .vocab.json + .npy)
model.wv.save("vectors")
wv = KeyedVectors.load("vectors", mmap="r")  # zero-copy memmap
vec16 = wv.get_vector("hello", level=16)

# Inspect vocabulary and the vector matrix
print(wv.index_to_key[:5])    # e.g., ['hello', 'computer', ...]
M = wv.vectors                # numpy.ndarray with shape (n_words, vector_size)
print(M.shape)
```

## What’s inside

- Word2Vec training: Skip‑gram or CBOW with Negative Sampling or Hierarchical Softmax
- Matryoshka levels: train multiple prefix dimensions and query by `level`
- Vector I/O: `KeyedVectors.save(base)` → `base.vocab.json` + `base.npy`; `KeyedVectors.load(base, mmap='r')` for fast, memory‑mapped reads
- Determinism: `set_seed(seed)` for reproducible runs

## Scope and disclaimer

- Not a drop‑in replacement for gensim. Only a minimal API is provided (`Word2Vec`, `KeyedVectors`, `wv[...]`, `most_similar`, `get_vector`).
- Built as a small practice project to explore PyO3/Rust performance and Matryoshka prefix training.
- For most production needs, start with gensim first, then evaluate if this project’s Matryoshka feature helps your use case.

## License

MIT — see `LICENSE`.
zetasketch-py 0.1.3
Python bindings for zetasketch-rs - unofficial Rust reimplementation of the ZetaSketch Java library for HyperLogLog++ implementation used by Google BigQuery and BigTable.
License: MIT
Requires Python: >=3.8
Keywords: zetasketch, hyperloglog, google, hll, sketch

# ZetaSketch RS

A native Rust re-implementation of the
[Zetasketch](https://github.com/google/zetasketch) Java library by Google.

The ZetaSketch library provides an implementation of the HyperLogLog++
algorithm used by several Google Cloud products, most notably BigQuery and
BigTable.

This library allows deserializing, modifying and serializing the HyperLogLog++
sketches used by these products.

You can learn more details about ZetaSketch and its HyperLogLog++
implementation in the
[README in the original
library](https://github.com/google/zetasketch/blob/master/README.md).

This reimplementation is based on version 0.1.0 of the original library, which
is the only version published.

## Compatibility

This library was implemented as a translation of the original Java code into
Rust, then refined here and there to improve performance and to be more
idiomatic. It strives to be 100% compatible with the Java library, which means
that for identical inputs, it should produce 100% identical sketches as the
Java library. Any deviation should be considered a bug.

We are using the [`j4rs`](https://github.com/astonbitecode/j4rs) crate for testing,
which allows to call us the original Java library from our Rust tests and compare
the behavior of both libraries.

## Python Bindings

The project also contains Python bindings for the Rust crate published on Pypi as
[`zetasketch-py`](https://pypi.org/project/zetasketch-py/).

The bindings are generated using [`maturin`](https://github.com/PyO3/maturin).

## Sponsorship

Porting the code over from Java to Rust was a fair bit of work that took me a fair
bit of time. If you find this library useful, please consider donating to help me
work on other similar projects in the future. Thank you!

## License

This project is licensed under the MIT license. See the [LICENSE](LICENSE) file
for details.

The original library is licensed under the Apache License 2.0.

> [!NOTE]
> This project is in no way affiliated with or endorsed by Google.