REDUP multilingual topic model

Multilingual topic model that maps a document to a 125-dimensional topic distribution. The repository contains inference weights and per-language BPE tokenizers for 100 languages.

The dump is built for BigARTM 0.9.2. It is not a Transformers / Diffusers checkpoint: AutoModel and AutoTokenizer are not supported.


Framework	BigARTM 0.9.2
Topics	125 (`topic_0` … `topic_124`)
Languages	100
Output	topic distribution (sums ≈ 1)
License	MIT

Related repositories

Role	Repository
Training / research pipeline	machine-intelligence-laboratory/text_categorization
Inference service (recommended)	redup-ai/redup.python.topicmodel

Use text_categorization as the historical training reference. For serving embeddings and explanations in production, prefer redup.python.topicmodel.

Repository contents

Path	Description
`artm/parameters.bin`	BigARTM model metadata and regularizers
`artm/p_wt.bin`	Topic–word matrix Φ used at inference
`tokenizers.json.gz`	BPE packs keyed by language: `{lang: {vocab, merges}}`
`config.json`	Machine-readable metadata
`LICENSE`	MIT

Total size is about 251 MB.

Languages

Each document must provide modality lang with an ISO-like code from the table below. Tokens are scored under BigARTM class id @{lang}.


`af`	`am`	`ar`	`av`	`az`
`ba`	`be`	`bg`	`bn`	`bs`
`ca`	`ce`	`cs`	`cv`	`cy`
`da`	`de`	`el`	`en`	`eo`
`es`	`et`	`eu`	`fa`	`fi`
`fr`	`gd`	`gl`	`gu`	`he`
`hi`	`hr`	`hu`	`hy`	`ia`
`id`	`inh`	`is`	`it`	`ja`
`jv`	`ka`	`kaa`	`kbd`	`kk`
`kl`	`km`	`ko`	`krc`	`ky`
`la`	`lez`	`lo`	`lt`	`lv`
`mg`	`mhr`	`mi`	`mk`	`ml`
`mn`	`mo`	`ms`	`my`	`myv`
`ne`	`nl`	`no`	`oc`	`os`
`pl`	`pt`	`rm`	`rn`	`ro`
`ru`	`sah`	`sh`	`si`	`sk`
`sl`	`sm`	`so`	`sq`	`sr`
`sv`	`sw`	`ta`	`tg`	`th`
`tk`	`tr`	`tt`	`udm`	`uk`
`ur`	`uz`	`vi`	`yi`	`zh`

How to download

pip install huggingface_hub

from huggingface_hub import snapshot_download

root = snapshot_download(repo_id="redup-ai/topicmodel-multilingual")

Usage

Recommended path: the inference helpers from redup.python.topicmodel.

pip install huggingface_hub bigartm==0.9.2
# install redup-topicmodel from the service repository / package index you use

import asyncio
from types import SimpleNamespace

from redup_topicmodel.topicmodel.worker import TopicModel


def document(document_id: str, tokens: list[str], lang: str):
    return SimpleNamespace(
        document_id=document_id,
        tokens=tokens,
        modalities={"lang": lang},
    )


async def main(root: str):
    model = TopicModel({
        "artifact_root": root,
    })
    pack = SimpleNamespace(documents=[
        document("doc-en", ["hello", "world"], "en"),
        document("doc-ru", ["привет", "мир"], "ru"),
    ])
    result = await model.get_documents_embedding("example", pack)
    for embedding in result["embeddings"]:
        # length-125 topic distribution
        print(len(embedding["values"]), sum(embedding["values"]))


asyncio.run(main(root))

gRPC service

Point the service config at the downloaded directory:

TopicModel:
  artifact_root: /path/from/snapshot_download

Tokenizer only

from redup_topicmodel.topicmodel.bpe import Tokenizers

tokenizers = Tokenizers.load(f"{root}/tokenizers.json.gz")
print(tokenizers["en"].encode("hello world"))

Limitations

Requires BigARTM and the redup inference stack; not compatible with transformers.AutoModel / AutoTokenizer.
parameters.bin is a BigARTM pickle — load only trusted artifacts.
tokenizers.json.gz stores independent BPE packs per language; it is not a Hugging Face tokenizer.json export.
Short or out-of-vocabulary inputs may produce degenerate topic mass (for example concentrated on topic_0).

License

MIT. See LICENSE.

Downloads last month: 48