syssec-utd/pylingual-shared-tokenizer-v1

Shared (all-version) PyLingual tokenizers for the universal backbone + per-version LoRA study. Each task has one frozen tokenizer; every known opcode is a single atomic token and 64 contiguous <future_opcode_NN> slots are reserved for instructions from future Python versions, bound per adapter.

• Fitted on dataset syssec-utd/pylingual-shared-tokenizer-corpus-v1 at revision 2dd36151733c372cc7409a77af7066b172ad6e76
• Total vocabulary budget 30000 (opcode markers and reserved slots included); 233 known opcodes
• Code: https://github.com/quinntyx/pylingual branch feat-shared-backbone-lora, commit 1c20bee50f2109032045c2ddc75301a38290f480

Task	Learned base vocab	Total vocab	Reserved IDs
statement	11280	11577	11513–11576
segmentation	29703	30000	29936–29999

Load a task with AutoTokenizer.from_pretrained("syssec-utd/pylingual-shared-tokenizer-v1", subfolder="statement").