Instructions to use marin-community/marin-tokenizer with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use marin-community/marin-tokenizer with Transformers:
# Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("marin-community/marin-tokenizer", dtype="auto", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Xet hash:
- 5de30f9b382a76d80a684575cc52cf4acba17b51e87f9dd450ebb96888d1fce4
- Size of remote file:
- 17.2 MB
- SHA256:
- 881c9c36c359e1617afef6f7583403567931b7b4f43f6552d2b2155a131650a2
·
Xet efficiently stores Large Files inside Git, intelligently splitting files into unique chunks and accelerating uploads and downloads. More info.