TS-Corpus WordPiece Tokenizer (32k, Uncased)
Overview
This repository contains a WordPiece tokenizer with a vocabulary size of 32,000, trained uncased on various datasets from the TS Corpus website. It is designed to handle Turkish text, leveraging rich and diverse sources to provide a robust tool for natural language processing tasks.
Dataset Sources
The tokenizer was trained using multiple corpora from the TS Corpus, specifically:
- TS Corpus V2
- TS Wikipedia Corpus
- TS Abstract Corpus
- TS Idioms and Proverbs Corpus
- Syllable Corpus
- Turkish Constitution Corpus
These diverse sources include a wide range of texts from encyclopedic articles to legal documents, providing a comprehensive linguistic foundation for the tokenizer.
Tokenizer Model
The tokenizer uses the WordPiece model, which is widely utilized in many modern NLP systems. It is particularly effective in handling languages with rich morphology like Turkish due to its subword segmentation approach. This tokenizer does not differentiate between uppercase and lowercase letters, ensuring uniformity in tokenization regardless of text casing.
Usage
To use this tokenizer, you can load it via the Hugging Face transformers library as follows:
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("tahaenesaslanturk/ts-corpus-wordpiece-32k-uncased")