README.md

---
datasets:
- mshojaei77/PersianTelegramChannels
language:
- fa
library_name: transformers
license: mit
pipeline_tag: text-generation
tags:
- 'Tokenizer '
- persian
- bpet
---
# PersianBPETokenizer Model Card

## Model Details

### Model Description
The `PersianBPETokenizer` is a custom tokenizer specifically designed for the Persian (Farsi) language. It leverages the Byte-Pair Encoding (BPE) algorithm to create a robust vocabulary that can effectively handle the unique characteristics of Persian text. This tokenizer is optimized for use with advanced language models like BERT and RoBERTa, making it a valuable tool for various Persian NLP tasks.

### Model Type
- **Tokenization Algorithm**: Byte-Pair Encoding (BPE)
- **Normalization**: NFD, StripAccents, Lowercase, Strip, Replace (ZWNJ)
- **Pre-tokenization**: Whitespace
- **Post-processing**: TemplateProcessing for special tokens

### Model Version
- **Version**: 1.0
- **Date**: September 6, 2024

### License
- **License**: MIT

### Developers
- **Developed by**: Mohammad Shojaei
- **Contact**: Shojaei.dev@gmail.com

### Citation
If you use this tokenizer in your research, please cite it as:
```
Mohammad Shojaei. (2024). PersianBPETokenizer [Software]. Available at https://huggingface.co/mshojaei77/PersianBPETokenizer.
```

## Model Use

### Intended Use
- **Primary Use**: Tokenization of Persian text for NLP tasks such as text classification, named entity recognition, machine translation, and more.
- **Secondary Use**: Integration with pre-trained language models like BERT and RoBERTa for fine-tuning on Persian datasets.

### Out-of-Scope Use
- **Non-Persian Text**: This tokenizer is not designed for languages other than Persian.
- **Non-NLP Tasks**: It is not intended for use in non-NLP tasks such as image processing or audio analysis.

## Data

### Training Data
- **Dataset**: `mshojaei77/PersianTelegramChannels`
- **Description**: A rich collection of Persian text extracted from various Telegram channels. This dataset provides a diverse range of language patterns and vocabulary, making it suitable for training a general-purpose Persian tokenizer.
- **Size**: 60,730 samples

### Data Preprocessing
- **Normalization**: Applied NFD Unicode normalization, removed accents, converted text to lowercase, stripped leading and trailing whitespace, and removed ZWNJ characters.
- **Pre-tokenization**: Used whitespace pre-tokenization.

## Performance

### Evaluation Metrics
- **Tokenization Accuracy**: The tokenizer has been tested on various Persian sentences and has shown high accuracy in tokenizing and encoding text.
- **Compatibility**: Fully compatible with Hugging Face Transformers, ensuring seamless integration with advanced language models.

### Known Limitations
- **Vocabulary Size**: The current vocabulary size is based on the training data. For very specialized domains, additional fine-tuning or training on domain-specific data may be required.
- **Out-of-Vocabulary Words**: Rare or domain-specific words may be tokenized as unknown tokens (`[UNK]`).

## Training Procedure

### Training Steps
1. **Environment Setup**: Installed necessary libraries (`datasets`, `tokenizers`, `transformers`).
2. **Data Preparation**: Loaded the `mshojaei77/PersianTelegramChannels` dataset and created a batch iterator for efficient training.
3. **Tokenizer Model**: Initialized the tokenizer with a BPE model and applied normalization and pre-tokenization steps.
4. **Training**: Trained the tokenizer on the Persian text corpus using the BPE algorithm.
5. **Post-processing**: Set up post-processing to handle special tokens.
6. **Saving**: Saved the tokenizer to disk for future use.
7. **Compatibility**: Converted the tokenizer to a `PreTrainedTokenizerFast` object for compatibility with Hugging Face Transformers.

### Hyperparameters
- **Special Tokens**: `[UNK]`, `[CLS]`, `[SEP]`, `[PAD]`, `[MASK]`
- **Batch Size**: 1000 samples per batch
- **Normalization Steps**: NFD, StripAccents, Lowercase, Strip, Replace (ZWNJ)

## How to Use

### Installation
To use the `PersianBPETokenizer`, first install the required libraries:
```bash
pip install -q --upgrade datasets tokenizers transformers
```

### Loading the Tokenizer
You can load the tokenizer using the Hugging Face Transformers library:
```python
from transformers import AutoTokenizer

persian_tokenizer = AutoTokenizer.from_pretrained("mshojaei77/PersianBPETokenizer")
```

### Tokenization Example
```python
test_sentence = "سلام، چطور هستید؟ امیدوارم روز خوبی داشته باشید"
tokens = persian_tokenizer.tokenize(test_sentence)
print("Tokens:", tokens)
encoded = persian_tokenizer(test_sentence)
print("Input IDs:", encoded["input_ids"])
print("Decoded:", persian_tokenizer.decode(encoded["input_ids"]))
```


## Acknowledgments
- **Dataset**: `mshojaei77/PersianTelegramChannels`
- **Libraries**: Hugging Face `datasets`, `tokenizers`, and `transformers`

## References
- [Hugging Face Tokenizers Documentation](https://huggingface.co/docs/tokenizers/index)
- [Hugging Face Transformers Documentation](https://huggingface.co/docs/transformers/index)