Hindi-Punk: Punctuation Prediction Model

Hindi-Punk is a fine-tuned model based on BERT MuRIL (Multilingual Representations for Indian Languages), specifically designed for adding punctuation to Hindi text. Leveraging the powerful capabilities of Google's MuRIL, which excels in understanding and representing multiple Indian languages, Hindi-Punk offers precise punctuation prediction for Hindi, making it a highly effective tool for natural language processing applications involving Hindi text.

Getting Started

To use the Hindi-Punk model, you'll need to have Python installed on your system along with PyTorch and the Hugging Face Transformers library. If you don't have them installed, you can install them using pip:

pip install torch transformers

Using the Model

Step 1: Import Required Libraries

Start by importing the necessary libraries:

import torch
import torch.nn as nn
from transformers import AutoTokenizer
from huggingface_hub import hf_hub_download
from transformers import BertModel

Step 2: Download and Load the Model

The model is hosted on Hugging Face, and you can download it directly using the following code:

# Define the repository name and filename
repo_name = "zicsx/Hindi-Punk"
filename = "Hindi-Punk-model.pth"

# Download the file
model_path = hf_hub_download(repo_id=repo_name, filename=filename)

Load the model using PyTorch:

# Define the model classes
class CustomTokenClassifier(nn.Module):
    # ...

class PunctuationModel(nn.Module):
    # ...

# Initialize and load the model
model = PunctuationModel(
    bert_model_name='google/muril-base-cased',
    punct_num_classes=5,
    hidden_size=768
)
model.load_state_dict(torch.load(model_path))

Step 3: Tokenization

Use the tokenizer associated with the model:

tokenizer = AutoTokenizer.from_pretrained(
    pretrained_model_name_or_path="zicsx/Hindi-Punk", use_fast=True,
)

Step 4: Define Inference Functions

Create functions to perform inference and process the model's output:

def predict_punctuation_capitalization(model, text, tokenizer):
    # ...

def combine_predictions_with_text(text, tokenizer, punct_predictions, punct_index_to_label):
    # ...

Step 5: Run the Model

You can now run the model on your input text:

text = "Your Hindi text here"
punct_predictions = predict_punctuation_capitalization(model, text, tokenizer)
combined_text = combine_predictions_with_text(text, tokenizer, punct_predictions, punct_index_to_label)
print("Combined Text:", combined_text)

Example

Here's an example of how to use the model:

example_text = "सलामअलैकुम कहाँ जा रहे हैं जी आओ बैठो छोड़ देता हूँ हेलो एक्सक्यूज मी आपका क्या नाम है तुम लोगों को बाद में देख लेता हूँ"
punct_predictions = predict_punctuation_capitalization(model, example_text, tokenizer)
combined_text = combine_predictions_with_text(example_text, tokenizer, punct_predictions, punct_index_to_label)
print("Combined Text:", combined_text)

---