This repository is created as part of the Flax/Jax community week by Huggingface. The aim of this project is to pretrain a language model using GPT-2 specifically for Tamil language.
To setup the project, run the following command,
pip install -r requirements.txt
Pretrained model on Tamil language using a causal language modeling (CLM) objective.
The GTP-2 model is trained on oscar dataset - ta
You can use the raw model for next sentence prediction, but it's mostly intended to be fine-tuned on a downstream task. See the model hub to look for fine-tuned versions on a task that interests you.
To perform training, do the following steps,
- Export the model directory (where you want to store the model artifacts like config, tokenizer, etc.)
- Create the config.json by running the following command,
- Create the tokenizer by running the following command,
- Once the config and tokenizer is created, run the following script to start training the flax model
To perform language generation using the model, pipeline can be used directly.
- First convert the flax model to pytorch using the following command,
- Use the following snippet to perform language generation,
from transformers import AutoTokenizer, AutoModelWithLMHead, pipeline model_name = 'abinayam/gpt-2-tamil' model = AutoModelWithLMHead.from_pretrained(model_name) tokenizer = AutoTokenizer.from_pretrained(model_name) set_seed(42) input_text = "ஒரு ஊரிலே ஒரு காக்கைக்கு" max_len = 300 no_seq = 5 generator = pipeline('text-generation', model=model, tokenizer=tokenizer) sequence = generator(input_text, max_length=max_len, num_return_sequences=no_seq)
- Downloads last month