shkbd/sheikh-bangla-coder-350m

Sheikh Bangla Coder 350M

A project to build a specialised 350 million parameter causal language model focused on Bangla text generation and coding assistance. The goal of this repository is to provide a full end‑to‑end pipeline for collecting data, training a custom tokenizer, preparing a dataset that mixes Bangla and English code, training a transformer model, evaluating it on both natural language and code generation tasks, and finally packaging and uploading the resulting artefacts to HuggingFace.

Because this environment does not have network access to download packages or scrape live websites, the scripts in this repository are written as templates and are not executed here. When run in a proper environment with Python dependencies installed (transformers, datasets, tokenizers, torch, sentencepiece, beautifulsoup4, etc.), the scripts will perform the following tasks:

1. Data collection: fetch Bangla text from Wikipedia, news sites, and literature; crawl code repositories for projects with Bangla comments; and construct synthetic code‑switching examples.
2. Data processing: clean and normalise Unicode text, deduplicate entries, and split the corpus into natural language, code, and mixed subsets.
3. Tokenizer training: train a Byte Pair Encoding (BPE) tokenizer with a 40 k vocabulary covering Bangla characters and common programming tokens.
4. Dataset creation: assemble the processed corpus into a datasets.Dataset for training the language model. Special tokens such as <code>/</code> and <bangla>/</bangla> mark code blocks and Bangla passages.
5. Model configuration: define a 350 M parameter transformer with 16 layers, 896 hidden size, 14 attention heads, context length 2048, and a vocabulary size of 40 k.
6. Training: implement a training loop using transformers.Trainer or a custom loop with fully sharded data parallel (FSDP) and mixed precision. The model is trained in phases: base language modelling, instruction tuning, and fine‑tuning for code‑switching.
7. Evaluation: compute Bangla BLEU scores, code execution accuracy on small programming tasks, and human evaluation metrics for code‑switching fluency.
8. Deployment: package the model weights and tokenizer as safetensor files, write a model card, and optionally build a Gradio demo for inference. Use the HuggingFace CLI to create a repository and upload the files.

The repository follows a clear directory structure:

sheikh-bangla-coder/
├── data/
│   ├── raw/         # unprocessed scraped text and code
│   ├── processed/   # cleaned and deduplicated corpus
│   └── tokenizer/   # tokenizer training files and vocabulary
├── training/
│   ├── scripts/     # Python scripts for training and preprocessing
│   ├── configs/     # model and training configuration files
│   └── checkpoints/ # model checkpoints
├── evaluation/      # evaluation scripts and metrics
└── deployment/
    └── model_card.md  # documentation and HuggingFace metadata

To get started, create a Python virtual environment, install the required dependencies, and adapt the scraping logic for your environment. Once the data has been collected and processed, run the tokenizer training script followed by the model training script. Finally, evaluate your model and use the deployment script to upload it to HuggingFace.