im-di4blo/NyayaGPT

Model Card — Indian Constitution & IPC Q&A (Fine-tuned Flan-T5-base)

This model is a fine-tuned version of google/flan-t5-base for legal Question-Answering tasks focused on the Indian Constitution and the Indian Penal Code (IPC). It was trained using Parameter-Efficient Fine-Tuning (PEFT) via LoRA adapters on the nisaar/Articles_Constitution_3300_Instruction_Set dataset.

Given an instruction or legal question as input, the model generates a relevant, context-aware legal answer grounded in the Indian Constitution and IPC.

---

Model Details

Model Description

google/flan-t5-base is a 250M-parameter instruction-tuned Sequence-to-Sequence (Encoder-Decoder) Transformer. This fine-tuned variant adapts the base model's general instruction-following capability to the specific domain of Indian constitutional and penal law by training on 3,311 curated instruction–response pairs sourced from the nisaar/Articles_Constitution_3300_Instruction_Set dataset.

LoRA adapters (rank 32, alpha 64) are applied to all attention and feed-forward projection layers (q, k, v, o, wi_0, wi_1, wo), enabling training of only ~5.19% of total parameters while preserving base model knowledge.

• Developed by: Blaze
• Model type: Sequence-to-Sequence (Encoder-Decoder) Text Generation with LoRA PEFT adapters
• Language(s) (NLP): English (en)
• License: Apache 2.0
• Finetuned from model: google/flan-t5-base

Model Sources

• Repository: https://huggingface.co/im-di4blo/NyayaGPT/tree/main
• Notebook: https://drive.google.com/file/d/1pHxIc_lCFkUxUzKVrlNcpklxweQgetSg/view?usp=sharing

---

Uses

Direct Use

This model can be used directly for legal question-answering tasks related to the Indian Constitution and IPC, without any further fine-tuning. It accepts a legal question or instruction as input and generates a relevant legal response.

Example usage:

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("[PLACEHOLDER — your HF repo ID]")
model = AutoModelForSeq2SeqLM.from_pretrained("[PLACEHOLDER — your HF repo ID]")

def ask(question):
    inputs = tokenizer(
        f"question: {question} answer:",
        return_tensors="pt",
        truncation=True,
        max_length=512
    )
    outputs = model.generate(
        **inputs,
        max_new_tokens=256,
        no_repeat_ngram_size=3,
        num_beams=4,
        early_stopping=True
    )
    return tokenizer.decode(outputs[0], skip_special_tokens=True)

print(ask("What is the significance of Article 32?"))
print(ask("What are the Fundamental Rights under the Indian Constitution?"))
print(ask("What does Article 21 say?"))

Downstream Use

This model can be integrated as a backend Q&A engine in:

• Legal information chatbots for Indian citizens
• Educational tools for law students studying the Indian Constitution and IPC
• Legal research assistants for practitioners needing quick statutory references
• Government/administrative portals providing automated legal guidance

Out-of-Scope Use

• Non-Indian legal systems: The model is trained exclusively on Indian constitutional and IPC law. It should not be used for legal advice pertaining to foreign jurisdictions.
• High-stakes legal decisions: This model is a research prototype and must not be used as a substitute for qualified legal counsel. All outputs should be reviewed by a licensed legal professional.
• Languages other than English: The training data is entirely in English; non-English queries will produce unreliable outputs.
• Real-time case law updates: The dataset is static and does not reflect legislative amendments or Supreme Court rulings after the dataset's collection date.

---

Bias, Risks, and Limitations

• Dataset Bias: The training dataset (nisaar/Articles_Constitution_3300_Instruction_Set) contains 3,311 samples curated from specific articles and cases. Topics not well-represented in this dataset may yield incomplete or hallucinated answers.
• Factual Hallucination: As with all generative language models, the model may generate plausible-sounding but factually incorrect legal statements. The BLEU score (6.91) and Exact Match (0.00%) indicate that outputs are fluent reformulations rather than verbatim legal text.
• Article Confusion: The model occasionally confuses article numbers (e.g., conflating Article 19 and Article 21), especially for closely related constitutional provisions.
• Limited Exact Recall: The model paraphrases legal provisions rather than quoting them verbatim, which may be insufficient in contexts requiring precise statutory text.
• Short Context Window: The maximum sequence length is capped at 512 tokens, which may truncate long case summaries or complex multi-clause queries.

Recommendations

• Always cross-verify model outputs against the official text of the Indian Constitution or the IPC before acting on them.
• Use beam search decoding (num_beams=4) with no_repeat_ngram_size=3 to improve response coherence and reduce repetition.
• For complex multi-part legal questions, break them into simpler sub-questions and aggregate the answers.
• Do not use outputs of this model in any formal legal document, petition, or proceeding without human expert review.

---

How to Get Started with the Model

Install dependencies:

pip install transformers peft torch sentencepiece

Run inference:

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

REPO_ID = "[PLACEHOLDER — your-username/indian-constitution-ipc-flan-t5]"

tokenizer = AutoTokenizer.from_pretrained(REPO_ID)
model = AutoModelForSeq2SeqLM.from_pretrained(REPO_ID)
model.eval()

questions = [
    "What is the significance of Article 32?",
    "Explain the right to equality under the constitution",
    "Who can declare national emergency?",
    "What are Fundamental Rights?",
    "What does Article 21 say?",
]

for q in questions:
    inputs = tokenizer(f"question: {q} answer:", return_tensors="pt", truncation=True, max_length=512)
    outputs = model.generate(**inputs, max_new_tokens=256, num_beams=4, no_repeat_ngram_size=3, early_stopping=True)
    print(f"Q: {q}")
    print(f"A: {tokenizer.decode(outputs[0], skip_special_tokens=True)}\n")

---

Training Details

Training Data

The model was fine-tuned on the nisaar/Articles_Constitution_3300_Instruction_Set dataset, which contains 3,311 instruction-response pairs covering:

• Articles of the Indian Constitution (Fundamental Rights, Directive Principles, Emergency Provisions, etc.)
• Indian Penal Code (IPC) clauses and offences
• Landmark Supreme Court case summaries with legal reasoning, precedents, and hypothetical legal advisories

Split used:

Split	Samples
Train	2,979
Validation	332

Training Procedure

Preprocessing

• Each training example's Instruction (possibly combined with a Context field) is prefixed as: "question: {Instruction} answer:"
• The Response field is tokenized separately as the decoder target label using text_target=.
• Both input and label sequences are truncated/padded to a maximum length of 512 tokens.
• Padding token IDs in label sequences are replaced with -100 so they are excluded from the cross-entropy loss computation.

Training Hyperparameters

Parameter	Value
Training regime	bf16 non-mixed precision (BFloat16)
Optimizer	adamw_torch_fused
Learning rate	2e-4
LR scheduler	Cosine annealing
Warmup steps	150
Weight decay	0.01
Gradient clipping	max_grad_norm = 1.0
Per-device batch size	16
Gradient accumulation	2 (effective batch size = 32)
Epochs	15
Evaluation strategy	Every 100 steps
Save strategy	Every 100 steps
Generation (eval)	predict_with_generate=True, greedy (num_beams=1)
Best model metric	eval_loss (lower is better)

LoRA Configuration

Parameter	Value
LoRA rank (r)	32
LoRA alpha	64
LoRA dropout	0.05
Target modules	q, k, v, o, wi_0, wi_1, wo
Bias	none
Task type	SEQ_2_SEQ_LM
Trainable params	13,565,952 / 261,143,808 (~5.19%)

Speeds, Sizes, Times

Detail	Value
GPU	NVIDIA A100-SXM4-40GB
Total VRAM	42.4 GB
VRAM allocated (pre-train)	~0.58 GB
Peak VRAM used	~34.05 GB
Total training steps	2,820 (over 30 epochs during development runs)
Training duration	~1 hour 22 minutes (on A100)
Final training loss	3.284952

---

Evaluation

Testing Data, Factors & Metrics

Testing Data

Evaluation was performed on the held-out validation split (332 samples, 10% of the full dataset), derived from the same nisaar/Articles_Constitution_3300_Instruction_Set dataset using a fixed seed=42 random split.

Factors

Evaluation covers the following legal instruction categories present in the validation split:

• Constitutional article explanation (e.g., Article 21, Article 32, Fundamental Rights)
• Legal case analysis (implications, precedents, legal reasoning)
• Statutory interpretation (IPC provisions)
• Hypothetical legal advisory drafting

Metrics

Metric	Description
Validation Loss	Cross-entropy loss on the validation set; primary training objective.
Perplexity	exp(validation_loss); measures how confidently the model predicts the target sequence.
ROUGE-L	Longest Common Subsequence overlap between prediction and reference.
ROUGE-1	Unigram overlap between prediction and reference.
ROUGE-2	Bigram overlap between prediction and reference.
BLEU	Corpus-level n-gram precision score (sacrebleu).
chrF++	Character n-gram F-score; robust to morphological variation in legal terminology.
Exact Match	Percentage of predictions matching reference exactly (case-insensitive).
Fluency (Auto)	Automated heuristic based on predicted response length relative to an ideal length of 50 words.
Coherence (Auto)	ROUGE-L between prediction and reference as a proxy for structural coherence.
Factual Correctness (Auto)	ROUGE-1 between prediction and reference as a proxy for factual overlap.
Overall Human Score (Auto)	Mean of Fluency, Coherence, and Factual Correctness, expressed as a percentage.

Results

Metric	Value
Validation Loss	1.485253
Perplexity (from loss)	4.416082
Perplexity (computed)	4.592739
ROUGE-L	0.2830 (28.30%)
ROUGE-1	0.3883 (38.83%)
ROUGE-2	0.2042 (20.42%)
BLEU	6.91
chrF++	25.43
Exact Match	0.00%
Fluency Score (Auto)	0.1208
Coherence Score (Auto)	0.2830
Factual Correctness (Auto)	0.3883
Overall Human Score (Auto)	26.40%

Summary

The model achieves a validation loss of 1.485 and perplexity of 4.416, indicating solid generalization to unseen legal instruction-response pairs. ROUGE-1 of 38.83% and ROUGE-L of 28.30% show meaningful lexical overlap with gold references. The 0% Exact Match is expected for open-ended legal generation tasks where paraphrasing is natural. The chrF++ score of 25.43 reflects reasonable character-level overlap with target legal text.

Training curve highlights (selected steps):

Step	Training Loss	Validation Loss	ROUGE-L	BLEU	chrF++
100	4.8997	2.0951	0.2395	4.06	17.94
500	3.6551	1.6131	0.2703	6.28	24.26
1000	3.3030	1.4485	0.2838	6.99	25.71
1500	3.0684	1.3673	0.2922	7.45	26.48
2000	2.9353	1.3335	0.2997	7.74	27.12
2820	2.9029	1.3208	0.2983	7.80	27.08

---

Model Examination

Sample Interactive Q&A Outputs (from trained model):

Q: What is the significance of Article 32?

Article 32 of the Constitution of India, which guarantees the right to equality and non-discrimination, is a fundamental principle of the Indian Constitution. It establishes the principle that all individuals, regardless of their caste, are entitled to equal protection of the laws and regulations enshrined in Article 32. This means that any law or regulation that prohibits discrimination on grounds of religion, race, caste or place of birth, is subject to the same constitutional protections as any other law.

Q: What does Article 21 say?

Article 21 of the Indian Constitution guarantees the right to equality and non-discrimination. Article 21 prohibits discrimination on various grounds, including religion, race, caste, sex, or place of birth. This means that all individuals, regardless of their religion, are entitled to equal protection of the laws and regulations enshrined in Article 21.

Q: What are Fundamental Rights?

The right to life and personal liberty is a fundamental right under Article 21 of the Constitution of India. It is the fundamental right of all individuals, regardless of their religion, race, caste, sex, or place of birth. Fundamental rights include the right to freedom of speech, expression, and expression, which are fundamental to the human rights system.

Note: The model occasionally conflates article numbers (e.g., describing Article 21 content under the heading of Article 32), which is a known limitation stemming from the high conceptual overlap in training data.

---

Environmental Impact

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

• Hardware Type: NVIDIA A100-SXM4-40GB (on Lightning.ai Cloudspace)
• Hours used: ~1.5 hours (training) + ~0.5 hours (evaluation & inference)
• Cloud Provider: Lightning.ai (underlying infrastructure: AWS / GCP)

---

Technical Specifications

Model Architecture and Objective

• Architecture: T5 (Text-To-Text Transfer Transformer) — Encoder-Decoder Transformer
• Base model size: 250M parameters (Flan-T5-base)
• Fine-tuning method: LoRA (Low-Rank Adaptation) via PEFT
• Trainable parameters: 13,565,952 (~5.19% of 261,143,808 total)
• Objective: Sequence-to-sequence cross-entropy loss on instruction-response pairs
• Precision: BFloat16 (bf16)
• Input format: "question: {legal instruction or question} answer:"
• Output format: Free-form legal answer text

Compute Infrastructure

Hardware

• Training GPU: NVIDIA A100-SXM4-40GB
• Total VRAM: 42.4 GB
• Peak VRAM consumed: ~34.05 GB
• Local development: Windows 10/11 with Python 3.10, CUDA 12.4, PyTorch 2.6.0+cu124

Software

Package	Version
Python	3.12 (Lightning.ai) / 3.10 (local)
PyTorch	2.6.0+cu118 (Lightning.ai) / 2.6.0+cu124 (local)
Transformers	≥ 4.40.0
PEFT	0.19.1
Datasets	≥ 2.18.0
Accelerate	≥ 0.28.0
SentencePiece	≥ 0.1.99
sacrebleu	≥ 2.3.1
rouge-score	≥ 0.1.2

---

Citation

BibTeX:

@misc{indian-constitution-flan-t5-2025,
  title        = {Indian Constitution \& IPC Q\&A: Fine-tuned Flan-T5-base},
  author       = {Blaze, FighterX},
  year         = {2026},
  note         = {Fine-tuned on nisaar/Articles\_Constitution\_3300\_Instruction\_Set using LoRA PEFT}
}

---

Glossary

• LoRA (Low-Rank Adaptation): A PEFT technique that injects small trainable rank-decomposed weight matrices into existing model layers, enabling efficient fine-tuning by updating far fewer parameters than full fine-tuning.
• ROUGE (Recall-Oriented Understudy for Gisting Evaluation): A set of metrics comparing n-gram overlaps between generated and reference text.
• BLEU (Bilingual Evaluation Understudy): A precision-based metric measuring n-gram overlap between generated and reference sequences, scaled by a brevity penalty.
• chrF++: A character n-gram F-score metric that is robust to morphological variation and works well across different text lengths.
• Perplexity: A measure of how well a probability model predicts a sample. Lower is better; perplexity = exp(cross_entropy_loss).
• BFloat16 (bf16): A 16-bit floating point format with the same exponent range as float32 but reduced mantissa precision. Natively supported by A100/L4 GPUs; avoids loss overflow issues common with fp16.
• Seq2SeqTrainer: A Hugging Face Trainer subclass specialised for sequence-to-sequence models, supporting generation-based evaluation and sequence-level metrics.

---

More Information

• Dataset card: nisaar/Articles_Constitution_3300_Instruction_Set
• Base model card: google/flan-t5-base
• PEFT library: huggingface/peft
• Lightning.ai setup guide: See LIGHTNING_AI_README.md in the project repository for instructions on resolving CUDA driver mismatches and memory optimizations specific to this project.

---

Model Card Authors

Blaze. FighterX

---

Framework Versions

• PEFT 0.19.1
• Transformers ≥ 4.40.0
• PyTorch 2.6.0+cu118 (Lightning.ai) / 2.6.0+cu124 (local Windows)