pkbloom-7b1-lora

pkbloom-7b1-lora is a parameter-efficient LoRA fine-tune of bigscience/bloom-7b1 trained on the Abirate/english_quotes dataset.

The model was adapted for causal language modeling using Hugging Face Transformers and PEFT. The original BLOOM weights were frozen and only LoRA adapter parameters were trained.

The training task converts each quote example into a single causal-language-modeling sequence of the form:

<quote> ->: <tags>

For example:

“Be yourself; everyone else is already taken.” ->: ['be-yourself', 'gilbert-perreira', 'honesty', 'inspirational', 'misattributed-oscar-wilde', 'quote-investigator']

This repository contains the LoRA adapter, not a standalone copy of the full BLOOM-7B1 base model.

Model Details

Model Description

• Developed by: pynk17
• Base model: bigscience/bloom-7b1
• Model type: Causal language model with LoRA adapters
• Task: Text generation / causal language modeling
• Fine-tuning method: PEFT LoRA
• Training precision: FP16
• Gradient checkpointing: Enabled
• Framework: Hugging Face Transformers + PEFT
• Adapter format: Safetensors
• Repository: pynk17/pkbloom-7b1-lora
• Visibility at creation: Private

Parameter-Efficient Fine-Tuning Configuration

The base model was loaded from:

model = AutoModelForCausalLM.from_pretrained(
    "bigscience/bloom-7b1",
    device_map="auto",
)

The tokenizer was loaded from the same base model:

tokenizer = AutoTokenizer.from_pretrained("bigscience/bloom-7b1")

The original model weights were frozen:

for param in model.parameters():
    param.requires_grad = False

One-dimensional parameters such as normalization parameters were cast to FP32 for stability:

if param.ndim == 1:
    param.data = param.data.to(torch.float32)

Gradient checkpointing was enabled to reduce stored activations during training:

model.gradient_checkpointing_enable()
model.enable_input_require_grads()

The language-model head output was cast to FP32:

class CastOutputToFloat(nn.Sequential):
    def forward(self, x):
        return super().forward(x).to(torch.float32)

model.lm_head = CastOutputToFloat(model.lm_head)

LoRA Configuration

The adapter was configured as:

from peft import LoraConfig, get_peft_model

config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)

model = get_peft_model(model, config)

Trainable Parameters

The training setup reported:

trainable params: 7,864,320
all params:       7,076,880,384
trainable %:      0.11112693126452029

Only about 0.111% of the total parameter count was trainable.

This is the central efficiency benefit of the LoRA setup: the full 7B-scale base model remained frozen while a much smaller adapter parameter set was optimized.

Training Data

The model was trained on:

Abirate/english_quotes

The dataset contained:

2,508 training examples

Each example originally contained fields such as:

• quote
• author
• tags

A new prediction field was constructed by concatenating the quote and its tag list:

def merge_columns(example):
    example["prediction"] = (
        example["quote"] + " ->: " + str(example["tags"])
    )
    return example

The resulting text was tokenized with the BLOOM-7B1 tokenizer:

data = data.map(
    lambda samples: tokenizer(samples["prediction"]),
    batched=True,
)

The training objective therefore teaches the model to continue quote text into its associated tag representation.

Training Procedure

Training used the Hugging Face Trainer API:

trainer = transformers.Trainer(
    model=model,
    train_dataset=data["train"],
    args=transformers.TrainingArguments(
        per_device_train_batch_size=4,
        gradient_accumulation_steps=4,
        warmup_steps=100,
        max_steps=200,
        learning_rate=2e-4,
        fp16=True,
        logging_steps=1,
        output_dir="outputs",
    ),
    data_collator=transformers.DataCollatorForLanguageModeling(
        tokenizer,
        mlm=False,
    ),
)

Before training:

model.config.use_cache = False

The source notebook notes that caching should be re-enabled for inference.

Training Hyperparameters

Hyperparameter	Value
Base model	bigscience/bloom-7b1
Dataset	Abirate/english_quotes
Dataset size	2,508 examples
Per-device train batch size	4
Gradient accumulation steps	4
Effective batch size	16 samples per optimizer update
Maximum training steps	200
Warmup steps	100
Learning rate	2e-4
Precision	FP16
Logging steps	1
Objective	Causal language modeling
MLM	Disabled
LoRA rank	16
LoRA alpha	32
LoRA dropout	0.05
LoRA bias	None
Gradient checkpointing	Enabled

Training Results

Training completed successfully at step 200.

The final Trainer output reported:

global_step: 200
train_runtime: 86.569 seconds
train_samples_per_second: 36.965
train_steps_per_second: 2.31
train_loss: 2.3122402310371397
epoch: 1.2743221690590112
total_flos: 1.3292485294620672e+16

Loss Behavior

The first logged loss was:

step 1: 3.409602

The final logged step reported:

step 200: 2.090332

The lowest individual logged loss in the supplied training trace was:

step 159: 1.572118

The per-step loss was noisy, but the run shows a clear overall reduction from the early 3.x range into the high-1.x / low-2.x range during later training.

The reported aggregate training loss was:

2.3122402310371397

No held-out validation loss or task-specific evaluation metric was supplied for this run.

Adapter Artifact

The uploaded adapter artifact was approximately:

31.5 MB

and was stored as:

adapter_model.safetensors

The adapter was pushed to the Hub with commit message:

basic training

Because this is a PEFT adapter repository, the base BLOOM-7B1 weights are not included in the adapter file.

Compute Infrastructure

Hardware

Training was run on:

GPU 0: NVIDIA RTX PRO 6000 Blackwell Server Edition

Software

The notebook installed and used:

• PyTorch
• Hugging Face Transformers
• Hugging Face PEFT
• Hugging Face Datasets
• Hugging Face Accelerate
• bitsandbytes
• loralib
• huggingface_hub

How to Load the Adapter

The source notebook reloads the PEFT configuration from the Hub and reconstructs the base model from config.base_model_name_or_path.

A standard PEFT loading pattern is:

import torch
from peft import PeftModel, PeftConfig
from transformers import AutoModelForCausalLM, AutoTokenizer

peft_model_id = "pynk17/pkbloom-7b1-lora"

config = PeftConfig.from_pretrained(peft_model_id)

base_model = AutoModelForCausalLM.from_pretrained(
    config.base_model_name_or_path,
    device_map="auto",
)

tokenizer = AutoTokenizer.from_pretrained(
    config.base_model_name_or_path
)

model = PeftModel.from_pretrained(
    base_model,
    peft_model_id,
)

model.config.use_cache = True
model.eval()

Example Inference

The training format suggests prompts that begin with a quote and ask the model to continue into tags.

import torch

prompt = '“So many books, so little time.” ->:'

inputs = tokenizer(
    prompt,
    return_tensors="pt",
).to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=40,
        do_sample=True,
        temperature=0.8,
        top_p=0.95,
    )

print(
    tokenizer.decode(
        outputs[0],
        skip_special_tokens=True,
    )
)

This inference example follows the training format.

Uses

Direct Use

This adapter is intended for experimentation with parameter-efficient fine-tuning of large causal language models.

The specific supervised pattern in this run is:

quote -> tags

It can therefore be used to explore whether the adapted model learns associations between quote text and tag-like descriptors.

Downstream Use

The repository can also serve as an example of:

• LoRA adaptation of a multi-billion-parameter causal language model
• PEFT training with a frozen base model
• FP16 adapter training
• gradient checkpointing for activation-memory reduction
• training large language models with a very small percentage of trainable parameters
• saving and distributing PEFT adapters through the Hugging Face Hub

Out-of-Scope Use

This model was not trained or evaluated as a general-purpose factual assistant, safety-critical system, or production classifier.

The supplied experiment does not establish:

• factual reliability
• robust quote attribution
• calibrated tag prediction
• generalization outside the English quotes dataset
• safety performance
• bias performance
• production readiness

Bias, Risks, and Limitations

The adapter inherits limitations from both the BLOOM-7B1 base model and the Abirate/english_quotes training data.

The training dataset contains quotes, author information, and user-defined tags. Those tags may contain subjective, noisy, incomplete, or attribution-dependent labels.

The training transformation does not explicitly include the author field in the generated prediction sequence, so this adapter should not be treated as an author-identification model.

The experiment includes no held-out evaluation set, no accuracy-style tag metric, and no human evaluation. Training loss therefore should not be interpreted as a direct measure of downstream usefulness.

Reproducibility Notes

Known configuration:

base model:                    bigscience/bloom-7b1
dataset:                       Abirate/english_quotes
training examples:             2,508
LoRA rank:                     16
LoRA alpha:                    32
LoRA dropout:                  0.05
LoRA bias:                     none
task type:                     CAUSAL_LM
trainable parameters:          7,864,320
total parameters:              7,076,880,384
trainable percentage:          0.11112693126452029%
per-device batch size:         4
gradient accumulation:         4
effective batch size:          16
warmup steps:                  100
maximum steps:                 200
learning rate:                 2e-4
precision:                     FP16
gradient checkpointing:        enabled
training runtime:              86.569 s
training loss:                 2.3122402310371397
reported epoch:                1.2743221690590112
adapter artifact:              ~31.5 MB
GPU:                           NVIDIA RTX PRO 6000 Blackwell Server Edition

Model Card Author

Priyanka / pynk17