Fine tuned model

.gitignore

@@ -6,5 +6,5 @@ FER/models/checkpoints
 FER/__pycache__
 FER/models/__pycache__
 Gemma2_2B/.cache
-Gemma2_2B/__pycache__
-Gemma2_2B/results
+**/*/wandb
+Gemma2_2B/outputs/

Gemma2_2B/finetune.ipynb

@@ -2,7 +2,7 @@
  "cells": [
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 1,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -25,9 +25,29 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 2,
    "metadata": {},
-   "outputs": [],
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Repo card metadata block was not found. Setting CardData to empty.\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "{'questionID': 0, 'questionTitle': 'Do I have too many issues for counseling?', 'questionText': 'I have so many issues to address. I have a history of sexual abuse, I’m a breast cancer survivor and I am a lifetime insomniac.    I have a long history of depression and I’m beginning to have anxiety. I have low self esteem but I’ve been happily married for almost 35 years.\\n   I’ve never had counseling about any of this. Do I have too many issues to address in counseling?', 'questionLink': 'https://counselchat.com/questions/do-i-have-too-many-issues-for-counseling', 'topic': 'depression', 'therapistInfo': 'Jennifer MolinariHypnotherapist & Licensed Counselor', 'therapistURL': 'https://counselchat.com/therapists/jennifer-molinari', 'answerText': 'It is very common for\\xa0people to have multiple issues that they want to (and need to) address in counseling.\\xa0 I have had clients ask that same question and through more exploration, there is often an underlying fear that they\\xa0 \"can\\'t be helped\" or that they will \"be too much for their therapist.\" I don\\'t know if any of this rings true for you. But, most people have more than one problem in their lives and more often than not,\\xa0 people have numerous significant stressors in their lives.\\xa0 Let\\'s face it, life can be complicated! Therapists are completely ready and equipped to handle all of the issues small or large that a client presents in session. Most therapists over the first couple of sessions will help you prioritize the issues you are facing so that you start addressing the issues that are causing you the most distress.\\xa0 You can never have too many issues to address in counseling.\\xa0 All of the issues you mention above can be successfully worked through in counseling.', 'upvotes': 3, 'views': 1971}\n",
+      "\n",
+      " Dataset({\n",
+      "    features: ['questionID', 'questionTitle', 'questionText', 'questionLink', 'topic', 'therapistInfo', 'therapistURL', 'answerText', 'upvotes', 'views'],\n",
+      "    num_rows: 2775\n",
+      "})\n"
+     ]
+    }
+   ],
    "source": [
     "from datasets import load_dataset\n",
     "dataset_name = \"nbertagnolli/counsel-chat\"\n",
@@ -40,9 +60,29 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 3,
    "metadata": {},
-   "outputs": [],
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      " \n",
+      "### System:\n",
+      "You are a Therapist Assistant, an LLM fine-tuned on Gemma 2 model by Google.\n",
+      "You provide safe and responsible support to users while encouraging them to visit a mental health professional if needed. \n",
+      "You are committed to promoting wellness, understanding, and support. Your responses should be clear, concise, and evidence-based, while maintaining a friendly and approachable tone.\n",
+      "\n",
+      "### User:\n",
+      "I have so many issues to address. I have a history of sexual abuse, I’m a breast cancer survivor and I am a lifetime insomniac.    I have a long history of depression and I’m beginning to have anxiety. I have low self esteem but I’ve been happily married for almost 35 years.\n",
+      "   I’ve never had counseling about any of this. Do I have too many issues to address in counseling?\n",
+      "\n",
+      "### Response:\n",
+      "It is very common for people to have multiple issues that they want to (and need to) address in counseling.  I have had clients ask that same question and through more exploration, there is often an underlying fear that they  \"can't be helped\" or that they will \"be too much for their therapist.\" I don't know if any of this rings true for you. But, most people have more than one problem in their lives and more often than not,  people have numerous significant stressors in their lives.  Let's face it, life can be complicated! Therapists are completely ready and equipped to handle all of the issues small or large that a client presents in session. Most therapists over the first couple of sessions will help you prioritize the issues you are facing so that you start addressing the issues that are causing you the most distress.  You can never have too many issues to address in counseling.  All of the issues you mention above can be successfully worked through in counseling.\n",
+      "\n"
+     ]
+    }
+   ],
    "source": [
     "gemma_prompt = \"\"\" \n",
     "### System:\n",
@@ -75,9 +115,17 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 4,
    "metadata": {},
-   "outputs": [],
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "(2220, 11) (555, 11)\n"
+     ]
+    }
+   ],
    "source": [
     "dataset = formatted_dataset.train_test_split(test_size=0.2, seed=42)\n",
     "print(dataset['train'].shape, dataset['test'].shape)"
@@ -101,17 +149,15 @@
     "    AutoModelForCausalLM,\n",
     "    AutoTokenizer,\n",
     "    BitsAndBytesConfig,\n",
-    "    HfArgumentParser,\n",
     "    TrainingArguments,\n",
-    "    logging,\n",
     ")\n",
-    "from peft import LoraConfig, PeftModel\n",
+    "from peft import LoraConfig\n",
     "from trl import SFTTrainer\n"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 6,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -122,7 +168,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 7,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -138,9 +184,17 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 8,
    "metadata": {},
-   "outputs": [],
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Setting BF16 to True\n"
+     ]
+    }
+   ],
    "source": [
     "# Check GPU compatibility with bfloat16\n",
     "if compute_dtype == torch.float16 and hyperparams['use_4bit']:\n",
@@ -154,9 +208,24 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 9,
    "metadata": {},
-   "outputs": [],
+   "outputs": [
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "3a112598cc9d4adf99116a9b19074886",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
    "source": [
     "model = AutoModelForCausalLM.from_pretrained(\n",
     "    hyperparams['model_name'],\n",
@@ -175,7 +244,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 10,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -192,9 +261,222 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 11,
    "metadata": {},
-   "outputs": [],
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "\u001b[34m\u001b[1mwandb\u001b[0m: Using wandb-core as the SDK backend.  Please refer to https://wandb.me/wandb-core for more information.\n",
+      "\u001b[34m\u001b[1mwandb\u001b[0m: Currently logged in as: \u001b[33mkausikremella\u001b[0m (\u001b[33mkausikremella-vit-ap\u001b[0m). Use \u001b[1m`wandb login --relogin`\u001b[0m to force relogin\n",
+      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m If you're specifying your api key in code, ensure this code is not shared publicly.\n",
+      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Consider setting the WANDB_API_KEY environment variable, or running `wandb login` from the command line.\n",
+      "\u001b[34m\u001b[1mwandb\u001b[0m: Appending key for api.wandb.ai to your netrc file: C:\\Users\\Nitin Kausik Remella\\_netrc\n"
+     ]
+    },
+    {
+     "data": {
+      "text/html": [
+       "Tracking run with wandb version 0.18.7"
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/html": [
+       "Run data is saved locally in <code>f:\\TADBot\\Gemma2_2B\\wandb\\run-20241115_192539-7eelojfi</code>"
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/html": [
+       "Syncing run <strong><a href='https://wandb.ai/kausikremella-vit-ap/TADBot/runs/7eelojfi' target=\"_blank\">eager-morning-3</a></strong> to <a href='https://wandb.ai/kausikremella-vit-ap/TADBot' target=\"_blank\">Weights & Biases</a> (<a href='https://wandb.me/developer-guide' target=\"_blank\">docs</a>)<br/>"
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/html": [
+       " View project at <a href='https://wandb.ai/kausikremella-vit-ap/TADBot' target=\"_blank\">https://wandb.ai/kausikremella-vit-ap/TADBot</a>"
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/html": [
+       " View run at <a href='https://wandb.ai/kausikremella-vit-ap/TADBot/runs/7eelojfi' target=\"_blank\">https://wandb.ai/kausikremella-vit-ap/TADBot/runs/7eelojfi</a>"
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/plain": [
+       "TrainingArguments(\n",
+       "_n_gpu=1,\n",
+       "accelerator_config={'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False},\n",
+       "adafactor=False,\n",
+       "adam_beta1=0.9,\n",
+       "adam_beta2=0.999,\n",
+       "adam_epsilon=1e-08,\n",
+       "auto_find_batch_size=False,\n",
+       "average_tokens_across_devices=False,\n",
+       "batch_eval_metrics=False,\n",
+       "bf16=True,\n",
+       "bf16_full_eval=False,\n",
+       "data_seed=None,\n",
+       "dataloader_drop_last=False,\n",
+       "dataloader_num_workers=0,\n",
+       "dataloader_persistent_workers=False,\n",
+       "dataloader_pin_memory=True,\n",
+       "dataloader_prefetch_factor=None,\n",
+       "ddp_backend=None,\n",
+       "ddp_broadcast_buffers=None,\n",
+       "ddp_bucket_cap_mb=None,\n",
+       "ddp_find_unused_parameters=None,\n",
+       "ddp_timeout=1800,\n",
+       "debug=[],\n",
+       "deepspeed=None,\n",
+       "disable_tqdm=False,\n",
+       "dispatch_batches=None,\n",
+       "do_eval=True,\n",
+       "do_predict=False,\n",
+       "do_train=False,\n",
+       "eval_accumulation_steps=None,\n",
+       "eval_delay=0,\n",
+       "eval_do_concat_batches=True,\n",
+       "eval_on_start=False,\n",
+       "eval_steps=0.2,\n",
+       "eval_strategy=IntervalStrategy.STEPS,\n",
+       "eval_use_gather_object=False,\n",
+       "evaluation_strategy=None,\n",
+       "fp16=False,\n",
+       "fp16_backend=auto,\n",
+       "fp16_full_eval=False,\n",
+       "fp16_opt_level=O1,\n",
+       "fsdp=[],\n",
+       "fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False},\n",
+       "fsdp_min_num_params=0,\n",
+       "fsdp_transformer_layer_cls_to_wrap=None,\n",
+       "full_determinism=False,\n",
+       "gradient_accumulation_steps=2,\n",
+       "gradient_checkpointing=False,\n",
+       "gradient_checkpointing_kwargs=None,\n",
+       "greater_is_better=None,\n",
+       "group_by_length=True,\n",
+       "half_precision_backend=auto,\n",
+       "hub_always_push=False,\n",
+       "hub_model_id=None,\n",
+       "hub_private_repo=False,\n",
+       "hub_strategy=HubStrategy.EVERY_SAVE,\n",
+       "hub_token=<HUB_TOKEN>,\n",
+       "ignore_data_skip=False,\n",
+       "include_for_metrics=[],\n",
+       "include_inputs_for_metrics=False,\n",
+       "include_num_input_tokens_seen=False,\n",
+       "include_tokens_per_second=False,\n",
+       "jit_mode_eval=False,\n",
+       "label_names=None,\n",
+       "label_smoothing_factor=0.0,\n",
+       "learning_rate=0.0002,\n",
+       "length_column_name=length,\n",
+       "load_best_model_at_end=False,\n",
+       "local_rank=0,\n",
+       "log_level=passive,\n",
+       "log_level_replica=warning,\n",
+       "log_on_each_node=True,\n",
+       "logging_dir=./outputs/google/gemma-2-2b-it--health-bot-1731678943/logs,\n",
+       "logging_first_step=False,\n",
+       "logging_nan_inf_filter=True,\n",
+       "logging_steps=50,\n",
+       "logging_strategy=IntervalStrategy.STEPS,\n",
+       "lr_scheduler_kwargs={},\n",
+       "lr_scheduler_type=SchedulerType.CONSTANT,\n",
+       "max_grad_norm=0.3,\n",
+       "max_steps=-1,\n",
+       "metric_for_best_model=None,\n",
+       "mp_parameters=,\n",
+       "neftune_noise_alpha=None,\n",
+       "no_cuda=False,\n",
+       "num_train_epochs=1,\n",
+       "optim=OptimizerNames.PAGED_ADAMW,\n",
+       "optim_args=None,\n",
+       "optim_target_modules=None,\n",
+       "output_dir=./outputs/google/gemma-2-2b-it--health-bot-1731678943,\n",
+       "overwrite_output_dir=False,\n",
+       "past_index=-1,\n",
+       "per_device_eval_batch_size=2,\n",
+       "per_device_train_batch_size=2,\n",
+       "prediction_loss_only=False,\n",
+       "push_to_hub=False,\n",
+       "push_to_hub_model_id=None,\n",
+       "push_to_hub_organization=None,\n",
+       "push_to_hub_token=<PUSH_TO_HUB_TOKEN>,\n",
+       "ray_scope=last,\n",
+       "remove_unused_columns=True,\n",
+       "report_to=['wandb'],\n",
+       "restore_callback_states_from_checkpoint=False,\n",
+       "resume_from_checkpoint=None,\n",
+       "run_name=google/gemma-2-2b-it--health-bot-1731678943,\n",
+       "save_on_each_node=False,\n",
+       "save_only_model=False,\n",
+       "save_safetensors=True,\n",
+       "save_steps=50,\n",
+       "save_strategy=IntervalStrategy.STEPS,\n",
+       "save_total_limit=None,\n",
+       "seed=42,\n",
+       "skip_memory_metrics=True,\n",
+       "split_batches=None,\n",
+       "tf32=None,\n",
+       "torch_compile=False,\n",
+       "torch_compile_backend=None,\n",
+       "torch_compile_mode=None,\n",
+       "torch_empty_cache_steps=None,\n",
+       "torchdynamo=None,\n",
+       "tpu_metrics_debug=False,\n",
+       "tpu_num_cores=None,\n",
+       "use_cpu=False,\n",
+       "use_ipex=False,\n",
+       "use_legacy_prediction_loop=False,\n",
+       "use_liger_kernel=False,\n",
+       "use_mps_device=False,\n",
+       "warmup_ratio=0.0,\n",
+       "warmup_steps=5,\n",
+       "weight_decay=0.001,\n",
+       ")"
+      ]
+     },
+     "execution_count": 11,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
    "source": [
     "import wandb\n",
     "import time\n",
@@ -237,9 +519,26 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 12,
    "metadata": {},
-   "outputs": [],
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "f:\\TADBot\\.venv\\Lib\\site-packages\\huggingface_hub\\utils\\_deprecation.py:100: FutureWarning: Deprecated argument(s) used in '__init__': dataset_text_field, max_seq_length, packing. Will not be supported from version '0.13.0'.\n",
+      "\n",
+      "Deprecated positional argument(s) used in SFTTrainer, please use the SFTConfig to set these arguments instead.\n",
+      "  warnings.warn(message, FutureWarning)\n",
+      "f:\\TADBot\\.venv\\Lib\\site-packages\\trl\\trainer\\sft_trainer.py:212: UserWarning: You passed a `packing` argument to the SFTTrainer, the value you passed will override the one in the `SFTConfig`.\n",
+      "  warnings.warn(\n",
+      "f:\\TADBot\\.venv\\Lib\\site-packages\\trl\\trainer\\sft_trainer.py:300: UserWarning: You passed a `max_seq_length` argument to the SFTTrainer, the value you passed will override the one in the `SFTConfig`.\n",
+      "  warnings.warn(\n",
+      "f:\\TADBot\\.venv\\Lib\\site-packages\\trl\\trainer\\sft_trainer.py:328: UserWarning: You passed a `dataset_text_field` argument to the SFTTrainer, the value you passed will override the one in the `SFTConfig`.\n",
+      "  warnings.warn(\n"
+     ]
+    }
+   ],
    "source": [
     "trainer = SFTTrainer(\n",
     "    model=model,\n",
@@ -264,9 +563,187 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 13,
    "metadata": {},
-   "outputs": [],
+   "outputs": [
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "b86eb0836dc64d2d929ef2f0b2f2bdf9",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "  0%|          | 0/1544 [00:00<?, ?it/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "{'loss': 2.4221, 'grad_norm': 0.682584822177887, 'learning_rate': 0.0002, 'epoch': 0.03}\n",
+      "{'loss': 1.9163, 'grad_norm': 0.5597965121269226, 'learning_rate': 0.0002, 'epoch': 0.06}\n",
+      "{'loss': 1.9249, 'grad_norm': 0.5598402619361877, 'learning_rate': 0.0002, 'epoch': 0.1}\n",
+      "{'loss': 1.9756, 'grad_norm': 0.6536526679992676, 'learning_rate': 0.0002, 'epoch': 0.13}\n",
+      "{'loss': 1.9548, 'grad_norm': 0.608141303062439, 'learning_rate': 0.0002, 'epoch': 0.16}\n",
+      "{'loss': 1.8867, 'grad_norm': 0.4548989534378052, 'learning_rate': 0.0002, 'epoch': 0.19}\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.\n",
+      "Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.\n"
+     ]
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "626464ae6db34340a6c248c021b3dfc8",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "  0%|          | 0/767 [00:00<?, ?it/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "{'eval_loss': 1.902209997177124, 'eval_runtime': 305.3236, 'eval_samples_per_second': 5.021, 'eval_steps_per_second': 2.512, 'epoch': 0.2}\n",
+      "{'loss': 1.9035, 'grad_norm': 0.43129104375839233, 'learning_rate': 0.0002, 'epoch': 0.23}\n",
+      "{'loss': 1.8868, 'grad_norm': 0.49856260418891907, 'learning_rate': 0.0002, 'epoch': 0.26}\n",
+      "{'loss': 1.7944, 'grad_norm': 0.4600728750228882, 'learning_rate': 0.0002, 'epoch': 0.29}\n",
+      "{'loss': 1.8076, 'grad_norm': 0.5697025656700134, 'learning_rate': 0.0002, 'epoch': 0.32}\n",
+      "{'loss': 1.8321, 'grad_norm': 0.7373968958854675, 'learning_rate': 0.0002, 'epoch': 0.36}\n",
+      "{'loss': 1.9213, 'grad_norm': 0.5277324318885803, 'learning_rate': 0.0002, 'epoch': 0.39}\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.\n",
+      "Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.\n"
+     ]
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "43e8d2b139874448be3241192f492b13",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "  0%|          | 0/767 [00:00<?, ?it/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "{'eval_loss': 1.852689266204834, 'eval_runtime': 71.3284, 'eval_samples_per_second': 21.492, 'eval_steps_per_second': 10.753, 'epoch': 0.4}\n",
+      "{'loss': 1.8277, 'grad_norm': 0.5442835688591003, 'learning_rate': 0.0002, 'epoch': 0.42}\n",
+      "{'loss': 1.7947, 'grad_norm': 0.4261704981327057, 'learning_rate': 0.0002, 'epoch': 0.45}\n",
+      "{'loss': 1.8975, 'grad_norm': 0.43769732117652893, 'learning_rate': 0.0002, 'epoch': 0.49}\n",
+      "{'loss': 1.8065, 'grad_norm': 0.6723660230636597, 'learning_rate': 0.0002, 'epoch': 0.52}\n",
+      "{'loss': 1.6969, 'grad_norm': 0.7517312169075012, 'learning_rate': 0.0002, 'epoch': 0.55}\n",
+      "{'loss': 1.7825, 'grad_norm': 0.5381327867507935, 'learning_rate': 0.0002, 'epoch': 0.58}\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.\n",
+      "Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.\n"
+     ]
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "700c8a3cdc694fd88e19ae4f442464d4",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "  0%|          | 0/767 [00:00<?, ?it/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "{'eval_loss': 1.81912362575531, 'eval_runtime': 71.971, 'eval_samples_per_second': 21.3, 'eval_steps_per_second': 10.657, 'epoch': 0.6}\n",
+      "{'loss': 1.7915, 'grad_norm': 0.6141555309295654, 'learning_rate': 0.0002, 'epoch': 0.62}\n",
+      "{'loss': 1.7635, 'grad_norm': 0.5057688355445862, 'learning_rate': 0.0002, 'epoch': 0.65}\n",
+      "{'loss': 1.728, 'grad_norm': 0.49006038904190063, 'learning_rate': 0.0002, 'epoch': 0.68}\n",
+      "{'loss': 1.8424, 'grad_norm': 0.4901270866394043, 'learning_rate': 0.0002, 'epoch': 0.71}\n",
+      "{'loss': 1.8308, 'grad_norm': 0.6117296814918518, 'learning_rate': 0.0002, 'epoch': 0.74}\n",
+      "{'loss': 1.8729, 'grad_norm': 0.5475451946258545, 'learning_rate': 0.0002, 'epoch': 0.78}\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.\n",
+      "Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.\n"
+     ]
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "7d41852d6bad4d65bdf1c972c7c86547",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "  0%|          | 0/767 [00:00<?, ?it/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "{'eval_loss': 1.786774754524231, 'eval_runtime': 71.1209, 'eval_samples_per_second': 21.555, 'eval_steps_per_second': 10.784, 'epoch': 0.8}\n",
+      "{'loss': 1.6851, 'grad_norm': 0.4951877295970917, 'learning_rate': 0.0002, 'epoch': 0.81}\n",
+      "{'loss': 1.7613, 'grad_norm': 1.3179290294647217, 'learning_rate': 0.0002, 'epoch': 0.84}\n",
+      "{'loss': 1.8753, 'grad_norm': 0.45116502046585083, 'learning_rate': 0.0002, 'epoch': 0.87}\n",
+      "{'loss': 1.7441, 'grad_norm': 0.550654411315918, 'learning_rate': 0.0002, 'epoch': 0.91}\n",
+      "{'loss': 1.8054, 'grad_norm': 0.4832320511341095, 'learning_rate': 0.0002, 'epoch': 0.94}\n",
+      "{'loss': 1.7869, 'grad_norm': 0.5937925577163696, 'learning_rate': 0.0002, 'epoch': 0.97}\n",
+      "{'train_runtime': 1964.5956, 'train_samples_per_second': 3.145, 'train_steps_per_second': 0.786, 'train_loss': 1.846395028069847, 'epoch': 1.0}\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "TrainOutput(global_step=1544, training_loss=1.846395028069847, metrics={'train_runtime': 1964.5956, 'train_samples_per_second': 3.145, 'train_steps_per_second': 0.786, 'total_flos': 9905705513385984.0, 'train_loss': 1.846395028069847, 'epoch': 0.9996762706377469})"
+      ]
+     },
+     "execution_count": 13,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
    "source": [
     "model.config.use_cache = False\n",
     "trainer.train()"
@@ -274,9 +751,77 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 14,
    "metadata": {},
-   "outputs": [],
+   "outputs": [
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "47c032db65ce47c6921c3087916cf02f",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "VBox(children=(Label(value='0.022 MB of 0.022 MB uploaded\\r'), FloatProgress(value=1.0, max=1.0)))"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "    <style>\n",
+       "        .wandb-row {\n",
+       "            display: flex;\n",
+       "            flex-direction: row;\n",
+       "            flex-wrap: wrap;\n",
+       "            justify-content: flex-start;\n",
+       "            width: 100%;\n",
+       "        }\n",
+       "        .wandb-col {\n",
+       "            display: flex;\n",
+       "            flex-direction: column;\n",
+       "            flex-basis: 100%;\n",
+       "            flex: 1;\n",
+       "            padding: 10px;\n",
+       "        }\n",
+       "    </style>\n",
+       "<div class=\"wandb-row\"><div class=\"wandb-col\"><h3>Run history:</h3><br/><table class=\"wandb\"><tr><td>eval/loss</td><td>█▅▃▁</td></tr><tr><td>eval/runtime</td><td>█▁▁▁</td></tr><tr><td>eval/samples_per_second</td><td>▁███</td></tr><tr><td>eval/steps_per_second</td><td>▁███</td></tr><tr><td>train/epoch</td><td>▁▁▁▂▂▂▂▂▃▃▃▃▄▄▄▄▄▅▅▅▅▅▅▆▆▆▆▇▇▇▇▇███</td></tr><tr><td>train/global_step</td><td>▁▁▁▂▂▂▂▂▃▃▃▃▄▄▄▄▄▅▅▅▅▅▅▆▆▆▆▇▇▇▇▇███</td></tr><tr><td>train/grad_norm</td><td>▃▂▂▃▂▁▁▂▁▂▃▂▂▁▁▃▄▂▂▂▂▂▂▂▂█▁▂▁▂</td></tr><tr><td>train/learning_rate</td><td>▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁</td></tr><tr><td>train/loss</td><td>█▃▃▄▄▃▃▃▂▂▂▃▂▂▃▂▁▂▂▂▁▂▂▃▁▂▃▂▂▂</td></tr></table><br/></div><div class=\"wandb-col\"><h3>Run summary:</h3><br/><table class=\"wandb\"><tr><td>eval/loss</td><td>1.78677</td></tr><tr><td>eval/runtime</td><td>71.1209</td></tr><tr><td>eval/samples_per_second</td><td>21.555</td></tr><tr><td>eval/steps_per_second</td><td>10.784</td></tr><tr><td>total_flos</td><td>9905705513385984.0</td></tr><tr><td>train/epoch</td><td>0.99968</td></tr><tr><td>train/global_step</td><td>1544</td></tr><tr><td>train/grad_norm</td><td>0.59379</td></tr><tr><td>train/learning_rate</td><td>0.0002</td></tr><tr><td>train/loss</td><td>1.7869</td></tr><tr><td>train_loss</td><td>1.8464</td></tr><tr><td>train_runtime</td><td>1964.5956</td></tr><tr><td>train_samples_per_second</td><td>3.145</td></tr><tr><td>train_steps_per_second</td><td>0.786</td></tr></table><br/></div></div>"
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/html": [
+       " View run <strong style=\"color:#cdcd00\">eager-morning-3</strong> at: <a href='https://wandb.ai/kausikremella-vit-ap/TADBot/runs/7eelojfi' target=\"_blank\">https://wandb.ai/kausikremella-vit-ap/TADBot/runs/7eelojfi</a><br/> View project at: <a href='https://wandb.ai/kausikremella-vit-ap/TADBot' target=\"_blank\">https://wandb.ai/kausikremella-vit-ap/TADBot</a><br/>Synced 4 W&B file(s), 0 media file(s), 3 artifact file(s) and 0 other file(s)"
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/html": [
+       "Find logs at: <code>.\\wandb\\run-20241115_192539-7eelojfi\\logs</code>"
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
    "source": [
     "wandb.finish()\n",
     "model.config.use_cache = True\n",

Gemma2_2B/gemma-2-2b-ft/README.md

@@ -0,0 +1,202 @@
+---
+base_model: google/gemma-2-2b-it
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.13.2

Gemma2_2B/gemma-2-2b-ft/adapter_config.json

@@ -0,0 +1,33 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "google/gemma-2-2b-it",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 16,
+  "lora_dropout": 0.1,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 64,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "q_proj",
+    "o_proj",
+    "k_proj",
+    "v_proj",
+    "up_proj",
+    "gate_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

Gemma2_2B/gemma-2-2b-ft/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9f3b214a7c09db55977481b72ff61a421f3bda54aaec0f9af471b3d492cafc80
+size 255632376

Gemma2_2B/inference.ipynb

@@ -277,6 +277,96 @@
     "outputs = model.generate(**input_ids, max_length=2048)\n",
     "print(tokenizer.decode(outputs[0]))"
    ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Model after fine tuning"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from peft import PeftModel\n",
+    "from transformers import AutoTokenizer, AutoModelForCausalLM\n",
+    "\n",
+    "# Load the base model and tokenizer\n",
+    "model_name = \"google/gemma-2-2b-it\"\n",
+    "device_map = {\"\": 0}  # Use GPU 0 for the model\n",
+    "\n",
+    "# Load the fine-tuned model\n",
+    "new_model = \"gemma-2-2b-ft/\"  # Replace with the path to your fine-tuned model"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "8bf9b158501544f092a784849b8e402d",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "base_model = AutoModelForCausalLM.from_pretrained(\n",
+    "    model_name, device_map=device_map, cache_dir=\".cache/\")\n",
+    "model = PeftModel.from_pretrained(base_model, new_model, cache_dir = \".cache/\")\n",
+    "model = model.merge_and_unload()\n",
+    "\n",
+    "# Reload tokenizer to save it\n",
+    "tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True, cache_dir = \".cache/\")\n",
+    "tokenizer.pad_token = tokenizer.eos_token\n",
+    "tokenizer.padding_side = \"right\"\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "<bos>I have so many issues to address. I have a history of sexual abuse, I’m a breast cancer survivor and I am a lifetime insomniac.    I have a long history of depression and I’m beginning to have anxiety. I have low self esteem but I’ve been happily married for almost 35 years.I’ve never had counseling about any of this. Do I have too many issues to address in counseling?\n",
+      "\n",
+      "### Response:\n",
+      "I would say absolutely not!  It is never too many issues to address in counseling.  It is actually quite common for people to come into therapy with a lot of issues and it is often the case that the issues are interconnected.  For example, a person who has experienced trauma may have difficulty sleeping, have low self esteem, and have anxiety.  It is important to remember that counseling is a collaborative process and the therapist will work with you to help you address all of your issues.\n",
+      "<eos>\n",
+      "CPU times: total: 16 s\n",
+      "Wall time: 17.3 s\n"
+     ]
+    }
+   ],
+   "source": [
+    "%%time\n",
+    "input_text = \"I have so many issues to address. I have a history of sexual abuse, I’m a breast cancer survivor and I am a lifetime insomniac.    I have a long history of depression and I’m beginning to have anxiety. I have low self esteem but I’ve been happily married for almost 35 years.I’ve never had counseling about any of this. Do I have too many issues to address in counseling?\"\n",
+    "input_ids = tokenizer(input_text, return_tensors=\"pt\").to(\"cuda\")\n",
+    "outputs = model.generate(**input_ids, max_length=2048)\n",
+    "print(tokenizer.decode(outputs[0]))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
   }
  ],
  "metadata": {

README.md

@@ -2,13 +2,13 @@
 
 ## Overview
 
-TADBot is small language model that is trained on the <input_data_set_name> dataset. It is a fine-tuned version of the Gemma 2 2B, which is a small language model with 2 billion parameters. TADBot is designed to assist people deal with mental problems and offer them advice based on the context of the conversation. It is not intended to replace professional mental health care, but rather to provide a supportive and empathetic resource for those who may be struggling with mental health issues. TADBot is still in development and is not yet available for public use.
+TADBot is small language model that is trained on the nbertagnolli/counsel-chat dataset. It is a fine-tuned version of the Gemma 2 2B, which is a small language model with 2 billion parameters. TADBot is designed to assist people deal with mental problems and offer them advice based on the context of the conversation. It is not intended to replace professional mental health care, but rather to provide a supportive and empathetic resource for those who may be struggling with mental health issues. TADBot is still in development and is not yet available for public use.
 
 ## Technology used
 
 - Gemma 2 2B: A small language model with 2 billion parameters that TADBot is fine-tuned on.
-- <input_data_set_name>: The dataset used to train TADBot on mental health and advice-giving tasks.
-- Hugging Face Transformers: A library used to fine-tune the Gemma 2 2B model on the <input_data_set_name> dataset.
+- nbertagnolli/counsel-chat: The dataset used to train TADBot on mental health and advice-giving tasks.
+- Hugging Face Transformers: A library used to fine-tune the Gemma 2 2B model on the nbertagnolli/counsel-chat dataset.
 - PyTorch: A library used for training and fine-tuning the language model.
 - Flask: A library used to create a server for TADBot.
 - Raspberry Pi: A small, low-cost computer used to host Test to Speech and Speech To Text models and TADBot server.
@@ -51,7 +51,10 @@ TADBot is small language model that is trained on the <input_data_set_name> data
 # How It Works
 
 ## Model
-
+TADBot uses a fine-tuned version of the Gemma 2 2B language model to generate responses. The model is trained on the nbertagnolli/counsel-chat dataset from hugging face, which contains conversations between mental health professionals and clients. The model is fine-tuned using the Hugging Face Transformers library and PyTorch.
+### Dataset
+The raw version of the dataset consists of 2275 conversation taken from an online mental health platform.
+- 
 # Implementation
 
 ## Deployment Instructions

pyproject.toml

@@ -28,6 +28,7 @@ dependencies = [
     "pyyaml>=6.0.2",
     "torch-tb-profiler>=0.4.3",
     "tensorflow>=2.18.0",
+    "wandb>=0.18.7",
 ]
 
 [tool.uv.sources]