Create README.md

README.md

@@ -0,0 +1,67 @@
+---
+datasets:
+- hivaze/emphatical_daily_dialogues
+---
+# Model Card for Model ID
+
+This model is a adapter for databricks/dolly-v2-3b, finetuned on hivaze/emphatical_daily_dialogues.
+Main goal of this model is to train model to create emphatical dialogues, which are controlled by instructions.
+
+## Model Details
+
+### Model Description
+
+Prompt template: `"{intro}\n\n### Instruction:\n{instruction}\n\n### Response:\n{response}\n"`\
+Example intro: "You are a kind and empathetic interlocutor. You are talking to a person. Below is an instruction that describes a task. Write a response that appropriately completes the request" \
+Example instruction: "You try to chit-chat. Complete a phrase, acting like an interlocutor."
+
+Training params:
+```
+train_args = TrainingArguments(
+    per_device_train_batch_size=8, # can be 4 with llama
+    per_device_eval_batch_size=8, # can be 4 with llama
+    gradient_accumulation_steps=4,
+    warmup_steps=20,
+    # max_steps=200,
+    optim="adamw_torch",
+    learning_rate=4e-5, # many possible values here from 1e-5 to 2e-4
+    # save_strategy="steps",
+    fp16=True,
+    # bf16=True,  # a100 required
+    num_train_epochs=2,
+    evaluation_strategy="steps",
+    eval_steps=50,
+    save_strategy="steps",
+    save_steps=400,
+    logging_strategy="steps",
+    logging_steps=10,
+    logging_dir=f"{local_output_dir}/runs",
+    report_to="tensorboard",
+    output_dir=local_output_dir
+)
+```
+
+LoRA config:
+```
+config = LoraConfig(
+    r=16, # can be 8 with llama
+    lora_alpha=32, # can be 16 with llama
+    # target_modules=["q_proj", "v_proj"],
+    target_modules=['query_key_value'],
+    lora_dropout=0.05,
+    bias="none",
+    task_type="CAUSAL_LM"
+)
+```
+
+- **Developed by:** hivaze
+- **Model type:** LoRA adapter for GPTNeoXForCausalLM
+- **Language(s) (NLP):** Primarly english
+- **Finetuned from model [optional]:** databricks/dolly-v2-3b
+- **Git repository**: https://github.com/hivaze/friendly_chatbot_task
+
+### Tensorboard
+
+![image.png](https://s3.amazonaws.com/moonup/production/uploads/648e72a866dcba8b5aaecbdc/ldy8tEvYPlTCo-2T1WElf.png)
+
+