Update README.md

README.md

@@ -67,6 +67,45 @@ We train on three sources of data to create this model:
 * [openchat/openchat_sharegpt4_dataset](https://huggingface.co/datasets/openchat/openchat_sharegpt4_dataset/resolve/main/sharegpt_gpt4.json) - 6,206 conversations
     * Multilingual conversations of humans talking to GPT-4.
 
+
+<details><summary>We prepare our data like so:</summary>
+
+```python
+import pandas as pd
+from datasets import Dataset, load_dataset, concatenate_datasets
+
+### Tagengo
+gpt4_dataset = load_dataset("lightblue/tagengo-gpt4", split="train")
+gpt4_dataset = gpt4_dataset.filter(lambda x: x["response"][1] == "stop")
+####
+
+### Megagon
+megagon_df = pd.read_json(
+    "https://raw.githubusercontent.com/megagonlabs/instruction_ja/main/data/data.jsonl",
+    lines=True,
+    orient="records"
+    )
+role_map = {"user": "human", "agent": "gpt"}
+megagon_df["conversations"] = megagon_df.utterances.apply(lambda x: [{"from": role_map[y["name"]], "value": y["text"]} for y in x])
+megagon_df["language"] = "Japanese"
+megagon_df = megagon_df[["conversations", "language"]]
+megagon_dataset = Dataset.from_pandas(df)
+###
+
+### Openchat
+openchat_df = pd.read_json("https://huggingface.co/datasets/openchat/openchat_sharegpt4_dataset/resolve/main/sharegpt_gpt4.json?download=true")
+openchat_df["conversations"] = openchat_df["items"]
+openchat_dataset = Dataset.from_pandas(openchat_df)
+###
+
+
+dataset = concatenate_datasets([gpt4_dataset, megagon_dataset, openchat_dataset])
+dataset = dataset.filter(lambda x: not any([y["value"] is None for y in x["conversations"]]))
+dataset.select_columns(["conversations"]).to_json("/workspace/llm_training/axolotl/llama3-multilingual/tagengo_openchat_megagon.json")
+```
+
+</details>
+
 # workspace/llm_training/axolotl/llama3-multilingual/output_tagengo_openchat_megagon_8B_llama3
 
 This model is a fine-tuned version of [meta-llama/Meta-Llama-3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) on the above described dataset.