In which way is this "multilang?
#3
by
maddes8cht
- opened
This model is named "multilang", but the Readme does not say anything about why.
Can you please say something about what is different with this model and what I can expect from it?
You can gladly point me to where I can find this information.
That would be very helpful.
Thanks for the question. This model was trained on all Q&A pairs from all languages in the OASST dataset (train_dataframe: data/user/oasst/train_full_multilang_allrank.pq).
It was prepared as this:
ds = load_dataset("OpenAssistant/oasst1")
train = ds["train"].to_pandas()
val = ds["validation"].to_pandas()
df = pd.concat([train, val], axis=0).reset_index(drop=True)
df_assistant = df[(df.role == "assistant")].copy()
df_prompter = df[(df.role == "prompter")].copy()
df_prompter = df_prompter.set_index("message_id")
df_assistant["output"] = df_assistant["text"].values
inputs = []
parent_ids = []
for _, row in df_assistant.iterrows():
input = df_prompter.loc[row.parent_id]
inputs.append(input.text)
parent_ids.append(input.parent_id)
df_assistant["instruction"] = inputs
df_assistant["parent_id"] = parent_ids
df_assistant = df_assistant[
["instruction", "output", "message_id", "parent_id", "lang", "rank"]
].rename(columns={"message_id": "id"})
df_assistant[(df_assistant["rank"] == 0.0) & (df_assistant["lang"] == "en")][
["instruction", "output", "id", "parent_id"]
].to_parquet(os.path.join(path, "train_full.pq"), index=False)
df_assistant[df_assistant["lang"] == "en"][
["instruction", "output", "id", "parent_id"]
].to_parquet(os.path.join(path, "train_full_allrank.pq"), index=False)
df_assistant[df_assistant["rank"] == 0.0][
["instruction", "output", "id", "parent_id"]
].to_parquet(os.path.join(path, "train_full_multilang.pq"), index=False)
df_assistant[["instruction", "output", "id", "parent_id"]].to_parquet(
os.path.join(path, "train_full_multilang_allrank.pq"), index=False
)