openaccess-ai-collective

/

manticore-13b

like 116

Follow

Open Access AI Collective 106

data process demo

    def generate_and_tokenize_prompt2(examples, CUTOFF_LEN=SEQ_LEN):
        instruction, input_text = examples['instruction'].strip(),examples['input'].strip()
        if len(input_text)==0:
            user_prompt = f"User:{instruction}\n###{input_text}\n\nAssistant: \n"
        else:
            user_prompt = f"User:{instruction}\n\nAssistant: \n"
        len_user_prompt_tokens = (len(tokenizer(user_prompt,truncation=True,max_length=CUTOFF_LEN + 1,)["input_ids"])- 1)  # no eos token
        full_tokens = tokenizer(user_prompt + examples["output"],truncation=True,max_length=CUTOFF_LEN + 1,padding="max_length")["input_ids"][:-1]
        labels = [-100]*len_user_prompt_tokens + [id if id!=0 else -100 for id in full_tokens[len_user_prompt_tokens:]  ]
        return {"input_ids": full_tokens,"labels": torch.LongTensor(labels) ,"attention_mask": torch.LongTensor([1] * len(full_tokens)) }

I use my data process to instruct tuning manticore-13b, the train loss initially very large, such as 78.0, but your train loss is very small ( https://wandb.ai/wing-lian/manticore-13b/runs/nq3u3uoh/workspace)