google

/

gemma-7b

like 2.93k

hello there! you should start from the base model, and use the same chat template that we used for our finetuned models (<start_of_turn>, <end_of_turn>, etc.).

for that specific domain dataset, I would suggest putting all of instruction, context, and question all in the user turn, and the response in the model turn. we don't yet support specific system instruction specification and context, but it's something we're thinking about.

let me know if this helps!

thanks @suryabhupa . Here, I have prepared a single sample. Can you please confirm whether I put everything in correct format?

'instruction': 'You're a helpful question answering bot, please answer the question precisely and politely'
'input': 'Who was Kyle Van Zyl playing against when he scored 36 of hisa teams 61 points?',
'context': "Van Zyl joined the Eastern Province Kings Academy, where he played for the Eastern Province U19 side in the 2010 Under-19 Provincial Championship. He was a key player for the Eastern Province U21 side in the 2012 Under-21 Provincial Championship, scoring 71 points in eight appearances. Van Zyl was under the Top SARU Performers, scoring the most tries at 6 in the 2012 Provincial Under 21 in the Rugby Junior Provincials.\n\nThis included a record and a remarkable personal haul in their opening match, when he scored 36 of his team's points in a 61–3 victory over Boland U21, consisting of four tries and eight conversions and was awarded Man of the Match.",
'response': 'Kyle Van Zyl was playing against Boland U21 when he scored 36 points, leading his team to victory in a 61-3 win.',

First prompt:

'text': '<start_of_turn> Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request. \n\n### Instruction:\nYou're a helpful question answering bot, please answer the question precisely and politely\n\n### Context:\n\nVan Zyl joined the Eastern Province Kings Academy, where he played for the Eastern Province U19 side in the 2010 Under-19 Provincial Championship. He was a key player for the Eastern Province U21 side in the 2012 Under-21 Provincial Championship, scoring 71 points in eight appearances. Van Zyl was under the Top SARU Performers, scoring the most tries at 6 in the 2012 Provincial Under 21 in the Rugby Junior Provincials.\n\nThis included a record and a remarkable personal haul in their opening match, when he scored 36 of his team's points in a 61–3 victory over Boland U21, consisting of four tries and eight conversions and was awarded Man of the Match.\n### Input:\nWho was Kyle Van Zyl playing against when he scored 36 of hisa teams 61 points? \n\n### Response:\nKyle Van Zyl was playing against Boland U21 when he scored 36 points, leading his team to victory in a 61-3 win.<end_of_turn>'

or

Second prompt:

'text': '<start_of_turn> Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request. \n\n### Instruction:\nYou're a helpful question answering bot, please answer the question precisely and politely\n\n### Context:\n\nVan Zyl joined the Eastern Province Kings Academy, where he played for the Eastern Province U19 side in the 2010 Under-19 Provincial Championship. He was a key player for the Eastern Province U21 side in the 2012 Under-21 Provincial Championship, scoring 71 points in eight appearances. Van Zyl was under the Top SARU Performers, scoring the most tries at 6 in the 2012 Provincial Under 21 in the Rugby Junior Provincials.\n\nThis included a record and a remarkable personal haul in their opening match, when he scored 36 of his team's points in a 61–3 victory over Boland U21, consisting of four tries and eight conversions and was awarded Man of the Match.\n### Input:\nWho was Kyle Van Zyl playing against when he scored 36 of hisa teams 61 points?<end_of_turn>\n\n### Response:\nKyle Van Zyl was playing against Boland U21 when he scored 36 points, leading his team to victory in a 61-3 win.'

could you please tell me which one is correct or can you please correct this format if mine is wrong.

Hi, some small suggestions, from https://huggingface.co/google/gemma-7b-it, the prompt should look like:

<bos><start_of_turn>user
INPUT 1<end_of_turn>
<start_of_turn>model
OUTPUT 1<end_of_turn>
<bos><start_of_turn>user
INPUT 2<end_of_turn>
<start_of_turn>model
OUTPUT 2<end_of_turn>

Note that the <bos> token may or may not be needed depending on how you are packing your examples.

thanks @suryabhupa , sorry for so many, I just want to get clarified. Because, I'm afraid to use any other new tokens or new special tokens during fine-tuning. Because, I don't want to use new special tokens which was not present in the training dataset. Hence why I'm clarifying with you. Again sorry for so many questions.

user
INPUT 1
model
OUTPUT 1

Format with instruction, context, input, and response

<bos><start_of_turn>user
Instruction + Context + Input <end_of_turn>
<start_of_turn>model
Reponse<end_of_turn>

Full example:

<bos><start_of_turn>user
Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request. \n\n### Instruction:\nYou're a helpful question answering bot, please answer the question precisely and politely\n\n### Context:\n\nVan Zyl joined the Eastern Province Kings Academy, where he played for the Eastern Province U19 side in the 2010 Under-19 Provincial Championship. He was a key player for the Eastern Province U21 side in the 2012 Under-21 Provincial Championship, scoring 71 points in eight appearances. Van Zyl was under the Top SARU Performers, scoring the most tries at 6 in the 2012 Provincial Under 21 in the Rugby Junior Provincials.\n\nThis included a record and a remarkable personal haul in their opening match, when he scored 36 of his team's points in a 61–3 victory over Boland U21, consisting of four tries and eight conversions and was awarded Man of the Match.\n### Input:\nWho was Kyle Van Zyl playing against when he scored 36 of hisa teams 61 points?<end_of_turn>
<start_of_turn>model
Kyle Van Zyl was playing against Boland U21 when he scored 36 points, leading his team to victory in a 61-3 win.<end_of_turn>

Yes, that looks great to me! Let us know how it goes :)

@suryabhupa quick clarification on this - are the special tokens (<bos>, <start_of_turn> etc) relevant for the base models as well? My understanding was that they are used while training the instruction tuned version

is this model as good as gpt4 and Claude Sonnet/Haiku for code completions and documentation fine tuning for a new programming language that did not make it to the general training datasets of all these models? (a new DSL basically, which none of the models understands, but I want to fine tune one of these, thinking about DeepSeek coder but then found out the high ranks on HumanEval and other benchmarks are unreliable because the training sets were contaminated, the StarCoder for example has 15% contamination of content in training set matching the HumanEval data... I've read some research paper recently about it and realized we have zero chance to actually know which of the models is really best for coding tasks right now). is this mode more of a 'general use case' or if fine tuned on a large data set of questions and answers, and then one more time on prompt-completion pairs, will become good for understanding the new DSL and able to both autocomplete and answer questions (to speed up work vs reading docs manually each time)?

@anihm136 They're not relevant for the base pretrained models when prompting it, but may be relevant if you're looking to finetune it.

@monkeyface Our models are much smaller than GPT4, most likely smaller than Sonnet, but may be comparable to Haiku sizes, and therefore likely comparable in performance to them (though I haven't tested myself).

Gemma models are meant to be more general use-case and support a wide number of downstream finetuning tasks; we didn't focus too much on code completion/documentation when creating our set of finetuned models, so I'm not too sure of what our performance there is like. In addition, there are other considerations when running code models, since the workflow of using them when writing code is different from chatting with it as a dialogue agent (so the evaluations that need to be done look quite different).

If you have a go at finetuning it on your dataset and want to report back numbers on standard benchmarks, we'd be really thrilled to hear about it! Let us know if you have any questions we can answer from our side.

Hello! I'm currently working on a project aiming to develop a chatbot capable of answering questions based on a provided context (imagine a large string or text file). For instance, in a marketplace scenario, where the chatbot is fed with comprehensive market information about products, customers can ask questions, and Gemma (or another model) responds accordingly, leveraging the context.

My query pertains to the approach outlined above: instead of passing the context with every question, is it feasible to train the chatbot directly using this contextual information and restrict it to generating responses solely based on that? Essentially, I aim to streamline the process and avoid passing the context with every individual question.
Thank you!

@mporra93 Sorry for the long delay in responding -- right now, our models don't support use of a system instruction or a special context area. For now, I would suggest placing all of the relevant inside the user prompt and prompting it to only use information here, and unfortunately having to include it in each question. Note that we find the models are not always perfect at following instructions, but we are working on improving this!

@suryabhupa the above prompt which I mentioned. It worked for me :)

Amazing! I'll close this for now, feel free to make more issues as needed.

@suryabhupa I am just curious why we cannot start with the checkpoints of instruction-tuned models, such as google/gemma-7b-it for Supervised Fine-tuning.

You could also start there, feel free to try that out!

You could also start there, feel free to try that out!

Well received with thanks!

Shuyue
May 16th, 2024

@suryabhupa , I was trying to finetuning the model using 's and /s' within <> (not displayed directly) as a single as bos and eos tokens respectively. I used [INST] and [/INST] to kind of separate the instruction from the output. But the models keep generating the [/INST] or a different token continuously. I am doing a full fine-tuning on Gemma 2b (base model). Do you think, this can be solved by using the template given in the discussion?