evaluation loss not calculated during during?
Hi @suryabhupa @ybelkada , I looked into the examples (https://huggingface.co/google/gemma-7b/tree/main/examples) I noticed only training dataset is passed during training, and does not include the evaluation dataset.
I would like to know is there any theoretical reason behind this ? for not evaluating the model during training.
Also, just wanted clarify on thing, what is the loss metrics used during the training? as it is generating training loss during training.
The loss during training is the usual log-likelihood loss. Sometimes we might track other things like the gradient or update norms to make sure things are working properly.
There's no theoretical reason not to evaluate the model; for finetuning jobs, we could consider evaluating perplexity on an evaluation set, but it turns out that perplexity may not be very indicative of downstream performance, e.g. human preferences.
thanks very much @suryabhupa