Mistral 7B produces different results when we hit via postman api
hi ,
we tried mistral model for Q and A but each time the response is different for same question how to get the consistent results . pls advice
Hi, well if you want deterministic results be sure to have temperature set to 1. I hope I understood your demand !
Thanks ! how about temperature set to 0 i could see some threads showing 0 to be set for same reproducible results
It depends but, usually, it's 1 the deterministic value for the temperature. You can try it out with the inference API for free, 1 should always give you the same response even with "use_cache": False.
To get consistent results, consider setting do_sample=False (in the section where you have defined temperature, top_p, top_k etc.) - this will, however, disable all these parameters, leading to the same answers from the model, which I presume is your goal.
Execuse me, when i can find documentation? I want to use serverless API but i don't found docs for body information