Mistral 7B produces different results when we hit via postman api

#124
by DivyaKanniah - opened

hi ,

we tried mistral model for Q and A but each time the response is different for same question how to get the consistent results . pls advice

Mistral AI_ org

Hi, well if you want deterministic results be sure to have temperature set to 1. I hope I understood your demand !

Thanks ! how about temperature set to 0 i could see some threads showing 0 to be set for same reproducible results

Mistral AI_ org
β€’
edited Apr 5

It depends but, usually, it's 1 the deterministic value for the temperature. You can try it out with the inference API for free, 1 should always give you the same response even with "use_cache": False.

To get consistent results, consider setting do_sample=False (in the section where you have defined temperature, top_p, top_k etc.) - this will, however, disable all these parameters, leading to the same answers from the model, which I presume is your goal.

Execuse me, when i can find documentation? I want to use serverless API but i don't found docs for body information

Thank you @pandora-s

Sign up or log in to comment