mistralai

/

Mistral-7B-Instruct-v0.1

like 1.44k

I have the exact same problem, I can only get around 20-30 tokens in the response. I wonder if it's model internal limitation

I have the exact same problem, I can only get around 20-30 tokens in the response. I wonder if it's model internal limitation

Please let me know if you find a solution.

Can you attach an image of the code you are using for generation? Its working fine for me.

Can you also try with model.generate(**inputs, max_new_tokens = 350) the default is 20 which might explain what's happening.

You can make POST request with inputs (and your query) and add "parameters" field. Increase max_new_tokens amount to get more text.

{
                "inputs": inputs,
                "parameters": {
                    "max_new_tokens": 100,
                    "temperature": 0.5,
                    "top_k": 40,
                    "top_p": 0.95,
                    "repetition_penalty": 1.1
                }
            }
``