togethercomputer

/

LLaMA-2-7B-32K

like 524

Update: I mistakenly posted this here. The issue is actually related to "Llama-2-7B-32K-Instruct" https://huggingface.co/togethercomputer/Llama-2-7B-32K-Instruct. I will repost to that thread.

Hi folks,
Not sure if this is the best place to ask this question. Please let me know if there is somewhere else. I'll also probably try your support/contact link.

When I use the playground with this model, it works fine.

I then copy the curl command that the playground generates and run that locally and it returns an empty text string for the answer. Here is a sample response. I've tried adjusting the temperature, top_p, top_k, max_tokens, stream_tokens and the prompt that I'm asking.

{"id":"83df4b8d3f94c39c-SEA","status":"finished","prompt":["[INST]  What is rain?  [/INST]  "],"model":"togethercomputer/Llama-2-7B-32K-Instruct","model_owner":"","num_returns":1,"args":{"model":"togethercomputer/Llama-2-7B-32K-Instruct","prompt":"[INST]  What is rain?  [/INST]  ","request_type":"language-model-inference","temperature":0.1,"top_p":0.1,"top_k":3,"max_tokens":1000,"stream_tokens":false,"stop":["[INST]","\n\n"],"negative_prompt":"","sessionKey":"906e58d5-91b6-4ec9-990e-315b47ec5184","update_at":"2023-12-31T03:00:29.376Z"},"subjobs":[],"output":{"usage":{"prompt_tokens":15,"completion_tokens":2,"total_tokens":17},"result_type":"language-model-inference","choices":[{"text":""}]}}%

This also happens when using the javascript code that is generated.

The generated curl command is working correctly for all other models that I have tried.

Thanks for any guidance,
Jeremy