togethercomputer

/

Llama-2-7B-32K-Instruct

like 160

Hi @LaferriereJC , thanks for your interest. Could you add "[INST]" as the stop sequence?
Additionally, the prompt to use is in the format of "[INST]\nInstruction\n[/INST]\n\n", which in your case should be:

"[INST]
What is the meaning of life?
[/INST]

"

Please note that there are two "\n" after the [/INST]. Please let us know if it works.

Yes this model does not seems to stop ..

prompt = """[INST]
What is the meaning of life in 20 words?
[/INST]

"""

The meaning of life is to seek happiness, fulfillment, and purpose in life. It is to live intentionally, to follow one's passions, and to make a positive impact on the world.

The meaning of life is to seek happiness, fulfillment, and purpose in life. It is to live intentionally, to follow one's passions, and to make a positive impact on the world.

The meaning of life is to seek happiness, fulfillment, and purpose in life. It is to live intentionally, to follow one's passions, and to make a positive impact on the world.

The meaning of life is to seek happiness, fulfillment, and purpose in life. It is to live intentionally, to follow one's passions, and to make a positive impact on the world.

The meaning of life is to seek happiness, fulfillment, and purpose in life. It is to live intentionally, to follow one's passions, and to make a positive impact on the world.

The meaning of life is to seek happiness, fulfillment, and purpose in life. It is to live intentionally, to follow one's passions, and to make a positive impact on the world.

The meaning of life is to seek happiness, fulfillment, and purpose in life. It is to live intentionally, to follow one's passions, and to make a positive impact on the world.

The meaning of life is to seek happiness, fulfillment, and purpose in life. It is to live intentionally, to follow one's passions, and to make a positive impact on the world.
the meaning of life is to seek happiness, fulfillment, and purpose in life. it is to live intentionally, to follow one's passions, and to make a positive impact on the world.

@monuminu , hmm this is what I got

What's the hyperparameter you are using?

(We recommend to use repetition_penalty = 1.1)

Ce

Was being ignorant and not paying attention.

I have the same problem

HI,
I have the same problem. The model never stops and generates follow-up questions.
I print the logits of the EOS token and define 'do_sample= False' & 'num_beams=1' which is supposed to get me the greedy answer.
But, the model predicts tokens with lower scores than EOS!
During dozens of experiments, the model never predicted EOS! I'll be happy for your help.

My code:

prompt_tokenize = tokenizer_llm('what is the capital of Israel?', add_special_tokens=True, return_tensors="pt")

output = model_llm.generate(**prompt_tokenize, do_sample = False,num_beams=1, max_new_tokens = 300,
temperature=0.7, repetition_penalty=1.1, top_p=0.7, eos_token_id=tokenizer_llm.eos_token_id, return_dict_in_generate=True, output_scores=True)

for i in range(3 * max_new_tokens):
print(f' EOS score is: {output.scores[i][0][2]}')
n = torch.squeeze(output.sequences)[i]
print(f' The chosen token is: {n} : {tokenizer_llm.decode(n)}' )
print(f' And its score is: {output.scores[i][0][n]}\n ')

output:
EOS score is: 5.02734375
The chosen token is: 5816 : what
And its score is: 1.7356178760528564

Hi @SAbrahamy , can you provide a full minimal example to reproduce this behaviour?

As a general hint, if you haven't tried this yet, and if you have problems with the model not stopping generation, you can also implement a stopping criteria (check out the docs here).

Hi @mauriceweber , thank you very much for the reply.
I have provided the code prompt and output, what other information would you like to reproduce the behavior?
Regarding stopping criteria, I don't have a better criterion than EOS, I want the model to stop when it sees fit.
Any idea why the model doesn't stop under the conditions I provided?

Sorry for the late answer here @SAbrahamy ! I had a closer look at the code you sent. I noticed that you are fetching the token id n from the sequences attribute of the output, while you fetch the EOS score from the scores attribute of the output. However, if you compare the two attributes, you will see that they have different shapes: sequences includes the prompt tokens, but scores does not. So you cannot compare the EOS token score to the generated token score -- they refer to two different locations in the generated sequence.

You can adjust your code to the following, in which case you should see that the EOS token has lower score than the predicted token:

for i, score in enumerate(output.scores):
    predicted_token = torch.argmax(score)
    token_score = round(float(torch.max(score)), 4)
    eos_score = round(float(score[0, tokenizer.eos_token_id]), 4)

    print(f"({i}) token: {predicted_token:<5}\t"
          f"token_score: {token_score:<5}\t"
          f"eos_score: {eos_score}")

@mauriceweber
I am unable to make the model stop generating tokens. Is there a solution to this problem ?

Hi @MohamedRashad -- what is your setup? which hyperparameters are you using?

@mauriceweber This is what i am using

tokenizer = AutoTokenizer.from_pretrained("togethercomputer/Llama-2-7B-32K-Instruct", use_fast=False)
model = AutoModelForCausalLM.from_pretrained(
    "togethercomputer/Llama-2-7B-32K-Instruct", trust_remote_code=True, torch_dtype=torch.float16, device_map="auto"
)
input_ids = tokenizer.encode(prompt, return_tensors="pt")
model.generate(
    input_ids,
    max_new_tokens=8192,
    temperature=0.7,
    repetition_penalty=1.1,
    top_p=0.7,
    top_k=50,
)

thanks for the details you sent! As a first step, you can try to play with the generation parameters, e.g., increasing / decreasing top_p and top_k or increase the repetition_penalty if your output appears to have too many repetitions.

Apart from that, you can also implement your own stopping criteria and ensure the model stops generating once it reaches a specific token (e.g. [INST]). The following template should get you started:

from transformers import StoppingCriteria
...

class MyStoppingCriteria(StoppingCriteria):
    def __init__(self, stop_token: int):
        super(MyStoppingCriteria, self).__init__()
        self._stop_token = stop_token

    def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor, **kwargs):
        stop_count = (self._stop_token == input_ids[0]).sum().item()
        return stop_count > 0

...

my_stop = MyStoppingCriteria(stop_token=tokenizer.encode(["[INST]"])[0])
output = model.generate(..., stopping_criteria=[my_stop])

@mauriceweber The generation stops after the first token with this code.