Model prone to endless repeating
I've struggled a lot with this model to continue repeating the same content, also in the spaces you find on HF the repeating issue happens.
A very high temperature made it more likely stop properly, a low temperature made it go into repeating until context is filled.
That might be a problem with the llama 3 base instruct model, I just hope that fine tuning can fix it ?
That's a problem with the eos token.
Go into config.json and generation_config.json and change
"eos_token_id": 128001,
to
"eos_token_id": 128009,
That's a problem with the eos token.
Go into config.json and generation_config.json and change
"eos_token_id": 128001,
to
"eos_token_id": 128009,
I already tried that, 128009 is not being generated either. Its a true repeating loop.
When going on high temperature I can see eot is more likely to be sampled. But low temp not. So something appears wrong
"<|start_header_id|>system<|end_header_id|>
<|eot_id|><|start_header_id|>user<|end_header_id|>
image
Describe this image<|eot_id|><|start_header_id|>assistant<|end_header_id|>"
Temperature: 0
I tried variants of the prompt, almost always endless repeating.
I modified the output of the client to include special tokens to make sure no variant of any ending token is missed (nothing comes).
When running it on high temperature it typically does not go into a loop, it samples the eot_id.
That's on a Q6K model locally but I also tested the same prompt and image on the huggingface space resulting in a similar endless repeat.
Example output:
"The image you've shared is a California driver's license issued to a woman named Lisa Cardenas. The license is valid from August 31, 2014 to August 31, 2017. It's a Class C license with a California state seal and a gold star on the top right corner. The license number is 1234568. The license is issued by the California Department of Motor Vehicles. The license photo is a close-up of Lisa Cardenas, and the license number and expiration date are printed on the bottom of the license. The license is issued by the California Department of Motor Vehicles. The license is valid from August 31, 2014 to August 31, 2017. The license number is 1234568. The license is issued by the California Department of Motor Vehicles. The license is valid from August 31, 2014 to August 31, 2017. The license number is 1234568. The license is issued by the California Department of Motor Vehicles. The license is valid from August 31, 2014 to August 31, 2017. The license number is 1234568. The license is issued by the California Department of Motor Vehicles. The license is valid from August 31, 2014 to August 31, 2017. The license number is 1234568. The license is issued by the California Department of Motor Vehicles. The license is valid from August 31, 2014 to August 31, 2017. The license number is 1234568. The license is issued by the California Department of Motor Vehicles. The license is valid from August 31, 2014 to August 31, 2017. The license number is 1234568. The license is issued by the California Department of Motor Vehicles. The license is valid from August 31, 2014 to August 31, 2017. The license number is 1234568. The license is issued by the California Department of Motor Vehicles. The license is valid f"
Another way to make it endless repeat is if you request a list of 20 comma separated features of an image. If it respects the prompt it likely starts with a few features and from there on it will not stop anymore.