meta-llama

/

Meta-Llama-3-8B

like 4.37k

I've found that LLama 3 runs about 5 to 10 times slower if you include a grammar statement.
EG If you include this in the check.
"grammar":" root ::= fullanswer \n fullanswer ::= "John: " answer \nanswer ::= sentence | "<|im_end|>" | sentence "\n"\nsentence ::= [a-zA-Z0-9.,?!' ]*\n"

I did the same comparison on LLama 2. One with grammar and one without. The 2 runs on LLama 2 were roughly the same.
I was running this through Koboldcpp submitting a request as json to it, and just removing this line massively increased it's speed.