Spaces:
Running
Is there any example for each LLM performance evaluation?
Hi all,
The llm-perf-leaderboard shows the performance but does not include where these numbers come from.
I check the optimum-benchmark repo, but there is no example configurations for llm shown in the leaderboard.
Another question is that I cannot find how to apply the optimizations to the llm using optimum-benchmark, such as LLM.int8(), Bettertransformer, etc. Should I apply them by myself if I want to reproduce the performance number ins the leaderboard?
If there is an example that reproduces the performance numbers in the leaderboard, that would be great for users to modify it to adjust to their own requirements.
Best
Hi all,
I just found that the results are from optimum/llm-perf-dataset in app.py.
However, from my side, I cannot see app view of llm-perf-leaderboard and cannot find optimum/llm-perf-dataset repo, either.
Is there anything wrong here?
Hi, llm-perf-dataset is still private for now, I'll make a public version soon.
You can reproduce the experiments on your hardware using optimum-benchmark.
I added an example template of the configurations I'm using in the about section.
For LLM.int8, set backend.laod_in_8bit to true in the config.