Spaces:
Running
on
CPU Upgrade
Add a column: average score per billion parameters
To help us gauge the average efficiency of each model, relative to its size, please add a column that shows the Average score divided by the size of the model, in billions of parameters. For example, falcon-40b-instruct, with an average score of 63.2, would show a "Avg/BP" value of 1.58. This would help researchers track the most-promising models (in terms of architecture, dataset, and training techniques), even if they were not as large as the largest open source models.
This would favor smaller models since a model with twice the parameters doesn't have twice the score.
You could instead do something like:
Efficiency Score = Score / log(Parameters)
Model 1: 55 / log(13 billion) = 5.44
Model 2: 60 / log(30 billion) = 5.72
Model 3: 62 / log(120 billion) = 5.59
(Equation would be adjusted based on what you value)
Hi! For performance evaluations, you can also take a look at the performance benchmark