Repository: mlfoundations/evalchemy
Issue: Please add LiveBench support!
Root cause: Unimplemented Feature Gap

Thanks so much for building this fantastic package! Can you please add support for the various past releases of [LiveBench](https://github.com/livebench/livebench) at some point?

Hi all, on your README it says that LiveBench support is added and can be found in https://github.com/mlfoundations/evalchemy/tree/main/eval/chat_benchmarks, but I don't see it there. Can you please explain? Thanks!

This was the PR:
https://github.com/mlfoundations/evalchemy/commit/76f72fea7613f776d2d5a14610f1a1b7d91f388a
I do see it.
[image: image.png]

On Mon, Jan 27, 2025 at 9:58 AM Ben Feuer ***@***.***> wrote:

> Hi all, on your README it says that LiveBench support is added and can be
> found in
> https://github.com/mlfoundations/evalchemy/tree/main/eval/chat_benchmarks,
> but I don't see it there. Can you please explain? Thanks!
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/mlfoundations/evalchemy/issues/41#issuecomment-2616521232>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAIJX24XDN2G4NNQBJEIFND2MZXWDAVCNFSM6AAAAABTP4YEAGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMJWGUZDCMRTGI>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>


I see LiveCodeBench in that PR, but not LiveBench. As your Readme correctly
states, they're entirely different benchmarks.

On Mon, Jan 27, 2025, 2:48 PM Mahesh Sathiamoorthy ***@***.***>
wrote:

> This was the PR:
>
> https://github.com/mlfoundations/evalchemy/commit/76f72fea7613f776d2d5a14610f1a1b7d91f388a
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_mlfoundations_evalchemy_commit_76f72fea7613f776d2d5a14610f1a1b7d91f388a&d=DwQFaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=cRVZDAsVf8YMy7ncQBvRsw&m=7Sswtrfq7sOJMi3BChc7Z-D8KHRGfXMJiFJE2wyRKkFpE_O7uQYzl-TLkvLvWtbT&s=DtDDsP4PPxpAkdsNZxtqOkDSEtlstrhVL3936EtJ9-w&e=>
> I do see it.
> [image: image.png]
>
> On Mon, Jan 27, 2025 at 9:58 AM Ben Feuer ***@***.***> wrote:
>
> > Hi all, on your README it says that LiveBench support is added and can
> be
> > found in
> >
> https://github.com/mlfoundations/evalchemy/tree/main/eval/chat_benchmarks
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_mlfoundations_evalchemy_tree_main_eval_chat-5Fbenchmarks&d=DwQFaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=cRVZDAsVf8YMy7ncQBvRsw&m=7Sswtrfq7sOJMi3BChc7Z-D8KHRGfXMJiFJE2wyRKkFpE_O7uQYzl-TLkvLvWtbT&s=7rXfh2nGSFLZlFSXDt-BNDj3yyv7RDVOfCgqGp6TkJk&e=>,
>
> > but I don't see it there. Can you please explain? Thanks!
> >
> > —
> > Reply to this email directly, view it on GitHub
> > <
> https://github.com/mlfoundations/evalchemy/issues/41#issuecomment-2616521232
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_mlfoundations_evalchemy_issues_41-23issuecomment-2D2616521232&d=DwQFaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=cRVZDAsVf8YMy7ncQBvRsw&m=7Sswtrfq7sOJMi3BChc7Z-D8KHRGfXMJiFJE2wyRKkFpE_O7uQYzl-TLkvLvWtbT&s=q1OPD3D1jGM8hLEVaQNGBXQxA9osWli-KFXGmvVM_Z4&e=>>,
>
> > or unsubscribe
> > <
> https://github.com/notifications/unsubscribe-auth/AAIJX24XDN2G4NNQBJEIFND2MZXWDAVCNFSM6AAAAABTP4YEAGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMJWGUZDCMRTGI
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AAIJX24XDN2G4NNQBJEIFND2MZXWDAVCNFSM6AAAAABTP4YEAGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMJWGUZDCMRTGI&d=DwQFaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=cRVZDAsVf8YMy7ncQBvRsw&m=7Sswtrfq7sOJMi3BChc7Z-D8KHRGfXMJiFJE2wyRKkFpE_O7uQYzl-TLkvLvWtbT&s=h6u0WHsSMJWmO4fnXsIvX22LqJWcx7y6uH48K0vuwwc&e=>>
>
> > .
> > You are receiving this because you are subscribed to this thread.Message
> > ID: ***@***.***>
> >
>
> —
> Reply to this email directly, view it on GitHub
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_mlfoundations_evalchemy_issues_41-23issuecomment-2D2616747369&d=DwMFaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=cRVZDAsVf8YMy7ncQBvRsw&m=7Sswtrfq7sOJMi3BChc7Z-D8KHRGfXMJiFJE2wyRKkFpE_O7uQYzl-TLkvLvWtbT&s=n1bOVaA73VYUwRps8p8RCWZNxzVWNBnUNW7GmpNRr-E&e=>,
> or unsubscribe
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AL4G7QCYKSDQHHHZABISRT32M2EPBAVCNFSM6AAAAABTP4YEAGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMJWG42DOMZWHE&d=DwMFaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=cRVZDAsVf8YMy7ncQBvRsw&m=7Sswtrfq7sOJMi3BChc7Z-D8KHRGfXMJiFJE2wyRKkFpE_O7uQYzl-TLkvLvWtbT&s=WwGPFUBokE0KI4Z3CPVkhSq3LztYRcJwvjtCBYT5hng&e=>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>


Oh my bad :)

Sent via Superhuman ( ***@***.*** )

On Mon, Jan 27, 2025 at 12:16 PM, Ben Feuer < ***@***.*** > wrote:

> 
> I see LiveCodeBench in that PR, but not LiveBench. As your Readme
> correctly
> states, they're entirely different benchmarks.
> 
> On Mon, Jan 27 , 2025, 2:48 PM Mahesh Sathiamoorthy ***@***.***>
> wrote:
> 
> > This was the PR:
> >
> > https:/ / github. com/ mlfoundations/ evalchemy/ commit/ 76f72fea7613f776d2d5a14610f1a1b7d91f388a
> (
> https://github.com/mlfoundations/evalchemy/commit/76f72fea7613f776d2d5a14610f1a1b7d91f388a
> )
> > < https:/ / urldefense. proofpoint. com/ v2/ url?u=https-3A__github. com_mlfoundations_evalchemy_commit_76f72fea7613f776d2d5a14610f1a1b7d91f388a&d=DwQFaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=cRVZDAsVf8YMy7ncQBvRsw&m=7Sswtrfq7sOJMi3BChc7Z-D8KHRGfXMJiFJE2wyRKkFpE_O7uQYzl-TLkvLvWtbT&s=DtDDsP4PPxpAkdsNZxtqOkDSEtlstrhVL3936EtJ9-w&e=
> (
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_mlfoundations_evalchemy_commit_76f72fea7613f776d2d5a14610f1a1b7d91f388a&d=DwQFaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=cRVZDAsVf8YMy7ncQBvRsw&m=7Sswtrfq7sOJMi3BChc7Z-D8KHRGfXMJiFJE2wyRKkFpE_O7uQYzl-TLkvLvWtbT&s=DtDDsP4PPxpAkdsNZxtqOkDSEtlstrhVL3936EtJ9-w&e=
> ) >
> > I do see it.
> > [image: image.png]
> >
> > On Mon, Jan 27, 2025 at 9:58 AM Ben Feuer ***@***.***> wrote:
> >
> > > Hi all, on your README it says that LiveBench support is added and can
> 
> > be
> > > found in
> > >
> > https:/ / github. com/ mlfoundations/ evalchemy/ tree/ main/ eval/ chat_benchmarks
> (
> https://github.com/mlfoundations/evalchemy/tree/main/eval/chat_benchmarks
> )
> > < https:/ / urldefense. proofpoint. com/ v2/ url?u=https-3A__github. com_mlfoundations_evalchemy_tree_main_eval_chat-5Fbenchmarks&d=DwQFaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=cRVZDAsVf8YMy7ncQBvRsw&m=7Sswtrfq7sOJMi3BChc7Z-D8KHRGfXMJiFJE2wyRKkFpE_O7uQYzl-TLkvLvWtbT&s=7rXfh2nGSFLZlFSXDt-BNDj3yyv7RDVOfCgqGp6TkJk&e=
> (
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_mlfoundations_evalchemy_tree_main_eval_chat-5Fbenchmarks&d=DwQFaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=cRVZDAsVf8YMy7ncQBvRsw&m=7Sswtrfq7sOJMi3BChc7Z-D8KHRGfXMJiFJE2wyRKkFpE_O7uQYzl-TLkvLvWtbT&s=7rXfh2nGSFLZlFSXDt-BNDj3yyv7RDVOfCgqGp6TkJk&e=
> ) >,
> >
> > > but I don't see it there. Can you please explain? Thanks!
> > >
> > > —
> > > Reply to this email directly, view it on GitHub
> > > <
> > https:/ / github. com/ mlfoundations/ evalchemy/ issues/ 41#issuecomment-2616521232
> (
> https://github.com/mlfoundations/evalchemy/issues/41#issuecomment-2616521232
> )
> > < https:/ / urldefense. proofpoint. com/ v2/ url?u=https-3A__github. com_mlfoundations_evalchemy_issues_41-23issuecomment-2D2616521232&d=DwQFaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=cRVZDAsVf8YMy7ncQBvRsw&m=7Sswtrfq7sOJMi3BChc7Z-D8KHRGfXMJiFJE2wyRKkFpE_O7uQYzl-TLkvLvWtbT&s=q1OPD3D1jGM8hLEVaQNGBXQxA9osWli-KFXGmvVM_Z4&e=
> (
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_mlfoundations_evalchemy_issues_41-23issuecomment-2D2616521232&d=DwQFaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=cRVZDAsVf8YMy7ncQBvRsw&m=7Sswtrfq7sOJMi3BChc7Z-D8KHRGfXMJiFJE2wyRKkFpE_O7uQYzl-TLkvLvWtbT&s=q1OPD3D1jGM8hLEVaQNGBXQxA9osWli-KFXGmvVM_Z4&e=
> ) >>,
> >
> > > or unsubscribe
> > > <
> > https:/ / github. com/ notifications/ unsubscribe-auth/ AAIJX24XDN2G4NNQBJEIFND2MZXWDAVCNFSM6AAAAABTP4YEAGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMJWGUZDCMRTGI
> (
> https://github.com/notifications/unsubscribe-auth/AAIJX24XDN2G4NNQBJEIFND2MZXWDAVCNFSM6AAAAABTP4YEAGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMJWGUZDCMRTGI
> )
> > < https:/ / urldefense. proofpoint. com/ v2/ url?u=https-3A__github. com_notifications_unsubscribe-2Dauth_AAIJX24XDN2G4NNQBJEIFND2MZXWDAVCNFSM6AAAAABTP4YEAGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMJWGUZDCMRTGI&d=DwQFaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=cRVZDAsVf8YMy7ncQBvRsw&m=7Sswtrfq7sOJMi3BChc7Z-D8KHRGfXMJiFJE2wyRKkFpE_O7uQYzl-TLkvLvWtbT&s=h6u0WHsSMJWmO4fnXsIvX22LqJWcx7y6uH48K0vuwwc&e=
> (
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AAIJX24XDN2G4NNQBJEIFND2MZXWDAVCNFSM6AAAAABTP4YEAGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMJWGUZDCMRTGI&d=DwQFaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=cRVZDAsVf8YMy7ncQBvRsw&m=7Sswtrfq7sOJMi3BChc7Z-D8KHRGfXMJiFJE2wyRKkFpE_O7uQYzl-TLkvLvWtbT&s=h6u0WHsSMJWmO4fnXsIvX22LqJWcx7y6uH48K0vuwwc&e=
> ) >>
> >
> > > .
> > > You are receiving this because you are subscribed to this
> thread.Message
> > > ID: ***@***.***>
> > >
> >
> > —
> > Reply to this email directly, view it on GitHub
> > < https:/ / urldefense. proofpoint. com/ v2/ url?u=https-3A__github. com_mlfoundations_evalchemy_issues_41-23issuecomment-2D2616747369&d=DwMFaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=cRVZDAsVf8YMy7ncQBvRsw&m=7Sswtrfq7sOJMi3BChc7Z-D8KHRGfXMJiFJE2wyRKkFpE_O7uQYzl-TLkvLvWtbT&s=n1bOVaA73VYUwRps8p8RCWZNxzVWNBnUNW7GmpNRr-E&e=
> (
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_mlfoundations_evalchemy_issues_41-23issuecomment-2D2616747369&d=DwMFaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=cRVZDAsVf8YMy7ncQBvRsw&m=7Sswtrfq7sOJMi3BChc7Z-D8KHRGfXMJiFJE2wyRKkFpE_O7uQYzl-TLkvLvWtbT&s=n1bOVaA73VYUwRps8p8RCWZNxzVWNBnUNW7GmpNRr-E&e=
> ) >,
> > or unsubscribe
> > < https:/ / urldefense. proofpoint. com/ v2/ url?u=https-3A__github. com_notifications_unsubscribe-2Dauth_AL4G7QCYKSDQHHHZABISRT32M2EPBAVCNFSM6AAAAABTP4YEAGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMJWG42DOMZWHE&d=DwMFaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=cRVZDAsVf8YMy7ncQBvRsw&m=7Sswtrfq7sOJMi3BChc7Z-D8KHRGfXMJiFJE2wyRKkFpE_O7uQYzl-TLkvLvWtbT&s=WwGPFUBokE0KI4Z3CPVkhSq3LztYRcJwvjtCBYT5hng&e=
> (
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AL4G7QCYKSDQHHHZABISRT32M2EPBAVCNFSM6AAAAABTP4YEAGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMJWG42DOMZWHE&d=DwMFaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=cRVZDAsVf8YMy7ncQBvRsw&m=7Sswtrfq7sOJMi3BChc7Z-D8KHRGfXMJiFJE2wyRKkFpE_O7uQYzl-TLkvLvWtbT&s=WwGPFUBokE0KI4Z3CPVkhSq3LztYRcJwvjtCBYT5hng&e=
> ) >
> > .
> > You are receiving this because you authored the thread.Message ID:
> > ***@***.***>
> >
> 
> 
> —
> Reply to this email directly, view it on GitHub (
> https://github.com/mlfoundations/evalchemy/issues/41#issuecomment-2616798774
> ) , or unsubscribe (
> https://github.com/notifications/unsubscribe-auth/AAIJX2YUMLKOB4L7UTV3RQT2M2HY5AVCNFSM6AAAAABTP4YEAGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMJWG44TQNZXGQ
> ).
> You are receiving this because you commented. Message ID: <mlfoundations/evalchemy/issues/41/2616798774
> @ github. com>
> 
> 
>

Thanks for your interest! We will add it in a couple days @penfever 

@penfever LiveBench is added. Please try it out and let us know! Thanks!