LM-PUB-QUIZ: A Comprehensive Framework for Zero-Shot Evaluation of Relational Knowledge in Language Models
Paper
•
2408.15729
•
Published
ML Chair Internal Org
university
AI & ML interests
None defined yet.
Recent Activity
View all activity
HU-Berlin-ML-Internal's activity
plonermaÂ
authored
2
papers
4 months ago
PatrickHallerÂ
updated
a
dataset
7 months ago
aynetdiaÂ
authored
a
paper
8 months ago
PatrickHallerÂ
posted
an
update
8 months ago
Post
1908
How Robust Is Your Model in Complex Code Generation Tasks? 🤔
We've launched the PECC benchmark to challenge chat models in code generation, drawing from the Advent of Code for programming tasks and the Euler Project for math-heavy challenges. This new task tests models with problems presented in both detailed prose and concise "leet code" styles, evaluating their ability to understand and solve complex coding issues and math problem in chat-based interactions.
It seems that the Claude 3 models outperforme ChatGPT:
Model / Avg. (pass@3)
Claude 3 Haiku / 27.67
GPT-3.5-Turbo / 23.75
Mixtral-8x22B-Instruct-v0.1 / 8.35
Read our Preprint📃: PECC: Problem Extraction and Coding Challenges (2404.18766)
Look at the dataset🔎: PatrickHaller/pecc
We also got accepted at LREC-COLING '24 🎉
We've launched the PECC benchmark to challenge chat models in code generation, drawing from the Advent of Code for programming tasks and the Euler Project for math-heavy challenges. This new task tests models with problems presented in both detailed prose and concise "leet code" styles, evaluating their ability to understand and solve complex coding issues and math problem in chat-based interactions.
It seems that the Claude 3 models outperforme ChatGPT:
Model / Avg. (pass@3)
Claude 3 Haiku / 27.67
GPT-3.5-Turbo / 23.75
Mixtral-8x22B-Instruct-v0.1 / 8.35
Read our Preprint📃: PECC: Problem Extraction and Coding Challenges (2404.18766)
Look at the dataset🔎: PatrickHaller/pecc
We also got accepted at LREC-COLING '24 🎉
PatrickHallerÂ
authored
2
papers
8 months ago
PatrickHallerÂ
authored
3
papers
9 months ago
OpinionGPT: Modelling Explicit Biases in Instruction-Tuned LLMs
Paper
•
2309.03876
•
Published
•
2
Fabricator: An Open Source Toolkit for Generating Labeled Training Data with Teacher LLMs
Paper
•
2309.09582
•
Published
•
4
BigBIO: A Framework for Data-Centric Biomedical Natural Language Processing
Paper
•
2206.15076
•
Published
•
3
aynetdiaÂ
authored
a
paper
10 months ago