Hugging Face's logo

openchat
/
openchat_3.5

Text Generation
Transformers
PyTorch
mistral
openchat
C-RLFT
conversational
Inference Endpoints
text-generation-inference
Model card Files Community
46

Why does this model perform so poorly on DROP compared to OpenHermes?

#29
by yahma - opened
Discussion
yahma

In the Huggingface Open LLM Leaderboard OpenChat performs really well on all the benchmarks except for DROP, where is scores 7.22 vs the 35.79 that OpenHermes-2.5-mistral scores.
Why such poor performance on DROP?

imone
 OpenChat org

Probably because open llm leaderboard doesn't use conversation templates and CoT, see discussion of gsm8k here. We manually tested the DROP examples and it worked really well.

imone changed discussion status to closed

Sign up or log in to comment