NousResearch

/

Hermes-2-Pro-Mistral-7B

like 467

Hi! Are the datasets used for function calling and JSON available somewhere?

Not yet

Why does this model perform less than capybara hermes 2.5 on json evaluation?
I get 1.8 on capybara and 2.9 at hermes 2 pro

what evaluation framework/dataset are you using for json?

I made a combination of my auto-gpt logs and the teknium json eval.parquet which I parsed into json than txt.
I am using oobabooga text gen training-pro plugin with perplexity evaluation.

The JSON from capybara is better mostly but if often forgets delimiters or ending brackets.
Hermes 2 pro though just stops the json mid generation or makes out of context ones.

Currently I use them both for gpt-pilot and my own mini auto-gpt.

你好