zhiminy/EvalEng huggingface dataset:
Repository: bigcode-project/bigcode-evaluation-harness
Issue: improve the prompt examples of one-shot setting in APPS evaluation
Root cause: Unimplemented Feature Gap

Models are usually evaluated on APPS after fine-tuning on the train split, but one can also do few-shot evaluation. It is already implemented in this evaluation harness: the prompt includes two shortened [examples](https://github.com/bigcode-project/bigcode-evaluation-harness/blob/main/lm_eval/few_shot_examples/apps_few_shot_prompts.json) from the train split one for each call type (Standard Input and Call based). 

We want to improve these examples: 
* analyse the different question types of APPS and build 2 or 3 examples to cover these types (make sure they aren't in the test set) 
* see how models behave given different examples (you can play with the model demos/spaces in this [org](https://huggingface.co/codeparrot) there's codeparrot, incoder and codegen)
* the prompt shouldn't end up being too long 

There are different levels of problem in the APPS dataset (e.g., introductory and interview levels). Do we want a particular prompt for each level, or do we want a level-agnostic prompt? Maybe both would be interesting. 

Hi, you can try both and see if it makes a difference, but I think the difference might depend on how the question is formulated rather than the difficulty level, question format can be different based on the source of the file (codeforces, hackerank..) even for similar difficulty levels. 

Any experiment are welcome to see what generalizes better.

Hello, regarding this I have run a couple of experiments on CodeGen-mono and multi after they have been finetuned on the train split of this [dataset](https://huggingface.co/datasets/giulio98/xlcost-single-prompt), I did the evaluation on the first half of test split(400 samples) on different scenarios:
* zero-shot
* few-shot: where two examples were taken randomly from the second half of the test.
What I noticed though is a drop in the performance for pass@1 and pass@10 metric in the few-shot setting
![image](https://user-images.githubusercontent.com/79860892/202536628-52efba8a-2ed3-4e64-a15a-86be051211ef.png)
![image](https://user-images.githubusercontent.com/79860892/202536696-75e14964-7e3e-47ce-bdb3-d303820b82a6.png)
could it be due to the fact that I randomly selected the examples to put in the context? or the model I'm using (350M) is too "small" to observe few-shot capability?



I think it could be either of the two possibilities, but few shot examples should be chosen carefully as well as the way you embed them in the prompt so that they help the model understand the context and task. 

You can try building some few shot prompts and test how the model behaves on them before considering using them for evaluation. For easy experimentation you can use this [demo](https://huggingface.co/spaces/codeparrot/codegen-subspace) for codegen-2B for example, or build one for codegen-350M-multi with Gradio.
Repository: bigcode-project/bigcode-evaluation-harness
Issue: Reproducing the performance of HumanEval on starcoder
Root cause: Documentation Deficiency

Thank you for providing an excellent evaluation toolkit! It is very convenient and flexible.

But when I used the evaluation tool to evaluate the HumanEval performance on the statcoder, I obtained the following results.
```
{
  "humaneval": {
    "pass@1": 0.3011280487804878,
    "pass@10": 0.41708568124396794,
    "pass@100": 0.5175640419344132
  },
  "config": {
    "model": "../ckpt/starcoder",
    "temperature": 0.2,
    "n_samples": 200
  }
}
```

It is lower than the paper result pass@1 is 33.6. Did I miss anything crucial? All parameters are default.

Thanks for the feedback! You're using StarCoder and not StarCoderBase, right? Can you share your execution command?  This should reproduce the result
```bash
accelerate launch  main.py \
  --model bigcode/starcoder \
  --max_length_generation 512 \
  --tasks humaneval \
  --n_samples 50 \
  --batch_size 50 \
  --temperature 0.2 \
  --precision bf16 \
  --allow_code_execution \
  --use_auth_token
  ```
  EDIT:
  I just run
  ```
  accelerate launch  main.py   --model bigcode/starcoder   --max_length_generation 512  --tasks humaneval   --n_samples 200   --batch_size 100   --temperature 0.2   --precision bf16   --allow_code_execution   --use_auth_token
```
  And it returns
  ```
  {
  "humaneval": {
    "pass@1": 0.3357317073170732,
    "pass@10": 0.4896174189684954,
    "pass@100": 0.6159876811517714
  },
  "config": {
    "model": "bigcode/starcoder",
    "temperature": 0.2,
    "n_samples": 200
  }
}
```
which matches the paper result

It works! Thanks for your patient response.
I have another question, what should I do if I want to evaluate humaneval using Starcoder's FIM mode? It seems that there is no corresponding CLI to activate it. 

I found that there are related operations in:

https://github.com/bigcode-project/bigcode-evaluation-harness/blob/e2072f2e444bfc8f726326cf38050ca9e721fa94/lm_eval/utils.py#L46

Here is based on the type of prompt, which means I need to make changes to the `get_prompt` function in 

https://github.com/bigcode-project/bigcode-evaluation-harness/blob/e2072f2e444bfc8f726326cf38050ca9e721fa94/lm_eval/tasks/humaneval.py#L46

Am I right ？

You will need to another  task (see [guide](https://github.com/bigcode-project/bigcode-evaluation-harness/blob/main/docs/guide.md)) for the FIM humaneval dataset . In particular, you would need to change the prompt and post-processing of the solution, check this issue https://github.com/bigcode-project/bigcode-evaluation-harness/issues/69 for pointers about FIM evaluation we did for SantaCoder in another codebase. We do support FIM mode in generation though through the code you shared and it is currently used in the [DS-1000](https://github.com/bigcode-project/bigcode-evaluation-harness/blob/main/lm_eval/tasks/ds1000.py) tasks)

Great, I see. Thanks for your help again !
Repository: bigcode-project/bigcode-evaluation-harness
Issue: santacoder fp16 causes NaN on humaneval?
Root cause: Resource Mishandling

Just wondering if we need to use fp32 for evaluation of santacoder?
I tried fp16 evaluation because I fine-tuned santacoder on the stack-dedup python dataset for 1000 steps with fp16 precision. But when I ran fp16 evaluation on humaneval, it leads to the following error (for both `--model=bigcode/santacoder` and `--model=myfp16_finetuned_santacoder`),  

```
File "/home/ywen/miniconda3/lib/python3.9/site-packages/transformers/generation/utils.py", line 2583, in sample
    next_tokens = torch.multinomial(probs, num_samples=1).squeeze(1)
RuntimeError: probability tensor contains either `inf`, `nan` or element < 0
```

The error went away if I use `--precision=fp32`, leading to 37.19% pass@100 on humaneval which is kinda close to the number reported in the paper. This is the command I used to run fp16 evaluation on humaneval.

```
accelerate launch main.py \
    --model bigcode/santacoder \
    --max_length_generation 368 \
    --tasks humaneval \
    --temperature 0.4 \
    --n_samples 100 \
    --batch_size 20 \
    --allow_code_execution \
    --trust_remote_code \
    --use_auth_token \
    --generation_only \
    --precision fp16 \
    --save_generations 
```

Yes SantaCoder had some issues with `fp16`so it's better to either use `bf16` or `fp32` with it

> Yes SantaCoder had some issues with `fp16`so it's better to either use `bf16` or `fp32` with it

I encountered the same error while evaluating Humaneval on [meta-llama/Llama-2-7b-chat-hf](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf/tree/main) using fp16 precision. However, I was able to complete the evaluation for the CodeLlama series model and [meta-llama/Llama-2-7b-hf](https://huggingface.co/meta-llama/Llama-2-7b-hf/tree/main) without issues. Could you please explain the reasons behind it and the impact of changing the precision setting from fp16 to fp32 on the pass@k score? Thank you very much for your assistance!
Repository: bigcode-project/bigcode-evaluation-harness
Issue: Publish the Docker images to ghcr.io?
Root cause: Unimplemented Feature Gap

It is possible to pre-build and publish the Docker images to ghcr.io, so that everyone doesn't need to rebuild them. I mean something like this:

https://github.com/orgs/huggingface/packages?repo_name=text-generation-inference

I am happy to do this. I think I have permissions to publish packages. But, I want to confirm with @loubnabnl  or someone before I do.

That would be great Arjun!

I need a project admin to enable this setting :) 

![image](https://github.com/bigcode-project/bigcode-evaluation-harness/assets/20065/da534b7a-df10-4095-9021-e32918960531)

I've pushed both containers here, but they are only available by members of the bigcode-project organization:

https://github.com/orgs/bigcode-project/packages

the packages are now public
Repository: bigcode-project/bigcode-evaluation-harness
Issue: Support `transformers.pipeline(model=...)` models like `HuggingFaceH4/starchat-beta`
Root cause: Environment Incompatibility

I couldn't load [`HuggingFaceH4/starchat-beta`](https://huggingface.co/HuggingFaceH4/starchat-beta) using our existing codebase on a single machine with multiple 40GB GPUs. 
See loading instructions on the [model card](https://huggingface.co/HuggingFaceH4/starchat-beta).

I believe this can be fixed by running `pip install --upgrade transformers`

Yes like arjun said this isn't an evaluation harness issue you need to upgrade your transformers version to `>=4.28.1` where the `gpt_bigcode` architecture was introduced to be able to load the model.
Repository: bigcode-project/bigcode-evaluation-harness
Issue: spider zero-shot evaluation with execution acuuracy metric
Root cause: Unimplemented Feature Gap

Spider is currently two-shot using BLEU score, we want to add zero-shot evaluation with execution accuracy metric similarily to [Picard](https://github.com/ServiceNow/picard) and this [paper](https://arxiv.org/pdf/2204.00498.pdf).
Repository: bigcode-project/bigcode-evaluation-harness
Issue: 8-bit models unsupported
Root cause: Validation Gap

Currently, the harness raises an exception when used with 8-bit models:
```
Traceback (most recent call last):
  File "bigcode-evaluation-harness/main.py", line 233, in <module>
    main()
  File "bigcode-evaluation-harness/main.py", line 216, in main
    results[task] = evaluator.evaluate(task)
  File "bigcode-evaluation-harness/lm_eval/evaluator.py", line 67, in evaluate
    generations, references = self.generate_text(task_name)
  File "bigcode-evaluation-harness/lm_eval/evaluator.py", line 45, in generate_text
    generations = parallel_generations(
  File "bigcode-evaluation-harness/lm_eval/generation.py", line 83, in parallel_generations
    model = model.to(accelerator.device)
  File "/root/venv/lib/python3.10/site-packages/transformers/modeling_utils.py", line 1873, in to
    raise ValueError(
ValueError: `.to` is not supported for `8-bit` models. Please use the model as it is, since the model has already been set to the correct devices and casted to the correct `dtype`.
```

for context, this is the model i've been trying to eval: https://huggingface.co/cassanof/santacoder-lua/tree/main

seems like a check is needed for every .to call... any suggestions?

The error comes from how we place models in gpus [here](https://github.com/bigcode-project/bigcode-evaluation-harness/blob/4eb71bb37e15605e803b25d0c47dbe76ed6bfaa9/lm_eval/generation.py#L94) which isn't supported for 8bit models, you can replace that line by also wrapping the model in `accelerator.prepare()` along with the dataloader when the model is in 8bit -now supported in accelerate [see](https://github.com/huggingface/accelerate/pull/1155)- (we avoid it for other precisions because it seemed to take a lot of memory for gradients.. see this [issue](https://github.com/huggingface/accelerate/issues/1336#issuecomment-1516696203))
cc @ArmelRandy 

Support for 8bit inference is added in this PR https://github.com/bigcode-project/bigcode-evaluation-harness/pull/95, however there are issues when using SantaCoder in fp16 for inference and that also happens in 8bit, top-sampling in particular throws the error mentioned in this [issue](https://github.com/bigcode-project/bigcode-evaluation-harness/issues/83) in fp16 and 8bit (doesn't happen in fp32 and bf16) we don't quite know why it happens, it could be instabilities due to the pre-training in fp16 of this model.


I see. Thanks! 
>however there are issues when using SantaCoder in fp16 for inference and that also happens in 8bit, top-sampling in particular

Is this SantaCoder with the original architecture or with StarCoder's architecture?

That was with SantaCoder's architecture, haven't tried gpt_bigcode-santacoder. (I also had the error with your fine-tuned model)
Repository: bigcode-project/bigcode-evaluation-harness
Issue: APPS dataset prompting seems wrong
Root cause: Algorithmic Error

In the original APPS paper and their [original code](https://github.com/hendrycks/apps/blob/main/eval/generate_gpt_codes.py#L69), Standard Input format is used when fn_name is not given.

But, in [here](https://github.com/bigcode-project/bigcode-evaluation-harness/blob/main/lm_eval/tasks/apps.py#L81), Standard is used when fn_name is given.

Thanks for the catch! It seems they were also swapped in the fine-tuning [code](https://github.com/bigcode-project/bigcode-evaluation-harness/blob/0fec5e99b4e0578ca50ed4c4b6b7a7744bf31610/finetuning/APPS/apps_dataset.py#L53) which is why we didn't observe performance discrepancy This [PR](https://github.com/bigcode-project/bigcode-evaluation-harness/pull/93) fixes it
Repository: bigcode-project/bigcode-evaluation-harness
Issue: Getting Zeros for StarCoder on multiple-js
Root cause: Environment Incompatibility

I am running the following : 
```
accelerate launch  main.py   \
  --model bigcode/starcoder   \
  --max_length_generation 512  \
  --tasks multiple-js   \
  --n_samples 120  \
  --batch_size 10  \
  --temperature 0.2  \
  --precision bf16  \
  --allow_code_execution   --use_auth_token
```

The results is : 

```
{
  "multiple-js": {
    "pass@1": 0.0,
    "pass@10": 0.0,
    "pass@100": 0.0
  },
  "config": {
    "model": "bigcode/starcoderbase",
    "temperature": 0.1,
    "n_samples": 120
  }
}
```


Is their any other parameters that I might be missing ?

Did you run the execution in a docker with all installed dependencies?

I'm going to guess that this is a Node version issue. The MultiPL-E JS benchmarks rely on deepEqual which require a fairly recent version of Node. I think the version in Ubuntu 20.04 is too old, but it works in Ubuntu 22.04.

I was trying it on my local setup with all the dependencies. Let me try and use the docker. 
Can you please confirm that the command/hyper-parameter shared above are correct. @loubnabnl @arjunguha 

Correct. You may get something slightly lower than what is reported in the paper. The original MultiPL-E code (github.com/nuprl/MultiPL-E) uses length 512, but interprets it as `len(prompt_tokens) + 512`. The evaluation harness I believe includes the prompt in the 512 tokens. So, you may need to increased it.

I was able to reproduce the results. Thanks
Repository: bigcode-project/bigcode-evaluation-harness
Issue: failed evaluation on GSM8K
Root cause: Documentation Deficiency

I try to run your code in a docker container from `ghcr.io/bigcode-project/evaluation-harness`.

The exact bash command is 

```bash
accelerate launch  main.py \
  --model bigcode/starcoder \
  --use_auth_token \
  --max_length_generation 512 \
  --tasks pal-gsm8k-greedy \
  --n_samples 1 \
  --temperature 0 \
  --batch_size 1 \
  --do_sample False \
  --allow_code_execution \
  --save_generations \
  --save_generations_path ./output/starcoder_on_gsm8k.json
```

However, it returns the following:

```
Evaluating generations...
{
  "pal-gsm8k-greedy": {
    "accuracy": 0.0,
    "num_failed_execution": 1319
  },
  "config": {
    "model": "bigcode/starcoder",
    "revision": null,
    "temperature": 0.0,
    "n_samples": 1
  }
}
```

where the saved generation contents are like:
<img width="1436" alt="Screenshot 2023-06-20 at 14 37 40" src="https://github.com/bigcode-project/bigcode-evaluation-harness/assets/36271700/35db9a7f-6043-4c90-bdf0-a2584f8dca6d">

Any solutions?

I believe you have to run with `--use_auth_token` as well as `--trust_remote_code` for models like starcoder since you need to agree to the terms to use them. I do believe it would be better for the evaluation to throw and error instead of running with these erroneous generations. 

> I believe you have to run with `--use_auth_token` as well as `--trust_remote_code` for models like starcoder since you need to agree to the terms to use them. I do believe it would be better for the evaluation to throw and error instead of running with these erroneous generations.

I've try these, but the problem remains same. I think it may result from the parallel_generation and it's the reproduction failure which shall be addressed by the official repo team.

It turns out to be max_length issue, where instead of 512, we should choose 2048 for this task. 

Many fresh guys don't know how to choose an appropriate max_length, maybe the document should provide a default length.

@tangzhy , Thank you for your inputs. The Prompts for GSM8K take up ~1500 tokens. The max_length has to be greater than that. We will update the docs to make this clear.
Repository: confident-ai/deepeval
Issue: Add LLMEvalMetric
Root cause: Unimplemented Feature Gap

Prompt to use for LLMEvalMetric 

```
We provide a question and the 'ground-truth' answer. We also provide \
the predicted answer.

Evaluate whether the predicted answer is correct, given its similarity \
to the ground-truth. If details provided in predicted answer are reflected \
in the ground-truth answer, return "YES". To return "YES", the details don't \
need to exactly match. Be lenient in evaluation if the predicted answer \
is missing a few details. Try to make sure that there are no blatant mistakes. \
Otherwise, return "NO".

Question: {question}
Ground-truth Answer: {gt_answer}
Predicted Answer: {pred_answer}
Evaluation Result: \
```

As featured from this guide: 

https://gpt-index.readthedocs.io/en/latest/examples/finetuning/knowledge/finetune_knowledge.html 


Done #161
Repository: confident-ai/deepeval
Issue: Add RAGAS metrics to DeepEval
Root cause: Unimplemented Feature Gap

Add RAGAS metrics to DeepEval. 
Key metrics that would be useful:
- Ragas Score (Harmonic mean) 
- Context Recall (for the retriever)

https://github.com/explodinggradients/ragas 

Hey guys Jithin from the Ragas team here
excited about this integration that you guys are doing and would love to offer any help to make this possible 🙂 

cheers

Thanks @jjmachan !! Would love your help in adding RAGAS Score metric here. 
One thing we need to do is provide a binary score threshold if it's a success/failure though. Would you be able to help submit a PR? We have a guide on writing a custom metric here: https://docs.confident-ai.com/docs/quickstart/custom-metrics

Here's an example for Bias you can take a look at if you want to also add ragas as a package to install: https://github.com/confident-ai/deepeval/blob/main/deepeval/metrics/bias_classifier.py 

When contributing, we just use black for now :) 

don't know when I'll get some cycles but when I do, I'll get to this :) . In the meantime, if getting on a call and helping design the implementation is useful to make it faster, I would love to!

No worries - I'd love to get on a call to discuss this! Do you have a calendly link? Otherwise - please feeli free to book something in my Calendly here: https://calendly.com/jacky-twilix/30min 

FYI - I started the PR here #121   @jjmachan :) 

awesome @ColabDog , excited to see that it is merged and try this out shortly. I've also booked a lot too

Sweet - looking forward to chatting. I wrote a bit of documentation here: https://docs.confident-ai.com/docs/measuring_llm_performance/ragas_score 

Would love for you to take a look and let me know if I missed anything! 

Closing this issue for now that since the documentation is written.
Repository: confident-ai/deepeval
Issue: [Question] What's the difference between Deepeval's RAG Evaluation and RAGAS

 Deepeval have rag evaluation https://docs.confident-ai.com/docs/guides-rag-evaluation
And another repo can do rag evaluation: https://github.com/explodinggradients/ragas

**What's the difference between the two in rag evaluation?**

@landhu similar implementation, more support into deepeval's ecosystem (concurrent execution, caching, verbose logs for debugging, integrate into Confident AI, etc)
Repository: confident-ai/deepeval
Issue: Relaxe grpc-io version 
Root cause: Architectural Constraint

Can you updated the following requirement?

`deepeval (>=0.21.61,<0.21.78 || >0.21.78,<0.22.0) requires grpcio (1.63.0)` or at least build the pip package from poetry instead of setup.py ?


Bumping this as grpcio 1.66 is required for python 3.13
https://github.com/grpc/grpc/issues/36201#issuecomment-2400217038

@kritinv any ideas if this is possible? it is required for python 3.13 support. I think you might be able to get away with removing `grpcio` from the requirements, since it is not directly used in your code 

https://github.com/search?q=repo%3Aconfident-ai%2Fdeepeval%20grpcio&type=code

linked: https://github.com/confident-ai/deepeval/pull/1228
Repository: confident-ai/deepeval
Issue: g_eval metric implicitly falls back to top score instead of sum of weighted top probabilities 
Root cause: Unimplemented Feature Gap

https://github.com/confident-ai/deepeval/blob/e49714ddd0fe523d27ef4c8b2477eeaeb7128ef2/deepeval/metrics/g_eval/g_eval.py#L315

It seems that the "except" clause here can cause unexpected results since the evaluation of the probabilities can fail, but the code will still return a score. 

In my opinion it would be good to give the user of the metric an ability to choose between:

1.  logprobs sum mode.  
2. top token mode (for cases where the logprobs vector is not transparent in the evaluator model, or if the user simply doesn't care about the quality guarantee in the paper, which is specified for the probability sum method). 



I understand that in some cases the evaluator model doesn't give access to the logprobs vector. In that case, if the user selected logprobs sum mode I would like them to see an error explaining that the probabilities are not available, and to use top token mode instead.


If there is agreement amongst the core team, I would be happy to contribute this functionality

Hey @mattany that would be extremely helpful, would really appreciate a PR :)
Repository: confident-ai/deepeval
Issue: Typo in docs/docs/benchmarks-introduction.mdx

**Describe the bug**
> A clear and concise description of what the bug is.

`promtps` shows up twice in the Markdown file located at https://github.com/confident-ai/deepeval/blob/main/docs/docs/benchmarks-introduction.mdx.

**To Reproduce**
Go to https://docs.confident-ai.com/docs/benchmarks-introduction#create-a-custom-llm. The `batch_generate` function contains the typos.

**Screenshots**
If applicable, add screenshots to help explain your problem.
![image](https://github.com/user-attachments/assets/48a40fa4-7875-4d0e-b301-d52eab975bcc)