zhiminy/EvalEng huggingface dataset:
Repository: Cloud-CV/evalai-cli
Issue: Pip install errors
Root cause: Environment Incompatibility

Hi EvalAI team,

I am trying to install the evalai-cli pip package in a virtual environment, but get the below error. I have tested this on Windows and WSL Ubuntu 22.04 with Python 3.10.12, and get the same issue with both OSs:

```
      Compile failed: command 'x86_64-linux-gnu-gcc' failed: Permission denied
      creating tmp
      cc -I/usr/include/libxml2 -I/usr/include/libxml2 -c /tmp/xmlXPathInitvrl2vdqq.c -o tmp/xmlXPathInitvrl2vdqq.o
      *********************************************************************************
      Could not find function xmlCheckVersion in library libxml2. Is libxml2 installed?
      *********************************************************************************
      error: command 'x86_64-linux-gnu-gcc' failed: Permission denied
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> lxml
```

I have tried to install various package from searching online via the commands below, but get the same issue still:
```
sudo apt-get install libpq-dev libxml2-dev libxmlsec1-dev python3-lxml libxml2-dev libxslt-dev 
```

Could you advise on how to properly install the `evalai-cli` pip package. 

Thank you!



Solved,

Not sure which install did the trick but here is what I did:

```
sudo apt-get install python3-pip python3-dev
sudo apt-get install libpq-dev libxml2-dev libxmlsec1-dev libxml2-dev libxslt-dev libxslt1-dev python3-lxml
sudo python -m pip install lxml
python -m pip install evalai
```
Repository: Cloud-CV/evalai-cli
Issue: Can not push leaderboard test docker to EvalAI platform
Root cause: External Dependency Breakage

Hi, I‘m trying to test my CARLA agent on the CARLA leaderboard 1.0 ([CARLA Autonomous Driving Challenge 2.0](https://eval.ai/web/challenges/challenge-page/2098/overview)), for which I have obtained permission from the organizers to compete. I have installed the latest version of the EvalAI package (1.3.16) and completed the "set token" operation.

However, when attempting to use the "evalai push" command to submit the required Docker image to the leaderboard, I encountered an error message "Error: None". Additionally, I tried other commands such as "evalai challenges," but received a "404 Client Error: Not Found for url: https://eval.ai/api/challenges/challenge/all" error.

I have attempted these operations on both a Linux machine and my personal Mac, and encountered the same issues on both platforms. Upon searching the EvalAI forum, I discovered that other users have also encountered similar problems, which can be found here: https://evalai-forum.cloudcv.org/t/evalai-push-error-or-am-i-just-a-noob-with-docker/1773

I tried to send an email to the EvalAI team but got no response, so I wonder if you could give me any advice, I would be very appreciate!

My commands for push the docker to the platform:

pip install evalai
evalai set_token {MY_EVALAI_TOKEN}
evalai push {MY_DOCKER_IMAGE} --phase carla-leaderboard-10-sensors-2098

<img width="565" alt="image" src="https://github.com/Cloud-CV/evalai-cli/assets/45144254/659c4dab-a706-4174-9495-1d8b19186511">


Commenting here to let you know -- I got around this by making a brand new account with a brand new email.
The CLI worked with the new token. Maybe old tokens lurking around the system are incompatible
Repository: Cloud-CV/evalai-cli
Issue: Storing the project settings in a config file.
Root cause: Unimplemented Feature Gap

Right now, there are some user-specific settings that need to be configured to use the CLI with different instances of the EvalAI. Some of them are:

* Host URL.
* Token File Location 
* Token File Name

The default for Host URL would be the `localhost:8000`, and the user could change it to their host to use different instances of EvalAI.

Solved #41
Repository: Cloud-CV/evalai-cli
Issue: Add proper exception handling 
Root cause: Validation Gap

the file :https://github.com/Cloud-CV/evalai-cli/blob/master/evalai/submissions.py.

here we have several places where we try to fetch entries from the HTTP responses.
We want to add proper exception handling in case we do not have the keys present in the response

@AishwaryJain07 This seems to be a good issue , I would like to work on this , please assign it to me
Repository: Cloud-CV/evalai-cli
Issue: Enhance Security Integration evalai-cli with OSS-Fuzz for Continuous Fuzz Testing

Hii @gchhablani

I hope this message finds you well! I’d like to propose integrating evalai-cli with OSS-Fuzz continuous fuzz testing platform designed to improve the stability and security of open-source project.

Why Integrate EvalAI with OSS-Fuzz?

Improved Stability: OSS-Fuzz can automatically detect edge-case bugs, crashes, and security vulnerabilities in evalai-cli codebase.
Enhanced Reliability: Continuous fuzzing ensures that untrusted inputs, such as data from sensors, communication protocols, or user-defined configurations, are handled robustly.
Proactive Bug Fixes: By identifying potential issues early, OSS-Fuzz helps maintain a stable and secure codebase.

Can I move Forward with this?
Repository: Cloud-CV/evalai-cli
Issue: Modify docstrings in functions
GitHub labels: enhancement

docs

**Current Scenario:** Tells the functioning of the function in a single line.

**Deliverables:**
We should add doc strings answering the following points:
1. Args : What argument a function accepts
2. Returns: What is the final output of the function
3. Raises: What exception it raises
4. Command (Optional): What is command used to invoke it.

@varunagarwal @deshraj What do you guys say?

Totally agree with what @RishabhJain2018 said. @isht3 can you please get this done quickly. This will be a bonus for our CLI. 

Yes, will get started on this.

#87 Fixes this.
Repository: Cloud-CV/evalai-cli
Issue: Add test cases to cover the complete codebase
GitHub labels: enhancement

priority-high

Since the coverage is `72%` which is quite low, hence we should add test cases in order to cover all the code, before merging the new PR's.

**Deliverables:** Increase the coverage to `90%`.

@isht3 Please try to fix this asap before working on the existing PR's as it would be hard to increase the coverage once the codebase increases and we will also not be able to figure out if things would broke while merging new PR's.
Repository: Cloud-CV/evalai-cli
Issue: Auto Docs Generation
GitHub labels: good first issue

dev-ops

By their very nature, CLI tools are harder to figure out compared to GUI tools. We will need really good docs to ensure the `evalai-cli` too gains traction among users.

It would be great to have automatic documentation generation from doc strings as a starting point so we can add more detailed documentation later.

Hi, should we use Sphinx for auto docs generation.

Both Sphinx (specifically `autodoc`) and `pydoc` should suffice.
I would check to see if Sphinx supports Markdown files since reStructuredText is not used.
Repository: Cloud-CV/evalai-cli
Issue: Documentation milestones.
Root cause: Documentation Deficiency

# Write docs for.

- [x] Welcome page.
- [x] Installation.
- [x] Getting the token.
- [x] Connecting CLI to the host.
- [x] Basic commands overview.
- [x] View Challenge, Phase, Phase Splits. 
- [x] Create a participant team.
- [x] Participate in a challenge.
- [x] Make a submission.
- [x] View status of a submission.
- [x] View all your submission.
- [x] View the leaderboard.
- [x] Directory Structure.
- [x] Contributing guidelines.
- [x] Setting theme for the doc.



#59 #77 resolves this. Closing it now.[](url)
Repository: Cloud-CV/evalai-cli
Issue: Create challenge configuration using CLI
Root cause: Unimplemented Feature Gap

Users should be able to create a challenge configuration zip using CLI. The idea is to have a similar setup that Sphinx provides when you want to generate the documentation configuration. It should work something like this:

Command: `evalai create_challenge`

```
$ evalai create challenge 

Enter the challenge name:
Test Challenge

Enter the Number of Phases:
2

Enter the Phase 1 name:
Test Phase

Enter the Phase 1 codename:
test-phase
.
.
.
.
.
.
```

CC: @isht3 @varunagrawal @RishabhJain2018 

This should be implemented only after the primary features are done and the CLI is finished, yes?

Yes, that's correct.

Hi @deshraj Can I work on this issue?
Repository: Cloud-CV/evalai-cli
Issue: Display Error properly when challenge does not exists.
Root cause: Validation Gap

This [line]( https://github.com/Cloud-CV/evalai-cli/blob/a23fdf05b116b39278f78b55cd1a4af55f7a2c1b/evalai/utils/challenges.py#L227) will give a `KeyError` upon running the command:
`evalai challenges 340 phases` where 340 is the pk of challenge that doesn't exists.

@isht3 Can you please verify this ?



#58 solves this
Repository: Cloud-CV/evalai-cli
Issue: Auth Token value gets reset
Root cause: Resource Mishandling

The token which is placed at `~/.evalai/token.json` gets reset whenever a user runs test cases on his local machine.

**Steps to reproduce:**
1. Add a valid token at `~/.evalai/token.json`.
2. Run the test cases using `py.test`.
3. Check the value of token at `~/.evalai/token.json`

The token `value at step 1 != value at step 3`.

#92 Fixes this.
Repository: Cloud-CV/evalai-cli
Issue: Development templates
GitHub labels: dev-ops

docs

We should have templates that describe to new contributors what the expectations of the project are. This requires putting the files in a `.github` folder to allow Github to render them appropriately. Below is a list of files that will be needed to close this issue.

1. [x] - `CONTRIBUTING.md`
2. [x] - `ISSUE_TEMPLATE.md`
3. [x] - `PULL_REQUEST_TEMPLATE.md`

@varunagrawal I think we should also add the other two remaining templates.

@RishabhJain2018  The issue and pull request template has been added in #107 & #108 .

Closing this since it is already fixed!
Repository: Cloud-CV/evalai-cli
Issue: Abstracting the request pattern.
GitHub labels: enhancement

help wanted

good first issue

GSOC-2019

Currently, most of the request function use the same format of http request pattern except a few. 

```
    headers = get_request_header()

    try:
        response = requests.get(url, headers=headers)
        response.raise_for_status()
    except requests.exceptions.HTTPError as err:
        if (response.status_code in EVALAI_ERROR_CODES):
            validate_token(response.json())
            echo(style("Error: {}".format(response.json()["error"]), fg="red", bold=True))
        else:
            echo(err)
        sys.exit(1)
    except requests.exceptions.RequestException as err:
        echo(err)
        sys.exit(1)
```
This could be simplified easily by abstracting this functionality as a function.

@isht3 @RishabhJain2018 Can I work on this issue?

@mahim23 Sure! Once you've completed your previous issue then you can work on this. Till then it is open for others to work on.
Repository: Cloud-CV/evalai-cli
Issue: Challenge Stats should not be shown to the participant users.
GitHub labels: bug

enhancement

good first issue

**Current Scenario:**
Challenge statistics is visible to both challenge hosts and challenge participants.


**Deliverabes**:
The challenge statistics should only be visible to challenge host.

@RishabhJain2018  is this issue open?


hey im unable to reproduce the issue. Could someone help me @RishabhJain2018  @isht3 

@darthgera123 Okay, please try requesting for the stats from both the host account and the participant account. (Use token from the participant/host accounts.)

The stats would give us the stats info for both participant and the host alike. We would like to stop showing the stats to the participant. 

Did you try this?

currently i can only get the api token from admin and not from any other source. Plus `evalai challenges 1 stats` doesnt work as well.


@darthgera123 That's because this feature is a work in progress on this branch https://github.com/Cloud-CV/evalai-cli/pull/86 Please feel free to start working on it. I'll help you get familiar with the parts, no worries. 😃 

@darthgera123 You can also get the auth token from the profile page of the user.

sure, ill look into it. Thanks a lot btw
Repository: Cloud-CV/evalai-cli
Issue: Remove challenge start & end date
GitHub labels: enhancement

help wanted

good first issue

Remove challenge start & end date when viewing the description for a single challenge.
<img width="1403" alt="screen shot 2018-08-28 at 11 54 45 am" src="https://user-images.githubusercontent.com/12206047/44734989-2fa7d500-aab9-11e8-922f-4daf5026fa97.png">




@RishabhJain2018 can i take up this issue ? 😄 

Sure! Please feel free to ask if you have any doubts. We will be happy to help.

@RishabhJain2018 Is this issue open, because PR #99 has resolved the issue.
Repository: Cloud-CV/evalai-cli
Issue: Documentation landing page Enhancements
Root cause: Documentation Deficiency

- [x] Please add sample responses with each command.
- [x] Please add options by taking the reference from here and commands can be https://docs.aws.amazon.com/cli/latest/reference/
https://docs.docker.com/engine/reference/commandline/cli/
- [x] Add review comments
- [x] Can `pip install evalai` automatically create `~/.evalai` ?

@RishabhJain2018 The last option can definitely be automated. https://stackoverflow.com/questions/15853058/run-custom-task-when-call-pip-install

@isht3 Can you please implement that?

@RishabhJain2018 Yup, I'll reach back today and I'll work on implementing it.

Closing this since it is already fixed.
Repository: EleutherAI/lm-evaluation-harness
Issue: Implement the WebQuestions evaluation
Root cause: Unimplemented Feature Gap

From the GPT-3 paper

> In this section we measure GPT-3’s ability to answer questions about broad factual knowledge. Due to the immense amount of possible queries, this task has normally been approached by using an information retrieval system to find relevant text in combination with a model which learns to generate an answer given the question and the retrieved text. Since this setting allows a system to search for and condition on text which potentially contains the answer it is denoted “open-book”. [RRS20] recently demonstrated that a large language model can perform surprisingly well directly answering the questions without conditioning on auxilliary information. They denote this more restrictive evaluation setting as “closed-book”. Their work suggests that even higher-capacity models could perform even better and we test this hypothesis with GPT-3. We evaluate GPT-3 on the 3 datasets in [RRS20]: Natural Questions [KPR+19], WebQuestions [BCFL13], and TriviaQA [JCWZ17], using the same splits. Note that in addition to all results being in the closed-book setting, our use of few-shot, one-shot, and zero-shot evaluations represent an even stricter setting than previous closed-book QA work: in addition to external content not being allowed, fine-tuning on the Q&A dataset itself is also not permitted.

- [X] Data processing code implemented
- [ ] Evaluation implemented

The evaluation code should be modeled after the interface in `lm_eval/base.py` and the example of the `BoolQ` task in `lm_eval/tasks/suerglue.py`

_Note_: HuggingFace includes this in its datasets package.

https://huggingface.co/datasets/web_questions
Repository: EleutherAI/lm-evaluation-harness
Issue: [big-refactor] Accelerate launch FSDP Runtime Error
Root cause: External Dependency Breakage

Is there someone  running accelerate launch with FSDP successfully? Please share the accelerate config, thx.

I run into the following error:

`RuntimeError: The tensor has a non-zero number of elements, but its data is not allocated yet. Caffe2 uses a lazy allocation, so you will need to call mutable_data() or raw_mutable_data() to actually allocate memory.`

I am running eval on 4 GPUs, the error message is replicate on both GPUs. Typically one batch is completed on one of the GPUs before erroring out.


Accelerate conf:
'''
compute_environment: LOCAL_MACHINE
debug: false
distributed_type: FSDP
downcast_bf16: 'no'
fsdp_config:
  fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
  fsdp_backward_prefetch_policy: BACKWARD_PRE
  fsdp_cpu_ram_efficient_loading: true
  fsdp_forward_prefetch: true
  fsdp_offload_params: false
  fsdp_sharding_strategy: 1
  fsdp_state_dict_type: SHARDED_STATE_DICT
  fsdp_sync_module_states: true
  fsdp_use_orig_params: true
machine_rank: 0
main_training_function: main
mixed_precision: fp16
num_machines: 1
num_processes: 4
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false
'''

Closing as we now recommend Tensor-parallel with vLLM or DDP with Huggingface rather than FSDP.
Repository: EleutherAI/lm-evaluation-harness
Issue: [New Task] Implement GPQA dataset
Root cause: Unimplemented Feature Gap

https://arxiv.org/abs/2311.12022
https://github.com/idavidrein/gpqa/

repository contains a number of prompts. 

Note that care should be taken to prevent the dataset from being easily accessible in an LM corpus / should potentially not be uploaded to HF datasets uncompressed, to prevent more contamination than necessary.

I believe @uanu2002 may be working on this!
Repository: EleutherAI/lm-evaluation-harness
Issue: [New Task Request] IFEval / Instruction-Following Eval
Root cause: Unimplemented Feature Gap

https://arxiv.org/abs/2311.07911
https://github.com/google-research/google-research/tree/master/instruction_following_eval

may be flawed in that it primarily measures expertise in following formatting, but worth supporting
Repository: EleutherAI/lm-evaluation-harness
Issue: Issue "No module named 'lm_eval'"
Root cause: Environment Incompatibility

When I follow the steps on the task guide and try to run the evaluation, I get this error:

ModuleNotFoundError: No module named 'lm_eval'

I can't seem to find this issue online. What am I missing? 

Hi! Have you run `pip install -e .` from the root directory of this cloned repository yet? You would get this error if you have not installed the lm_eval package yet. 

> Hi! Have you run `pip install -e .` from the root directory of this cloned repository yet? You would get this error if you have not installed the lm_eval package yet.

Thanks for the response! I did pip install it properly but still get the same issue after uninstalling it and installing it again just to be sure. 

What is the command you run when you get this error? what directory do you run it from? Also, what branch of the repo are you using?

Closing as I can't replicate this--please reopen if this issue persists and you have a minimal working example of it!

also get the same issue
Repository: EleutherAI/lm-evaluation-harness
Issue: can we pass specific configs for specific tasks while running multiple benchmarks?
Root cause: Unimplemented Feature Gap

suppose i want to run hellaswag and arc_challenge tasks, and want to pass num_fewshot=10, batch_size=16 for hellaswag and num_fewshot=25,batch_size=16 for arc_challenge. how can we do that? As i understand, if we use this in the cli:
python main.py     --model hf-causal-experimental     --model_args pretrained=mistralai/Mistral-7B-v0.1   --tasks hellaswag,arc_challenge     --device cuda:0 --batch_size 16 --num_fewshot 25

in this it would take num_fewshot = 25 and batch_size = 16 for both the tasks. i want to specify seperate configs for both benchmarks


This is not currently exposed to the user in full generality via the CLI. However you can do this using the Python library, and there are some hacks that will get you most of the way there with the CLI.

In the case of batch_size, presumably your need is driven by the fact that tasks with smaller context lengths can fit a larger batch size on your GPU. You can set batch_size to `auto:2` to have the code automatically calculate the maximum batch size. If you run more tasks, change `2` to be the total number of tasks.

Note that you can also achieve your needs for num_fewshot in the big-refactor branch by creating a new task with a hard-coded value for num_fewshot.

We do not currently have plans to expose this in full generality via the CLI.

Closing because this is supported via setting parameters in tasks' YAML configs, and we do not plan to expose this via the CLI currently.
Repository: EleutherAI/lm-evaluation-harness
Issue: Is there a way to check if each sample in the dataset is correct or incorrect?
Root cause: Documentation Deficiency

Thank you in advance

Hi!, you can use `--log_samples` flag on the `big_refactor` branch to save the model outputs to file. Think there is an analogous option on the main branch as well but not sure.

The option on the `main` branch only logs *inputs* not *completions* unless there's been a PR I've missed.
Repository: EleutherAI/lm-evaluation-harness
Issue: [New Task] CommonsenseQA
Root cause: Unimplemented Feature Gap

https://github.com/jonathanherzig/commonsenseqa
https://huggingface.co/datasets/commonsense_qa

Evaluated on by Llama 2 + Mistral

Hi. I just want to know if there is any update on this? Thanks

Hi! There has not been yet--we'd still welcome a contribution!

I may look at it at some point in the future but have been prioritizing framework/library improvements.

@haileyschoelkopf I'm interested in taking this on if no one is currently working on it.

> @haileyschoelkopf I'm interested in taking this on if no one is currently working on it.

Thank you! Please do.

Can anyone mention which set are you guys using to evaluate?

The test set does not contain the answer key.
Repository: EleutherAI/lm-evaluation-harness
Issue: [big-refactor] Adding Flash Attention 2 to HF Model
Root cause: Unimplemented Feature Gap

Hey, thanks for this amazing repo.

I noticed that currently the repo does not support Flash Attention 2. I tried simply turning on the use_flash_attention_2=True flag in the huggingface model module and it worked out of the box, and provided speedups and memory usage reduction (especially for long tasks / few-shot, which makes sense).

Would it be possible to add native support for FA2 via HF? I am also happy to work on a PR myself if you could please let me know what needs to be done :) 

Hi! We'd be happy to merge a PR with this flag!

If you're willing to do so, even more preferable would be to allow for support of arbitrary extra kwargs passed to the HF model or tokenizer init, in order to support new kwargs such as this one out of the box and to minimize the bloat in the number of kwargs listed in the `HFLM` init() method.

Will do, thanks! Closing this issue and will open a PR once I have something.