zhiminy/EvalEng huggingface dataset:
Repository: Cloud-CV/EvalAI
Issue: `docker-compose up --build` throws an error
Root cause: Environment Incompatibility

I am trying to follow the instructions on README.md to build up the docker images. But it seems that some Dockerfile uses an out-of-date system: https://github.com/Cloud-CV/EvalAI/blob/master/docker/dev/nodejs/Dockerfile this file tries to install google-chrome-stable on debian jessie, which can't be satisfied:

```
The following packages have unmet dependencies:
 google-chrome-stable : Depends: libdrm2 (>= 2.4.60) but 2.4.58-2 is to be installed
                        Recommends: libu2f-udev but it is not installable
                        Recommends: libvulkan1 but it is not installable
E: Unable to correct problems, you have held broken packages.
ERROR: Service 'nodejs' failed to build: The command '/bin/sh -c apt-get update && apt-get install -yq google-chrome-stable' returned a non-zero code: 100
```

Is there anyone who knows how to fix this? Thank you.



Hello, try replacing this Dockerfile step with apt-get update -qqy --no-install-recommends && apt-get install -qqy --no-install-recommends google-chrome-stable

I'm having a similar issue but with a completely other project. Build succeeds on Ubuntu, but fails on WSL with this error 

```
The following packages have unmet dependencies:
 google-chrome-stable : Depends: libgbm1 (>= 17.1.0~rc2) but 13.0.6-1+b2 is to be installed
                        Recommends: libu2f-udev but it is not installable
                        Recommends: libvulkan1 but it is not going to be installed
``` 

Did you find any solution for this issue ? I am facing the same 

I am facing similar issue but with python 3.7-stretch image. 
**Docker version 20.10.17, build 100c70180f on Ubuntu 22.04.1 LTS**

Dockerfile as follows

```
FROM library/python:3.7-stretch
RUN apt-get update

RUN wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -
RUN sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-chrome.list'
RUN apt-get -y update
RUN apt-get install -y google-chrome-stable
```
Repository: Cloud-CV/EvalAI
Issue: Phase is in random order in leaderboard
Root cause: Algorithmic Error

Phases are in random order in the leaderboard (undesirable), unlike the sorted behavior in the phases tab.

This is due to the backend API [`challenge_phase_split_list`](https://github.com/Cloud-CV/EvalAI/blob/master/apps/challenges/views.py#L824-L850) not ordering the filtered results by `pk`, unlike other APIs like [`challenge_phase_list`](https://github.com/Cloud-CV/EvalAI/blob/master/apps/challenges/views.py#L717) which are doing the sorting.

Will create a pull request to fix this.
Repository: Cloud-CV/EvalAI
Issue: build stopped at step 2/20: gpg: no valid OpenPGP data found
Root cause: Environment Incompatibility

Here is the issue I get when running `docker-compose up --build`

```bash
$ docker-compose up --build
Building nodejs
Sending build context to Docker daemon  99.75MB
Step 1/20 : FROM node:14.20.0
 ---> 326f034bd14b
Step 2/20 : RUN wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -
 ---> Running in 12c9919d2e8d
Warning: apt-key output should not be parsed (stdout is not a terminal)
gpg: no valid OpenPGP data found.
The command '/bin/sh -c wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -' returned a non-zero code: 2
ERROR: Service 'nodejs' failed to build : Build failed
```

I am running it under WSL2. My WSL image is running ubuntu 22.04.
From this image I can run without problem 
```bash
$ wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub
-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v1.4.2.2 (GNU/Linux)

mQGiBEXwb0YRBADQva2NLpYXxgjNkbuP0LnPoEXruGmvi3XMIxjEUFuGNCP4Rj/a
```

I guess this is due to my network configuration but any help would be much appreciated.

This is because I am behind a corporate firewall. One option is to pass --no-check-certificate option to wget.
But anyway I cannot go further because apt command cannot go through it. I have to ask my corporate teams to add google apt repo to artifactory. Will take a while.
Repository: Cloud-CV/EvalAI
Issue: How to build docker images on a machine and execute to another one
Root cause: Documentation Deficiency

This is a question not an issue.

For a reason I cannot build evalai docker images on my host machine.
It is behind a corporate firewall. But I would like to run evalai on this host, and connect internally to our corporate gitlab.

I have successfully built docker images on a standard linux machine, out of this network. And I would like to 'move' these images on my final host and just start evalai. And I am not sure how to do it.

From my standard linux machine, I am able to list and extract all evalai images with commands like `sudo docker save -o /tmp/evalai_nodejs.tar evalai_nodejs` . Copy these images to my host machine.
And from this host machine, I can then load these images with `sudo docker load -i evalai_nodejs.tar` and they are visible from `docker images`
But I am not sure how to start evalai. If I try `docker-compose up --build` from evalai, it tries to rebuild them and it fails.

Sorry it is likely a very basic docker question. But any help would be appreciated.

Hello, do a docker-compose up without --build

Thank you, it works now with nodejs image. I am going to transfer all images, then start evalai. Will let you know the results.
Repository: Cloud-CV/EvalAI
Issue: To make it run with gitlab

This is not an issue but a question.

Is there successful configuration of EvalAI using gitlab instead of github?
I have a local gitlab server and have no access to internet / github.
Specifically in process-challenge.yml, I am not sure how to translate github action context calls (such as ``` GITHUB_CONTEXT: ${{ toJson(github) }}
```

any help would be great

I would be interested in this as well!

Hey, it was a long time ago but i would be interested as well. Did you find a solution?

no progress on my side

Closing this since it is a very old issue.
Repository: Cloud-CV/EvalAI
Issue: Submission status stays at "submitted" and no results show up on the leaderboard
Root cause: Resource Mishandling

We are hosting a challenge here: https://eval.ai/web/challenges/challenge-page/1788/overview It is a prediction-based challenge (submitting a single result file) and it has been approved. However, after submitting results using 2 different user accounts, we found that the submissions did not generate any result entry on the leaderboard. Their status stayed at "submitted," and it did not seem like any scores were generated. 

Here is the worker log:
```
[2022-08-25 16:51:18] INFO WORKER_LOG Using /tmp/tmp_t7n3iky as temp directory to store data
[2022-08-25 16:51:18] INFO No custom requirements for challenge 1788
[2022-08-25 16:51:38] INFO WORKER_LOG Processing message body: {"challenge_pk": "1788", "phase_pk": "3492", "is_static_dataset_code_upload_submission": false, "submission_pk": 219300}
[2022-08-25 16:51:38] INFO SUBMISSION_LOG [x] Received submission message {"challenge_pk": "1788", "phase_pk": "3492", "is_static_dataset_code_upload_submission": false, "submission_pk": 219300}
```

Here is the screenshot of the submissions. (I reran one of the submissions so there was a canceled entry)
![Screen Shot 2022-08-25 at 12 59 33 PM](https://user-images.githubusercontent.com/5194487/186725937-c3d4fcf7-57bc-462f-99d8-694033c099ca.png)

I have written and tested the evaluation script. Are there missing steps from my end? I understand that I do not need to run any submission worker on my own server if the submissions are not code-based. Is that right?



Hi, I have the same symptoms now.
If I submit it by last week, I got a score after running and finished.
However, if you submit it from yesterday, submiited is coming out.
Can you figure out how to solve it?

I asked for more compute resources from the EvalAI team and that solved the problem.
Repository: Cloud-CV/EvalAI
Issue: Submission stuck at "Submitted" for TextCaps and NoCaps
Root cause: External Dependency Breakage

Hello,

Looking over the previous issues, I believe this is a recurrent issue. This time this issue has been there for NoCaps and TextCaps at least 3-4 days ago.

Thanks for your attention,

I just met the same problem on VQA Benchmark. Wondering if is there any expected time to fix this? @gchhablani Really appreciate your time and help.
Repository: Cloud-CV/EvalAI
Issue: Create endpoint to get details about a particular team
GitHub labels: enhancement

GSOC-2019

easy_to_fix

Details about the endpoint:
- Url: /api/teams/<team_id>/
- Header: Authorization: Token <token>
- Method: GET
- Response: 
  - status on success: HTTP_200_OK
  - status on exception:  HTTP_406_NOT_ACCEPTABLE
  - status on Unauthorized access: HTTP_203_NON_AUTHORITATIVE_INFORMATION

For more details, feel free to discuss on thread or gitter channel https://gitter.im/Cloud-CV/EvalAI

Fixed in #63
Repository: Cloud-CV/EvalAI
Issue: AttributeError: 'MultiOut' object has no attribute 'flush'
Root cause: Validation Gap

Found an issue mentioned in one of the messages:
```AttributeError: 'MultiOut' object has no attribute 'flush'.``` The VizWiz 2022 challenge evaluation is failing apparently because of this.

This is probably due to the MultiOut class added recently. 

I have encountered the same problem. Is there any solution?

> Found an issue mentioned in one of the messages: `AttributeError: 'MultiOut' object has no attribute 'flush'.` The VizWiz 2022 challenge evaluation is failing apparently because of this.
> 
> This is probably due to the MultiOut class added recently.


Hi, sorry to bother you. I have encountered the same problem. Is there any solution?



@xinyu1205 This issue has been fixed in staging and the fix will be moved to prod soon. Sorry for the delay.

CC @Ram81 

> @xinyu1205 This issue has been fixed in staging and the fix will be moved to prod soon. Sorry for the delay.
> 
> CC @Ram81

Really appreciate for your reply! I'm look forward to an early resolution of this issue.
Repository: Cloud-CV/EvalAI
Issue: Installed on Docker Desktop, getting login screen but cannot go further
Root cause: Configuration Error

Login doesn't work.

After deploying Eval AI via docker compose everything seems fine. However when I attempt to log in or sign up nothing happens at all. It seems like api error. May be I'm missing some configuration, because Im deploying on remote server with its own open ip. Please, help me out.

@artnuke 

Try to replace localhost on your IP:
https://github.com/Cloud-CV/EvalAI/blob/09202ad46e31bd721c3cbb0c2fa31e740cd6d19e/frontend/src/js/config.json#L4

If it doesn't help, do the same here:
https://github.com/Cloud-CV/EvalAI/blob/09202ad46e31bd721c3cbb0c2fa31e740cd6d19e/docker/dev/docker.env#L11

Then deploy one more time. Don't forget to rebuild the Docker images.

Hi @Ilyushin, I'm facing the same issue on my end, I'm using a Windows 11 computer and have already instealled Docker Desktop on it. I'm following the documentation and by executing `docker-compose up --build`, an error raised as following: 
```
evalai_statsd-exporter_1 is up-to-date
evalai_db_1 is up-to-date
evalai_nodejs_1 is up-to-date
evalai_sqs_1 is up-to-date
Recreating evalai_django_1 ... done
Recreating evalai_worker_py3_8_1 ... done
Recreating evalai_worker_1       ... done
Attaching to evalai_statsd-exporter_1, evalai_db_1, evalai_nodejs_1, evalai_sqs_1, evalai_django_1, evalai_worker_1, evalai_worker_py3_8_1
django_1           | /usr/bin/env: ‘bash\r’: No such file or directory
```
also, I checked the Docker Desktop, the status of `evalai_django_1`, `evalai_worker_1` and `evalai_worker_py3_8_1` are all "Exited" due to the error `/usr/bin/env: ‘bash\r’: No such file or directory`:

![image](https://user-images.githubusercontent.com/59240680/211394079-4f1cec27-8622-4a09-bba6-2ed3e9d5d7de.png)

I was wondering if this is due to the Windows OS or something else? Unfortunately I dont have a linux machine to test this.

Looking forward to your help! Thank you so much!



I experienced this problem, and guessed it to be a problem with the encoding of line endings. To experiment I deleted the Git repo, executed `git config --global core.autocrlf true` then cloned the repo again and it then worked okay.

Hi I've tried using git config --global core.autocrlf true and re cloning the repo, still have the same error as above " /usr/bin/env: ‘bash\r’: No such file or directory" Could anyone give me some guidance on this issue? I've tried it on windows and on my linux wsl. 

Closing this since it is a very old issue. Please re-open if the issue persists.
Repository: Cloud-CV/EvalAI
Issue: uWSGI listen queue of socket full
Root cause: Environment Incompatibility


<img width="1512" alt="Screen Shot 2022-11-26 at 12 29 59" src="https://user-images.githubusercontent.com/63837696/204073777-f765c574-3a0f-4bd9-bd63-38ba5d3780ba.png">
<img width="1512" alt="Screen Shot 2022-11-26 at 12 29 14" src="https://user-images.githubusercontent.com/63837696/204073781-785f2b4e-ce7c-47f8-a64c-7ca7a5ee07d3.png">

I'm facing this issue when i run `docker-compose up --build`. I am using a MacBook M1. Any help would be appreciated.

Closing this since #4498 solves this.
Repository: Cloud-CV/EvalAI
Issue: BDD100K Object Detection on EvalAI submission status failed
Root cause: Configuration Error

I submit my validation and testing result of BDD100K Object Detection Challenge on EvalAI on Monday, and get the result of validation but the status of testing is "Failed" with the following error message:

"Traceback (most recent call last):
File "/code/scripts/workers/submission_worker.py", line 504, in run_submission
submission_output = EVALUATION_SCRIPTS[challenge_id].evaluate(
AttributeError: module 'challenge_data.challenge_1857' has no attribute 'evaluate' "

Please help me to resolve the problem and get the testing result, thanks!

Thanks for pointing out this issue. There is a problem with the challenge configuration. We have reached out to the hosts and are trying to get this issue resolved as quickly as possible.
 
Thanks,
Gunjan

Thank you for your reply. If the issue is resolved, please inform me. Thank you very much!!

Best Regards,
Lucy
Repository: Cloud-CV/EvalAI
Issue: BDD100K Object Detection on EvalAI submission FileNotFoundError
Root cause: Validation Gap

I submitted my testing result of BDD100K Object Detection Challenge on EvalAI on Monday, and get  the status of testing is "Failed" with the following error message:

Traceback (most recent call last):
  File "/code/scripts/workers/submission_worker.py", line 508, in run_submission
    submission_metadata=submission_serializer.data,
  File "/tmp/tmpebqg16sx/compute/challenge_data/challenge_1857/main.py", line 76, in evaluate
    results = load(user_submission_file, nprocs=1).frames
  File "/usr/local/lib/python3.7/site-packages/scalabel/label/io.py", line 81, in load
    raise FileNotFoundError(f"{inputs} does not exist.")
FileNotFoundError: /tmp/tmpebqg16sx/compute/submission_files/submission_242106/b0ca67e6-1b64-4bd8-a0d1-c774eb179ad5.json does not exist.

Please help me to resolve the problem and get the testing result, thanks!
Repository: Cloud-CV/EvalAI
Issue: Easiest way to develop a challange

Hey guys,

The [doc explains](https://evalai.readthedocs.io/en/latest/host_challenge.html#host-challenge-using-github) how to fork and setup CI to get the challenge on the official website. I was wondering if you can point me out to the best practices to develop a challenge locally. I have `eval.ai` up un running locally, I can clone any challenge template I want but I don't know how to properly develop locally. 

Thanks in advance.

Cheers,


Fra

Hi @FrancescoSaverioZuppichini

You can use this script to generate challenges locally. [Repo Link](https://github.com/AyushR1/EvalAI-Starters/tree/local)  

Clone the repo, edit config.json in the local folder to enter auth and user id and then on the root directory just run the script using
 `python local/challenge_processing_script.py`
 



Closing this since this is an old issue.
Repository: Cloud-CV/EvalAI
Issue: Package not installed even if correctly build and in requirements
Root cause: Environment Incompatibility

Worker's logs says this package is not installed even if is in `github/requirements.txt`

```
from fast_coco_eval import COCOeval_fast
ModuleNotFoundError: No module named 'fast_coco_eval'
```

`github/requirements.txt`

```
PyGithub===1.53
requests==2.24.0
pandas
torch
fast_coco_eval
```

Should I force install using `sys` inside the script?

Thanks a lot

Fra

Oh, just fund in the `__init__.py` the custom code to install packages, closing :)
Repository: Cloud-CV/EvalAI
Issue: Remove test_annotation field from ChallengePhase serializer
GitHub labels: bug

GSOC-2019

medium-difficulty

backend

We should remove the field `test_annotation` from all the APIs related to Challenge Phase since this data has to be only accessed by the Challenge Organizer. 

@deshraj  can I try this one?

is anyone doing this??

yeah I want to work on this one!

yooo...no issue...i was just asking

yeah its fine :-)

Assigning to @Ram81 :-) 

@deshraj  I've removed ' test_annotation ' field from ChallengePhaseSerializer's meta class is there any other dependency I should look for?

Can you please write test for the same and check if this field is returned or not in response from the APIs?

Sure will do that :+1: 

@deshraj Do I have remove other tests from CreateChallengePhaseTest & UpdateChallengePhaseTest  as we removed field from Serializer?

@Ram81 No, you don't need to remove the tests. Just make changes to the existing tests to check if the `test_annotation` field is present or not in all those APIs. Does that make sense?

I've to just revert the tests i.e if field is not present  ok response will be there else a bad request,am I right?

I think it would be a good idea to create a PR and then I can suggest the changes. 

@deshraj Okay will do & let you know

@deshraj do I have to just add a test in GetChallengePhaseTest class in test_view.py or some other ones too or am I doing it totally wrong, please guide?

Yeah, you are correct. 

@deshraj so there I've to remove test_annotations field from 'expected[]' ,right?

Yup. You might also need to add one or more test cases. Can you open a PR once you are done with the changes and then we can see what other changes do we need to make? 

Okay for now I will remove the field & generate a PR & then you suggest changes !

@deshraj  what about tests you wanted me to write?

Fixed in #431
Repository: Cloud-CV/EvalAI
Issue: Fix manage tab visibility access on challenge page
Root cause: Validation Gap

### Description
Currently, the challenge page manage tab for any challenge is visible to anyone if they try to directly access the link. This should not be the case. Ideally, it should be redirected to `404` page directly.

https://user-images.githubusercontent.com/29076344/226559437-d0c7fd99-f5d4-4cdb-84d2-7e0dcdd01b62.mov

I made an attempt in #3883 but it is slow as we use our Django API to fetch whether the user is challenge host or not. This leads to a 1 second delay in redirecting to the 404 page which is undesired. The hosts should be able to access this link but the participants should not.

Note that this is a **critical** issue.

### Pre-requisites/Learnings
This issue essentially requires background on a few things:
- An overview understanding of [EvalAI challenges](https://evalai.readthedocs.io/en/latest/host_challenge.html).
- An understanding of the [Challenge Model](https://github.com/Cloud-CV/EvalAI/blob/master/apps/challenges/models.py) on EvalAI.
- An understanding of the [EvalAI frontend](https://github.com/Cloud-CV/EvalAI/tree/master/frontend).

### How to Approach
- You can start by setting up EvalAI locally. Set up the dev version locally (preferred) using the steps [here](https://github.com/Cloud-CV/EvalAI/tree/master#installation-instructions).
- Play around with the the frontend, and create a PR. Please add screenshots/GIFs of the changes.
- Reach out to us regarding any issues that you face or if you need any help with understanding any of the above.



@gchhablani I can also work on this one, If possible assign it to me! Thanks!

@gchhablani  I understand this issue and would like to work on it .

@AyushR1 As I said, let's get your PR merged. I will provide review on that one asap and then you can take up some other issue. If there are no new issues left, we will definitely find something for you to work on, so don't worry :) 

@CodeDetector Let's get the other issue fixed first and then move on to the next one. Thanks! ^_^

@gchhablani I'd like to work on this! Can it be assigned? 

@imperorrp I have assigned this issue to @CodeDetector. Please check if any other issues are open, I will assign you to them.

Hi @gchhablani I am working on this issue . Was facing issues with docker setup that is resolved . 

@CodeDetector Please ping me on the slack if you face any issues. Thanks!

@CodeDetector Are there any updates on this?
Repository: Cloud-CV/EvalAI
Issue: Highlight entry in My Submissions Tab
Root cause: Unimplemented Feature Gap

### Description
This issue is a revival of #2142. 
The idea is that the participants are not able to see the highlighted submissions on their "My Submissions" page which correspond to the entry shown in the leaderboard. This feature helps the participants easily get to their corresponding submission

### Pre-requisites/Learnings
This issue essentially requires background on a few things:

- An overall understanding of [EvalAI challenges](https://evalai.readthedocs.io/en/latest/host_challenge.html).
- An understanding of the models related to [Challenges](https://github.com/Cloud-CV/EvalAI/blob/master/apps/challenges/models.py) on EvalAI.
- An understanding of the [EvalAI frontend](https://github.com/Cloud-CV/EvalAI/tree/master/frontend).

### How to Approach
- You can start by setting up EvalAI locally. Set up the dev version locally (preferred) using the steps [here](https://github.com/Cloud-CV/EvalAI/tree/master#installation-instructions).
- There can be various ways to approach this. Please feel free to make an attempt at this and create a PR (with screenshots).
- Reach out to us regarding any issues that you face or if you need any help with understanding any of the above.

Getting started and asking for help is the best way for this one :)


Hi @gchhablani  can I work on this issue as I am beginner so it would be good to start while trying to solve this issue.

Hi @gchhablani can I work on this issue as I have a bit experience on this ,i have worked on more than 10 projects 

@anandkr47 Can you please take this up? I have assigned @Adi9235 to another issue. Thanks!

> @anandkr47 Can you please take this up? I have assigned @Adi9235 to another issue. Thanks!

Yes 👍 i will 

failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to create LLB definition: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed


i am getting this error when i do the docker-compose -- build


@anandkr47 are you still working on this?

hello can i take up this ?