zhiminy/EvalEng huggingface dataset:
Repository: Cloud-CV/EvalAI
Issue: In Docs Change link in Host Challenge using Github in step 1.

When we click on **EvalAI-Starters** it goes to different page.
<img width="1440" alt="ss2" src="https://user-images.githubusercontent.com/60038994/148992353-d80f0008-645a-4cf4-bdc1-9cc4fe5abadf.png">



I have send Pull Request.

@haldendmello that is the expected behavior.

Closing this issue
Repository: Cloud-CV/EvalAI
Issue: Enable code upload challenges in development.
Root cause: Documentation Deficiency

Hello,

I'm setting up my own forked version of EvalAI and I'm having troubles setting code upload challenges. If I understand correctly the code (in the DEV environment), there is no worker that is able to receive a code upload submission, right?

So, for the worker `Dockerfile` instead of this:
```
CMD ["./docker/wait-for-it.sh", "django:8000", "--", "python", "-m", "scripts.workers.submission_worker"]
```
I'm doing this:
```
CMD ["./docker/wait-for-it.sh", "django:8000", "--", "python", "-m", "scripts.workers.code_upload_submission_worker"]
```

That raise a bunch of errors, that I'm trying to figure out. 
Is that the "correct" way to enable code upload on development?

p.s.: I'm only interested in code upload challenge, not remote evaluation.

Thanks!

Actually, I just found that there is a different worker Dockerfile in `EvalAI/docker/dev/code-upload-worker/Dockerfile`. Is that the one that should be used?

There is some documentation on all this?

Hi @gustavofuhr,

We don't support running code upload challenges locally, the pipeline is tightly coupled with AWS infrastructure so setting up locally would be difficult. 

>Actually, I just found that there is a different worker Dockerfile in EvalAI/docker/dev/code-upload-worker/Dockerfile. Is that the one that should be used?

Yes, that's correct

>There is some documentation on all this?

No, currently we don't have any documentation on how to set it up locally. 

Closing the issue as I have addressed the queries.


Got it @Ram81, but there is any doc to set up code upload eval remotely then?

@gustavofuhr unfortunately we don't have any documentation on how to set it up remotely but we do support spawning the required infrastructure in a remote AWS account on EvalAI. The way it works is, challenge hosts create a remote evaluation code upload challenge on EvalAI and share an IAM role with us that we use to spawn the required infrastructure in remote AWS account. In this flow, challenge hosts don't have to manage the infrastructure as everything is automated.
Repository: Cloud-CV/EvalAI
Issue: [Frontend] Make navbar sticky on privacy policy page
GitHub labels: easy_to_fix

frontend_v2

GSOC-2022

### Description

Currently, the navbar on the privacy policy page is not stick on large screens. As part of fix of this issue we need to make the navbar sticky on all resolutions of the screen. See the screenshot for reference of current issue.


![Screenshot from 2022-01-17 17-57-55 (1)](https://user-images.githubusercontent.com/16323427/149845661-c216dd18-6c83-487d-b3de-a092425c0c44.png)


Hey, I am interested in contributing to this issue? can anyone help me?


Hi @Ayush-Raghuwanshi, sure you can work on this issue. Let us know if you have any questions

Hey, can i contriburte to this issue?

Seems like Ayush isn't working on this issue so can i take it?

Yeah I am so sorry, @Ram81 you can assign other. I cant work on it right now, I am sorry...


@Ram81 @RishabhJain2018 Does this issue/PR look relevant? I think the PR fixes the issue.

Hi I am keenly interested in addressing specific issues and am earnestly hopeful of having the opportunity to engage with them further.

Closing this since we are not working on Frontend_v2
Repository: Cloud-CV/EvalAI
Issue: [Frontend] Fix the twitter feed loading when user navigates back to homepage
GitHub labels: easy_to_fix

frontend_v2

GSOC-2022

### Description

The twitter feed homepage fails to load when a users visits some other page after visiting the homepage and then comes back to the homepage. Steps to reproduce:

1. Open EvalAI homepage on FrontendV2
2. Navigate to login/signup page
3. Navigate back to homepage and check the twitter feed

Screenshot for reference:

![Screenshot from 2022-01-17 18-02-21](https://user-images.githubusercontent.com/16323427/149845858-cf83d18a-07a2-44f2-b455-109189c32178.png)

Expected behavior:

![Screenshot from 2022-01-17 18-03-28](https://user-images.githubusercontent.com/16323427/149845934-4e5ea244-acc1-4731-b1cc-d280d44dae57.png)



Hi, can I work on this? I'm a beginner though, would need some help if that's okay?

@Ram81 can I work on the issue. I know the basics of Angular. 

> @Ram81 can I work on the issue. I know the basics of Angular.
I have send Pull request
Fix #3676 [Frontend] Fix the twitter feed loading when user navigates back to homepage 

Hi @ayushvrma, sure you work on this issue. You can post your doubts on our gitter channel if you are stuck.


>@Ram81 can I work on the issue. I know the basics of Angular.

Hi @haldendmello, thanks for your interest. We usually follow first come first serve approach to assign issues as @ayushvrma has already requested this issue you'll have to look for another issue. If he isn't able to make progress in a week then we can reassign this issue to you.



Hi @ayushvrma if you want any help you can contact me.

Hey @Ram81 can I work on this issue as I don't think ayush has made any progress as there's no pull request linked to this issue


@Ram81 @haldendmello I'm sorry to not being able to figure out how to fix the code yet. You can assign it to @bridyash13 

@ayushvrma @bridyash13 This issue is fixed.



Closing this as it is fixed
Repository: Cloud-CV/EvalAI
Issue: [Backend] Submission validation absent in challenges for unsupported files
Root cause: Validation Gap

I noticed that the user can upload non .txt files as a submission in the challenges. Resources can be saved on such a file by just adding a validation.

Ideal Behaviour:
Throw 400 Bad Request for unsupported files

Current Behaviour:
No validation written in views for this.


https://user-images.githubusercontent.com/89455838/151967012-aad876d9-604d-4742-a2ce-c12b0023eb6b.mp4


If this is a potential issue. I would like to be assigned to it and work on it. Thank You!


Hello @GeekGawd  it seems there is already a issue for this error. if i am correct it is #3358 

Closing this issue as it is a duplicate
Repository: Cloud-CV/EvalAI
Issue: [Question] Is it possible to run multiple workers?
Root cause: Documentation Deficiency

Could you explain how to run multiple workers in a single machine for parallel evaluation? For example, I run a challenge on my local server in docker, then several participants submit their solutions. As I understand, the worker run submissions one-by-one and therefore participants wait in a queue. It will be great to run multiple workers to speed up the evaluation. Thanks!

Please spin up multiple worker containers. Closing this for now, please open if you need anything else.
Repository: Cloud-CV/EvalAI
Issue: [FrontendV2] Fix overlapping controls on My submissions page when you change screen width 
GitHub labels: frontend_v2

GSOC-2022

### Description

The controls on `My submissions` page overlap when a user changes the screen width. The ideal behavior is that the controls don't overlap at different screen widths.

Current screenshot for reference:

![Screenshot from 2022-02-11 17-42-47](https://user-images.githubusercontent.com/16323427/153680813-7084c70c-484b-465a-9c70-fad4ae2c045b.png)

Expected behavior: Controls shouldn't overlap at different screen width


@Ram81 i would like to work on this issue

@Suryansh5545 sure, go ahead

Closing this since we are not working on Frontend_v2
Repository: Cloud-CV/EvalAI
Issue: [FrontendV2] Add validation for team URL when a user edits team details
GitHub labels: easy_to_fix

frontend_v2

GSOC-2022

### Description

Currently, when a user edits the team details and submits the updates we don't check if the team URL is a valid URL. As part of this task you will add a validation check to ensure the entered value in edit form is a valid URL.

Current behavior:

![Screenshot from 2022-02-11 17-47-32](https://user-images.githubusercontent.com/16323427/153681160-e7aa5c02-898d-4b39-b774-509b29550099.png)

Expected behavior: We should throw a error before update if the value is not a valid URL



@Ram81  Can I work on this issue.

Can i work on this?

Hi @haldendmello , as @Suryansh5545  already opened a PR for this task it would be good if you could pick up some other task.

Thanks @Suryansh5545 for the quick PR. In future, it would be great if you could start working on a task only after you get it assigned. We would like to avoid redundant effort from multiple contributors.

@Ram81 Sorry for that.

Hi I am keenly interested in addressing specific issues and am earnestly hopeful of having the opportunity to engage with them further.

Closing this since we are not working on Frontend_v2
Repository: Cloud-CV/EvalAI
Issue: [FrontendV2] Disable submit button on submit page when a selected file is of incorrect type
GitHub labels: easy_to_fix

inactivity

frontend_v2

GSOC-2022

### Description

Currently, in our Frontend V2 submit page on the challenge page user can submit a file even if it has a extension which is not allowed by challenge hosts in (i.e. not in `allowed_submission_file_types` for a challenge phase). If a user uploads a file on the frontend which has a invalid extension we should disable the submit button and show a message that selected file type is invalid.

Screenshot for reference:

![Screenshot from 2022-02-12 12-05-57](https://user-images.githubusercontent.com/16323427/153720823-2babbc7d-4e16-403d-8d61-b6525d74daf7.png)


Expected behavior: Disable the submit button when file uploaded using `Upload file` option has a invalid extension.

@Ram81 want to work on this issue.


Is this issue still open?

@Ram81  Can you please assign me this issue? im intrested to work upon it .

I am intrested in solving this issue. Can you please assign me on this.

@SumeelaMadusankha This issue is old. We are not building `frontend_v2` currently.
@Ram81 I am closing this one for now.
Repository: Cloud-CV/EvalAI
Issue: What is the stable working version with bug fixes?
Root cause: Documentation Deficiency

The last released version is from 4 years ago (1.1) - is there a "stable" recent version with bug fixes? Should one use the master branch? It's not clear from here.

Hi @surak, the current master is the latest release which is stable.

It's hard for institutions to deploy things in production without a clear release schedule - is "master" from 2020 the same as 2022? Would it be possible to create a tag so packagers and people working with software installs can work with? Otherwise, we will simply lock into a commit. 

Thanks!
Repository: Cloud-CV/EvalAI
Issue: Create endpoint to delete a team
GitHub labels: enhancement

GSOC-2019

easy_to_fix

Details about the endpoint:
- Url: /api/teams/<team_id>/
- Header: Authorization: Token <token>
- Method: DELETE
- Response: 
  - status on success: HTTP_200_OK
  - status on exception:  HTTP_406_NOT_ACCEPTABLE
  - status on Unauthorized access: HTTP_203_NON_AUTHORITATIVE_INFORMATION

For more details, feel free to discuss on thread or gitter channel https://gitter.im/Cloud-CV/EvalAI

Fixed in #65
Repository: Cloud-CV/EvalAI
Issue: [FrontendV2] Add search option on all challenges and hosted challenges page
GitHub labels: frontend

new-feature

good first issue

javascript

frontend_v2

GSOC-2022

### Description

Add support to search challenges by name on the `All challenges` and `Hosted challenges` page.

@Ram81 I would like to take this issue

Hi I am keenly interested in addressing specific issues and am earnestly hopeful of having the opportunity to engage with them further.

Closing this since we are not working on Frontend_v2
Repository: Cloud-CV/EvalAI
Issue: [FrontendV2] Show progress bar when a participant is uploading a submission from UI
GitHub labels: frontend_v2

GSOC-2022

### Description

Currently, when a participant is uploading a submission they don't see the upload progress. As part of this task we have to add support for showing a progress bar for a submission being uploaded on the UI.

@Ram81 want to work on this issue.


![progressbar](https://user-images.githubusercontent.com/60038994/155369466-ab8b9062-0dcd-4f8a-b8ae-6d4a56dd6b4d.gif)

@Ram81 when we submit there are some dots as a progress bar there in GIF. I should replace that with some round progress bar.

If this issue is open, Can i work on this

@Sourav-Kumar-Panda currently development of frontend_v2 is off the roadmap.

ok

Closing this since we are not working on Frontend V2.
Repository: Cloud-CV/EvalAI
Issue: Expand team to show team members
GitHub labels: enhancement

GSOC-2019

medium-difficulty

frontend

critical

In team view, there is an option to show members of team. Need to implement it such that clicking on card expand with its team member also add toggle functionality to cards.

Let me know in case of any query .

@aka-jain  can you assign this to me?


@Ram81 please try to do one issue at a time. Otherwise it blocks the overall development. We have enugh number of issues for you to solve. :-) 

This issue will apply for both the participant team view and the challenge host team view. 

@deshraj  I will do this one now. 
@aka-jain  any hints for this one?

@Ram81: I think you should do #372 first, it will be helpful for you to give an overview of whole system. 
If you still want to do this then please let me know. I'll make #372 free for others.

@aka-jain Okay as you say I will try #372 first

@aka-jain can I work on this?

@aka-jain @deshraj Is someone working on this issue ??

@rishab please work on any other issue, I'm assigning it to @jayantsa.
@jayantsa please go ahead, and let me know if you need any help

@aka-jain okk sure :-)

@aka-jain Is there any API endpoint that returns participants based on team name ?

@deshraj : can you help @jayantsa  here?

@jayantsa Please look at the http://localhost:8000/docs and you can find an API endpoint with url ` /api/participants/participant_team/<pk>` which gives you the details of a challenge based on the primary key of the team which is `<pk>` here. For eg. for a challenge with primary key `1`, the url will be ` /api/participants/participant_team/1`. 

Please let me know if you have more queries. 

@deshraj  This API returns object of form -> **Object { id: 1, team_name: "team2", created_by: "jayant" }**  but inorder to show Team members in particular team we should return **Object { count: 3, next: null, previous: null, results: Array[3] }** where array contains all the Participants from participant db having team_name="team2". Am I right ?

Sorry @jayantsa to catching you late. @deshraj any idea about API response?

Dependent on #476 

Also Dependent on #515 

@aka-jain solved #476 and #515. Will work on this issue. Can we use bootstrap for making drop-down list??


@jayantsa we are planning to use materializecss throughout. So, it would be good if you can use materializecss for drop down list. 

@deshraj  ok sure.

@deshraj  **Please assign this issue to some one else** because due to some reason won't be able to contribute for some time. **Solved #476, #515, #527, #528** that were required for this. 
Now when call to **/api/participants/participant_team**  is made then within results we will have team details along with its members and same is the case when call to **/api/hosts/challenge_host_team** is made

Thanks for informing @jayantsa :-) 

@aka-jain Assigning it to you. 

@aka-jain : I am on it.

Okay 👍

Fixed in #896. Thanks @spyshiv :) 

Cool :)
Repository: Cloud-CV/EvalAI
Issue: An circular reference error in `submission_worker.py` during evaluation
Root cause: Validation Gap

Hi EvalAI team,

We're trying to host a challenge for a question answering shared task at our NAACL Workshop 2022, but all of the submissions I tried by myself to test evaluation scripts failed because of a circular reference detected in [`submission_worker.py`](https://github.com/Cloud-CV/EvalAI/blob/master/scripts/workers/submission_worker.py). I run the evaluation script by myself following [the section in README](https://github.com/Cloud-CV/EvalAI-Starters#test-your-evaluation-script-locally) instruction and it worked perfectly. 
Please see the error message below. 
```
[2022-03-17 20:14:13] INFO WORKER_LOG Processing message body: {"challenge_pk": 1638, "phase_pk": 3235, "submission_pk": 184979, "is_static_dataset_code_upload_submission": false}
[2022-03-17 20:14:13] INFO SUBMISSION_LOG [x] Received submission message {"challenge_pk": 1638, "phase_pk": 3235, "submission_pk": 184979, "is_static_dataset_code_upload_submission": false}
[2022-03-17 20:14:24] ERROR SUBMISSION_LOG Exception while receiving message from submission queue with error Circular reference detected
Traceback (most recent call last):
File "/code/scripts/workers/submission_worker.py", line 712, in process_submission_callback
process_submission_message(body)
File "/code/scripts/workers/submission_worker.py", line 679, in process_submission_message
user_annotation_file_path,
File "/code/scripts/workers/submission_worker.py", line 609, in run_submission
submission_result = json.dumps(submission_result)
File "/usr/local/lib/python3.7/json/__init__.py", line 231, in dumps
return _default_encoder.encode(obj)
File "/usr/local/lib/python3.7/json/encoder.py", line 199, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/usr/local/lib/python3.7/json/encoder.py", line 257, in iterencode
return _iterencode(o, 0)
ValueError: Circular reference detected
```
Our challenge: [challenge-page/1638/overview](https://eval.ai/web/challenges/challenge-page/1638/overview)
Github repository: [MIA-Shared-Task-2022_EvalAI](https://github.com/mia-workshop/MIA-Shared-Task-2022_EvalAI)
Could you help us to figure out what happens? Our competition is already kicked off and we will release our test data soon, so we need some help immediately. 
We really appreciate your help! 

The error has been solved and I close this issue.
Repository: Cloud-CV/EvalAI
Issue: Show count in Dashbboard view cards
Root cause: Unimplemented Feature Gap

Add count to all mentioned cards in Dashboard. 
Example: for card participated challenges show number of participated challenges of user just above the text written and over the image.

@Ram81 : Sure, go ahead.

@aka-jain any hints?

Go thorugh the API doc first at /docs on your django server. From that, you can get the API's that will be needed here to call count of each card. If count field doesn't exist then you can use length to find count of an Object.
Got it? 
You can take help related to API calls from other controllers. In case you want to know more about API services in angular then let me know, happy to help you :)

@deshraj : Can you list all the API's here related to cards in Dashboard?



@Ram81 if you have started working on this, then can you please check the /docs and let me know if you can find the relevant API endpoints. 

@deshraj  I'll see & let you know if any doubts!

@deshraj @aka-jain for create challenge card I have to use API Endpoint from challenges App,am I right?

@aka-jain  are the API's ready for this issue,so that I can start to work on it?

@Ram81 we are not ready with this. Once we have the APIs ready for this, I will let you know. 

We are not planning to show the current cards on the dashboard. It will be replaced by a news feed. hence closing this issue.
Repository: Cloud-CV/EvalAI
Issue: Modify the distance of the footer from the top of the screen
GitHub labels: enhancement

GSOC-2019

easy_to_fix

frontend

The footer on the dashboard should start from the height equal to the height of the screen. Please let us know if you have doubts. 

@deshraj Can I work on this ?

Sure. Assigning it to you. 

Fixed in #420
Repository: Cloud-CV/EvalAI
Issue: Problem with run python migrate
Root cause: Configuration Error

Hi
I want to run EvalAI on Ubuntu 22.
But when I want to run the below command: 
`python manage.py migrate`

It raises an error like this:
_File "/evalai/venv/lib/python3.8/site-packages/django/conf/__init__.py", line 176, in init
raise ImproperlyConfigured("The SECRET_KEY setting must not be empty.")_

Currently I am using python 3.10

Can anyone help me with this problem?
Thanks in advance

To solve this problem open the _manage.py_ file
and add .env extenstion to settings. so replace this:
`os.environ.setdefault("DJANGO_SETTINGS_MODULE", "settings")`
with this:
`os.environ.setdefault("DJANGO_SETTINGS_MODULE", "settings.env")`

Closing this since it is a very old issue and is already fixed.
Repository: Cloud-CV/EvalAI
Issue: Blank screen is coming while running django server
GitHub labels: bug

enhancement

GSOC-2019

easy_to_fix

backend


![image](https://cloud.githubusercontent.com/assets/16155501/21747806/a3188e16-d528-11e6-8071-170111bd8aca.png)

localhost:8000 is blank and django server is throwing warnings.

update repo. and remember to pip install the requirement files too.

@anweshknayak Still not working

update the bower components too

and there were some recent changes in README have a look

@jainarunarunjain is the issue resolved or not?

i had already done this but nothing happened


Can you make request to any API endpoint? 

@deshraj yes, api request is working

@jainarunarunjain do gulp dev and gulp connect and runservser simultaniously and goto localhost/8888.....there is the actual website..... i think localhost/8000 is just the backend part and the views r not directly linked to html thts wht it is so......not so sure abt it.......but the site is at localhost/8888 not localhost/8000 thts for sure

@deshraj i dont think this is a bug, this is how it is i guess....(just guessing :-P )

No, you are right. I have discussed with @jainarunarunjain the same on Gitter. Thanks for your input @parismita :-) Glad that you are trying to help others. 

@deshraj : Making template path as frontend/dist is solving my problem. Is it a right way to solve this issue?

error still persists? Please reopen @dexter1691 @deshraj @aka-jain 

@deshraj : Please have a look to this issue. 

Sure, It's already on my todo list. Will fix it soon. 

Fixed in #615. Now, we have to make sure that the 500 and 404 page of django has its own static files and should not interfere with the angular files since both are independent.
Repository: Cloud-CV/EvalAI
Issue: Submission worker exits after checking for requirements.txt
Root cause: Validation Gap

Hi,

I'm trying to host a competition using github, which seemed like a really easy and fast way to set one up. 
Currently though, I'm having the submission worker on evalAI exit after checking for requirements.
The following is what I see under  Challenge Settings > Evaluation > Submission worker logs

```
[2022-06-02 19:40:01] INFO WORKER_LOG Using /tmp/tmpelrmcff0 as temp directory to store data
[2022-06-02 19:40:01] INFO No custom requirements for challenge 1759
[2022-06-02 19:40:54] INFO WORKER_LOG Quitting Submission Worker.
```

and under "My submissions", I see my last submission in the "Running" state for the past many hours.
Can someone please help me figure out what's wrong?

Thanks!
  


Closing this since it is a very old issue.
Repository: Cloud-CV/EvalAI
Issue: How to store large amounts of test annotation data?
Root cause: Documentation Deficiency

Hello,

we are trying to host a challenge which would compare user's submissions with ground truth image data.
We have roughly 10GB of such ground truth data which needs to sit on the evaluation server.
It is not obvious where to put these files, I assume that pushing those into the template github repository is not the option here?

Thanks!

Please use this to push your large challenge annotations: https://cli.eval.ai/#challenge-annotations