zhiminy/EvalEng huggingface dataset:
Repository: Cloud-CV/EvalAI
Issue: insecure/Unsupported Django Version
Root cause: External Dependency Breakage

Present:-
We are currently on Django 2.2 LTS and the Extended support for this version has been ended in April 2022
![image](https://user-images.githubusercontent.com/34577232/230734776-bc6206b4-8802-4077-bc17-014ca7575bc5.png)
So, this version will no longer get any security patch.

Future:-
            We should consider upgrading to Django 3.2 or 4.2 LTS . 
![image](https://user-images.githubusercontent.com/34577232/230735240-a4c88d6a-04a6-4656-b320-edf0a77a10d8.png)

This will require a major change as in django 4.2 they have introduced many changes few of the major once are
Change in URL Routing since 2.2
Support for some databases dropped.
Several new settings and some removed in 4.2


 @Suryansh5545  
I am at a beginner level right now and would love to take up this ticket. can you please assign it to me? thanks in advance!



@Jaishree2310 maintainers haven't commented on this yet so I am not sure about this. Also this will be a lot of work and I believe will require more than one PR.

@Suryansh5545 Thanks for pointing this out. What you are proposing will be an entire project of its own. Can you please shed some light on what will be the issues with continuing the current version for the time being?

@gchhablani The main issue currently will be that with the version 2.2 being deprecated , we will no longer receive security patchs for the vulnerabilities discovered after the april 2022 period. This isn't that big of a issue for now to justify to upgrade our django version immediately as it will be a major change and we can continue to use our current version for the time being. But perhaps should consider upgrading after the completion of current projects.
Repository: Cloud-CV/EvalAI
Issue: Can not submit the result parquet file (305 M) successfully on evalai
Root cause: Environment Incompatibility

Hello. Thanks for your great contributions!!!
I can not submit the result parquet file (305 M) successfully on evalai.
When I using CLI, the uploading process always terminated with following error. Even though I switched many networks.

“”“
evalai challenge 1719 phase 4020 submit --file ./submit.parquet  --large
19%|███████████████▌ | 3/16 [04:51<21:02, 97.15s/it]

There was an error while uploading the file: HTTPSConnectionPool(host='evalai.s3.amazonaws.com', port=443): Max retries exceeded with url: /media/submission_files/submission_259211/820b0aa5-3194-4b44-92f2-fc3d64745531.parquet?uploadId=w46EJjKLs.PoGxdOs7TTDYBgXrKVk4MT7Jeqqtk9lvKxxtW.1dhu8z72SdEa8FGdtkBgdSZYXQ2GUAZvuCGTPMA8G67.7uVbOa5sEhh825QUQk64QAfjHanPJ04rd1UD&partNumber=4&AWSAccessKeyId=AKIAIVU6WOHPESDHAVZA&Signature=NynjwkCqKzTuVvAgRYqWgwCe%2FSU%3D&Expires=1681144663 (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2396)')))
Repository: Cloud-CV/EvalAI
Issue: Increase scrollbar width
GitHub labels: easy_to_fix

frontend

GSoC-2023

minor issue

Currently, the scroll bar is too thin for the users to use it properly. We want to increase the width of the scroll bar just enough to help users.


![image](https://user-images.githubusercontent.com/29076344/231877873-d1fef42b-09c7-4a5d-8ddd-a2aff2f5cf98.png)



@gchhablani I can take this one. Please assign it to me.

@gchhablani I can do better design thinking and user experience ,please assign it to me 


@b9aurav I am assigning this to you.
@Jaishree2310 There is another frontend issue that I just posted. Can you please check that?

is this issue closed?
Repository: Cloud-CV/EvalAI
Issue: Add auto-collapse/expand for metrics on the leaderboard
Root cause: Unimplemented Feature Gap

Add `...` for loading more metrics on the leaderboard.
The metrics should get shown when the users scroll down but not unless they have reached there.



@gchhablani can I work on it?

Hey @gchhablani can I work on this?

Hi @gchhablani, can I contribute to this issue? If yes, please let me know how can I start. 
Thanks

Hi @gchhablani can I take this token? Please assign this to me.
Repository: Cloud-CV/EvalAI
Issue: Allow empty start and end dates on hosted challenges
GitHub labels: easy_to_fix

frontend

good first issue

javascript

GSoC-2023

Currently, the hosted challenges page assumes that the challenge will have start and end dates. However, this is not true as we allow users to change start and end dates later, so they can be empty at a given time.

We need to fix this to show boards without start and end dates.

See the relevant frontend code here: https://github.com/Cloud-CV/EvalAI/blob/923fbf92a9d08c8659e9c548b28ba8196ffe7578/frontend/src/views/web/hosted-challenges.html#L19-L21

@gchhablani I would like to take it up. 

@AyushR1 We are still not sure about this one. Will reopen if we decide to do this.
Repository: Cloud-CV/EvalAI
Issue: There is no documentation on deployment best practices and how to set up for debugging
Root cause: Documentation Deficiency

I'm struggling to find in the documentation any information about how to set up the dev environment for debugging the Django back end. Also, there is no information about deployment best practices, what are the configuration decisions for the monitoring, staging, production, etc.

Thanks for taking the time to report a bug in the EvalAI project, or propose a new feature! Before creating a new issue, please do a quick search to check that it hasn't already been filed on the [issue tracker](https://github.com/Cloud-CV/EvalAI/issues)._

If you'd like to report a bug, please include the following information:
- the expected behaviour
- the observed behaviour
- steps to reproduce the problem
- which operating system and/or browser you're using, if relevant
- (if possible) copies of the console/server logs

On the other hand, if you'd like to propose a new feature:
- please describe, as clearly as possible, what you'd like to see
- (if possible) include mockups/screenshots, if these would help convey the meaning better

Thanks!


Helllo, its a different topic, but maybe you can help me. Through a research I came across the following website, which shows the simulation
simulation environment of a tractor is shown (https://carla.org/2023/04/21/avl-off-road-simulation/).
The description states that the data from the simulation can be provided on request.
can be made available on request. We are very interested in the data and would
ask whether it is possible for you to make this data available
make this data available for research purposes?

Many thanks in advance and I look forward to your feedback. 

Silas (sila.scholz@ostfalia.de)
Repository: Cloud-CV/EvalAI
Issue: EvalAI status stuck on "Runnung"(challenge 1989 phase 3962)
Root cause: Resource Mishandling

Hi, I've been stuck with submissions on the test set for days, but it works fine on the validation set.
I submit it on Apr 16, 2023 6:59:47 PM but I forgot my submission ID.
Can anyone help me with this please?(challenge 1989 phase 3962)
![](https://user-images.githubusercontent.com/56541797/232544971-f2ac339a-0817-417c-99bf-0ac67cacd737.png)

I am seeing the same thing. One of my phases never finishes while other three finished with no problem.

I cannot submit more now.

Hey Everyone, Closing this since this isn't the issue anymore. Please feel free to open a new issue if it persists.
Repository: Cloud-CV/EvalAI
Issue: Challenge Images are not appearing in the UI

Bug:

- Expected: Challenge images should be rendered
- Observed: The images are not showing because the GET call for the images is failing with a 404 error

- steps to reproduce the problem:
        1. clone the repo and build services
       
- Operating system: Ubuntu 22.04.2 LTS
- Browser: Google Chrome and Firefox


![image](https://user-images.githubusercontent.com/49990128/233170157-76078d0f-a156-4312-87af-e457835f50aa.png)
![image](https://user-images.githubusercontent.com/49990128/233170288-e91c9e0a-540c-4f65-a1bf-41b10c3706d6.png)



@TriveniBhat This issue is resolved already. Its closed now.

Closing this since it is very old.
Repository: Cloud-CV/EvalAI
Issue: Rectify redirection in evaluation script documentation
Root cause: Documentation Deficiency

- **Observed behavior**
The link to starter templates for static code-upload challenge evaluation in the EvalAI docs is directed to the wrong location.

![doc](https://user-images.githubusercontent.com/77075449/233830728-9ea01300-c72d-49f8-9a54-7db683fe4747.gif)

- **Expected behavior**
The correct link can be found [here](https://github.com/Cloud-CV/EvalAI-Starters/blob/master/evaluation_script/main.py).

Additionally, there is a minor typo in the syntax.
![Screenshot from 2023-04-23 14-50-38](https://user-images.githubusercontent.com/77075449/233831095-13916f7b-cbe3-48cf-9668-1d75420a5ed2.png)
Repository: Cloud-CV/EvalAI
Issue: Fix reset password message for inactive users

Users who are inactive get "Resent Password Link Sent to Email" message but do not receive emails.
We should not send them emails but at the same time show appropriate message here.
![Screenshot 2023-04-28 at 2 42 43 PM](https://user-images.githubusercontent.com/29076344/235228078-64ff6a54-af62-41a3-b65e-d6f3023aefbf.png)
Repository: Cloud-CV/EvalAI
Issue: Failing get all challenges endpoints
Root cause: Validation Gap

```
{{host_name}}/api/challenges/challenge/all/all/all
{{host_name}}/api/challenges/challenge/all/all/public
{{host_name}}/api/challenges/challenge/all/unapproved/all
{{host_name}}/api/challenges/challenge/all/unapproved/public
```

These four API calls are failing with 500. I am not exactly sure why. Adding try-except does not help.
Needs further investigation.

@gchhablani Can you assign it to me? I would be happy to work on it.
Repository: Cloud-CV/EvalAI
Issue: Submission stuck at "Submitted"
Root cause: Others

Submission stuck at "Submitted" at webqa competition(https://eval.ai/web/challenges/challenge-page/1255/overview).
![image](https://github.com/Cloud-CV/EvalAI/assets/24732802/ada3dcc6-c2c6-4586-be96-c880d466f9c6)
The situation has been going on for two days, making it difficult to get evaluations.


Same problem

@balabooooo Which challenge do you face this problem for?

> @balabooooo Which challenge do you face this problem for?

@gchhablani BDD100K Multiple Object Tracking Challenge, thanks. And I have submitted for about two days.

@gchhablani Hi :), is there any solution?

@balabooooo I have reached out to the hosts. They need to update things on their end.
Thanks for your patience.

@gchhablani Thanks

Hey Everyone, Closing this since this isn't the issue anymore. Please feel free to open a new issue if it persists.
Repository: Cloud-CV/EvalAI
Issue: Modify the challenge configuration after the competition starts
Root cause: Documentation Deficiency

Can I modify the challenge configuration after the competition starts on eval.ai?  Currently, the results of Test-Standard2023 will not be displayed on the leaderboard. 
Can I make Test-Standard2023 public on the leaderboard after the competition starts?

Yes, you can. Just make changes in your github configuration and it will be updated.
Repository: Cloud-CV/EvalAI
Issue: My file stuck at “Submitted” at VizWiz Captions 
Root cause: Others

Hello @ram81 @David_Novotny @Rishabh_Jain @rishabh ,
the last 2 days, I noticed that the VizWiz Caption Challenge 2021 ( ID 739), 
fail to process my results and the status is always "Submitted". 
The team name is Chris Morfopoulos.
Can you please have a look ?

Thanks in advance!

@chmorfop Fixed.

Thank you @gchhablani !! Much appreciated!

Hey Everyone, Closing this since this isn't the issue anymore. Please feel free to open a new issue if it persists.
Repository: Cloud-CV/EvalAI
Issue: cannot submit/evaluate to eval.ai
Root cause: External Dependency Breakage

In couple days, I found I cannot submit to eval.ai or finish evaluation there. 

- I tried uploading files to website however it gets stuck there and never shows "successfully".
- I tried to use evalai cli however it complains "Could not establish a connection to EvalAI. Please check the Host URL."
- At the same time, my phase 2 never finish evaluating and always shows "running". It should takes ~240 seconds to finish.

Our challenge is: https://eval.ai/web/challenges/challenge-page/2089/overview

I suspect file uploading/downloading system is affected so that I cannot upload and evaluation script cannot download. Maybe caused by aws crash? 


Hey Everyone, Closing this since this isn't the issue anymore. Please feel free to open a new issue if it persists.
Repository: Cloud-CV/EvalAI
Issue: pip install -r requirements/dev.txt breaks
Root cause: Environment Incompatibility

Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-YodbBn/twisted/


Faced this while running a fresh install on different Ubuntu 14.04 PC.

Can you please post a screenshot ?

![screenshot from 2017-01-10 16 10 02](https://cloud.githubusercontent.com/assets/8139783/21803243/3d5d3210-d74f-11e6-8841-935588f4a09a.png)


@anweshknayak Please try this [link](http://stackoverflow.com/questions/35991403/python-pip-install-gives-command-python-setup-py-egg-info-failed-with-error-c). May be the setuptools are not upgraded.

Tried that too before. Doesn't work

@anweshknayak please update the python version to solve this problem. 

I think the issue is solved now. Closing it. Please feel free to reopen if you are still facing this problem. 

refer this for solving (http://stackoverflow.com/questions/22938679/error-trying-to-install-postgres-for-python-psycopg2)
```

sudo apt-get update
sudo apt-get install -y build-essential
sudo apt-get install -y python3.4-dev
sudo apt-get install -y libpq-dev

```
Repository: Cloud-CV/EvalAI
Issue:  how to do search for challenge by keywords

may you help to understand how to do search for challenge by keywords
in
https://eval.ai/web/challenges/list
or github 
https://github.com/Cloud-CV/EvalAI

for example 
find changes related to short text classification
or 
find changes related to long  text classification



@Sandy4321 Thanks for reaching out regarding this.
Curretnly there is no way to do so.
We are working on some tags which the hosts can add to search challenges using.

Adding keyword search will be a bit challenging because we have a lot of metadata regarding each challenge, but we will keep this in mind.


so no way at to search 
maybe at lest 
some how to get list of projects?

@Sandy4321 Best way right now would be using the API endpoints to get all challenges.

Closing this since it is a very old issue.
Repository: Cloud-CV/EvalAI
Issue: Incorrect link to remote evaluation script in docs
Root cause: Documentation Deficiency

The page https://evalai.readthedocs.io/en/latest/evaluation_scripts.html#writing-a-remote-evaluation-script needs to be updated as it currently contains a link to https://github.com/Cloud-CV/EvalAI-Starters/blob/master/remote_challenge_evaluation/evaluation_script_starter.py 
which no longer exists as it is now a couple of scripts in the same folder.

Hello,I would like to work on this issue!

This issue has been fixed with PR #4536
Repository: Cloud-CV/EvalAI
Issue: Create endpoint to get list of all the challenges
Root cause: Unimplemented Feature Gap

Details about the endpoint:
- Url: /api/challenges/
- Header: None
- Method: GET
- Response:
  - status on success: HTTP_200_OK
  - status on exception:  HTTP_406_NOT_ACCEPTABLE
  - status on Unauthorized access: HTTP_203_NON_AUTHORITATIVE_INFORMATION

For more details, feel free to discuss on thread or gitter channel https://gitter.im/Cloud-CV/EvalAI

Fixed via #122
Repository: Cloud-CV/EvalAI
Issue: My file stuck at “Submitted” at BDD100k MOT
Root cause: Others

Hello @Ram81 @David_Novotny @Rishabh_Jain @rishabh   @gchhablani,
I’m sorry to bother you. I found that the BDD100K Multiple Object Tracking Challenge,
fail to process my results and the status is always "Submitted"（test set）.
The team name is xiong.
Can you please have a look ?

I really need this result and would very much appreciate your help and reply. Thanks in advance!

Solved. I don't know who helped me, but thanks a lot !

@qwomeng  楼主你好，请问你的MotionTrack论文中的github链接打不开，请问有开源打算吗？什么时候开源呀？
Repository: Cloud-CV/EvalAI
Issue: Leaderboard incorrectly marked as private
Root cause: Configuration Error

Hi team, I recently started a new challenge "Brain-to-Text Benchmark '24", but I can't seem to get the leaderboard to be publicly visible even though it is clearly marked as such in the config file. 

Here's what it looks like: [https://eval.ai/web/challenges/challenge-page/2099/leaderboard/4944](https://eval.ai/web/challenges/challenge-page/2099/leaderboard/4944). 

How can I get the leaderboard to be publicly visible? 

Resolved - the "visibility" parameter in the challenge_phase_splits section of the config file should be set to 3 not 1. If it's set to 1, the leaderboard will be private even if the leaderboard_public parameter is set to True.
Repository: Cloud-CV/EvalAI
Issue: I can no longer participate in any challenges.
Root cause: Environment Incompatibility

Recently, I found that my EvalAI account can no longer participate in any challenges. At the bottom of the challenge result submission webpage, it shows: Sorry, the participant team is not approved by host yet. The same is true for the accounts of people around me (their accounts are registered and used by me). What's going on?

@18979705623 please clear the cookies of your browser . That should fix the issue.

> 

This method works! Amazing.

Closing since it is resolved!
Repository: Cloud-CV/EvalAI
Issue: Evaluations stuck for VizWiz VQA test-dev set.
Root cause: External Dependency Breakage

Hello!

My submissions to the [VizWiz VQA challenge](https://eval.ai/web/challenges/challenge-page/1911/overview) have been stuck in 'Running' or 'Submitted' for the past two days. Looking at previous issues, it seems like this is a common problem. Does anyone know how we can resolve this?


Thank you.
Repository: Cloud-CV/EvalAI
Issue: Duplicate Challenge Phase split
Root cause: Validation Gap

When creating a challenge phase split, we are able to create challenge phase split with same dataset split id,  challenge phase id and leaderboard id, this shouldn't be the case as it creates a duplicate entry for no use.

To maintain a well-organized and clear competition environment, each challenge phase split should have a unique combination of identifiers:

1. Dataset split ID: This identifier designates a specific split of the dataset, such as "train," "validation," or "test."
2. Challenge phase ID: This identifier represents the particular challenge phase, such as "Phase 1" or "Phase 2."
3. Leaderboard ID: This identifier is used to differentiate between multiple leaderboards, for instance, "Leaderboard 1" or "Leaderboard 2."
Repository: Cloud-CV/EvalAI
Issue: Unable to distinguish between different leaderboards
Root cause: Unimplemented Feature Gap

We offer the flexibility to create multiple leaderboards for a challenge, each containing specific information about the labels for the metrics. However, the current setup only displays the challenge phase split options in the leaderboard dropdown. There is no way for the user to find out if the two phase split have different leaderboards.

To fix this we should either display the leaderboard name/id first then for selection then show the challenge phase split associated with the leaderboard

![image](https://github.com/Cloud-CV/EvalAI/assets/34577232/e6621702-3843-44e3-9779-038449b60fa5)


Hello
 @gchhablani 
 It's the first time I'm contributing to Cloud-CV, and I am interested in working on this issue. 
 I will review and understand the code base and begin working on this 
 If I face any issues i will reach out
 Thank You

I have completed my first issue can I take this next issue
@gchhablani 
and if passed will merge it

Can I begin working on this issue

Thank you


I'm interseted in this, please assign me this issue.

Hi I am keenly interested in addressing specific issues and am earnestly hopeful of having the opportunity to engage with them further.

Hi, I would like to contribute
Please assign this issue to me
@gchhablani @Suryansh5545 

is the issue still open if yes can you please link the resources i could refer for the same
thank you

hey @Suryansh5545 , please let me know is this issue need to fix?


@Imkkrish I guess @Akshat453 is already working on it. Feel free to take up other open issues

@gautamjajoo i will update my pr and let you know 


Hello! I'm Rahul, a beginner looking to make my first contribution to CloudCV for GSoC 2025. I'm interested in working on this issue. Could I please be assigned to it?


 @Suryansh5545 Hi if this issue is still active I would love to contribute!
Repository: Cloud-CV/EvalAI
Issue: Leaderboard id replaced in challenge phase split
Root cause: Algorithmic Error

Currently when a challenge host tries to update his challenge using the config and he has more than one leaderboard then the other leaderboards are removed and only one remains. This might be due to the https://github.com/Cloud-CV/EvalAI/blob/master/apps/challenges/views.py#L3890.
Repository: Cloud-CV/EvalAI
Issue: Low code coverage

We have a Current cove coverage of around 67% which is quite low. we should aim to increase the coverage upto 90%.
Below are the specfic functions and areas of code which remain uncovered and are causing low coverage.

**Base: utils.py**

    encode_data
    decode_data
    send_email
    get_url_from_hostname
    get_or_create_sqs_queue

**Challenge: aws_utils.py**

    register_task_def_by_challenge_pk
    create_service_by_challenge_pk
    update_service_by_challenge_pk
    delete_service_by_challenge_pk
    service_manager
    start_workers
    stop_workers
    scale_workers
    delete_workers

**admin.py**

    start_selected_workers
    scale_selected_workers
    restart_selected_workers
    delete_selected_workers

**admin_filters.py**

    lookups
    queryset

**Jobs: utils**

    is_url_valid
    get_file_from_url

**Jobs: aws_utils.py**

    generate_aws_eks_bearer_token

**tasks.py**

    download_file_and_publish_submission_message

**admin.py**

    make_submission_private
    change_submission_status_to_cancel

Additionally, I've noticed that some files under the "Scripts/worker" directory have a code coverage of 0%. It would be beneficial to address this and include appropriate tests for these files as well:

**Scripts/worker - remote_submission_worker.py**

    download_and_extract_zip_file
    load_challenge
    extract_challenge_data
    process_submission_callback
    process_submission_message
    extract_submission_data
    read_file_content
    run_submission

**Scripts/worker - submission_worker.py**

    extract_challenge_data
    load_challenge
    run_submission
    process_add_challenge_message

**Scripts/worker - worker_utils.py: (0% coverage)**

**Scripts/worker - rl_submission_worker.py: (0% coverage)**


Hey @Suryansh5545, I would like to contribute towards better test coverage. And also I believe that's a good starting point to contribute =)

I was thinking about starting to write some unit tests for `Jobs:utils`. 
Specially `is_valid_url`, sounds like a pretty much standalone method and a good starting point. Do you have any other suggestions on where to start?

I would appreciate any guidance on how to set up the dev environment for this (maybe I missed it in the docs).
I noticed that there isn't any test for `Jobs:utils`, so I also would appreciate any other information to get started, like how to properly structure the test suite and libs to use.