zhiminy/EvalEng huggingface dataset:
Repository: Cloud-CV/EvalAI
Issue: check for code formatting
GitHub labels: medium-difficulty

GSOC-2025

Check if the code is formatted. Pre-commit hooks by black and flake8 exist, but I don't think we have been checking the score. 

- `pyproject.toml` isn't updated.
- `.pylint` & `.flake8`: all these files present in the home directory are not updated. 

Solution:

Check the code to get the pylint score. Then, update the code using black and flake8 to maximise the score.

Hi @gautamjajoo , I'd like to work on this issue

Sure, please take this.

Hey @gautamjajoo  @RishabhJain2018 Can I work on this issue since @Zahed-Riyaz  is already working on #4539

Hey @Akshat453 , #4539 is paused for a while, hence @Zahed-Riyaz picked this one.

Hi, @RishabhJain2018 @gautamjajoo 
So as of now simply running `black .` bumps up the score from 7.11 to 7.12. 

- `pre-commit-config.yaml` didn't recognise flake8 as a hook in specified repository so I updated it's repo to `https://github.com/pycqa/flake8`.
- The initial maximum score is `7.12` 

Are there any changes you'd like me to make to increase this further? 

For now I'm going to run hooks and make specific changes in the directories flake8 suggests has scope for improvement


```
pylint --recursive=y . | grep "Your code has been rated"
output : Your code has been rated at 7.23/10 (previous run: 7.23/10, +0.00)
```
After resolving the code quality issues in the codebase and with the errors `.flake8` and `.pylintrc` have been set to ignore ,the maximum Pylint score that can be achieved is `7.23/10`.
Repository: Cloud-CV/EvalAI
Issue: Migrate from Travis CI/CD to Github CI/CD
GitHub labels: CI

GSOC-2025

hard

### Issue:

Currently, we use TravisCI for our CI/CD builds, which costs us $73/month. As a cost-cutting initiative, move to Github CI/CD since it's cheaper.


### Deliverables:

- [ ] Working CI/CD pipeline with Github CI/CD.

@RishabhJain2018 can you please assign this one?

Sure @lazyperson1020 !

@RishabhJain2018 I have written a `ci-cd.yml` workflow file which I tested on my own fork. It builds and runs the frontend and backend tests. For deployment stage there are a lot of secrets and I haven't tested that for now seprately. Do I raise a PR for build and test jobs?

<img width="1014" alt="Image" src="https://github.com/user-attachments/assets/70ab6a05-fc66-4598-81ac-6757d54fb9df" />

Sure, please open a PR.

Hey @lazyperson1020 Any PR?

@RishabhJain2018 apologies for late response. Will raise soon.
Repository: Cloud-CV/EvalAI
Issue: Add missing organization Images
GitHub labels: bug

enhancement

easy_to_fix

frontend

good first issue

GSOC-2025

### Issue:
During the recent deployment to staging server https://staging.eval.ai on the landing page, the following organizations images are missing.
<img width="1219" alt="Image" src="https://github.com/user-attachments/assets/a4081fbf-b35e-4ac0-91b7-686dbaafd6e0" />

### Deliverable:
- Find and add the images for the organizations.

@RishabhJain2018 could you please assign this to me?

Sure!

Hi, can I be assigned to this too or is it already almost complete?

Hey! @RishabhJain2018 I’ve started working on issue #4557 and making good progress. I was wondering if I could also take up another small issue in parallel to contribute more. Let me know if that’s okay!

Hey @dlegend4000 @abhishekaj28 , @aditi-dsi is already working on it. Please pick other issues.

Hey @abhishekaj28 
> I’ve started working on issue https://github.com/Cloud-CV/EvalAI/issues/4557 and making good progress.

Please work on one issue at a time. I have a bunch of things to be implemented.

@aditi-dsi Are you stil working on this or should i go for it ?


@itsmanasdash on it.
Repository: Cloud-CV/EvalAI
Issue: Server EvalAI Not Updating Evaluation Script
Root cause: Configuration Error

My challenge hosted on EvalAI (prediction based) has an evaluation script that runs whenever a new submission is submitted. However, when I have been attempting to change this evaluation script, the error in the worker logs report OLD code that is removed from the codebase (the GitHub repo that is used to host the evaluation script code has a successful build.) I'm wondering why the code on the server side is not being updated when the GitHub builds are successful? Thanks! 

Closing this since the issue is already communicated!
Repository: Cloud-CV/EvalAI
Issue: API Documentation Not Loading - HTTP ERROR 500
GitHub labels: bug

enhancement

docs

GSOC-2025

### Description
The documentation for the API hosted at `https://127.0.0.1:8000/api/docs/docs.yaml` is not loading and throws an HTTP ERROR 500.

### Stack Trace
```
Error: Error downloading https://127.0.0.1:8000/api/docs/docs.yaml HTTP ERROR 500
    at https://evalai.s3.amazonaws.com/static/drf-yasg/redoc/redoc.min.js:55:4725
    at https://evalai.s3.amazonaws.com/static/drf-yasg/redoc/redoc.min.js:108:35852

Error: HTTP ERROR 500
    at https://evalai.s3.amazonaws.com/static/drf-yasg/redoc/redoc.min.js:55:4725
    at https://evalai.s3.amazonaws.com/static/drf-yasg/redoc/redoc.min.js:108:35432
```

**ReDoc Version:** 2.0.0-rc.14  
**Commit:** 121bf64  

### Steps to Reproduce
1. Navigate to `https://127.0.0.1:8000/api/docs/docs.yaml`.
2. Observe the HTTP ERROR 500 response.

### Expected Behavior
The API documentation should load successfully and display the interactive ReDoc interface.

### Actual Behavior
A "Something went wrong..." error page is displayed with the HTTP ERROR 500 message.

Please assign me this issue
Repository: Cloud-CV/EvalAI
Issue: Expire CloudWatch Logs Based on Challenge End Date
GitHub labels: enhancement

medium-difficulty

backend

GSOC-2025

### Description:

Currently, our CloudWatch logs are not set to expire, which leads to unnecessary log retention and potential cost accumulation over time. To manage our AWS resources more efficiently, we should configure the CloudWatch logs to expire in alignment with each challenge's end date.

### Proposed Solution:

Identify the CloudWatch log groups associated with each challenge.

Set a retention policy on these log groups based on the respective challenge's end date.

Please assign me this issue 

Done!

Sir could we add tests as per the PR https://github.com/Cloud-CV/EvalAI/pull/4406 to effectively test `aws_utils.py`

Yes, please.
Repository: Cloud-CV/EvalAI
Issue: Follow-up: Challenge Hosting Request - Few-Shot Industrial Defect Detection

Dear EvalAI Team,

I'm following up on a challenge hosting request I submitted via email to `team@cloudcv.org` several days ago. The challenge is titled:

**Few-Shot Screen Industrial Defect Detection Challenge**

### 📌 Submission Summary:
- Focus: Few-shot object detection in industrial screen defect inspection
- Format: Pascal VOC (XML)
- Metric: mAP@0.95
- Public leaderboard: Yes
- Daily submission limit: 2
- Duration: April 15, 2025 – May 10, 2025

### 📎 Attachments in Email:
I attached the following two zip files:
- `evalai_challenge_config.zip` (contains `challenge_config.yaml`, `phases.yaml`, `evaluation_script/evaluate.py`)
- `evalai_markdown.zip` (contains `overview.md`, `evaluation.md`, `submission_guidelines.md`, `terms_and_conditions.md`)

The email was sent from: **245053998@qq.com**  
Sender name: **Ting Wang**

I would appreciate it if you could confirm the receipt of my submission and let me know if any further action is needed to proceed with hosting the challenge.

Thank you for your time and support!

Best regards,  
**Ting Wang**


Hi @image-based-screen-industrial-detect , A team member has responded to you on the email. Please check your inbox. Please feel free to re-open if the issue still persists.
Repository: Cloud-CV/EvalAI
Issue: Add Github Icon to link with challenge repository
GitHub labels: enhancement

frontend

GSOC-2025

### Description

We need to add a GitHub icon to the challenge interface that links to the repository. This icon should be placed to the left of the stars box in the header area of the challenge page.

### Requirements

- [ ] Add a GitHub icon adjacent to the stars counter (left side)
- [ ] Link the icon to the challenge repository URL
- [ ] Set visibility permissions so the icon is only visible to:
    - Challenge host
    - Host team members

### Technical Details

- Use standard GitHub icon from your icon library
- Implement proper permission checks to restrict visibility
- Ensure the icon maintains proper spacing and alignment with existing UI elements

Hey @RishabhJain2018 can i work on this issue ?

Hey @Akshat453 , Sure!

Hey @RishabhJain2018 Which link should open when the github icon is clicked?

It should be linked to the github challenge repository of the challenge.
Repository: Cloud-CV/EvalAI
Issue: BUG: Negative Time Displayed in Submission Countdown Timer
GitHub labels: bug

enhancement

frontend

GSOC-2025

**Description:**

We encountered a UI bug where the countdown timer for the next set of submissions displays a negative value for seconds. This happened right after hitting the submission limit for the day.

**Steps to Reproduce:**
1. Submit until the daily limit is reached.
2. Observe the countdown timer for when submissions will be available again.

**Expected Behavior:**
The countdown timer should not display negative values. It should reset cleanly to `0 seconds` or roll over appropriately.

**Actual Behavior:**
The seconds field briefly or persistently displays a negative number, e.g., `-1 seconds`.

**Screenshot:**
![Image](https://github.com/user-attachments/assets/fdb06437-88cb-46d5-b998-d7933152965d)

Hello @RishabhJain2018 I'm new to this codebase, but I'm excited to give this a try. Let me know if that's okay!

Sure, please give it a try!

@RishabhJain2018 only the PR from the first contributor who commented is considered acceptable, but it looks like he made his submission without knowing that.

> [@RishabhJain2018](https://github.com/RishabhJain2018) only the PR from the first contributor who commented is considered acceptable, but it looks like he made his submission without knowing that.

@Amarta113 Sorry as this was not assigned to you and it was not solved i thought it would be fine if i take this one

Hi @Sekhar-Kumar-Dash, the issue was actually assigned to me. It seems you may have missed the earlier comment and the announcement in the channel:

Issues will be assigned on a first-come, first-served basis. The first person to comment on an issue will be assigned to it, and any other PRs for the same issue will be closed.
Please take only one issue at a time and focus on completing your assigned task before picking another.
There are plenty of issues available for everyone, so no rush—take it easy! 

hey @Amarta113 I get that . Actually i meant the issue  is not assigned to you meaning this issue does not have any assignees thats the reason i got confused and thought this may be open for everyone.
My bad sorry again for this
Repository: Cloud-CV/EvalAI
Issue: 500 Internal Server Error when validating challenge config via GitHub Actions
Root cause: Validation Gap

Hi EvalAI team,

I'm currently setting up a challenge using GitHub Actions, following the [official documentation](https://github.com/Cloud-CV/EvalAI/blob/master/docs/source/host_challenge.md#host-challenge-using-github). I've pushed the challenge files to the challenge branch of my repository:
🔗 https://github.com/image-based-screen-industrial-detect/industrial-defect-challenge/tree/challenge

However, the GitHub Action fails during validation with the following error:

Following errors occurred while validating the challenge config:  
500 Server Error: Internal Server Error for url: https://eval.ai/api/challenges/challenge/challenge_host_team/4014/validate_challenge_config/

Exiting the challenge_processing_script.py script after failure  
Error: Process completed with exit code 1.

Exiting the challenge_processing_script.py script after failure  
Error: Process completed with exit code 1.
What I’ve already verified:

The challenge_config.yaml file exists and passes YAML validation

All required files exist, including:

challenge_details/*.html

evaluation_script/evaluation.py (with a valid evaluate() function)

test_annotations/test.json

Paths in the config are correct

The repo secrets (EVALAI_HOST_URL, EVALAI_AUTH_TOKEN, HOST_TEAM_PK) are properly set

Still, the API call to validate the config results in a 500 error, which makes it hard to debug further on my side.

Could you help investigate what might be causing this issue on your backend or guide me on how to further debug?

Thank you!

Hi @image-based-screen-industrial-detect,
 I think you are missing in these details:

![Image](https://github.com/user-attachments/assets/e3d2530c-d36d-4a5c-87b9-cb4e90ce1b46)



I would suggest you to take this repository private before you add these.
Repository: Cloud-CV/EvalAI
Issue: 📛 Issue: host_config.json contains placeholder values in EvalAI Starters
Root cause: Configuration Error

Issue described here: https://github.com/Cloud-CV/EvalAI-Starters/issues/105#issue-3006534271

@RishabhJain2018 could you please assign this to me?

Sure!

@RishabhJain2018 raised a PR here: [https://github.com/Cloud-CV/EvalAI-Starters/pull/106](https://github.com/Cloud-CV/EvalAI-Starters/pull/106)

https://github.com/Cloud-CV/EvalAI-Starters/pull/109 fixes the issue.
Repository: Cloud-CV/EvalAI
Issue: Correct pagination count in team page.
Root cause: Algorithmic Error

Initially when we have less than 10 results on existing teams then its page count is shown as 0 but it should be 1. 

Refer to #462 

@aka-jain  can you assign this one to me?

Sure, go ahead. Assinging this to you.

any details?

Check existing teams card in team view, you can see the current page number 0 but it should be 1 if we have atleast 1 team in existing team. It gets 1 when you crossed limit of 10 teams.

Okay will do that & let you know.

Closing it since now it is a part of #488
Repository: Cloud-CV/EvalAI
Issue: Bug: Incorrect URL Validation on First Name, Last Name and Affiliation section

**Description:**
When attempting to update the profile, if changes are made to the First Name, Last Name, or Affiliation fields, the update fails.
An error message is shown (as seen in the image below). Other fields (like username or social links) update correctly. This issue only occurs when one of the above fields (First Name, Last Name, or Affiliation) is modified.

**Steps to Reproduce:**

-  Go to the "Update Profile" page.

-  Modify the First Name, Last Name, or Affiliation field.

-  Click the "Submit" button.

-  See an error message, shown in the image.

**Expected Behavior:**
The profile should update successfully when valid First Name, Last Name, or Affiliation values are provided.

**Actual Behavior:**
The profile update fails with a general error message whenever First Name, Last Name, or Affiliation is changed.

Additional Notes:
Other fields (like username and social links) work fine when updated.


![Image](https://github.com/user-attachments/assets/ccf20687-747a-46fd-bdd3-262f8615975b)

Hello @RishabhJain2018 , please take a look into it, I am working on it.

Hello @RishabhJain2018 can you assign this issue to me

@Pranay096  it is already done.
Repository: Cloud-CV/EvalAI
Issue: Pagination not working in teams view.
Root cause: Algorithmic Error

Need to solve it for both participant and host team view.

@aka-jain shall i try this?? please give some details

Sure, in team view there are buttons for next and previous which are not working. Logic is already there. You only have to check and make it correct.

One update, this is working fine when you reload the page. Basically when we create a team it appends to existing team at run time but it is not affecting pagination logic, which works on every page reload. 
Please correct this logic so that without reloading if we exceed limit of 10 then pagination buttons should get activated.

@aka-jain vm.existTeam is not updating with click, thts why this is hapenning, nothing is updating in the js file....what is to be done now?

@parismita : I think it gets updated with success function of create team not with click. Isn't it?


@aka-jain but i did console.log() of response....it was not updating...if team is created then it should

@aka-jain same as #488 right ?

@anweshknayak : you can work on this. I think there is some error with this issue. Just try  and let me know if you find any error?

@parismita : Thanks for the efforts, I'm assigning this to @anweshknayak for now.

sure @aka-jain 

@aka-jain by any chance this maybe cache related issue is it ? and $state.reload() would this even help in the controller. I don't think just asking.

@anweshknayak  Yes, there is nothing like cache thing issue. And don't use reload too. Let me know where you are exactly stucked?

@aka-jain pagination logic is all fine. can you give some hints?

@anweshknayak @aka-jain the updating is done after sucess function right?....then the sucess function not called after submit...possibly....even when i console logged only once i got 200 response

@parismita I think the pagination part is not being updated while creating new team. You pointed out the real issue I guess.  anyway @aka-jain is working on this. 

Thanks guys for helping. 
I think there is some confusion related to this task. Let us handle this.

@aka-jain I have fixed this issue. Shall I put up a PR ? 

Sure @anweshknayak, please go ahead. I will check it :)

@aka-jain @anweshknayak is this issue resolved? 

@deshraj Only challenge participation team pagination is left. Let me complete that too then we can close this issue.

@aka-jain all the paginations done? 

@aka-jain @anweshknayak Is this resolved? If yes, please add a reference to the PR.
Repository: Cloud-CV/EvalAI
Issue: Been in submitted for a long time
Root cause: External Dependency Breakage

![Image](https://github.com/user-attachments/assets/41ada0fb-38fc-4020-b5bd-47449451a566)

nuscenes detect

![Image](https://github.com/user-attachments/assets/26ecc29b-547f-4920-a68a-03cee1ae8977)

The same problem, I hope it can be solved quickly.

Hey guys, I am having the same issue. According to their FAQ, you should contact them: https://eval.ai/contact#:~:text=If%20you%20encounter%20this%20message,stuck%20at%20submitted%20or%20running.

> Hey guys, I am having the same issue. According to their FAQ, you should contact them: https://eval.ai/contact#:~:text=If%20you%20encounter%20this%20message,stuck%20at%20submitted%20or%20running.
Pls have you solved this problem？


Same problem...
Someone managed to solve it?

Hi @ctyfang @uzielroy @dodeokwangyooo @Jctrp, This should be resolved now. Please check and confirm.

Thanks for getting back to us Rishabh. My jobs are still in submitted status. I tried to resubmit, but now the portal just freezes when submitting. I'm submitting to Argoverse 2 and not NuScenes.

> Hi [@ctyfang](https://github.com/ctyfang) [@uzielroy](https://github.com/uzielroy) [@dodeokwangyooo](https://github.com/dodeokwangyooo) [@Jctrp](https://github.com/Jctrp), This should be resolved now. Please check and confirm.

My job is still in running status (nuscenes detect)

Hi @dodeokwangyooo , The nuscenes issue should be fixed.

Hi @ctyfang , We have emailed the hosts of Argoverse challenge since they manage the evaluation setup for the challenge.

nuScenes detection challenge status always running, how can I resolve it? Thank you. @RishabhJain2018 

![Image](https://github.com/user-attachments/assets/85561935-2e63-4521-a8f9-22a41131c612)

> > Hi [@ctyfang](https://github.com/ctyfang) [@uzielroy](https://github.com/uzielroy) [@dodeokwangyooo](https://github.com/dodeokwangyooo) [@Jctrp](https://github.com/Jctrp), This should be resolved now. Please check and confirm.
> 
> My job is still in running status (nuscenes detect)

Hello, how do you solve this problem?

I also submitted twice, and both are stuck in the 'running' state. What should I do?

my nuscenes detect task is also stuck at “submited”. @RishabhJain2018
Repository: Cloud-CV/EvalAI
Issue: Cannot Access Participant Management for Private Challenge
Root cause: Unimplemented Feature Gap

Hi EvalAI Team,

I’m the host of a private challenge named **"Few-shot Industrial Screen Defect Detection Challenge 2025"**. I've successfully submitted the approval request, and everything works fine, including submissions and evaluations.

However, in the "Manage" section, I cannot find the **Participants** or **Add Participant** tab, so I'm unable to invite users to join the challenge while it’s still private.

Could you please check if participant management is enabled for this challenge, or advise on how to resolve it?

Thanks in advance!

Best,  

Challenge ID (if available): 2514


Hi @image-based-screen-industrial-detect , I recommend reaching out to the team at team@eval.ai.
Repository: Cloud-CV/EvalAI
Issue: Feature Request: Support Docker Image Uploads for Testing Environments
Root cause: Documentation Deficiency

During the evaluation phase, libraries such as PyTorch3D are required. If we install these dependencies in the init of the evaluation_script, each submission will need to reinstall the dependencies, which takes a lot of time. Therefore, we hope to submit a pre-configured Docker container instead. 
In issue(https://github.com/Cloud-CV/EvalAI/issues/4134) you said "create a docker container based on our worker image and share the URL." Is this service still supported? Best regards, thank you!

Hi @jzuern @rbler1234, Yes, I think this feature is avallable. Please share the public worker_image_url with the team@eval.ai over the email along with the challenge details and you should be able to use it.

Hi @rbler1234 Please let me know if you need more information on this?
Repository: Cloud-CV/EvalAI
Issue: Update Leaderboard Description

# **Description:**

The current description text under the Leaderboard section uses placeholder text (Lorem ipsum dolor sit amet...). 

This should be replaced with a meaningful description.

# **Based on the EvalAI docs suggested description is:**

The leaderboard displays a scoreboard listing participating teams and their current scores based on evaluation metrics. Each challenge has its own leaderboard, which can be filtered by phase and dataset split. Results are typically ranked using metrics such as mAP@0.95.

This change will improve the user experience and ensure clarity for participants engaging with AI challenges on the platform.

![Image](https://github.com/user-attachments/assets/1e9d32c4-cc8a-4ea8-afe4-053925f9c800)

Hello @RishabhJain2018 Please take a look!


hey @Amarta113 , this text is replaced by the challenge hosts for their challenge.
Repository: Cloud-CV/EvalAI
Issue: The nuscenes detect task has been in subbimt for a long time
Root cause: Others

can anyone solve the problem？

same problem @RishabhJain2018 

same problem

Hey @dingmiaomiao @xuanmingShang @dodeokwangyooo , I believe the problem is resolved now. Please check and confirm.

> Hey [@dingmiaomiao](https://github.com/dingmiaomiao) [@xuanmingShang](https://github.com/xuanmingShang) [@dodeokwangyooo](https://github.com/dodeokwangyooo) , I believe the problem is resolved now. Please check and confirm.

Now my task is stuck in the running phase. I want to know how to avoid this problem. @RishabhJain2018 

| Method | Method description | Project URL | Publication URL | Status | Execution time (sec.) | Submitted file | Result file | Stdout file | Stderr file | Submitted at

1 | 514 | None | None | None | running | None | Link | None | None | None | May 14, 2025 10:41:37 AM


> Hey [@dingmiaomiao](https://github.com/dingmiaomiao) [@xuanmingShang](https://github.com/xuanmingShang) [@dodeokwangyooo](https://github.com/dodeokwangyooo) , I believe the problem is resolved now. Please check and confirm.

i dont why my submit always has problem can u tell me why？ 

@RishabhJain2018 My task is stuck in the running phase too. I submitted it three days ago, but the status is still running now. Could you please check and fix it? 

<img width="1430" height="607" alt="Image" src="https://github.com/user-attachments/assets/20a18ad6-8adc-4a1e-a00e-7aed57564ecd" />
I submitted on Jul 24.