zhiminy/EvalEng huggingface dataset:
Repository: Cloud-CV/EvalAI
Issue: Fix the selected metric display label
Root cause: Architectural Constraint

### Expected behaviour
The selected metrics should not be visible in the metrics label.

### Observed behaviour
The metrics overflow when a significant number are selected. 

<img width="1139" alt="Screenshot 2024-05-02 at 12 32 43 PM" src="https://github.com/Cloud-CV/EvalAI/assets/24366008/f88f3505-629a-4ac3-a09b-69a5a558257a">



Hey @gautamjajoo , I want to look into this issue. Could you please elaborate it?

@gautamjajoo Can I work on this issue?

Sure @chiruu12 .

> Sure [@chiruu12](https://github.com/chiruu12) .

Thank you so much sir can you also mention the files for the same if possible?

:) I think that's the part that you need to figure out.
Repository: Cloud-CV/EvalAI
Issue: Decorator to check that Challenges are approved by admin or not.
GitHub labels: bug

enhancement

medium-difficulty

backend

Add a decorator to check whether a challenge is being `approved_by_admin` or not. Also add the test cases regarding the same.

@deshraj Can I work on this ?

Looks like you have a good insight about the APIs. Thanks for pointing this issue. 

You can surely work on this. Assigning it to you. :-) 

We are planning to include it in the near future after Version 1.0

Closing it for now.
Repository: Cloud-CV/EvalAI
Issue: Evaluation script working locally but not on server
Root cause: Environment Incompatibility

Hi,

I've followed the instructions to submit the challenge for EvalAI and evaluation script is working fine locally, but still has some errors on the server side:
`"/tmp/tmp8biownt_/compute/challenge_data/challenge_2325/__init__.py", line 49, in <module> from .main import evaluate File "/tmp/tmp8biownt_/compute/challenge_data/challenge_2325/main.py", line 5, in <module> from phase_1.main import evaluate_accuracy ModuleNotFoundError: No module named 'phase_1'`
What can we the problem? I even don't have these lines in the code now so I don't understand why does it use the old info.

Best,
Mariia

Hey,

I have similar issues. I am getting errors in the worker logs from the web interface that state that a module can not be found although I uncluded it in the requirements and it worked locally. Did you find a solution?

@mariiak2021 Did you find a solution for that? I am facing the same, and I cannot see any answer on the server. How could we get a faster response from the repo maintainers? Maybe @taranjeet @dexter1691 @deshraj  @krtkvrm. Could you please point me to a solution? or maybe a person can help me with that? We will receive submissions once I launch the evaluation dataset tomorrow; I would like to see a real-time leaderboard. That was the main reason I selected Eval.AI over other platforms. Appreciate your help on this. 

Hey @paularamo @mariiak2021 , Apologies for so delayed response. Were you able to resolve and fix the issue?

Closing this for now! Please reopen if the issue persists.
Repository: Cloud-CV/EvalAI
Issue: Fix leaderboard inconsistencies
Root cause: Unimplemented Feature Gap

- [ ] Highlighting the tab when phase is selected
- [ ] Clear the leaderboard option

Hey @gautamjajoo is this issue still pending? Can I take this issue as it is been 2 weeks?
Thanks!! 

@gautamjajoo can i get assigned to this issue?

@gautamjajoo can I get this issue assigned to me, if it's still pending?

Hey @Divyanjali-Mudili , Sure you can work on this.

Hey @ANNNA106 , Please pick any other issue.

@RishabhJain2018 the challenges are getting none. can you please tell me how to resolve this problem?

![Image](https://github.com/user-attachments/assets/8e906d43-2bb1-4d0d-b71f-ab8f5157836f)

Hey @Divyanjali-Mudili, Try rebuilding your django container.

@RishabhJain2018 assign me this issue, I'm working on this.


@capricorn-32 I'm working on it 

@RishabhJain2018 if this issue is still open I would love to contribute, if no one else is working on it already.

@Divyanjali-Mudili, are you still on this ?

Hi,I would like to work on this issue and contribute, can it be assigned to me?

Hello @RishabhJain2018 , I’m interested in tackling this issue. Is it available to pick up? Please assign it to me if it’s free; otherwise, could you point me to other good starter issues? Thank you!



Hi 👋

I’d like to work on this issue and add the MongoDB version check instructions to the README for Ubuntu users.
I have 2+ years of industry experience in a product-based company, have contributed to multiple open-source projects, and have hands-on experience working with MongoDB.

Please assign this issue to me so I can start working on it and submit a PR.

Thanks!

I tested this on latest master and couldn’t reproduce the issue.
Phase tab highlights correctly on selection and page reload.
Seems resolved already — please confirm if I’m missing a specific case.
Repository: Cloud-CV/EvalAI
Issue: Profile Section is not fixed when page is scrolled

When we open My Profile link from dashboard & try to scroll the position of profile section isn't fixed.

@aka-jain I will work on this for now.

@Ram81 : Sure.

@aka-jain  should I make the profile section fixed like dashboard-sidebar when opened or some other change if you want?

Yes, give it a fixed position for now, and integrate it with inner scroll in Profile section.

"integrate it with inner scroll in Porfile section." didn't quite get that?

@Ram81 : Make profile section scrollable in case when screen's height becomes less then the height of the content in Profile Bar. 

@aka-jain  should I Integrate inner scroll to profile section in this issue or do you want me to close this issue & wait for next version of the issue?
Repository: Cloud-CV/EvalAI
Issue: The evaluation stuck on "Submitted" status
Root cause: Resource Mishandling

@gchhablani HI, my evaluation stuck on the status of "Submitted", I see you helped lots of guys, can you help me with that? It's been 2 days after I submitted.

I evaluate on subtask of [KILT](https://eval.ai/web/challenges/challenge-page/689), Open Domain QA - Natural Questions & TriviaQA & HotpotQA.

Team: wenyang

Thank you!

It's been 6 days? Anyone help?

Hi, have you solved the problem?

Yes. I remember that I waited for 1 week, and suddenly it executed the running. There's nothing I can do but wait.
Repository: Cloud-CV/EvalAI
Issue: [Bug]: Editing of submission meta attributes not allowed if not provided during submission
Root cause: Validation Gap

**Description:**

Currently, EvalAI does not allow editing of submission meta attributes if they were not provided during the initial submission. This is a bug, as users should be able to add or edit meta attributes even if they were not provided initially.

**Steps to reproduce:**

Make a submission without providing a specific field (e.g. a text field).
Attempt to edit the submission.
Observe that editing is not allowed due to the missing field.

**Expected behavior:**

Editing should be allowed even if a field was not provided during the initial submission. The user should be able to add or update the field during the editing process.
Repository: Cloud-CV/EvalAI
Issue: [Enhancement]: Accepting files of incorrect formats instead of rejecting them
Root cause: Validation Gap

**Description:**

Currently, when submitting to EvalAI, the system is accepting files of incorrect formats instead of rejecting them. This can cause issues with the evaluation process

![image](https://github.com/user-attachments/assets/8e2901e7-f741-4af4-ad1f-13b8077ffcca)

**Steps to reproduce:**

Go to the submission page for a challenge on EvalAI
Attempt to submit a file of a format that is not accepted by the challenge (e.g. an image file for a challenge that requires a CSV file)
Observe that the system accepts the file and allows the submission to proceed

**Expected behavior:**

The system should reject files of incorrect formats and display an error message to the user, indicating that the file format is not accepted.

Hello, @Harshit28j To address the issue of accepting incorrect file formats, I suggest implementing a file validation mechanism on either server-side or client-side so that only the correct file format is submitted.
Repository: Cloud-CV/EvalAI
Issue: Frontend(Auth): Prevent unnecessary api calls 
GitHub labels: enhancement

frontend

- [x] Handle form's required field error on client side
- [x] check length of password (min. 8 characters) field before api call in login, signup etc.
- [x] in signup prevent api call if both passwords aren't same 

Verify form's parameters as much as possible on the client side.  No need to send bad request on server.

@spyshiv @aka-jain Can I work on this?

Yeah sure. Go ahead. 
assigning you

@spyshiv I think we should this?

Can't close this now. Need to add this feature to reset password and change password too. 

@spyshiv okk. So, can I work on that ?

As for now i am continuing this. Let me set some features then I will let you know when you have to work on this

okk sure. no problem. :-)

refer to #483 
Closing this issue
Repository: Cloud-CV/EvalAI
Issue: ​ Problem with remote evaluation worker
Root cause: Validation Gap

Hello everyone,

I am trying to set up a remote evaluation challenge and facing issues. When I submit a file in the web interface, the fetching as well as the evaluation script are working but at the end, I am getting this error:

Error: 400 Client Error: Bad Request for url: https://eval.ai/api/jobs/challenge/2389/update_submission/

I also printed out the data that is passed to the update_submission_data function:

{‘challenge_phase’: 4723, ‘submission’: 477497, ‘stdout’: ‘’, ‘stderr’: ‘’, ‘submission_status’: ‘FINISHED’, ‘result’: [{‘train_split’: {‘final_train_loss’: 0.00010908210573358123, ‘final_val_loss’: 0.0009243950532566357, ‘Metric3’: 123, ‘Total’: 123}}, {‘test_split’: {‘final_train_loss’: 0.00010908210573358123, ‘final_val_loss’: 0.0009243950532566357, ‘Metric3’: 123, ‘Total’: 123}}], ‘metadata’: ‘’}

Can someone help me whats wrong?

Thanks for your time and help


Does anyone working on this? If not interested to work on this issue. Please assign it to me.

Just to give an update. So I changed the structure of my results dictionary and now it runs without errors. However while the result files for the submissions are now present, the leaderboard remains empty, so I am not sure whether something with my format is still wrong. One result file in the web interface looks e.g. like this:

[{'final_train_loss': 6.025793465431145e-05, 'final_val_loss': 0.00013760417953432937, 'Metric3': 123, 'Total': 123}]

And this is my leaderboard setting in the config:

leaderboard:
  - id: 1
    schema:
      {
        "labels": ["final_train_loss","final_val_loss","Metric3","Total"],
        "default_order_by": "final_val_loss",
        "metadata": {
          "final_val_loss": {
            "sort_ascending": True,
            "description": "The last validation loss from training the linear probing layer",
          }
        }
      }
Repository: Cloud-CV/EvalAI
Issue: Error during Installation on m1 Macbook
Root cause: Environment Incompatibility

I tried installing Evalai on my m1 Macbook pro running macos 15.1 and docker compose version v2.29.2-desktop.2 (I copied the version from the about docker desktop section in the docker application). I followed the instruction for installation on the readme.md file and after cloning the repository when I run `docker-compose up --build` I get the following error.

_failed to solve: process "/bin/sh -c apt-get update && apt-get install -yq google-chrome-stable" did not complete successfully: exit code: 100_

I have checked if docker is working as expected.

I am still a beginner and i'm sorry if i have missed something.

The error you are encountering suggests that there is an issue with installing google-chrome-stable in the Docker container during the build process. This is often caused by outdated or missing repositories in the base image, or incompatibility issues between the image and the software being installed.  
@Vishesh-Mistry can i help you in fixing this


@RishabhJain2018 is this issue still persistent or we have some fix for this ?

Closing this since #4498 solves this.
Repository: Cloud-CV/EvalAI
Issue: Security Integration EvalAI with OSS-Fuzz for Continuous Fuzz Testing

Hello @gchhablani 

I hope this message finds you well! I’d like to propose integrating EvalAI with  OSS-Fuzz continuous fuzz testing platform designed to improve the stability and security of open-source project.

Why Integrate EvalAI with OSS-Fuzz?

Improved Stability: OSS-Fuzz can automatically detect edge-case bugs, crashes, and security vulnerabilities in EvalAI codebase.
Enhanced Reliability: Continuous fuzzing ensures that untrusted inputs, such as data from sensors, communication protocols, or user-defined configurations, are handled robustly.
Proactive Bug Fixes: By identifying potential issues early, OSS-Fuzz helps maintain a stable and secure codebase.


Can I move Forward with this?

Closing this since we don't need it now.
Repository: Cloud-CV/EvalAI
Issue: This is the new issue

_Thanks for taking the time to report a bug in the EvalAI project, or propose a new feature! Before creating a new issue, please do a quick search to check that it hasn't already been filed on the [issue tracker](https://github.com/Cloud-CV/EvalAI/issues)._

If you'd like to report a bug, please include the following information:
- the expected behaviour
- the observed behaviour
- steps to reproduce the problem
- which operating system and/or browser you're using, if relevant
- (if possible) copies of the console/server logs

On the other hand, if you'd like to propose a new feature:
- please describe, as clearly as possible, what you'd like to see
- (if possible) include mockups/screenshots, if these would help convey the meaning better

Thanks!
Repository: Cloud-CV/EvalAI
Issue: problem when submit on Argoverse Motion Forecasting Competition
Root cause: External Dependency Breakage

After uploading the results, the status remains' submitted ', has been 10days. How can I solve this problem? Anyone can help me?

submitted at Dec 23, 2024 1:37:29 PM

ID:hewyouw
[email:wanwk@stu.xidian.edu.cn](mailto:email:wanwk@stu.xidian.edu.cn)
team name is:wsgz


It has been 16 days since there were any new submissions on the Argoverse leaderboard.

same problem

@gchhablani  @RishabhJain2018  @deshraj  I have met the same question, is there any solutions?

Unable to submitted now

> Unable to submitted now

I also tried, it stuck when upload the h5 file.

And there are new recode(20hours) in other challenges, I don't know what make the difference

Hey Everyone, Closing this since this isn't the issue anymore. Please feel free to open a new issue if it persists.
Repository: Cloud-CV/EvalAI
Issue: File got stuck at in Argoverse Motion Forecasting Competition
Root cause: Others

### Challenge Name: Argoverse Motion Forecasting Competition
### Challenge URL:  https://eval.ai/web/challenges/challenge-page/454/overview
After I submitted the file, the status keeps “submitted”
submitted at Jan 3, 2025 8:19:53 PM
id: Dandeklion
email: [kainanyaoo@gmail.com](mailto:kainanyaoo@gmail.com)
team name is AET
@deshraj @gchhablani  @RishabhJain2018 

@aroundabout did you find a solution for this? I am facing the same issue.

@deshraj @gchhablani @RishabhJain2018
Hi, I meet the same issue, my team name is try111. Can anyone help me?

Hey guys, this shouldn't be an issue. Please feel free to reopen if it is the issue again.

@RishabhJain2018 Hi I meet the same issue again, can you checking it? Thank you!
Repository: Cloud-CV/EvalAI
Issue: Replace pagination code from utility function paginated_queryset in participants app
GitHub labels: easy_to_fix

backend

`paginated_queryset` is a function in `base utils.py`.

Idea is to remove paginator code with a function named `paginated_queryset`

```
paginator = PageNumberPagination()
paginator.page_size = settings.REST_FRAMEWORK['PAGE_SIZE']
result_page = paginator.paginate_queryset(participant_teams, request)
```

For reference, look at PR #422 

@anweshknayak I think you are already working on #300. Please try to finish that first. 

@deshraj @TroJan shall i try this??


Sure. Assigning it to you. 

@deshraj please check if its fine

Great work @parismita. :-) 

Fixed in #447. Closing it.
Repository: Cloud-CV/EvalAI
Issue: cannot receive an verify email
Root cause: External Dependency Breakage

i tried many times, using  incognito mode or clear your browser cache. even tried 5 accounts, when i was trying to evaluate my result.json in the VQAv2 ,link is here https://eval.ai/web/challenges/challenge-page/830/my-submission. anyone could help me? i guess is the website eval.ai's wrong.


I also encountered the same issue where I couldn't receive the verification email from EvalAI, even after multiple attempts. Have you managed to solve it? If so, could you share how you fixed it?

> 我也遇到了同样的问题，即使尝试多次，我也无法收到来自 EvalAI 的验证电子邮件。你成功解决了吗？如果解决了，你能分享一下你是如何解决的吗？

There seems to be an issue with the registration on the EvalAI platform.

@liudean998 Is there an issue during registration? The first time I tried to register, there was no response. When I tried to register again, it showed that the account was already registered. I haven't received the email verification when using this account to log in.


> [@liudean998](https://github.com/liudean998) Is there an issue during registration? The first time I tried to register, there was no response. When I tried to register again, it showed that the account was already registered. I haven't received the email verification when using this account to log in.

It seems to be an issue with the platform's page. In reality, your registration was successful, but the page does not automatically redirect. You can log in using the username and password you set during registration.

@liudean998 Yes, I have the same issue. I tried several accounts, but the problem persists. Have you registered an account before?

I also registered yesterday and tried several email addresses, but none of them worked. The problem is still not resolved today.

@liudean998 The official response today said they will try to update and resolve the issue.
Repository: Cloud-CV/EvalAI
Issue: My file stuck at “Submitted” VQA Challenge 2021
Root cause: Others

Hi, I've submitted the verification file for the **VQA Challenge 2021** several times, but it has been stuck in the submitted state. Could you help fix this? Thanks!

I've encountered the same issue. Could you help fix this? Thanks!

I've encountered the same issue. Could you help fix this? Thanks! jijijiji

same with me. I also submitted to VQA Challenge 2021.

I encountered the same problem. Is it possible to be fixed recently? Thank you for your contributions to the community.

I encountered the same problem. Could you help fix this? 

Thanks! 

the same!

I encountered the same problem. Is it possible to be fixed recently? Thank you for your contributions to the community

same problem

stuck for almost a month

Same issue! We would greatly appreciate it if this issue could be resolved promptly.
Thank you for your hard work and support! @RishabhJain2018 @gchhablani 


Hi Everyone, Closing this since the issue is fixed. Please feel free to reopen if the issue still persists.

I also submitted to VQA Challenge 2021 and I encountered the same problem. Is it possible to be fixed recently? Thank you for your contributions to the community

Hi @AlenjandroWang , Checking it. Thanks for reporting.
Repository: Cloud-CV/EvalAI
Issue: Several questions about testing the remote evaluation setup before publishing the challenge

Hi @RishabhJain2018,

I have two questions regarding the setup of a remote evaluation challenge that we want to run in March

1) We have several members in the Host Team, however currently only I, who created the challenge, sees the "submit" field on the web interface. Since we want to check the evaluation process also with other submissions, how could we enable the other members to submit via the web interface ?

2) In general we would also like to test our evaluation with submissions that are not from the host team before publishing the challenge. Is it possible to do so before sending the approval request or is it indended to do afterwards ?

Thanks so much for your time and support!

update, @rishabhbhardwaj , we are at the stage where we have made public our 2025 CVPR EARTHVISION data challenge https://eval.ai/web/challenges/challenge-page/2465 , but it seems it becomes accessible to the public after admin approval, only?

FYI, issue related: #4464

@rishabhbhardwaj , FYI, the challenge launched and the above questions are not relevant, feel free to close, and thx for all your hard work!

Closing this since this resolved!
Repository: Cloud-CV/EvalAI
Issue: Replace pagination code from utility function paginated_queryset in hosts app
GitHub labels: easy_to_fix

backend

`paginated_queryset` is a function in `base utils.py`.

Idea is to remove paginator code with a function named `paginated_queryset`

```
paginator = PageNumberPagination()
paginator.page_size = settings.REST_FRAMEWORK['PAGE_SIZE']
result_page = paginator.paginate_queryset(participant_teams, request)
```

For reference, look at PR #422 


@deshraj @TroJan  Can i work on this

Sure, go ahead. Assigning it to you. 

Fixed in #450
Repository: Cloud-CV/EvalAI
Issue: submission issue on FALCON challenge
Root cause: Validation Gap

Hey,
I encountered an issue while trying to submit my Docker image using the evalai push command. The command returned the following output:
status: Pushed  id: <my_docker_image_id>
There were no error messages, but the command did not output a submission ID. Additionally, I noticed that a new Docker image was created locally with the following name:
905418259932.dkr.ecr.us-east-1.amazonaws.com/few-shot-algorithms-for-consistent-neural-decoding-falcon-2319-participant-team-<my_team_id>  
This image has the same ID as the one I attempted to submit.
Could anyone help with this? Thanks!
Repository: Cloud-CV/EvalAI
Issue: The registration on the EvalAI platform seems to be having issues, and it's not functioning properly.
Root cause: Validation Gap


Hi, Thank you for reporting this. We are looking into it. We will update once it is resolved.

@RishabhJain2018 Thank you for your contribution to the website.

I cannot receive the approval mail 

@muktilin It seems that there was no redirection during registration, but when I clicked the register button again, it showed that the account is already registered.

@muktilin Do you have an account that works properly?

Same problem
Repository: Cloud-CV/EvalAI
Issue: Accessing submission metadata in Remove Evaluation
Root cause: Documentation Deficiency

Hi, is it possible to access submission metadata, particularly the `participant_team_name` field, when running remote evaluation?
We need to populate a custom leaderboard that is not ranked directly by the outcome by `evaluate()` and therefore need the team names.

The message_body retrieved from the `eval_ai_interface` seem to contain only the submission and challenge phase keys.
Is it possible to pull submission metadata to the remote worker?

Found it in the `submission` object.