zhiminy/EvalEng huggingface dataset:
Repository: Cloud-CV/EvalAI
Issue: Fail status for the submission on the nuScenes Challenge
Root cause: Validation Gap

Hi,  @gchhablani @Suryansh5545  I am really sorry to bother you, but I really need this result urgently because the deadline for CVPR is coming orz.  When I submit my result to the [nuScenes LiDAR Semantic Segmentation Challenge](https://eval.ai/web/challenges/challenge-page/720/participate), the final status is  Failed. Although I can see the evaluation results on the stdout,  I cannot make my submission public.

The content in the stderr is as follows:

```
Traceback (most recent call last):
  File "/code/scripts/workers/submission_worker.py", line 599, in run_submission
    LeaderboardData.objects.bulk_create(leaderboard_data_list)\
  File "/usr/local/lib/python3.7/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 474, in bulk_create
    ids = self._batched_insert(objs_without_pk, fields, batch_size, ignore_conflicts=ignore_conflicts)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 1204, in _batched_insert
    ignore_conflicts=ignore_conflicts,
  File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 1186, in _insert
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/sql/compiler.py", line 1377, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python3.7/site-packages/raven/contrib/django/client.py", line 127, in execute
    return real_execute(self, sql, params)
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.7/site-packages/django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.IntegrityError: null value in column "is_disabled" violates not-null constraint
```


I find the problem of "Failed status" seems to be related to the recent [commit](https://github.com/Cloud-CV/EvalAI/commit/ae017b46fbb406dadfd2bb44da025336022d9e60).  Could you please help me? Thanks! 



Fixed. Please send me an email in case this happens again.
Thanks!
Repository: Cloud-CV/EvalAI
Issue: My file stuck at “Submitted” at nocaps.
Root cause: Resource Mishandling

Hello,

@gchhablani 
I’m sorry to bother you. My submissions on the [nocaps](https://eval.ai/web/challenges/challenge-page/355/leaderboard/1011) keep getting stuck at 'submitted' status for a long time (~10hours).  Can you help me to solve it ? 
<img width="1142" alt="image" src="https://github.com/Cloud-CV/EvalAI/assets/19266420/0b0b21c6-3e78-49ae-b15a-5f1f02129891">


I meet the same problem, and I have submitted my results two days.@gchhablani 
![image](https://github.com/Cloud-CV/EvalAI/assets/95663989/c1365bbb-280a-4658-b7e3-656b406b8270)


@Pefect96 Fixed.

@gchhablani I have not obtain my results after submitting my file six days. It is always runing. Can you help me?
![image](https://github.com/Cloud-CV/EvalAI/assets/95663989/d655b80c-32c0-43cd-a52c-d1c227a2c84a)
Repository: Cloud-CV/EvalAI
Issue: stuck at TextVQA Challenge 2021
Root cause: Resource Mishandling

please help me.

I also encountered the same issue. Submission in Test-Standard and Test-Violation Standard are stuck at "Submitted" without evaluation

![Capture](https://github.com/Cloud-CV/EvalAI/assets/33834357/444c2ec9-8a99-45e7-b7bf-7ed67d792efd)
Please help fix the challenge. Thank you

@gchhablani Could you help fix this challenge. Many thanks for your help

@gchhablani Thanks for fixing it but it seems my new submission is stuck at the "Running" stage for few hours
![image](https://github.com/Cloud-CV/EvalAI/assets/33834357/c76acad4-337c-462c-9715-a7d2c9c5f7d8)
Repository: Cloud-CV/EvalAI
Issue: Hosting a code-upload based challenge locally
Root cause: Architectural Constraint

Hello,
I'm trying to use EvalAI to host a code-upload based challenge locally. The main issue I encountered is that everything is set up to run on AWS. 
Is there an official way to run EvalAI on production on a linux machine? If not, how do I uncouple the platform from AWS and introduce my machine instead?
Is there anything I should set up on my linux machine to host the challenge?

Are there any AWS-dependent components I should completely remove? If so, what should be my starting point?


I am not worried about the performance and/or memory limitations of my local machine. 
Thanks in advance!

Closing this since we are sunsetting code-upload challenge feature for now.
Repository: Cloud-CV/EvalAI
Issue: Submission Stuck on submitted for eBay 2023 ML challenge
Root cause: Resource Mishandling

My submissions to the 2023 eBay ML challenge keep getting stuck on submitted for hours. I am nowhere near my submission limits. My team name is rj-ai-de2. Please fix this so that I can submit and compete in the last couple weeks of the competition.

@gchhablani 
![IMG_6419](https://github.com/Cloud-CV/EvalAI/assets/77995559/78ebbff8-f910-4822-bb3b-a4da727920cf)
Repository: Cloud-CV/EvalAI
Issue: Improve docs for code-upload/static code-upload 
GitHub labels: easy_to_fix

docs

GSoC-2024

We need to add AWS links/tutorials needed for mounting EFS and adding data, etc.

Hi @gchhablani, can I contribute to this issue if its open? Thanks


@Bhavay-2001 Go for it. Not sure how much how understand EFS and AWS but sure!

@gchhablani Hey there, as a part of GSoC even I would like to contribute and if the issue is still open and no one is working on it, I would like to take this up.


Hello @gchhablani, I am interested in taking up this issue.

Hi, @gchhablani I can solve this issue please assign this issue to me.

Closing this since we are sunsetting code uplaod challenge feature.
Repository: Cloud-CV/EvalAI
Issue: Always runing on nocaps-XD
Root cause: Resource Mishandling

@gchhablani I have not obtain my results after submitting my file six days. It is always runing. Can you help me?
![image](https://github.com/Cloud-CV/EvalAI/assets/95663989/1c436e81-d1e1-4920-bc22-4dce26c261aa)
Repository: Cloud-CV/EvalAI
Issue: Check challenge phase is corresponding to challenge in update submission API
Root cause: Validation Gap

Here, we do check for whether the host is accessing the challenge: https://github.com/Cloud-CV/EvalAI/blob/master/apps/jobs/views.py#L1117-L1121
```
    if not is_user_a_staff(request.user) and not is_user_a_host_of_challenge(request.user, challenge_pk):
        response_data = {
            "error": "Sorry, you are not authorized to make this request!"
        }
        return Response(response_data, status=status.HTTP_400_BAD_REQUEST)

```

But, we don't verify whether the other details (challenge phase) are corresponding to that challenge PK:
https://github.com/Cloud-CV/EvalAI/blob/8bf1c3dca1f753bcb59dbe12b1f340383755eb60/apps/jobs/views.py#L1124
```
        challenge_phase_pk = request.data.get("challenge_phase")
```

We need to fix this.

Hey @gchhablani, looked into this issue a bit, could you please explain some more? Can other users access the `update_submission` API except for host users? How can challenger_phase_pk  be used here to further improve the API?

Hey @gchhablani Can I work on this issue?


Hello I am interested in this issue can I fix this?
@gchhablani 

I have already completed on issue assigned to me and submitted a PR review pending and want to work on this issue.

Thank You
Rahul Yadav

@gchhablani Please assign this issue to me
Repository: Cloud-CV/EvalAI
Issue: Add detailed documentation on auto scaling and what to expect
GitHub labels: enhancement

docs

good first issue

easy-medium

GSoC-2024

We want to provide a documentation of our auto scaling script and auto cancel scripts and how they work.
We want to warn users that we are not liable in case it fails, and there are scenarios where they should check their AWS usage and accordingly scale it down if auto scaling is failing. They should contact us in case that happens.

I am interested for this work can u assign me this with details expectations!

@nickhil-verma Go over `auto_scale_*` scripts in `scripts/deployment` and prepare a PR with documentation.

i am interested for this work can you assign me if this work is not done?

@gchhablani I am interested in this
Additionally, I only see auto_deploy scripts in deploy. sh in the Master Branch 
Are you referring to those scripts?

@Rahulxx01 Check `monitoring`.
Create a PR and let me know on Slack.

@gchhablani Can you provide me link to join Slack group  I am not able to find it on Cloud CV or GitHub ?

Sorry for inconvenience just new to this project

@gchhablani 
I have begun working on the documentation by understanding the **auto_***  scripts in the scripts/monitoring.
Will try to finish it as soon as possible and will keep the team updated on the issue channel 
Once completed will message  in the issue tab and create a PR accordingly
Thank You

@gchhablani Hey there, as a part of GSoC even I would like to contribute and if the issue is still open and no one is working on it, I would like to take this up.

@jahnvisahni31 I am working on it and will submit a PR by EOD.

Created a PR @gchhablani currently under review now

Additionally can you also provide me with Slack workspace invite link

Thank You
Rahul Yadav

is this issue still active? if in case it is will i be assigned please?

@Rahulxx01  hi if this issue is active, I would love to contribute!
Repository: Cloud-CV/EvalAI
Issue: Argoverse 2: Motion Forecasting Competition always stays in submitted status
Root cause: Validation Gap

_Thanks for taking the time to report a bug in the EvalAI project, or propose a new feature! Before creating a new issue, please do a quick search to check that it hasn't already been filed on the [issue tracker](https://github.com/Cloud-CV/EvalAI/issues)._

If you'd like to report a bug, please include the following information:
- the expected behaviour
- the observed behaviour
- steps to reproduce the problem
- which operating system and/or browser you're using, if relevant
- (if possible) copies of the console/server logs

On the other hand, if you'd like to propose a new feature:
- please describe, as clearly as possible, what you'd like to see
- (if possible) include mockups/screenshots, if these would help convey the meaning better

Thanks!
Repository: Cloud-CV/EvalAI
Issue: Clear input field after creating a Participant Team
GitHub labels: bug

enhancement

GSOC-2019

easy_to_fix

frontend

On the `My Participant Team` view, once the user creates a participant team, then the input field should become empty. 

Please see the screenshot below for more information. 
<img width="1439" alt="screen shot 2017-01-12 at 11 27 17 pm" src="https://cloud.githubusercontent.com/assets/2945708/21918384/b2cde00a-d91e-11e6-977e-942fb0d38886.png">


@deshraj  Shall i try this?


Yup sure. Assigning it to you. 

@deshraj after creating new team why does it automatically redirects to challenge-page.html?? 

Actually, the files are not refreshed since you last pulled the code from master branch. Please run `gulp dev` in the root directory of the project and it should work fine. 

@deshraj if the PR is fine then i can try #425 

@parismita I would like to make @aka-jain call on this. @aka-jain can you please review the PR?

Fixed in #429. Closing it.
Repository: Cloud-CV/EvalAI
Issue: Travis Build Status to Reflect Deployment Script Failures
GitHub labels: enhancement

GSoC-2024

CI

We've observed that in certain scenarios, such as the recent [example](https://github.com/Cloud-CV/EvalAI/pull/4239), when our deployment or push script fails, TravisCI reports the build as successful. This can be misleading and may lead to confusion.

We need the TravisCI to report it as failed if the scripts have failed to push or deploy

Hello, I am Shivesh. I came here for GSOC 2024.
I have worked with languages like *Python, TypeScript, etc* and am pretty much familiar with the stack this project is using.
I am interested in contributing to this organization.
Can you help me get started with this?

Hi @gchhablani I want to take this issue up !

@gchhablani please check this PR, it might fix this issue: https://github.com/Cloud-CV/EvalAI/pull/4276
Repository: Cloud-CV/EvalAI
Issue: Fix environment log file in code upload challenges
Root cause: Unimplemented Feature Gap

The issue is that the environment logs are not being populated in the environment log files.
We need to figure out the exact reason and address it.
For static-code upload challenges, we also need to track the `sidecar-container` logs which will become the environment log file.

I have experience in Python and have worked with AWS, which I believe could be helpful in addressing this issue. I'm eager to explore the codebase and understand the current implementation of the logging system. Could you provide any further insights or specific areas in the code that might need attention? This would help me to get a clearer picture and start contributing more effectively.

Hi @Suryansh5545 and @gchhablani, I would like to confirm that does 'code upload challenge' mean uploading challenge using a zip file? I think this feature is not present in the latest version?

@Suryansh5545 Could you please tell me how to reproduce the bug?

Closing this since we are sunsetting the code upload challenge feature.
Repository: Cloud-CV/EvalAI
Issue: Keep showing "Running" status after submitting
Root cause: Documentation Deficiency

Hi, I participated in the Argoverse 2: Motion Forecasting Competition and submitted my result. However the status of my submission is still running after 5 days. So I would like ask that Is this situation normal, and is there any issue in my submission
<img width="820" alt="1" src="https://github.com/Cloud-CV/EvalAI/assets/93180406/ca7044b4-4d29-41c2-a37d-5036c25699cc">
This is my submissions.
Thanks!
.

@YEEN1NG
This challenge doesn't run on EvalAI compute. Please contact the hosts.

Closing as it is very old. Please reopen if it isn't resolved.
Repository: Cloud-CV/EvalAI
Issue: Clear input field after creating a Host Team
GitHub labels: bug

enhancement

GSOC-2019

easy_to_fix

frontend

On the 'My Hosts Team' view, once the user creates a host team, then the input field should become empty.

Please see the screenshot below for more information.
<img width="1440" alt="screen shot 2017-01-12 at 11 29 55 pm" src="https://cloud.githubusercontent.com/assets/2945708/21918420/28f82b3c-d91f-11e6-9892-d5cd55a99714.png">


@deshraj can I work on this ? even #424 is related. 

@anweshknayak looks like you are already working on #300. Please try to finish that first. 

Fixed in #430
Repository: Cloud-CV/EvalAI
Issue: My result stuck at “Submitted” at "VizWiz VQA Challenge 2023"
Root cause: Others

<img width="1231" alt="image" src="https://github.com/Cloud-CV/EvalAI/assets/106020747/c2768ddc-fb11-469b-832e-6fe1f9954a24">


Hi, I encountered the same issue, have you solved it?
Repository: Cloud-CV/EvalAI
Issue: Add better error handling for getting AWS credentials for a challenge
Root cause: Validation Gap

- In case the API doesn't work.
- In case it works but there is an error in getting the credentials.

can you please share the video or something to recreate this error


@Nsdevil

This is a sample output from a Postman request to `participant_team/aws/credentials/`:


```
<!DOCTYPE html>
<html>

<head>
    <title>EvalAI- Home Page</title>
</head>

<body>
    <h1>Error 500 - Internal Server Error</h1>
</body>

</html>
```

@gchhablani Hey there, as a part of GSoC even I would like to contribute and if the issue is still open and no one is working on it, I would like to take this up.

Hi, @gchhablani without being assigned can we fix the issue and raise the PR?
I just wanted to know so that I can start working on the issues.

@gchhablani , Is this fixed or needs improvement?
Repository: Cloud-CV/EvalAI
Issue: Remove the radio button from the My Participant Team View 
GitHub labels: enhancement

GSOC-2019

easy_to_fix

frontend

See the line: https://github.com/Cloud-CV/EvalAI/blob/master/frontend/src/views/web/teams.html#L20

We need to remove the radio buttons and replace them by https://github.com/Cloud-CV/EvalAI/blob/master/frontend/src/views/web/challenge-host-teams.html#L19

Let me know if you have concerns. 

I would like to try this, but i get errors thrown when installing frontend :disappointed: 
`npm install
npm WARN evalai@1.0.0 No repository field.`


@gauthamzz : That's not the error, your client side should work fine. Moreover have you read Readme again, it has been updated. Also try to run gulp dev and see if errors are coming or not.

hi,I want to work on this issue,could you assign it to me

@Rishav09 - Sure! Go ahead. 

@Rishav09 any update on this? 

Hi,I just fixed this and it got merged too.Sorry ,I didn't mention it here

Thanks for informing. :-)
Repository: Cloud-CV/EvalAI
Issue: Add sidecar container logs and fix environment container logging
Root cause: Unimplemented Feature Gap


Hi @gchhablani ! I want to try to fix the issue. It's the first time I'm contributing to Cloud-CV, and I am interested in working on this issue. After reviewing the contribution guidelines([here](https://github.com/Cloud-CV/EvalAI/blob/master/.github/CONTRIBUTING.md)), I think I'll be able to work on this. Could you please give some more description?

Hi @snehalchaudhari98!
Glad to know you're interested in solving this.
Can you please share your understanding of this issue?

Hi @gchhablani 

I'm implementing a sidecar container for logging enhancement in our Docker setup. Should it be limited to production, or include staging? Also, any recommendations for sidecar containers?
Your guidance on these matters would greatly assist me in moving forward with the implementation effectively

Hi is this issue still open? If yes, is there any recommendation for which logging agent to use?


Hi @gchhablani  @RishabhJain2018  I've implemented a sidecar container with a fluentd image, If I could get some input on how you would want those logs to be presented that would be great.  I've currently got the logs of each container to just output in the fluentd container. 
![image](https://github.com/Cloud-CV/EvalAI/assets/73216889/82870e2f-41d9-46d5-b283-030360296461)


Hi, @gchhablani,

I am interested in working on this issue as well. As shivas1516 mentioned, could you provide more details about the conditions? Specifically, I'm looking for clarification on production logging and staging logging.

Additionally, I am keen on participating in the 2024 GSoC Cloud-CV project. Is resolving at least one issue a prerequisite for selection?
Repository: Cloud-CV/EvalAI
Issue: initial build error
Root cause: Validation Gap

I'm working off of the EvalAI-Starters template. I've been trying to initialize a competition but it's not working. I had a previous competition initialize successfully, but updates stopped working today due to the same error. 
![erorr](https://github.com/Cloud-CV/EvalAI/assets/24941864/b5cadbe0-e691-4c91-a57f-8ca826e9429b)


issue:  #4263

To fix these issues:

Check Challenge Configuration URL: Double-check the URL specified for validating the challenge configuration. Ensure that it includes the correct schema (e.g., http:// or https://) and does not contain any extra spaces or incorrect characters.

Verify GitHub Authentication Token: Verify the GitHub authentication token provided in the configuration. Make sure it is correctly formatted and has the necessary permissions to create or update issues in the repository. You may need to regenerate the token if it is incomplete or incorrect.

Update Configuration: Once you have corrected the URL and verified the GitHub authentication token, update the challenge configuration with the corrected values.

Retry Validation: After updating the configuration, retry the validation process to see if the issues have been resolved. If the error persists, review the configuration and logs again to identify any remaining issues.
Repository: Cloud-CV/EvalAI
Issue: Allow cilium policies to request model download in EKS
Root cause: Unimplemented Feature Gap


Hi @gchhablani can you elaborate this issue ! I can see custom cilium network policy exist, but it is not used and currently, EvalAI is only installing Cilium using: [Official Installation - Cilium](https://raw.githubusercontent.com/cilium/cilium/v1.9/install/kubernetes/quick-install.yaml)
Repository: Cloud-CV/EvalAI
Issue: Fix static-code upload challenge - pod exits after submission container runs
Root cause: Resource Mishandling

The current system experiences abrupt crashes in the pod, specifically when the submission.csv file is being created within the submission container. The creation time of submission.csv varies based on participant or host team submissions, leading to synchronization issues with the monitoring container. 

As a temporary solution, hosts are manually adding a sleep command after submission file creation to allow the monitoring container sufficient time to pick up the file. However, this approach is considered hacky and not a sustainable solution.

**Expected Behavior**:
The system should handle the creation of the submission.csv file more robustly, ensuring that the monitoring container doesn't encounter issues or crashes when the file becomes available. A more elegant and automated solution is sought to replace the current workaround.

**Current Behavior**:
Abrupt crashes in the pod occur when the submission.csv file is being created. The monitoring container is killed, likely due to the file not being immediately available for processing.

We need to investigate and implement a cleaner solution to handle the timing discrepancies between the submission file creation and the monitoring container's activity. This may involve enhancing the synchronization mechanism or introducing a more robust way for the monitoring container to detect the availability of submission.csv.

Hi @shivas1516, are you still working on this issue?

No, Feel free to assign this issue to you 

Hi @poju3185 , if you are not working on this issue. I would love to take it

Closing this since we will be sunsetting this feature.