zhiminy/EvalEng huggingface dataset:
Repository: Cloud-CV/EvalAI
Issue: Add a new status option in Submission model
GitHub labels: enhancement

GSOC-2019

easy_to_fix

backend

For reference, see this https://github.com/Cloud-CV/EvalAI/blob/master/apps/jobs/models.py#L25

We want to add a new status option with name `SUBMITTING = "submitting"`

@deshraj  as I am not working on any issue currently can I work on this?

Sure, go ahead. Assigning it to you. :-) 

@deshraj  I just have to add the option name submitting & add it in status_Options in the Submission model ,am I right ? or do I have to change any other code?

You would also need to add an entry similar to this https://github.com/Cloud-CV/EvalAI/blob/master/apps/jobs/models.py#L32

Okay Understood @deshraj will do it & let you know?

Sounds good. 

@deshraj  sorry for this I will just remove directive.js from it !

@deshraj : please make sure that migrations(named please) are also there in the PR.

Sure, will take of it. 

@Ram81 : thanks for the quick fix :)
Repository: Cloud-CV/EvalAI
Issue: Reduce size of deregister button

The size of deregister button is too much compared to surrounding text.

Also
- Team name is not shown properly.
<img width="518" alt="Screenshot 2023-07-31 at 2 50 20 PM" src="https://github.com/Cloud-CV/EvalAI/assets/29076344/5f7afc63-bb82-4288-b2f2-2cd2d21c1968">

- Use lowercase `t` in team.

- After deregister, I see this:
<img width="642" alt="Screenshot 2023-07-31 at 2 51 04 PM" src="https://github.com/Cloud-CV/EvalAI/assets/29076344/0d35cf3b-1e8a-43fd-8b7e-2f95178521c1">


<img width="1171" alt="Screenshot 2023-07-31 at 2 52 28 PM" src="https://github.com/Cloud-CV/EvalAI/assets/29076344/500bfa1e-72d0-4522-a5ee-61d30c2d44cf">


- This URL is still accessible, it should not be:  https://eval.ai/web/challenges/challenge-page/2077/submission
- Redirect them to overview page of the challenge or the dashboard.
Repository: Cloud-CV/EvalAI
Issue: Add warning for submission regarding deregistering
Root cause: Unimplemented Feature Gap

Whenever a submission is made for the first time by a participant team, they should get a warning that they will not able to deregister after that.

This issue is currently open. Please check this PR https://github.com/Cloud-CV/EvalAI/pull/4270 and verify if it works correctly or not.

I would like to be assigned to this issue

@gchhablani Sir! Could you please assign this issue to me? I want to give it a shot.

Hi I am keenly interested in addressing specific issues and am earnestly hopeful of having the opportunity to engage with them further.

Hi @gchhablani, I would like to work on this if possible. As this is going to be my first contribution.

hii @RishabhJain2018 , I would love to work on this issue, could you please assign it to me?

Sure, asigning it to you!

@RishabhJain2018 is this issue still open?

Hey @Akshat453 , Yes , it is.

Can you assign it to me ?

Done!
Repository: Cloud-CV/EvalAI
Issue: Problems with EC2 deployment
Root cause: Validation Gap

- `sudo` does not work on the instance after ssh.
- `/root` contains EvalAI. I think root should be the default thing when we ssh. Usually we have `Projects/EvalAI` on there, but it is fine if it is not possible. Just make sure we are able to see EvalAI after `ls`.
- Echo commands are needed for each step with some demarcations so we are aware of the step it is at:
    -  `echo "Step 5/X: Pulling docker image"`.
 - `groupadd: group 'docker' already exists` on `groupadd`. So maybe remove this.
 - Name of the instance is `staging-<pk> Instance`. It should be `Worker Instance staging-<pk>`.
 - If I stop the instance, container is killed. If I restart it, image is present, but container is not.
Repository: Cloud-CV/EvalAI
Issue: Create endpoint for updating the challenge
Root cause: Unimplemented Feature Gap

Details about the endpoint:
- Url: /api/challenges/<challenge_id>/
- Header: Authorization: Token <token>
- Method: PUT/PATCH
- Response: 
  - status on success: HTTP_200_OK
  - status on exception:  HTTP_406_NOT_ACCEPTABLE
  - status on Unauthorized access: HTTP_203_NON_AUTHORITATIVE_INFORMATION

For more details, feel free to discuss on thread or gitter channel https://gitter.im/Cloud-CV/EvalAI

Fixed in #126
Repository: Cloud-CV/EvalAI
Issue: Crontab Fix: Currently crontab is created for root user, it should be for ubuntu user ($USER after cd into /home/ubuntu).
Root cause: Configuration Error


@gchhablani  I would like to work on this issue. Can I get necessary access to work on this? (I have worked with EC2 instances in the past)

@gchhablani  I suggest making these changes to line 69 of scripts/deployment/deploy_ec2_worker.sh

`sudo -u ubuntu crontab workercron`
or
`crontab -u ubuntu workercron`
Repository: Cloud-CV/EvalAI
Issue: Fix manage EC2 instance API

If instance ID does not exist, the APIs throw error.

```
Traceback (most recent call last):
  File "/home/ubuntu/Projects/EvalAI/scripts/monitoring/auto_scale_ec2_workers.py", line 123, in <module>
    start_job()
  File "/home/ubuntu/Projects/EvalAI/scripts/monitoring/auto_scale_ec2_workers.py", line 113, in start_job
    start_or_stop_workers_for_challenges(response, metrics, evalai_interface)
  File "/home/ubuntu/Projects/EvalAI/scripts/monitoring/auto_scale_ec2_workers.py", line 100, in start_or_stop_workers_for_challenges
    start_or_stop_workers(challenge, metrics[str(challenge["id"])], evalai_interface)
  File "/home/ubuntu/Projects/EvalAI/scripts/monitoring/auto_scale_ec2_workers.py", line 91, in start_or_stop_workers
    stop_instance(challenge, evalai_interface)
  File "/home/ubuntu/Projects/EvalAI/scripts/monitoring/auto_scale_ec2_workers.py", line 41, in stop_instance
    response = evalai_interface.stop_challenge_ec2_instance(challenge["id"])
  File "/home/ubuntu/Projects/EvalAI/scripts/monitoring/evalai_interface.py", line 165, in stop_challenge_ec2_instance
    response = self.make_request(url, "PUT")
  File "/home/ubuntu/Projects/EvalAI/scripts/monitoring/evalai_interface.py", line 42, in make_request
    response.raise_for_status()
  File "/home/ubuntu/venv/lib/python3.6/site-packages/requests/models.py", line 960, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 406 Client Error: Not Acceptable for url: https://eval.ai/api/challenges/1989/manage_ec2_instance/stop/
```

Hello, is it viable to simply log the information whenever an HTTPError occurs if instance ID doesn't exist, as done in line 36?

 
```
 def make_request(self, url, method, data=None):
        headers = self.get_request_headers()
        try:
            response = requests.request(
                method=method, url=url, headers=headers, data=data
            )
            response.raise_for_status()
        except requests.exceptions.RequestException:
            logger.info(
                "The worker is not able to establish connection with EvalAI"
            )
            raise
        return response.json()?

```

We can add an extra else statement to return the response (if it exists) so that it doesn't throw an error entirely or maybe return a default response?

I'm a bit new to opensource but I'd love to contribute to this project as it seems really cool and within grasp of what I've learnt so far. Let me know if there's anything that I can work on!
Repository: Cloud-CV/EvalAI
Issue: Modify EC2 autoscaling to not check for end dates challenges
GitHub labels: enhancement

easy_to_fix

GSoC-2024


Hi @gchhablani, I want to clarify this issue. So I need to remove this line of code for not checking the end dates?
[https://github.com/Cloud-CV/EvalAI/blob/bacbf89c5fa7e6155183525ddfef0f7a2e953af6/scripts/monitoring/auto_scale_ec2_workers.py#L89](url)
Repository: Cloud-CV/EvalAI
Issue: My submission is stuck at 'Submitted'.
Root cause: Configuration Error

My submission for the VQAv2(https://eval.ai/web/challenges/challenge-page/830/overview) test-dev phase keeps getting stuck at 'submitted' status.
my team name is DC

![image](https://github.com/Cloud-CV/EvalAI/assets/101086211/fdb9268c-5f55-4cf4-93ca-4338e397f928)


The same question with you.
My team name is happy623 

Same here for VQAv2. I have waited for one day but the status is still submitted.

@gchhablani I saw that your magical hand addressed this issue for a lot of people. It would be great if you could help us look into this issue. Thank you so much!

@ziqipang @PengLongkun @DonghaChoiGIST I have resolved this issue. Hopefully all submisssions will get evaluated soon.

@gchhablani Thank you! It works now
Repository: Cloud-CV/EvalAI
Issue: How to reactivate a use account?

I would like to know how to reactivate an account that has been deactivate?

Thanks!


@yifeisu Please share your user over an email, I will reactivate it.

@gchhablani I also need to reactivate my account, email is leyu@mail.ustc.edu.cn, thanks!

@ceresOPA  Done
Repository: Cloud-CV/EvalAI
Issue: Evaluations stuck for VQAv2 test dataset
Root cause: Others

_Thanks for taking the time to report a bug in the EvalAI project, or propose a new feature! Before creating a new issue, please do a quick search to check that it hasn't already been filed on the [issue tracker](https://github.com/Cloud-CV/EvalAI/issues)._

If you'd like to report a bug, please include the following information:
- the expected behaviour
- the observed behaviour
- steps to reproduce the problem
- which operating system and/or browser you're using, if relevant
- (if possible) copies of the console/server logs

On the other hand, if you'd like to propose a new feature:
- please describe, as clearly as possible, what you'd like to see
- (if possible) include mockups/screenshots, if these would help convey the meaning better

Thanks!
Repository: Cloud-CV/EvalAI
Issue: My code stuck at evaluating nuscenes test set detection 
Root cause: Resource Mishandling

_Thanks for taking the time to report a bug in the EvalAI project, or propose a new feature! Before creating a new issue, please do a quick search to check that it hasn't already been filed on the [issue tracker](https://github.com/Cloud-CV/EvalAI/issues)._

If you'd like to report a bug, please include the following information:
- the expected behaviour
- the observed behaviour
- steps to reproduce the problem
- which operating system and/or browser you're using, if relevant
- (if possible) copies of the console/server logs

On the other hand, if you'd like to propose a new feature:
- please describe, as clearly as possible, what you'd like to see
- (if possible) include mockups/screenshots, if these would help convey the meaning better

Thanks!

![image](https://github.com/Cloud-CV/EvalAI/assets/65105015/ea73b066-982d-40ce-81c3-67cd11448312)


Hello @Ram81 @David_Novotny @Rishabh_Jain @rishabh @gchhablani, 
I am really sorry to bother you, but my nuscenes test set submission stays at 'Submitted' status for 3 days from September 1st. May I ask if there is problem in the server? Thanks for hard work! 

@mincheoree This is fixed. Please let us know in case of issues. Thanks!

@gchhablani but mine still hangs at 'Submitted' even I resubmitted results.json. From my previous experience, isn't it supposed to turn into 'Processing' status in few minutes?
![image](https://github.com/Cloud-CV/EvalAI/assets/65105015/d53bbecb-3877-4d0f-9d68-2fede9eda69d)
Repository: Cloud-CV/EvalAI
Issue: Team Validation error, while building a custom challenge
Root cause: Documentation Deficiency

I get following error while I fill config file correctly. What might be the possible cause of this error? Thanks in advance.

Following errors occurred while validating the challenge config: 500 Server Error: Internal Server Error for url: https://eval.ai/api/challenges/challenge/challenge_host_team/2797/validate_challenge_config/

@asparius You will need to add your GitHub auth token to your secrets.

> @asparius You will need to add your GitHub auth token to your secrets.

@gchhablani Where exactly? I did it manually by adding it to the repository.
Repository: Cloud-CV/EvalAI
Issue: My file stuck at “Running” at TextVQA Challenge 2021
Root cause: Resource Mishandling

I found that the TextVQA Challenge 2021 failed to process my results and the status is always "Running"（test set）.
The team name is 2023test.
Can you please have a look?
![image](https://github.com/Cloud-CV/EvalAI/assets/44249380/11002bbe-2b97-4ef9-87c6-118b3d788f02)


Hello @Ram81 @David_Novotny @Rishabh_Jain @rishabh @gchhablani,
I am really sorry to bother you, but I really need this result urgently. May I ask if there is a problem with the server? Thanks for your hard work!
Repository: Cloud-CV/EvalAI
Issue: My file stuck at “Submitted” at Nuscenes Detection challenge
Root cause: Others

Sorry to bother, but my nuscenes test set submission stays at 'Submitted' status and it looks it won't change.
My team name is pak.
![1212](https://github.com/Cloud-CV/EvalAI/assets/113985787/8703f3ab-38bb-4681-976a-cead375e0aa9)


Hi, I think you have the wrong rishabh ;)

Sorry i fixed it :)

Does it work now?

Still doesn't work ;)

Now it works! thank you
Repository: Cloud-CV/EvalAI
Issue: Dependency Issue in worker-Eval AI starters
Root cause: Environment Incompatibility

The repo is built, and displayed in eval.ai. When I start worker, it gives error due to some external libraries needed in the evaluation script such as NLTK. I added nltk in requirements.txt file in github folder however the issue still persists. It still does not work due to "no module found error"
Repository: Cloud-CV/EvalAI
Issue: [Bug] Unable to update challenge due to challenge validation
Root cause: Validation Gap


- the expected behaviour
The challenge should be updated without a issue
- the observed behaviour 
After deleting challenge phase 1 , we get this error https://github.com/Suryansh5545/TestPhase/actions/runs/6113014452/job/16591633990
- steps to reproduce the problem
Delete the challenge phase 1 then try to update the challenge.


Hi @gchhablani, I am also facing a similar issue https://github.com/Cloud-CV/EvalAI-Starters/issues/82. Is there a workaround available until the issue is resolved? Thanks!

Hey i am interested to work on this issue please assign me this issue


Hi @Suryansh5545 @gchhablani 
The link shared in the description has expired. Could you provide some more details on the issue and how to reproduce it?

hey @gchhablani will start working on this issue. Are there any suggestions from your side?

can i work on it ?


hey @Suryansh5545, I have implemented the fix for Issue #4166 (allowing challenge phase deletion) and verified the unit tests locally. My build is currently failing due to the global Travis CI issue tracked in #4846 (Sass deprecation errors). Ready for review once the CI is fixed!
Repository: Cloud-CV/EvalAI
Issue:  “Submitted” at Nuscenes Detection challenge
Root cause: Others

It has been past 18 hours. The team name is dkliang.

<img width="899" alt="image" src="https://github.com/Cloud-CV/EvalAI/assets/23117298/bc55131b-6316-4c56-b008-9932047e9cc8">
Repository: Cloud-CV/EvalAI
Issue: Job names - submission IDs get repeated for code upload submissions
Root cause: Algorithmic Error

Example:
```
submission-15570, submission-16144, submission-16170, submission-16533, submission-16562, submission-16564, submission-16565, submission-16566, submission-16567, submission-16568, submission-16571, submission-16571, submission-16613, submission-16613, submission-16621, submission-16621, submission-16622, submission-16622, submission-16623, submission-16623, submission-16624, submission-16624, submission-16625, submission-16625, submission-16626, submission-16626, submission-16627, submission-16627, submission-16628, submission-16628, submission-16629, submission-16629, submission-16630, submission-16630, submission-16631, submission-16631, submission-16632, submission-16632, submission-16633, submission-16633, submission-16634, submission-16634, submission-16636, submission-16636, submission-16637, submission-16637, submission-16638, submission-16638, submission-16639, submission-16639, submission-16640, submission-16641, submission-16641, submission-16642, submission-16643, submission-16643, submission-16645, submission-16645
```

I want to fix this issue! can I get more info regarding this??

@gchhablani can you please give more details on this!


Hi @gchhablani  I want to try to fix the issue. It's the first time I'm contributing to Cloud-CV, and I am interested in working on this issue. After reviewing the contribution guidelines([here](https://github.com/Cloud-CV/EvalAI/blob/master/.github/CONTRIBUTING.md)), I think I'll be able to work on this. Could you please give some more description?

@gchhablani I think it has 2 cases, if number of submissions are less - then it can be solved using set and deleting the set instance after it has been used, but for large number of submissions it has to be done using SQL commands like UNIQUE ofc using orm.
please provide more clarifications for this issue.
thank you

Closing this since we are sunsetting code uplaod challenge feature.
Repository: Cloud-CV/EvalAI
Issue: Evaluations stuck for VQA Challenge 2021 and GQA Real-World Visual Reasoning Challenge
Root cause: Resource Mishandling

I’m sorry to bother you. My submissions for the [VQAv2](https://eval.ai/web/challenges/challenge-page/830/overview) Phase: Test-Standard Phase and [GQA](https://eval.ai/web/challenges/challenge-page/225/overview) Phase: test2019 keep getting stuck at 'submitted' status.
My team name is hh_23333. 

<img width="1078" alt="image" src="https://github.com/Cloud-CV/EvalAI/assets/37319709/57afe6e1-5fb8-4b12-8df5-ac5ec6e88eee">

<img width="1067" alt="image" src="https://github.com/Cloud-CV/EvalAI/assets/37319709/59767e4f-0b70-4019-851a-648db979d5d4">



@gchhablani Could you please have a look? I really need these results and would appreciate your help and reply. Thanks in advance!
Repository: Cloud-CV/EvalAI
Issue: Create endpoint for deleting a challenge
Root cause: Unimplemented Feature Gap

Details about the endpoint:
- Url: /api/challenges/<challenge_id>/
- Header: Authorization: Token <token>
- Method: DELETE
- Response: 
  - status on success: HTTP_200_OK
  - status on exception:  HTTP_406_NOT_ACCEPTABLE
  - status on Unauthorized access: HTTP_203_NON_AUTHORITATIVE_INFORMATION

For more details, feel free to discuss on thread or gitter channel https://gitter.im/Cloud-CV/EvalAI

Fixed via #127
Repository: Cloud-CV/EvalAI
Issue: Failed
Root cause: Validation Gap


In VQA Challenge 2021, submitting an answer file prompts Failed. is this due to an error in the assessment process or is it an error in my answer file?

I have the same issue. Even if i use the result file evaluated in the past, it fails now.
The stderr shows some errors on sql.
psycopg2.errors.NotNullViolation: null value in column "is_disabled" violates not-null constraint

@ssyze ，Is this the issue you are talking about that shows up on the assessment site? May I ask if you have resolved it now?

I also meet the same question. Have you solved the problem?

Same here. Files that were valid submissions in the past lead to failures. However, it seems that the stdout file already contains the final evaluation results. 

> Same here. Files that were valid submissions in the past lead to failures. However, it seems that the stdout file already contains the final evaluation results.

Yes, i will obtain the final results but it takes a lot of time. Now I have submitted for nearly 20 hours, and the result is not yet available. The state is always submitted.

Same again, I had files queueing for around 30hrs. I bet there isn't much we can't do unless some EvalAI contributor has some tricks up his sleeve @gchhablani   

same question for 3 days.

@FarinaMatteo Thanks for the heads up.“it seems that the stdout file already contains the final evaluation results.” I've found it so far.

@LinB203 You can read FarinaMatteo's comments. Files that were valid submissions in the past lead to failures. However, it seems that the stdout file already contains the final evaluation results.

Same question on nuScenes challenge. Hello @gchhablani  @Suryansh5545 @Ram81 @RishabhJain2018  @deshraj I am really sorry to bother you, but I really need this result urgently because the deadline for CVPR is coming. 
Could you please take a look at this issue? Thanks for your hard work! 

I find the problem of  "Failed status" seems to be related to the recent [commit](https://github.com/Cloud-CV/EvalAI/commit/ae017b46fbb406dadfd2bb44da025336022d9e60). The status of submissions on [nuScenes LiDAR semantic segmentation track](https://eval.ai/web/challenges/challenge-page/720/participate) are always Failed. 

Hi all,
Thanks for pointing to this issue.
Due to a recent merge, for some challenges the evaluation was failing.
I have fixed this.
If you continue to see this for any challenge, please send us a contact message through the forum or send me an email.

Thanks,
Gunjan
Repository: Cloud-CV/EvalAI
Issue: Create API endpoint to get all the submissions made by user for a challenge phase
Root cause: Unimplemented Feature Gap


@deshraj Can I work on this?

Yeah Sure. Assigning it to you. 

Let me know if you need some help. 

ok sure.

@deshraj In **[this](https://github.com/Cloud-CV/EvalAI/blob/master/apps/jobs/views.py#L27)** function, shouldn't we check the flag value for `approved_by_admin` field [here](https://github.com/Cloud-CV/EvalAI/blob/master/apps/challenges/models.py#L34) for submitting to a challenge ?

Yeah, great thought. We will add a decorator for the same in another PR. I am keeping a note of the same. 

So, Can I submit the PR without adding this decorator?

Sure, go ahead. 

Fixed in #427. Closing it.