zhiminy/EvalEng huggingface dataset:
Repository: Cloud-CV/EvalAI
Issue: Sandbox or late submissions
Root cause: Unimplemented Feature Gap

Hi!
Great framework that has virtually everything that is needed for hosting AI competitions!

The one feature I am looking for is a so called sandbox. Is it possible to enable late submissions after the deadline? E.g., kaggle allows to submit predictions even after the competition has ended. The leaderboard doesn't change of course but  the participant see where his/her score.

I have searched an admin panel for some parameters that could be toggled, but no luck.

Would appreciate if someone clears this for me. Probably, I was looking into a wrong direction.

Hi @maya-ami ,

Thanks for reaching out.

> The one feature I am looking for is a so called sandbox. Is it possible to enable late submissions after the deadline? E.g., kaggle allows to submit predictions even after the competition has ended. The leaderboard doesn't change of course but the participant see where his/her score.

No, we don't allow late submissions after the challenge deadline. One thing you can do is to have two phases within the challenge and you can use one phase for the deadline and another one after the deadline.

Please feel free to ask if you have further questions.

@RishabhJain2018 I see, thank you!
Repository: Cloud-CV/EvalAI
Issue: Build should take less time
Root cause: Architectural Constraint

Wouldn't it reduce the build time by installing the `prod.txt` dependencies before adding all the code (to leverage caching)? Or is there any specific reason we are not doing this?


https://github.com/Cloud-CV/EvalAI/blob/aa0f4ffc464fc7c934f2702f02ab20c1a2c3e022/docker/prod/django/Dockerfile#L4-L9

This is fixed after moving to travis.com
Repository: Cloud-CV/EvalAI
Issue: Fix cyclic import dependency in challenges/aws_utils.py
Root cause: Architectural Constraint

### Description

Currently, we can't import `Challenge` model or any method from `challenges/utils.py` in [challenges/aws_utils.py](https://github.com/Cloud-CV/EvalAI/blob/master/apps/challenges/aws_utils.py) because `aws_utils.py` is being imported in `challenges/model.py`. If we try to import any util methods or models from `challenges/utils.py` or `challenges/models.py` in `challenges/aws_utils.py` it fails because of cyclic imports.

### Fix

Refactor `challenges/models.py` to fix cyclic imports

@Ram81 can I pick up this issue? I'll make a quick PR

Sure @burnerlee !
So, I am unassigning you from #2994 .

oh can i take up only one issue at a time?

Yes, you can finish this one and then pick that one, if it is not assigned :) 

oh ,cool

￼Opened PR #3148 to fix the issue. Please review @RishabhJain2018 @Ram81 

Please also reassign me #2994 

@burnerlee it seems like you didn't test the changes by fixing imports in `aws_utils.py`. After fixing cyclic imports we want to place all `from .models` import at the start of the file in `aws_utils.py` and not in the methods. 

If you check [this](https://github.com/Cloud-CV/EvalAI/blob/master/apps/challenges/aws_utils.py#L356) you'll see the workaround for handling cyclic import is still there. If you move all such imports to start of the file we still see the same issue. If you're still interested you can pick this issue after you are done with your current task.

yeah, I'll pick this issue, I've my all PRs updated now and made a PR for the issue in eval-cli
Repository: Cloud-CV/EvalAI
Issue: Optimize Travis CI Build during checks on making a Pull Request

When a pull request is made, Travis CI runs a single test to check for all errors/warnings in the code, and with an exit status after the complete check is finished. Even if some part of the check fails, it continues to run the check further and notifies the user about the failure only at the end of the complete test which wastes a lot of time. 
For e.g. the flake8 test occurs at the very beginning of the whole test. Even if it fails, the complete test continues to run and after around 20-30 minutes the user is notified of an error.
The single test also does not clearly mentions which part of the test exactly failed. One has to open up the entire logs of the boot machine of Travis CI and look for the error.
I propose that we can break all these tests into a number of tests that could run simultaneously on different boot machines and would give faster and clear results. 
Also, we can return a particular check with a status code as soon as a single error is found in a check so that the build only checks ahead once that particular error is sorted out.

@RishabhJain2018 @Ram81 I would love to work on this if this seems useful. I have completed my previous issues and made a PR

@burnerlee setting up multiple boot machines also takes time and recently the boot time has increased. So I am not sure if we  start multiple boot machines we will get any benefit from it.

@Ram81, I understand your point. Can we at least return the build with an error as soon as one of the checks fails so that the user does not have to wait for a long time?

Yeah that change is good to have.

@burnerlee I would suggest you first complete the tasks you are working on before picking up any new task. I can see that you already have 3 PRs open which needs some work before they can be approved and merged. Once you are done with them then you can pick this up or if you are interested in a backend related task you can look at this [issue](https://github.com/Cloud-CV/evalai-cli/issues/305)

@Ram81 @RishabhJain2018 completed the already assigned issues/ requested reviews. Should I start working on this change? Or is there some other issue on a higher priority than this? Could you please assign it to me? :) Thank you

This is fixed after moving to travis.com
Repository: Cloud-CV/EvalAI
Issue: [FrontendV2] Fix error message on My Submissions page refresh 
Root cause: Validation Gap

### Description

In frontend v2 when a challenge host goes to the `My Submissions` tab on the challenge page and refreshes the page the UI shows error notification with message `Success`.

Steps to reproduce the error:

1. Select any active challenge
2. Go to `My Submissions` tab
3. Refresh the page

Here's the screenshot of the error notification.

<img width="1440" alt="Screen Shot 2021-01-20 at 8 55 00 PM" src="https://user-images.githubusercontent.com/16323427/105197165-57080c80-5b62-11eb-9fa9-2e0c22200e20.png">

Expected behavior:

No error notification should be shown.


Can I work on it?

@gautamjajoo Sure, go ahead
Repository: Cloud-CV/EvalAI
Issue: [FrontendV2] Allow toggling Show on leaderboard when submission is submitted/runnning
Root cause: Validation Gap

### Description

When a participant goes to `My Submission` page and select a submission they can't toggle `Show on leaderboard` checkbox when status of submission is `Submitted/Running`. We should allow toggling of this checkbox even when submission is not in `Finished` state.

Steps to reproduce:

1. Select a challenge
2. Go to Submit tab and make a submission
3. Go to My Submissions tab
4. Select a submission with `Submitted/Running` status and try to toggle `Show on Leaderboard` checkbox.

Here's a screenshot for reference:

<img width="1440" alt="Screen Shot 2021-01-20 at 9 17 55 PM" src="https://user-images.githubusercontent.com/16323427/105199635-f9c18a80-5b64-11eb-951c-d1c62fc9bb4f.png">



@Ram81 can I make a PR for this issue?

@burnerlee Sure, go ahead

#3160 made
Repository: Cloud-CV/EvalAI
Issue: [FrontendV2] Pagination not working on All submissions page
Root cause: Configuration Error

### Description

On the `All submissions` tab of challenge page pagination is not working. If there are multiple pages for the submissions when a users tries to navigate to the next page UI throws an error notification with `Success` message.

Expected behavior:

Pagination should work as expected and user should be able to navigate between pages. If there is an error then a valid error message should be displayed instead of `Success`.

Steps to reproduce:

1. Select a challenge
2. Make submissions greater than the pagination limit
3. Go to `All submissions` page and select a phase
4. Try navigating between pages

Here's a screenshot for reference:

<img width="1440" alt="Screen Shot 2021-01-20 at 9 33 18 PM" src="https://user-images.githubusercontent.com/16323427/105201964-76edff00-5b67-11eb-8315-615377809473.png">


@Ram81 the number of submissions shown on a single page currently are 100. Should we change this too? This is creating a problem in testing as there are not enough submissions for a single phase

@burnerlee  no we don't need to change the limit. But you can reduce it to do testing and revert the limit before pushing the PR. Also, it would be good if you work only on 1 issue at a time. 

@Ram81 Can you please assign me this issue. I have almost solved it, I just need to make a PR. 

The issue is at [line 365](https://github.com/Cloud-CV/EvalAI/blob/e884666e789fec9b44da1d8974c7244f099b99f3/frontend_v2/src/app/components/challenge/challengeviewallsubmissions/challengeviewallsubmissions.component.ts#L365) as the `environment.api_endpoint` is not set correctly.

For testing purposes `environment.api_endpoint` should be set to `http://localhost:8000/api/`, and for production it should be `http://eval.ai/api/` (which it is already set to), but it is using the `environment.staging.ts` for testing purposes, instead of `environment.ts`

@Ram81 Can I contribute to this issue. I have started contributing to open-source projects

Closing this since we are not working on Frontend_v2.
Repository: Cloud-CV/EvalAI
Issue: [FrontendV2] Show selected submission details in 3 columns
Root cause: Unimplemented Feature Gap

### Description

When a submission is selected on the `My Submissions` tab we currently divide the fields into two columns. The metadata fields should be shown in 3 columns for meta fields in each column.

Here is the current screenshot for reference

<img width="1440" alt="Screen Shot 2021-01-20 at 9 38 18 PM" src="https://user-images.githubusercontent.com/16323427/105202465-07c4da80-5b68-11eb-8681-db1f008097b8.png">

Expected:

Column 1 fields: Team Name, Method Name, Method Description, Project Url, Publication URL
Column 2 fields: Status, Submitted At, Execution Time
Column 3 fields: Show on Leaderboard, Baseline, Edit Submission details



@Ram81 in the latest master pull, there Baseline is not visible. Does this need to be fixed too?

Are you using host account or participant account? Baseline option is only available to challenge hosts

Oh yes, thank you :)

![Screenshot from 2021-01-21 00-23-35](https://user-images.githubusercontent.com/55936223/105227995-060b0f00-5b88-11eb-9352-a02da8f4f5d2.png)


#3163 made.
Repository: Cloud-CV/EvalAI
Issue: [FrontendV2] Fix private split tag css on the leaderboard
GitHub labels: easy_to_fix

frontend_v2

GSOC-2021

### Description

`Private` tag for a challenge phase split should appear next to the phase split name in the dropdown on the `Leaderboard` tab of the challenge page. Because of inconsistent css the `private` tag for a challenge phase split is aligned below the phase split text in dropdown.

Here's a screenshot for reference

<img width="1440" alt="Screen Shot 2021-01-20 at 9 47 11 PM" src="https://user-images.githubusercontent.com/16323427/105203684-6a6aa600-5b69-11eb-8b20-f0c6d91f79fe.png">
 

Hi @Ram81 !
I would like to give it a shot, if it is okay with you and if no one else is already working on it :)

After playing with the code a little I think that `frontend` is the default one that is being used, could you guide me on how to use `frontendV2`.


@Devanshu24 sure, go ahead.

>After playing with the code a little I think that frontend is the default one that is being used, could you guide me on how to use frontendV2.

You can access it on `http://localhost:4200`  endpoint.


![Screenshot from 2021-01-21 15-34-20](https://user-images.githubusercontent.com/56106207/105335576-368d9000-5bfe-11eb-87d8-3cbce5ce2585.png)



This is not exactly the issue you had pointed out, but when I was on the leaderboard page I could see that phase selector was sort of bumping into the Precision Value selector (the image is a little small, so please open it in a new tab), this is to my understanding because of the following lines
https://github.com/Cloud-CV/EvalAI/blob/c4899ac9f5916e72d5ddb4e01b48ffbe35e80d16/frontend_v2/src/styles/base.scss#L215-L219
I did not understand the reason for that specific value (`478px`)

I could override the `width` property in the component's scss file here (by setting the `ViewEncapsulation` to `None` more info [here](https://angular.io/api/core/ViewEncapsulation) )
https://github.com/Cloud-CV/EvalAI/blob/c4899ac9f5916e72d5ddb4e01b48ffbe35e80d16/frontend_v2/src/app/components/challenge/challengeleaderboard/challengeleaderboard.component.scss#L169-L171

First I wanted to run the idea by you once :)

Fixed in #3225

@Devanshu24  the issue of dropdown overlaying on precision value is fixed in #3225.
Repository: Cloud-CV/EvalAI
Issue: [FrontendV2] Show selected submission details in 3 columns All submissions page
GitHub labels: easy_to_fix

frontend_v2

GSOC-2021

### Description

When a submission is selected on the `All submissions` tab we currently divide the fields into two columns which makes it look cluttered. The metadata fields should be shown in 3 columns for meta fields in each column.

Here is the current screenshot for reference
<img width="1440" alt="Screen Shot 2021-01-20 at 9 52 56 PM" src="https://user-images.githubusercontent.com/16323427/105204235-f67ccd80-5b69-11eb-90be-a34174c31b4c.png">

Expected:

Column 1 fields: Submission Number, Team Members , Team Members Email Id, Submitted At
Column 2 fields: Stdout file, Stderr file, Metadata file
Column 3 fields: Show on Leaderboard, Unflagged, Re-run submission, Delete submission



@Ram81 please assign the issue to me. Made a PR for the last assigned issue

@burnerlee sure, but please complete the requested changes on your previous PRs before picking up new issues.

Sure, #3160 has been updated. #3146 needs review from @RishabhJain2018 and #3150 needs further actions to be decided.

@burnerlee Please do complete https://github.com/Cloud-CV/evalai-cli/issues/305 before starting with new issues. We definitely need that feature :)
cc: @Ram81 

> @burnerlee Please do complete [Cloud-CV/evalai-cli#305](https://github.com/Cloud-CV/evalai-cli/issues/305) before starting with new issues. We definitely need that feature :)
> cc: @Ram81

I thought that issue would take a bit time and I have my exams going on in my college. I would get a prep leave on 23 and complete the issue on 23 itself. Till that just thought to give a go on these small issues

Okay. Thanks for letting us know.

I'll surely submit it on 23, please please do not unassign me :(

Sure :)

Will complete this as #3163 is merged. Will use same CSS classes

Is this issue still open? @burnerlee are you working on it?

I think it was completed and merged

Closing this as this was done
Repository: Cloud-CV/EvalAI
Issue: Login/Signup not shown even after login 
Root cause: Configuration Error

### Description
Once a user has login, the signup/login button is shown on refreshing the home page instead it should show the name and log out button should be visible.

### Steps to Reproduce : 
1. Login and go on the homepage.
2. Refresh the page.

The same problem occurs If you redirect to home page through already login page.

### Would you like to work on the issue?
Yes
@RishabhJain2018 @Ram81 

![Screenshot from 2021-01-23 19-01-57](https://user-images.githubusercontent.com/24366008/105579695-f66d0f80-5dad-11eb-8a06-1734c0ad4c24.png)
Repository: Cloud-CV/EvalAI
Issue: Chrome suggests password in `Login Page` password field 

Currently, the Password field of the login page receives a password suggestion from chrome, which should not happen as you wouldn't want to be recommended a password when you login

![Screenshot from 2021-01-24 21-26-34](https://user-images.githubusercontent.com/56106207/105635794-f0992c00-5e8a-11eb-9647-a115ccfd553f.png)


This can be fixed by changing the `autocomplete` attribute in the password field on the login page from `"new-password"` to `"current-password"`
The line for reference:
https://github.com/Cloud-CV/EvalAI/blob/6bd9c14e29b396c1853e7c0e662c6efbfd40eb35/frontend_v2/src/app/components/auth/login/login.component.html#L58

[MDN](https://developer.mozilla.org/en-US/docs/Web/HTML/Attributes/autocomplete#values) on this topic for reference

I'd be happy to send a PR if you want :)

Thanks!

cc @Ram81 


@Ram81 @RishabhJain2018 Is this something you would want me to implement?

Sorry, I was not sure if I should tag you or not, I understand that you would be busy. Just wanted to ensure this hasn't skipped your sight, sorry again if this issue is not important :)

@Devanshu24 Can I take up this issue? 

Hi! Sure, I am not working on it. Although I am not sure if the team wants it. So best to ask them :)

@Ram81 @RishabhJain2018 can you review the PR ? It solves the bug of the login page.

Closing this since it is a very old issue.
Repository: Cloud-CV/EvalAI
Issue: More space between chip and View Details in Hosted Challenges card (Frontend).

When logging in with host account,
In "Hosted Challenges" section Cards will look better if it had a little spacing between "Approved, Published" chip and "View Details" .

Please have a look at the image for better understanding - 
![Cloudcv](https://user-images.githubusercontent.com/57143358/105754790-7e9b1280-5f70-11eb-80bc-a159a051e435.PNG)


I would like to work on this issue. Please assign this issue to me.

@RishabhJain2018 Please have a look at this PR.
Repository: Cloud-CV/EvalAI
Issue: No module named 'rest_framework_simplejwt' error on running after latest master pull
Root cause: Environment Incompatibility

Error:
```
worker_1     | wait-for-it.sh: timeout occurred after waiting 45 seconds for django:8000
worker_1     | Traceback (most recent call last):
worker_1     |   File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main
worker_1     |     "__main__", mod_spec)
worker_1     |   File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code
worker_1     |     exec(code, run_globals)
worker_1     |   File "/code/scripts/workers/submission_worker.py", line 43, in <module>
worker_1     |     django.setup()
worker_1     |   File "/usr/local/lib/python3.7/site-packages/django/__init__.py", line 24, in setup
worker_1     |     apps.populate(settings.INSTALLED_APPS)
worker_1     |   File "/usr/local/lib/python3.7/site-packages/django/apps/registry.py", line 91, in populate
worker_1     |     app_config = AppConfig.create(entry)
worker_1     |   File "/usr/local/lib/python3.7/site-packages/django/apps/config.py", line 116, in create
worker_1     |     mod = import_module(mod_path)
worker_1     |   File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
worker_1     |     return _bootstrap._gcd_import(name[level:], package, level)
worker_1     |   File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
worker_1     |   File "<frozen importlib._bootstrap>", line 983, in _find_and_load
worker_1     |   File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
worker_1     | ModuleNotFoundError: No module named 'rest_framework_simplejwt'
```

@burnerlee you have to rebuild the docker containers. New packages were added recently.

Closing this as it is not a bug.

Yes, worked! Thank you @Ram81 :)
Repository: Cloud-CV/EvalAI
Issue: Setup RabbitMQ Worker for Job Submission
Root cause: Unimplemented Feature Gap

A python worker(as of now) to get submission from queue, execute it with the evaluation script of the challenge(challenge id will be passed in the message) and do further tasks like sending mail, saving stdout and stderr outputs.

When a user makes a submission for a challenge, then a api to save his submission detail(like time at which the submission is made, participant team id, etc) will be requested. This api in turn will publish  a message to the queue. Assume that this message has key `submission.*.*`. This is actually the left part of the RabbitMq exchange

At the right side of the exchange, there will be python worker(will be definitely more than 1). These workers will listen on key `submission.*.*` on the exchange. Upon receiving a new message, the worker will execute the evaluation script of the desired challenge with the arguments passed in the message. stdout and stderr output will be captured and db updates, mail etc such tasks will be performed afterwards.

As discussed with @deshraj , we came up with two approaches . Listing down both approaches as separate comments

NOTE :: 
* Message key values may change 

Approach 1:

when the worker starts for the first time, fetch all the active challenges and its corresponding uploaded evaluation script. Dynamically require this evaluation script in a global variable named `EVALUATION_SCRIPTS`. So whenever any request comes for submission, the worker will fetch challenge id from that message and then simply call

EVALUATION_SCRIPTS[challenge_id].main(args)

This approach will work fine if at once all the active challenges are uploaded(this wont happen :P)
So the problem that comes is how this worker will be made aware of any new active challenge being added. One way to do this can be we can write a small worker which will poll the db at every X (6 hrs ?) interval of time and then publish a message with key `newchallenge.*.*`. This message when received at the exchange wil be delivered to each worker's queue.

So in a nutshell this approach brings

* 1 worker listening on 2 keys `newchallenge.*.*` and `submission.*.*`. 
* 1 worker polling db to fetch new active challenges.


__Problem/ Not Clear ??__

How will we redirect stdout and stderr when calling the evaluation script for a particular challenge.


Approach 2:

Worker only listens on a single key `submission.*.*`. Upon receiving a message of this type, it extracts the challenge id from the message, then queries the db to url of the evaluation script. Fetches script from there and then use subprocess to call the script.

__Problem__

For each submission need to query database, query can be definitely cached, but still a headache.

@deshraj @dexter1691 : your input please

@TroJan : this issue  is solved, right? 
Should we close?

The first version of worker is ready. Closing it now.
Repository: Cloud-CV/EvalAI
Issue: [frontendV2] Profile screen is not showing properly with screen-width more than 1440px

Profile screen is not showing properly with screen-width more than 1440px .

"Side bar" is overlapping with "Dashboard" .

For better understanding please have a look at this Image --
![cloudcv_profile](https://user-images.githubusercontent.com/57143358/106051508-8a244000-610e-11eb-8ace-de958768156d.PNG)

I want to work on this issue , Please assign it to me .




Thanks @Alabhya268 for catching this. Assigned you!