zhiminy/EvalEng huggingface dataset:
Repository: Cloud-CV/EvalAI
Issue: Setup Throttling in drf backend
Root cause: Unimplemented Feature Gap


@RishabhJain2018 you showed interest in working on this. Are you still interested in working on this issue?

@deshraj Yeah sure.

@deshraj Please provide with some details and references on how to proceed with it.

@RishabhJain2018:
[This resource](http://www.django-rest-framework.org/api-guide/throttling/#example) seems to be a good starting point to setup throttling. 

The motivation to setup throttling is to prevent the server from being over-burdened by certain API calls that might be resource intensive. Like the submission related endpoints.  A concrete example might be that we want to throttle the number of submissions a user can make to a particular competition per minute? So that one user doesn't keep retrying if the submission is failing. 

@deshraj: What else? Can we list them here for easy reference. 

The leaderboard and all the other things that an unsigned viewer can see will probably not update a lot and will be cached. So might be okay to not throttle those, right? 

@dexter1691 Thank you for the resource.

yeah. It would be okay not to throttle the leaderboard. Moreover, I also want to ask about the throttling on login attempts, Do we also need to throttle that with a time limit ?




@deshraj @dexter1691 I also found a open source load testing tool http://locust.io/ . Please have a look.

@deshraj I think we should close this issue now?

Yup.
Repository: Cloud-CV/EvalAI
Issue: Error when submitting evaluation script update without uploading file
Root cause: Validation Gap

### Description

Currently, there's no check on UI side while uploading evaluation script for a challenge. If you submit without uploading an evaluation script API will throw error mentioned [here](https://sentry.io/organizations/cloudcv/issues/1878006951/?referrer=slack). We should handle this on UI side an not allow updates without file upload.
Repository: Cloud-CV/EvalAI
Issue: Exception when model not found in `get_model_object` utils
Root cause: Validation Gap

### Description

Currently, the `get_model_object` util raises an exception when a model object with queried primary key is not found. This exception is not being handled in partial submission evaluation API. See [this](https://sentry.io/organizations/cloudcv/issues/1883775184/?project=152529&referrer=slack) error for an example

@Ram81 @RishabhJain2018, can I please take up this issue? I'll make a PR quickly to sort this out

Sure!

@RishabhJain2018 the above error link does not work for me. Could you please also tell me the API where this error takes place?

``` 
url(
        r"^challenges/(?P<challenge_pk>[0-9]+)/update_partially_evaluated_submission/$",
        views.update_partially_evaluated_submission,
        name="update_partially_evaluated_submission",
    )
```
Is it the above API we are taking about?

@RishabhJain2018 please reassign this issue to me when possible. I have solved the other assigned issue and made a PR for the same. Also, please provide some insights on the error by answering the above query too

Hey @burnerlee , You can work on this now. 

Regarding your query, I'll let @Ram81 respond to this.

@burnerlee  yes this issue is for `update_partially_evaluated_submission` API

Oh cool, I'll fix this. Thanks

@Ram81 @RishabhJain2018 made a PR #3150 to fix this. Please review

Closing this as well as we will throw `404` for not found error.
Repository: Cloud-CV/EvalAI
Issue: Auto deployment script failing
Root cause: Environment Incompatibility

### Description

The `./scripts/deployment/deploy.sh auto_deploy` is failing in the travis build. For reference see the build [here](https://travis-ci.org/github/Cloud-CV/EvalAI/builds/726818184#L6908). It is failing with following error

```
bash: docker-compose: command not found
```

It was the docker-compose path issue on the server. It is fixed now!
Repository: Cloud-CV/EvalAI
Issue: Leaderboard takes a long time to load due to vendor.js exceptions 
Root cause: Configuration Error

Whenever I load the main screen I receive a bath or errors of missing links in vendor.js this takes about 5 seconds and delays my page loading.
![evalai](https://user-images.githubusercontent.com/874774/94306918-2f08f380-ff4a-11ea-90ac-40b1d159b0fe.png)
Anyone else ever experienced this or have any clue about how  to address the issue?

Hi @vfbsilva , Thanks for opening the issue. Can you please give a bit more context on how you are hosting the EvalAI server at your end? Is it the development version or did you try to set up the production version at your own?

We are using a production environment I did some more research and seems the call to:
`/api/jobs/challenge_phase_split/18/leaderboard/?page_size=1000`
Is taking a very long time. iS this normal? Can this value be changed?

Hey @vfbsilva ,

> We are using a production environment I did some more research and seems the call to:
> /api/jobs/challenge_phase_split/18/leaderboard/?page_size=1000
> Is taking a very long time. iS this normal? Can this value be changed?

It shouldn't take long to load. The value can be changed from the Controller file.
Moreover, can you please give a bit more detail of where you are trying to host it in production mode?

Closing this since it is very old.
Repository: Cloud-CV/EvalAI
Issue: Dedup environment variables for EvalAI django server endpoint
Root cause: Architectural Constraint

### Description

We are using multiple environment variables for EvalAI django server endpoint, `API_HOST_URL` `EVALAI_API_SERVER` `DJANGO_SERVER`. Ideally we should only have a common environment variable `DJANGO_SERVER` for the endpoint which should be used by all the features.

Refer: 
1. [dev/docker.env](https://github.com/Cloud-CV/EvalAI/blob/master/docker/dev/docker.env)
2. [prod/docker_staging.env](https://github.com/Cloud-CV/EvalAI/blob/master/docker/prod/docker_staging.env)
3. [prod/docker_production.env](https://github.com/Cloud-CV/EvalAI/blob/master/docker/prod/docker_production.env)
Repository: Cloud-CV/EvalAI
Issue: Add a field max_submissions_per_day and max_submissions in ChallengePhase model
Root cause: Unimplemented Feature Gap

Both the fields should be positive integer field with default value of 100K. 100K here means number of submissions are unlimited.

Adding these fields will involve creating a migration. Also please create a named migration.

Can I try if unassigned? 

Yeah sure :)


 Close this ?

@gauthamzz : sure. Thanks for the help :)
Repository: Cloud-CV/EvalAI
Issue: A11y Support

Hi there! I like this initiative, congrats for this amaze project! You have in the backlog add support for internationalization?

> You have in the backlog add support for internationalization?

Hey @brunowego, At the moment we are not focussing on it but would be happy to accept any PR for it.

This is an interesting idea! I think we would need to consider somethings,

1. Is this a very important issue right now? We can make a general issue that lists down the work required for different languages.
2. What languages to prioritize for, as in, which group of users should we aim for first? 
3. How are we going to let users choose a different language?

Closing this since it is not a priority for now.
Repository: Cloud-CV/EvalAI
Issue: LDAP or/and SSO Auth
Root cause: Unimplemented Feature Gap

Looking forward, do you have planning for add this functionarlity?

Hi @brunowego, At the moment the team is not focussing on adding this functionality but we would be happy to accept any PR from the contributors. 

@rishabhbhardwaj  can I work on this issue?

Closing this since it is a very old issue.
Repository: Cloud-CV/EvalAI
Issue: Rename TestEnvironment model to ChallengePhase model
GitHub labels: easy_to_fix

backend

This will involve updating references in places liks api(views, urls), model, serializers and test case.
Also a named migration should be created for this.

Why this model name change ?

Because pyflakes is generating the following warning.
```
cannot collect test class 'TestEnvironment' because it has a __init__ constructor
```

Reason for this warning is pyflakes looks for class or functions starting with `Test*` and here by chance model name is `TestEnvironment`.


Merged via #304
Repository: Cloud-CV/EvalAI
Issue: 2099 End Date Issue


![Screenshot from 2020-11-02 01-57-00](https://user-images.githubusercontent.com/24211231/97813516-c3374a80-1cae-11eb-819a-c6c6b77a739f.png)

Some End Dates in Challenges are showing 2099.
Expected Error in Date/Time Local code in the front-end.

Hi @vivektalwar13071999, Thanks for opening the issue. The challenges that have `2099` as the end date are never-ending challenges, hence the date is in the future.

I am closing this thread as this is not an issue. Please feel free to reopen if you have any doubts.
Repository: Cloud-CV/EvalAI
Issue: Challenge Start Date is showing year 2020 and End Date is showing 2019
GitHub labels: bug

easy_to_fix

backend

![Screenshot from 2020-11-02 02-06-54](https://user-images.githubusercontent.com/24211231/97814839-1fe73500-1cb0-11eb-91bd-fe32b82a792f.png)

A front-end issue in the Past-Challenge section where challenge year is 2019 and start year is 2020.

@vivektalwar13071999 thanks for catching this. Would you like to work on this?

Yes will work and resolve the issue and ask if any help would be needed. 

Closing the issue as it is resolved in the production enviornment.
Repository: Cloud-CV/EvalAI
Issue: Docker Compose fails - Cannot download base worker container
Root cause: Environment Incompatibility

Following installation instruction from https://evalai.readthedocs.io/en/latest/installation.html:

The command:
`
docker-compose up --build
`
Fails with the following error 

```
Building worker
Step 1/6 : FROM evalai_django
ERROR: Service 'worker' failed to build : pull access denied for evalai_django, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
```

This is from a fresh pull

Connection issue was present from 10:00AM EDT, 11/03/2020
Connection issue self-resolved as of 2:00 PM EDT, 11/03/2020
Could have been an issue with docker servers.
Repository: Cloud-CV/EvalAI
Issue: Warnings generated by Docker-Compose on fresh build 
Root cause: Configuration Error

# Problem

Cloning a fresh version of EvalAI and running `docker-compose up --build`, outputs warnings

## Warning List

1. [ ]  django-import-export 2.0.2 has requirement Django>=2.0, but you'll have django 1.11.23 which is incompatible.
2. [ ] djangorestframework-expiring-authtoken 0.1.4 has requirement djangorestframework<=3.4.6,>=3.2.3, but you'll have djangorestframework 3.9.3 which is incompatible.
3. [ ] docker-compose 1.21.0 has requirement PyYAML<4,>=3.10, but you'll have pyyaml 4.2b1 which is incompatible.
4. [ ] docker-compose 1.21.0 has requirement requests!=2.11.0,!=2.12.2,!=2.18.0,<2.19,>=2.6.1, but you'll have requests 2.24.0 which is incompatible.
5. [ ] google-auth 1.23.0 has requirement setuptools>=40.3.0, but you'll have setuptools 39.2.0 which is incompatible.
6. [ ] aws-sam-translator 1.28.1 has requirement jsonschema~=3.0, but you'll have jsonschema 2.6.0 which is incompatible.
7. [ ] cfn-lint 0.40.0 has requirement importlib-resources~=1.4; python_version < "3.7" and python_version != "3.4", but you'll have importlib-resources 3.3.0 which is incompatible.
8. [ ] cfn-lint 0.40.0 has requirement jsonschema~=3.0, but you'll have jsonschema 2.6.0 which is incompatible.
9. [ ] You are using pip version 10.0.1, however version 20.3b1 is available.You should consider upgrading via the 'pip install --upgrade pip' command.

## Notes

For `9`, this is thrown when in actual the pip3 / pip version of the host is `20.2.4`.

# Steps to reproduce 

```bash
git clone https://github.com/Cloud-CV/EvalAI
cd EvalAI
sudo docker-compose up --build
```

# What should happen instead

The versions specified are old and should be updated. I suggest we use this issue to maintain a list of what warnings need to be updated, and which might remain still.


This issue has been inactive for a while, so we're closing it for now. If this is still relevant, please feel free to reopen or create a new issue with updated details. Thanks!
Repository: Cloud-CV/EvalAI
Issue: Add bot for tracking stale issues

It would nice to add a bot that can check, mark, and close issues that are stale. There are threads under 'Issues' which haven't been touched for weeks. Do we want to do that for this repository? I'll be happy to open a PR if needed. 

@RishabhJain2018 @Ram81 may I add this bot?

We already have this bot configured, hence closing this one!
Repository: Cloud-CV/EvalAI
Issue: Certain items not correctly aligned in mobile view

- **the expected behaviour**
The icons and the buttons should be centrally aligned so that the page looks aesthetically clean
- **the observed behaviour**
An extra right-margin is set which is fine for desktop view but doesn't look good in the mobile view of the page
- **steps to reproduce the problem**:-
Login and see the page using Inspect Element feature of Chrome. Alternatively the same issue should be present while logging in from smartphones as well.

**Here are the screenshots**
![1](https://user-images.githubusercontent.com/59053357/100602617-22c65b80-332a-11eb-864a-bb6088473818.JPG)
![2](https://user-images.githubusercontent.com/59053357/100602876-7c2e8a80-332a-11eb-926a-4df94d0284f9.JPG)


Hi! I would like to work on this issue!

I saw that this issue is still on the live site, can you assign this task to me? 

can i work on this issue?
Repository: Cloud-CV/EvalAI
Issue: Submit button leads to freeze/hang, same observed over CLI
Root cause: Environment Incompatibility

Hello EvalAI, thanks for all your work building this platform.

I'm having trouble submitting to the Argoverse 3d Detection challenge (challenge 725 phase 1554) over the online interface or over CLI.

The CLI hangs forever at
```
evalai challenge 725 phase 1554 submit --file myfile.zip --private
...
Publication URL []: 
Uploading the file...
  0%|                                                                                                                | 0/1 [00:00<?, ?it/s]
```

The online interface turns grey and hangs forever when I press "submit" for a 80 MB zip file.

Could you please take a look to see what might be going wrong?

was successful on another machine, seems to be a network issue

Hi @johnwlambert, Glad that it got resolved. Please feel free to ping us back, if the issue appears again.
Repository: Cloud-CV/EvalAI
Issue: Shifting from 'django-rest-auth' to 'dj-rest-auth'
Root cause: External Dependency Breakage

We have to shift from `django-rest-auth` to `dj-rest-auth` as `django-rest-auth` has stopped updating. `dj-rest-auth` is almost similar to `django-rest-auth` and has some updating feature that we can use also it's an active project so it gets updated quite often.
You can find `dj-rest-auth` documentation [here](https://dj-rest-auth.readthedocs.io/en/latest/introduction.html)


Hello @Ram81 , I was thinking to get started with this issue and when #3069 will be done we can update this. Also, I just want to ask that what version I should mention in `requirements/common.txt`? I'm getting a bit confused about that, [here](https://github.com/jazzband/dj-rest-auth/releases) is the list of their releases.
Thanks
(Off topic) One more question I want to ask is that, where I can find the signup part, actually I checked the `django-rest-auth`, in `rest_auth.urls` but I wasn't able to find it, as the signup part is on `https://eval.ai/auth/signup` so it must be on `rest_auth.urls` but I couldn't find it. I want to report and work on an issue regarding that.
Thanks

Hello @Ram81 and @RishabhJain2018  , will you please review it.
Thanks
Repository: Cloud-CV/EvalAI
Issue: Listing contributors to README page

What we can do is we can list the main contributors and maintainers on the README page, this way it will be more attractive and it will encourage our main contributors. Either we can list all the contributors of this project or we can simply list the main contributors.
We can do it by simply using [all-contributors](https://allcontributors.org/docs/en/bot/usage).
It will look like this - 
![Screenshot from 2021-01-03 03-57-39](https://user-images.githubusercontent.com/44300735/103467780-7d633500-4d78-11eb-8b47-8d5a746a34d4.png)

A process on how to work on this issue - For this issue, the maintainer of the project have to [configure](https://allcontributors.org/docs/en/bot/installation) the repository with the bot So that the bot can automatically list the contributors on the readme file. We can do it here and follow the steps - 
![Screenshot from 2021-01-03 04-14-40](https://user-images.githubusercontent.com/44300735/103467948-3bd38980-4d7a-11eb-9400-34ed5df7abc8.png)
Just after that, we can simply list main contributors just like this - 

![Screenshot from 2021-01-03 04-15-48](https://user-images.githubusercontent.com/44300735/103467969-60c7fc80-4d7a-11eb-94c5-50ff4f99059b.png)

This task will hardly take 1 minute to complete but it will make our project look more attractive, Views are most welcome.
Thanks

Closing this since it is a very old issue and we already have this in Readme.
Repository: Cloud-CV/EvalAI
Issue: Add swagger api documentation for get_all_entries_on_public_leaderboard API
GitHub labels: backend

docs

good first issue

GSOC-2021

### Description

Hosts can use EvalAI APIs to get details of the leaderboard entries. The get leaderboard entries API documentation is missing documentation on EvalAI [docs](http://eval.ai/api/docs/).


### Deliverables
Add documentation for using `get_all_entries_on_public_leaderboard` API using swagger. An example of using swagger for API documentation is available [here](https://github.com/Cloud-CV/EvalAI/blob/master/apps/jobs/views.py#L87). The documentation should provide a list of query and header params described briefly and an example.

hello @Ram81, can I please work on and complete the issue?

@burnerlee yes, go ahead. Assigned the issue

thank you so much

@Ram81, made a PR #3140 to solve this issue, please review. Also please suggest some other issues which I can work on for further contribution

Fixed using #3140
Repository: Cloud-CV/EvalAI
Issue: Add swagger api documentation for download_all_submissions API
GitHub labels: backend

docs

good first issue

GSOC-2021

### Description

Hosts can use EvalAI APIs to download all submission details. The download all submissions API documentation is missing documentation on EvalAI [docs](http://eval.ai/api/docs/).


### Deliverables
Add documentation for using `download_all_submissions` API using swagger. An example of using swagger for API documentation is available [here](https://github.com/Cloud-CV/EvalAI/blob/master/apps/jobs/views.py#L87). The documentation should provide a list of query and header params described briefly and an example.

I am new to open source contribution, Can I help to resolve the issue.
I used swagger and know how it works.
Can you assign an issue?

@mukul20-21 sure, you can work on the issue.

Hello sir, I am Nikhilesh I have  experience in writing swagger documentation for two REST api's I created.Can I work on this issue if its still open?

@Ram81 , Seems like nobody is working on this issue, Could you please assign it to me.
Repository: Cloud-CV/EvalAI
Issue: Add swagger api documentation for remove_participant_team_from_challenge API
Root cause: Documentation Deficiency

### Description

Hosts can use EvalAI APIs to remove participant teams from an active challenge. The remove participant team from challenge API documentation is missing documentation on EvalAI [docs](http://eval.ai/api/docs/).


### Deliverables
Add documentation for using `remove_participant_team_from_challenge` API using swagger. An example of using swagger for API documentation is available [here](https://github.com/Cloud-CV/EvalAI/blob/master/apps/jobs/views.py#L87). The documentation should provide a list of query and header params described briefly and an example.

Hello Sir,
I have deployed an machine learning model using Swagger API, So, I know how it works.
I am also new to open source contribution .
Can you please assign this issue to me ?
I want to help in resolving it and want to contribute to open source community.


@17Rajat yes, go ahead. 

> @17Rajat yes, go ahead.

Thanks


Hello Sir, I am Nikhilesh. I have experience in writing swagger documentation can I work on it if it is still open?

Hi @Ram81 , Can I work on this issue?
Thanks

Hello @Ram81 @RishabhJain2018 
Small doubt,
I'm working on this issue but I'm stuck at one point. I was trying to test the API using postman then I got an error "Authentication credentials were not provided." So I passed the Token to the authorization part but still I'm getting this error. Can you provide some insights?
I tried searching on google but got nothing.
Thanks
Repository: Cloud-CV/EvalAI
Issue: Fix About us page css inconsistencies in frontend v2
GitHub labels: good first issue

frontend_v2

GSOC-2021

### Description

Fix the about us page on large screens. Currently, the about us page css is not consistent between frontend v1 and v2. The page css is broken for larger screens in frontend v2. Page css should be same as frontend v1

Frontend V1:
![Screen Shot 2021-01-10 at 7 30 04 PM](https://user-images.githubusercontent.com/16323427/104125095-92445780-537a-11eb-9b05-35c597f41664.png)


Frontend V2
![Screen Shot 2021-01-10 at 7 30 06 PM](https://user-images.githubusercontent.com/16323427/104125100-983a3880-537a-11eb-9d27-20ba16fb0f7e.png)

 Also, make sure that the content css is the same as that of the frontend V1. 



@Ram81 can I work on this please. I'll make a PR to fix this quickly

Hi @burnerlee, I think PR #3133 already fixes this issue. You can pick up some other issue to work on
Repository: Cloud-CV/EvalAI
Issue: Fix Our team page css inconsistencies in frontend v2
GitHub labels: good first issue

frontend_v2

GSOC-2021

### Description

Fix the our team page css. Currently, our team page css is not consistent between frontend v1 and v2. The page css is broken for larger screens in frontend v2. Page css should be same as frontend v1.

Frontend V1:
![Screen Shot 2021-01-10 at 7 35 17 PM](https://user-images.githubusercontent.com/16323427/104125204-565dc200-537b-11eb-9164-6635ea4b45a5.png)


Frontend V2:
![Screen Shot 2021-01-10 at 7 35 20 PM](https://user-images.githubusercontent.com/16323427/104125228-68d7fb80-537b-11eb-85e4-6722e8375c93.png)


 Also, make sure that the content css is the same as that of the frontend V1. 



Working on it!

Should there be a box-shadow in the background and division of Core Team and Contributor in v2? Or it should be same as it is currently.

Let's make it the same as it is now.

@RishabhJain2018 This can also be closed as #3133 has been merged.

Closing this. Thanks for pointing it out.
Repository: Cloud-CV/EvalAI
Issue: Show past challenges in frontend v2 dashboard
Root cause: Unimplemented Feature Gap

### Description

On the frontend v2 dashboard, we show a list of current and upcoming challenges but the list of past challenges is not being populated. 

![Screen Shot 2021-01-10 at 7 44 32 PM](https://user-images.githubusercontent.com/16323427/104125371-514d4280-537c-11eb-811f-59c797b646ef.png)

The above screenshot shows the list of challenges populated challenges in current and upcoming tabs but not in the past challenges tab.

### Deliverables

1. Show the list of cards for past challenges returned by the challenge API