zhiminy/EvalEng huggingface dataset:
Repository: Cloud-CV/EvalAI
Issue: Create an API endpoint to get all the challenges participated by the user
GitHub labels: GSOC-2019

easy_to_fix

backend

URL: `/api/challenges/get_all_challenges_participated_by_user/`
Method: GET

@deshraj Can I work on this?

Sure, let me know if you need some help. 

okk.

@deshraj As discussed with **TroJan** in #349 , please assign this to TroJan.

FIxed in #503
Repository: Cloud-CV/EvalAI
Issue: Create an API endpoint to get all the challenges hosted by the user
Root cause: Unimplemented Feature Gap

URL: /api/challenges/get_all_challenges_hosted_by_user/
Method: GET

@deshraj Can I work on this ?

Sure. Assigning it to you. 

@deshraj As discussed with **TroJan** in #350 , please assign this to TroJan.

Fixed in #503
Repository: Cloud-CV/EvalAI
Issue: 2 proposed feature requests for leaderboard
Root cause: Unimplemented Feature Gap

Hi EvalAI team, thanks for all your hard work on an excellent platform. I have two proposed feature requests:

1. When I click on an entry in a leaderboard (say the 5th entry), the leaderboard re-centers on the object, and hides the objects above, which is a bit counter-intuitive. I would prefer behavior where that entry is actually highlighted.

2. Could we have an option to show on the leaderboard which columns have "higher is better" or "lower is better"? (via arrows)

(e.g. as shown in Table 4 of [this paper](https://arxiv.org/pdf/2006.11275.pdf))
<img width="1130" alt="Screen Shot 2021-06-19 at 1 53 25 PM" src="https://user-images.githubusercontent.com/16724970/122651271-ba906900-d105-11eb-9825-53fdd69e5af8.png">



Hi @johnwlambert, Thank you for the feature requests. We will try to release these ASAP.


Closing as this is fixed!
Repository: Cloud-CV/EvalAI
Issue: Remote evaluation doesn't work
Root cause: Interface Contract Mismatch

Running eval.ai with docker-compose and trying to connect with a remote worker does not work. It seems the remote worker script is a bit out of date with the current API. The [API documentation](https://eval.ai/api/docs/#operation/jobs_challenge_submission_list) helped out a lot in figuring out what the issue was.

These are the issues I had to fix to get the script to work:

* Rename `Token` to `Bearer` ([L329](https://github.com/Cloud-CV/EvalAI/blob/96c2b5e02bfbcceba6d1c672d20529a059fd6af8/scripts/workers/remote_submission_worker.py#L329)).
* Remove return statement ([L527](https://github.com/Cloud-CV/EvalAI/blob/96c2b5e02bfbcceba6d1c672d20529a059fd6af8/scripts/workers/remote_submission_worker.py#L527-L528)). I don't understand why this is there.
* Fix the `submission_data["result"]` data structure format. 
  The evaluation script is expected to return an object in the format of :
  ```
  [ 
    { "train-split" : { 
      "Metric" : 100 
    } } 
  ]
  ``` 
  but the API expects :
  ```
  [ 
    {
      "split": "train-split", 
      "show_to_participant" : True, 
      "accuracies" : { "Metric" : 100 } 
    }
  ]
  ```
  So a little bit of post-processing is needed. 

I will update #3496 to contain these fixes.

@Ram81: After I update the PR, can you review my changes?

Closing this since it is a very old issue.
Repository: Cloud-CV/EvalAI
Issue: Unable to paste passwords on signup form

Since I use a password manage (as I would argue most if not everyone should), I prefer to be able to paste in passwords in sign up forms since they are randomly generated. There is a lot of debate around this issue, but the consensus seems to be that disallowing pasting passwords is bad for security since it encourages users to choose weak passwords among other reasons.

Some references below:

[The “Cobra Effect” that is disabling paste on password fields](https://www.troyhunt.com/the-cobra-effect-that-is-disabling/)
[Let them paste passwords](https://www.ncsc.gov.uk/blog-post/let-them-paste-passwords)
[Don’t Disable Copying / Pasting Passwords](https://medium.com/@EricJFisher/dont-disable-copying-pasting-passwords-a2198428f8f)
[Is there any reason to disable paste password on login?](https://security.stackexchange.com/questions/131106/is-there-any-reason-to-disable-paste-password-on-login)

hey just wanted to know if this topic has come to conclusion or not ?


Hey. Is this issue not solved yet? I would love to contribute to this. 

Hey , I would like to contribute to this issue, which files should be changed?


@michaelmior Is this issue still there? I would like to work on this.  

I don't think this is an issue any more as I am able to paste in a password now.
Repository: Cloud-CV/EvalAI
Issue: Create endpoint to get all the teams
Root cause: Unimplemented Feature Gap

Details about the endpoint:
- Url: /api/teams/challenge/<challenge_id>
- Header: Authorization: Token <token>
- Method: GET
- Response:
  - status: HTTP_200_OK

Also, please add the pagination for the same. 

For more details, feel free to discuss on thread or gitter channel https://gitter.im/Cloud-CV/EvalAI

Fixed in #60
Repository: Cloud-CV/EvalAI
Issue: Integrate Eslint with the frontend code
GitHub labels: enhancement

GSOC-2019

easy_to_fix

frontend


For reference, see this http://eslint.org/docs/user-guide/configuring

@deshraj Can I work on this??

@deshraj should I create the configuration file at `EvalAI/.eslintrc.json` this location or should I add it in `package.json` file?

@RishabhJain2018 : Do some research and find the best way to integrate Eslint by your own.

@aka-jain okk sure.

Assigning to @aka-jain now. 

@RishabhJain2018: This resource might be very useful: https://github.com/airbnb/javascript
It's the gold standard guidelines used by a lot of javascript repositories. 

@dexter1691 : Airbnb guidelines are really awesome but we are already using John Papa Guidelines for our Javascript (Angular1), also currently we are not working on ES6 version which is highly recommendable for production now a days so I think at this point this repo may not be useful for us as many guidlines are related to ES6. But to make code quality good, John Papa guidelines are also fair enough for JS and Angular. https://github.com/johnpapa/angular-styleguide/blob/master/a1/README.md

@RishabhJain2018 : You can go thorugh the Airbnb repo, it will be very helpful to understand the structure of Javascript :) And for our project, try to go through ES5 section.

@aka-jain : This is done? Should we close?

@dexter1691 : Need to do some improvements on this. I'm working on it.

@aka-jain : Assigning this to me since you have other tasks to do.

@spyshiv Sure.

@spyshiv Is it done with Gulp improvement?

@aka-jain : Almost done. There is seperate branch `js_linting` for this purpose. Need minor improvements in it.
However checkout to that branch and run `gulp lint` to see the errors and warnings. 

Fixed with #650
Repository: Cloud-CV/EvalAI
Issue: settings/dev.sample.py changed but not updated in Readme.md

no USER and PASSWORD field in dev.sample.py as written in Readme

This is how it is supposed to look. 


'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'evalai',
        'USER': 'your username',
        'PASSWORD': 'your password',
        'HOST': 'localhost',
        'PORT': '5432',
    }

Shoudn't this be fixed @deshraj 

Sorry, my bad. Thanks @parismita for opening this issue. Do you want to take lead on this and fix it? 

yeah! what is to be changed Readme or dev.sample.py?


You should change the `dev.sample.py`

Fixed.
Repository: Cloud-CV/EvalAI
Issue: Challenge_pk and phase_pk change type from string to int when resubmitting
Root cause: Interface Contract Mismatch

Hello everyone,
while processing submissions for a challenge, I noticed that the `phase_pk` and `challenge_pk` keys change type.
When a submission is sent the first time, my evaluation script receives them as a `string`, e.g.
```
b'{"body":{"challenge_pk":"1134","phase_pk":"2326","is_static_dataset_code_upload_submission":false," ....
```
However, if the submission is resent again by clicking on "Rerun submission" on EvalAI All submissions tab, the `challenge_pk` and `phase_pk` become `int` type:
```
b'{"body":{"challenge_pk":1134,"phase_pk":2326,"submission_pk":154550,"is_static...
```

I discovered this as my evaluation script had to do different things depending on the `phase_pk` and having it change from `string` to `int` raised an exception.


hey, I am new to open source, can I work on this issue

Hi @emilio-cartoni, Thanks for pointing this out. We are looking into it.
cc: @Ram81
Repository: Cloud-CV/EvalAI
Issue: Not able to see scores for LVIS challenge
Root cause: Environment Incompatibility

Hi!

We have submitted our file and it has been over 24 hours but we are not able to see our scores for LVIS 2021 val phase. Is this normal? What is the expected delay? Also is there any way to know if something went wrong and if so, what went wrong?

Now it shows cancelled. With execution time None.

I also meet the same problem? have you solved it?

Nope

Hi @sudz123, there was a issue with our evaluation server due to which submissions were not processed. We have fixed the issue, your submissions should be done evaluating in couple of hours.

Closing it as it is resolved. Please feel free to reopen in case it is still not resolved.
Repository: Cloud-CV/EvalAI
Issue: Create endpoint to update the team details
GitHub labels: enhancement

GSOC-2019

medium-difficulty

Details about the endpoint:
- Url: /api/teams/<team_id>/
- Header: Authorization: Token <token>
- Method: PUT/PATCH
- Response: 
  - status on success: HTTP_200_OK
  - status on exception:  HTTP_406_NOT_ACCEPTABLE
  - status on Unauthorized access: HTTP_203_NON_AUTHORITATIVE_INFORMATION

For more details, feel free to discuss on thread or gitter channel https://gitter.im/Cloud-CV/EvalAI

Fixed in #64
Repository: Cloud-CV/EvalAI
Issue: Cost for hosting challenges?

Hi there

Is there any cost associated with hosting a challenge on EvalAI?

Thanks

Hi @danielamassiceti, no there is no cost associated with hosting a challenge on EvalAI. You can go to [eval.ai](https://eval.ai/) and follow the instructions on our [starters template](https://github.com/Cloud-CV/EvalAI-starters) to get started.

Closing this, as this is not a issue
Repository: Cloud-CV/EvalAI
Issue: Add a timeout field in Submission model 
Root cause: Unimplemented Feature Gap

Add a new field `timeout` in `Submission` model in jobs app. Set the default to `300 seconds`. 

@deshraj can I work on this?

Sure, assigning it to you. 

any hint for this ?

Anwesh, see this https://docs.djangoproject.com/en/1.10/ref/models/fields/#timefield

@deshraj I have one doubt. You mentioned timeout? I couldn't actually get how to proceed with this. TimeField as it says in docs time( represented in Python by a datetime.time instance ) How about setting default to 300s ?

See this https://docs.python.org/2/library/datetime.html#datetime.timedelta.total_seconds. This might help. 

For eg. you can create a 300 secs datetime object using following code: 
```
from datetime import timedelta
a = timedelta(seconds=300)
```

@deshraj : what if we create this as a positive integer field with default value of 300 ? Also can we please keep the name as execution_time_limit. This is actually self explanatory.

@deshraj it is done should be closed

Merged in #391 . thanks @anweshknayak :)
Repository: Cloud-CV/EvalAI
Issue: docker-compose error
Root cause: External Dependency Breakage

Hello, 

Upon running the docker-compose command, I am receiving the following error:

 => CANCELED [evalai_django internal] load metadata for docker.io/library/python:3.7.5                             8.3s
 => CANCELED [evalai_nodejs_v2 internal] load metadata for docker.io/library/node:12.2.0                           8.2s
 => CANCELED [evalai_nodejs internal] load metadata for docker.io/library/node:8.11.2                              8.2s
 => ERROR [evalai_worker internal] load metadata for docker.io/library/evalai_django:latest                        8.1s
failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to create LLB definition: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

Any ideas why this might be happening/how to resolve? 

Thank you.

Same Issue. Can't set up the project. Seems like the image `pakohan/elasticmq` is removed.

Hey @gshippee @Aditya-Gupta1 , Thanks for bring this to our notice. We will look into this and get back to you.

This issue was fixed. Closing this PR
Repository: Cloud-CV/EvalAI
Issue: Django rest framework packages version conflicts 
Root cause: External Dependency Breakage

Building the project fails due to python package incompatibility issue.

**Expected behaviour**
  Builds successfully 

**Observed behaviour**
Build fails with version conflicts error on pip install.
djangorestframework-expiring-authtoken==0.1.4 requires djangorestframework to be less than 3.4.6
But drf-yasg==1.17.0 requires djangorestframework to be greater than 3.8

**Operating System**
Ubuntu (tried on Rhel 7 too but was facing the same error)

Thank you!


Hello,have you solved it?

hello, i want to solve this issue, can i take this one?@ShanmukhaSahiti 

hello, I have meet this same problem, too. had any one found ways to solve it?

The current configuration builds correctly. Please feel free to re-open if the issue persists.
Repository: Cloud-CV/EvalAI
Issue: Docker compose error on Apple Silicon
Root cause: Environment Incompatibility

Running `docker-compose up --build` results in the following when building the nodejs container. Please let me know how I can get past this!

```
 => ERROR [ 4/18] RUN apt-get update && apt-get install -yq google-chrome-stable                                                            6.2s
------
 > [ 4/18] RUN apt-get update && apt-get install -yq google-chrome-stable:
#7 2.667 Get:1 http://security.debian.org jessie/updates InRelease [44.9 kB]
#7 3.667 Ign http://deb.debian.org jessie InRelease
#7 3.667 Get:2 http://deb.debian.org jessie-updates InRelease [16.3 kB]
#7 3.667 Get:3 http://dl.google.com stable InRelease [1811 B]
#7 3.678 Get:4 http://deb.debian.org jessie Release.gpg [1652 B]
#7 3.693 Get:5 http://deb.debian.org jessie Release [77.3 kB]
#7 5.206 Get:6 http://dl.google.com stable/main amd64 Packages [1099 B]
#7 6.205 W: Failed to fetch http://deb.debian.org/debian/dists/jessie-updates/InRelease  Unable to find expected entry 'main/binary-arm64/Packages' in Release file (Wrong sources.list entry or malformed file)
#7 6.205
#7 6.205 W: Failed to fetch http://security.debian.org/debian-security/dists/jessie/updates/InRelease  Unable to find expected entry 'main/binary-arm64/Packages' in Release file (Wrong sources.list entry or malformed file)
#7 6.205
#7 6.205 W: Failed to fetch http://deb.debian.org/debian/dists/jessie/Release  Unable to find expected entry 'main/binary-arm64/Packages' in Release file (Wrong sources.list entry or malformed file)
#7 6.205
#7 6.205 E: Some index files failed to download. They have been ignored, or old ones used instead.
#7 6.205 Fetched 143 kB in 3s (40.2 kB/s)
------
executor failed running [/bin/sh -c apt-get update && apt-get install -yq google-chrome-stable]: exit code: 100
ERROR: Service 'nodejs' failed to build : Build failed
```

Thanks

I did some more digging. First off, it seems that the nodejs image required updating - I used node:14. The package google-chrome-stable was also not available, so I had to comment that but still run 'apt-get update' in the Dockerfile.

I also had trouble installing matplotlib - in the end I had to comment it out in worker.txt due to 'Running cythonize failed!' errors. This allowed me to get the app up and running!

Closing this issue.

Is there a solution?

Updating node (e.g., `node:14`) also worked for me.

???
Repository: Cloud-CV/EvalAI
Issue: Changing the port numbers

Is it possible to change the ports from 8888 and 8000 to other numbers?

Hi @seokhwankim, Yes, you can modify the port numbers when launching the docker containers in [docker-compose.yml](https://github.com/Cloud-CV/EvalAI/blob/master/docker-compose.yml). 

Closing this, as this is not a issue.
Repository: Cloud-CV/EvalAI
Issue: [Installation] `pakohan/elasticmq` Error During Installation
Root cause: External Dependency Breakage

If you'd like to report a bug, please include the following information:
- expected behaviour
Successful Installation
- observed behaviour
Error
```❯ sudo docker-compose up --build
Pulling sqs (pakohan/elasticmq:)...
ERROR: The image for the service you're trying to recreate has been removed. If you continue, volume data could be lost. Consider backing up your data before continuing.

Continue with the new image? [yN]n
ERROR: pull access denied for pakohan/elasticmq, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
```

- steps to reproduce the problem
`git clone https://github.com/Cloud-CV/EvalAI.git evalai && cd "$_"`
`docker-compose up --build`
- which operating system and/or browser you're using, if relevant
Elementry OS (Based on Ubuntu 18.04)
- (if possible) copies of the console/server logs
![Screenshot from 2021-11-23 12-56-10](https://user-images.githubusercontent.com/72073401/142984878-134df7aa-66c3-4220-954b-7373cb355618.png)

PS: I'm already logged into Docker via terminal

It seems that [pakohan/elasticmq](https://hub.docker.com/r/pakohan/elasticmq) image is removed from docker hub and is the root cause of this problem
https://hub.docker.com/r/pakohan/elasticmq

@uzair-ali10 hello sir can i help you to solve this ?


@uzair-ali10 assign this to me

@uzair-ali10 fixed check 

@uzair-ali10 sir can u please accept my pull request i wanna contribute

Closing this issue. Fixed in PR #3638
Repository: Cloud-CV/EvalAI
Issue: Set django cache backend for the development environment
Root cause: Configuration Error

Currently, we are using memcached for caching in the development environment also. We should change it to default django cache backend. 

Let me know if you have questions. 

@deshraj Can I work on this?

Sure. Assigning it to you. :-) 

@deshraj For a suggestion, can we use [this](https://docs.djangoproject.com/en/1.10/topics/cache/#dummy-caching-for-development) cache backend for development ?

Sounds good to me.

Fixed.
Repository: Cloud-CV/EvalAI
Issue: Submission metadata feature requests
Root cause: Unimplemented Feature Gap

Hi! I have two proposed feature requests for submission metadata:
1. The "View Metadata" bubble on the leaderboard currently only displays the `submission_meta_attributes` from the config and not `default_submission_meta_attributes`. I would prefer `default_submission_meta_attributes` (e.g. project URL and publication URL) to be visible as well, or at least for there to be an option to make them visible.
2. When I make a submission and decline to provide information for `submission_meta_attributes`, there is currently no place to add/edit those fields after submission. The Edit option on the My Submissions page only displays the `default_submission_meta_attributes` if no `submission_meta_attributes` were included with the original submission (images below). I'd like it to be possible to fill those fields in after submission.

Edit submission pop-up after submitting *without* `submission_meta_attributes` for challenge 1256:
<img width="165" alt="no_optional_meta" src="https://user-images.githubusercontent.com/64850082/144514830-e1748869-44d4-427d-8356-2676c278787d.png">

Edit submission pop-up after submitting *with* `submission_meta_attributes` for challenge 1256:
<img width="253" alt="with_optional_meta" src="https://user-images.githubusercontent.com/64850082/144514911-1f54a825-200a-4fa5-ad5d-0fe2f1026578.png">

Hi @felixp8 , Thanks for the feature requests. We will get them added to the platform soon.

@RishabhJain2018 can I work on this?
Repository: Cloud-CV/EvalAI
Issue: Create is_public field in Phase model
GitHub labels: enhancement

GSOC-2019

easy_to_fix

backend

We would like to add a new field `is_public` in phase model to check if the particular phase is set active by the host team or not. Set the default value to `False`. 

Can you assign this to me , do we need a named migration after this ?

@gauthamzz I am glad to see such enthusiasm but can you please finish other issues before I assign you any new ? 

Fixed in #373.
Repository: Cloud-CV/EvalAI
Issue: [Backend] Add support to recreate fargate workers for active challenges from django admin
GitHub labels: medium-difficulty

backend

GSOC-2022

### Description

To push latest changes to active farget workers we need to recreate the service so that it uses the latest docker image of submission worker. Currently, we recreate worker service for each challenge manually by deleting the worker and starting it again using django admin individually.

We need to add support for recreating the fargate service (i.e. delete service and create) for all active challenges from django admin.

Deliverables:

1. Show a option on Django admin for recreating fargate services for all active challenges
2. Add Django post hooks to recreate services for active challenges

@Ram81 Hi ! I have a fair understanding of Django and the codebase . Is this issue open for contribution ? 

Hi @Ram81  Can I work on this issue.



Hi @Ram81 please check. This is the expected behavior?. If not then please can I get some help?

**Steps to Follow**
1. Select "Ongoing" from "By Challenges" filter for all active challenges to get selected.
2. Select all the Challenges.
3. Go to Action and select "Delete and Start all selected challenge workers". 

![workersGIF](https://user-images.githubusercontent.com/60038994/151308596-4ffa0705-110b-4d1f-a02f-863e2859c3ba.gif)





Hi @haldendmello yes, that is correct.

@Ram81 I have send pull request.
https://github.com/Cloud-CV/EvalAI/pull/3685


@Ram81 I don't think we need this one. Can we please close this?
Repository: Cloud-CV/EvalAI
Issue: [Backend] Add support to recreate workers on EC2 instance for active challenges
Root cause: Unimplemented Feature Gap

### Description

To push latest changes to active workers on EC2 instance we need to recreate the service so that it uses the latest docker image of submission worker. Currently, we recreate worker service for each challenge manually by deleting the worker and starting it again using [worker deployment script](https://github.com/Cloud-CV/EvalAI/blob/master/scripts/deployment/deploy.sh#L210) individually.

We need to add support for recreating the fargate service (i.e. delete service and create) for all active challenges in the deployment script.

Deliverables:

1. Add API calls to fetch all active challenges
2. Use the fetched challenges to recreate the workers on EC2 instance

@Ram81 I would like to work on this issue, can you please guide me to get started to contributing to CloudCV. Thanks!

@Ram81 Hi ! I am interested in working this issue . Is this still open ?
Repository: Cloud-CV/EvalAI
Issue: RuntimeError: Click will abort further execution because Python 3 was configured to use ASCII as encoding for the environment. 
Root cause: Environment Incompatibility


Traceback (most recent call last):
  File "/usr/local/bin/evalai", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 676, in main
    _verify_python3_env()
  File "/usr/local/lib/python3.6/site-packages/click/_unicodefun.py", line 118, in _verify_python3_env
    'for mitigation steps.' + extra)
RuntimeError: Click will abort further execution because Python 3 was configured to use ASCII as encoding for the environment.  Consult http://click.pocoo.org/python3/for mitigation steps.

Closing this issue as it doesn;t have sufficient details