zhiminy/EvalEng huggingface dataset:
Repository: Cloud-CV/EvalAI
Issue: On Firefox browser, counter for host teams is not accurate
GitHub labels: bug

GSOC-2019

frontend

dependent

good first issue

inactivity

Steps to reproduce, as tested on evalai.cloudcv.org (April 6, 2018)

1. Create login, go to dashboard
2. Head to "create challenge", create a team
3. Check the dashboard again, but no team count is present
4. NOTE: This is not an issue for participant teams.

![image](https://user-images.githubusercontent.com/12480297/38429365-be7dc08e-3983-11e8-9f11-16895b6c0706.png)

![image](https://user-images.githubusercontent.com/12480297/38429464-130a87d6-3984-11e8-8eac-d101402a6464.png)



Hi @ezavesky I tested it and it seems to be working fine for me. Can you please try it again. Please feel free to ping us if you face issues in it. 

@ezavesky can you please also check the network tab on the browser console? You will see red colored API endpoint if there any error exists.

Very odd. It seems to work in Chrome (bottom) but not Firefox (top).  Please close or deprioritize as needed.

![image](https://user-images.githubusercontent.com/12480297/38436295-dcd0363c-3999-11e8-98ad-294af941e1b0.png)


@ezavesky thanks for pointing out. It seems that the behavior that you mentioned is correct. After doing some debugging, it looks like Firefox is sending an `OPTIONS` request type instead of `GET` to fetch the challenge hosts teams hence this is the error. 

@spyshiv can you please look into this issue?  

@deshraj I'm on it. 

@RishabhJain2018 , is the issue still open? If yes, I would like to take it up.

@supra08 Would you like to still work on it?

![error](https://user-images.githubusercontent.com/19167324/52170401-71772e80-276f-11e9-8ba0-ce1d548819d8.png)
@RishabhJain2018 Its working fine on my end.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


This issue has been automatically closed as there is no further activity. Thank you for your contributions.
Repository: Cloud-CV/EvalAI
Issue: KeyError: 'image'
Root cause: Validation Gap

https://sentry.io/cloudcv/evalai/issues/519375153/

```
KeyError: 'image'
(5 additional frame(s) were not displayed)
...
  File "rest_framework/views.py", line 477, in dispatch
    response = self.handle_exception(exc)
  File "rest_framework/views.py", line 437, in handle_exception
    self.raise_uncaught_exception(exc)
  File "rest_framework/views.py", line 474, in dispatch
    response = handler(request, *args, **kwargs)
  File "rest_framework/decorators.py", line 52, in handler
    return func(*args, **kwargs)
  File "challenges/views.py", line 565, in create_challenge_using_zip_file
    image = yaml_file_data['image']

KeyError: 'image'
```
Repository: Cloud-CV/EvalAI
Issue: Add tests for challenge creation using zip file.
GitHub labels: enhancement

easy_to_fix

backend

## Description
The unit tests for the function `create_challenge_using_zip_file` are missing in `apps/challenges/views.py`.

## Deliverables

- [ ] Add the required tests to increase the coverage.

i'd like to work on this

@lenniezelk Sure go ahead!

i started writing a test but i ran into a problem. `zip_challenge.yaml` inside the sample zip `examples/example1/test_zip_file.zip` refers to `evaluation_script: data.py` which is missing in zip file. I found a `evaluation_script.zip` in the sample zip and renamed the `main.py` to `data.py` and created a new sample zip archive. On running test i get error: `{u'error': [u'The submitted data was not a file. Check the encoding type on the form.']}`. I am a bit stuck here. Any assistance is highly appreciated. @RishabhJain2018 

Please read the instructions [here](https://evalai.readthedocs.io/en/latest/challenge_creation.html#challenge-creation-using-zip-configuration) to create a sample zip archive, you need give the the path of evaluation script zip folder relative to the `yaml` file. So, please give the path as `evaluation_script.zip` and try running the tests again. Feel free to ask if you have any further questions.

added a pr for this @RishabhJain2018
Repository: Cloud-CV/EvalAI
Issue: Change the privacy level of files on AWS S3
GitHub labels: enhancement

GSOC-2019

medium-difficulty

backend

critical

priority-high

## Current Scenario

Currently, anyone can access the files using the S3 links. This creates a potential threat of getting the test annotations being exposed to the public if someone gets the URL. We want to protect the files to be only accessible from the particular hostnames by setting the policy in S3 and modify the Django views to give permission to the particular set of users.

## Deliverables

- [ ] Add policy on S3 to restrict the access from particular hostnames (see this https://blog.botreetechnologies.com/aws-s3-file-upload-access-control-using-boto3-with-django-web-framework-11114de2d928 for more details)
- [ ] Update the Django view using which any file served as a media has to pass through a decorator which will check whether to give access of this file to the user or not (see this https://stackoverflow.com/questions/28364935/amazon-s3-and-django-allow-only-the-users-from-my-website-and-not-the-anonymou?answertab=active#tab-top to know how to implement this) 

Sir can i take up this issue! @deshraj 

@deshraj I assume this is still open. I will work on this issue.

Sure @yongzx You can work on this.

@deshraj @RishabhJain2018 For the first task, are the files referring to the submitted files in `My Submission` and `View All Submission`? And we are only restricting the access to the users who have logged in to the dashboard right?


@yongzx Are you working on this? If not, I would like to do this one. @RishabhJain2018 @deshraj

You can work on this one.


Is work on this issue done?
Or can I take up this issue? @RishabhJain2018 @deshraj 

I would like to take the issue up if it's not done yet.

@deshraj  as the issue is still open, can I work on it?

Is this issue free to work on? I'd like to work on it if so! @RishabhJain2018
Repository: Cloud-CV/EvalAI
Issue: Configuring test settings for py.test.
GitHub labels: enhancement

backend

# Proposed changes.

* Adding `settings/test.sample.py`
* `cp settings/dev.sample.py settings/dev.py` and adding `dev.py` to .gitignore so that we don't have to stage the changes everytime we are running tests and commiting.
* Deciding between `py.test` and `pytest`.

Closing this issue, since this is no needed now!
Repository: Cloud-CV/EvalAI
Issue: error: [Errno 11] Resource temporarily unavailable
Root cause: Resource Mishandling

https://sentry.io/cloudcv/evalai/issues/531162663/

```
error: [Errno 11] Resource temporarily unavailable
  File "gunicorn/workers/sync.py", line 68, in run_for_one
    self.accept(listener)
  File "gunicorn/workers/sync.py", line 27, in accept
    client, addr = listener.accept()
  File "python2.7/socket.py", line 206, in accept
    sock, addr = self._sock.accept()

No evaluation script is present in the zip file. Please add it and then try again!
```

Since this is resolved. Hence, I am closing this one.
Repository: Cloud-CV/EvalAI
Issue: Error: NodeNotFoundError when running python manage.py migrate command using python 3.4 or above
Root cause: Validation Gap

`raceback` (most recent call last):
File "manage.py", line 22, in <module>
execute_from_command_line(sys.argv)
File "/home/sachin/evalai/venv/lib/python3.5/site-packages/django/core/management/init.py", line 367, in execute_from_command_line
utility.execute()
File "/home/sachin/evalai/venv/lib/python3.5/site-packages/django/core/management/init.py", line 359, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/sachin/evalai/venv/lib/python3.5/site-packages/django/core/management/base.py", line 294, in run_from_argv
self.execute(args, **cmd_options)
File "/home/sachin/evalai/venv/lib/python3.5/site-packages/django/core/management/base.py", line 345, in execute
output = self.handle(args, **options)
File "/home/sachin/evalai/venv/lib/python3.5/site-packages/django/core/management/commands/migrate.py", line 83, in handle
executor = MigrationExecutor(connection, self.migration_progress_callback)
File "/home/sachin/evalai/venv/lib/python3.5/site-packages/django/db/migrations/executor.py", line 20, in init
self.loader = MigrationLoader(self.connection)
File "/home/sachin/evalai/venv/lib/python3.5/site-packages/django/db/migrations/loader.py", line 52, in init
self.build_graph()
File "/home/sachin/evalai/venv/lib/python3.5/site-packages/django/db/migrations/loader.py", line 268, in build_graph
raise exc
File "/home/sachin/evalai/venv/lib/python3.5/site-packages/django/db/migrations/loader.py", line 238, in build_graph
self.graph.validate_consistency()
File "/home/sachin/evalai/venv/lib/python3.5/site-packages/django/db/migrations/graph.py", line 261, in validate_consistency
[n.raise_error() for n in self.node_map.values() if isinstance(n, DummyNode)]
File "/home/sachin/evalai/venv/lib/python3.5/site-packages/django/db/migrations/graph.py", line 261, in <listcomp>
[n.raise_error() for n in self.node_map.values() if isinstance(n, DummyNode)]
File "/home/sachin/evalai/venv/lib/python3.5/site-packages/django/db/migrations/graph.py", line 104, in raise_error
raise NodeNotFoundError(self.error_message, self.key, origin=self.origin)
django.db.migrations.exceptions.NodeNotFoundError: Migration challenges.0019_added_leaderboard_data_table dependencies reference nonexistent parent node ('jobs', '0004_submission_output')'
Repository: Cloud-CV/EvalAI
Issue: Add new logo in home page
GitHub labels: enhancement

easy_to_fix

good first issue

### Deliverables
Add Draper logo in the home page
Logo: http://www.draper.com/sites/all/themes/draper/img/Draper-Primary-Logo-RGB.png

I would like to work on this.

Sure @gautamjajoo. Go ahead.
Repository: Cloud-CV/EvalAI
Issue: We should not expose the PK for any phase or challenge_phase_split in the url.
GitHub labels: enhancement

medium-difficulty

backend

## Current Scenario
The pk of the challenge phase and challenge phase split appears in the url.

## Deliverables
### Step 1: 
write a script to modify the value of id for different challenge phases of the same challenge. For VQA Challenge with pk 80, it has 3 phases with id = pk = (100, 101, 102) and 4 challenge phase splits with id = pk = (41,42,43,44)

What we might want to do here is that:
1. add a new field with name `index` with default as None
2. Write script to modify the field for each challenge phase based on the previous script that we shared
3. Modify the challenge creation API to add support for populating this field

After step 1,
  `challenge phase pk = (100,101,102) and challenge phase id = (1,2,3)
  challenge phase split pk = (41,42,43,44) and challenge phase split id = (1,2,3,4)`

### Step 2:

  Modify all the URLs that work like this. See the below example:

  2.1 Old URL: `/api/80/submit/phase/<pk>`
  2.2 New URL: `/api/80/submit/phase/<index>`
  2.3 Modify the API definition to take the index for filtering instead of pk for challenge phase.


Closing this for now since we converged upon other design decisions.
Repository: Cloud-CV/EvalAI
Issue: Add support for setting submission limit  per month per team for a challenge
Root cause: Unimplemented Feature Gap

## Current Status

Currently, we only provide the capability to set the daily submission limit and total submission limit throughout the challenge. For evaluation of tasks envisioning evaluation of models on EvalAI as long-running test server and not challenge with finite time, we would like to add the feature to limit the number of submissions to a challenge per month per team which eventually limits the number of submissions to test-set per publication.

Thanks @jcjohnson for the feature request.

## Deliverables

- [ ] Add new field `max_submission_per_month` for each challenge phase
- [ ] Set default value to `max_submission_in_total`
- [ ] Add check for monthly submission limit 
- [ ] Add tests
- [ ] Update challenge creation API
- [ ] Modify API to get the remaining submissions and add support to show remaining monthly submissions
- [ ] Update yaml documentation
- [ ] Update EvalAI-Starters to add support for monthly max submission limit

I am working on this issue

Sure go ahead!
Repository: Cloud-CV/EvalAI
Issue: Update submission worker to create challenge specific queues.
Root cause: Unimplemented Feature Gap

### Deliverables:

The submission worker should be updated so that challenge specific queues can be created instead of one single queue for all the challenges.

**Challenge Queue name:** submission_challenge_<challenge_pk>

Make sure to use the exchange type as **direct**.(refer [here](https://www.rabbitmq.com/tutorials/tutorial-four-python.html))
Repository: Cloud-CV/EvalAI
Issue: gulp-deprecation error while running docker-compose command
Root cause: External Dependency Breakage

While running the `docker-compose up --build` command to install and compose the docker container to set up the project the command is exiting with an error. I'm attaching the screenshots of the errors that were occurring even after so many tries.

![docker_gulp](https://user-images.githubusercontent.com/22341493/40987613-9c4663e2-6906-11e8-91c6-49e07cf77e46.png)

![docker_gulp_2](https://user-images.githubusercontent.com/22341493/40987625-a6f54e20-6906-11e8-9dca-b757cc02d94d.png)

![screenshot from 2018-06-05 21-18-31](https://user-images.githubusercontent.com/22341493/40987639-b1bcf4b6-6906-11e8-9180-b7c9b00253e1.png)

Please resolve the above issue as it might be a hurdle for new contributors to set up the project.

 Currently I am working on it. Will update the gulp tasks soon. 

Ah. Ohkay. And the problem persists even while following trivial installation procedure(non-docker approach). Thanks :)

@HargovindArora the gulp deprecation is a warning and that is not affecting the project in any way as of now. We are working on it as @spyshiv mentioned. Your error seems like while building docker container, your machine is not able to download the package due to some internet issues. Can you please try removing the containers and do it again? It might work then or try changing the internet connection that you are using. 

Else try restarting the docker daemon. That usually works for me when docker is having a hard time connecting to the internet.

@deshraj @varunagrawal I'm trying your given solutions and will let you know if it worked. But "gulp" warning surely did trouble me a lot while following the non-docker installation procedure.
Anyways @spyshiv fixed it! :-)


You should check out #1697 for the gulp problems. @spyshiv has already fixed it and the pull request is in the review. 

Please do let us know once your error is resolved. 

@deshraj As @varunagrawal stated there was an error with docker daemon. I tried restarting it but it didn't help. Therefore I completely removed and reinstalled docker and it finally worked.
But the gulp error isn't letting me to setup using non-docker approach. I'll try it again once #1697 is merged.
Thanks :)

I think this is resolved. Hence, closing this one.
@HargovindArora Feel free to open up if you find some issues.
Repository: Cloud-CV/EvalAI
Issue: Unnecessary command to execute, listed in README for docker installation instructions 
Root cause: Documentation Deficiency

In README for docker installation, there is a command listed `cp settings/dev.sample.py settings/dev.py` but on inspection, it is found that there is no such file "settings/dev.sample.py" and it's throwing the following error:

![cp_command](https://user-images.githubusercontent.com/22341493/41037075-c65499e8-69af-11e8-98bf-8e5d56cfc6b9.png)

If this issue is valid can I work on this?
Thanks

This seems to be an inconsistency in the README. We'll have to update that since when I wrote the docker instructions, we definitely had a sample dev settings file.
The last commit to have the sample dev settings is a1a93c192898fb1190d8dc11945e4fe70928967d so my recommendation would be to update the README and submit a PR :slightly_smiling_face: 

@varunagrawal Sure. I'm on it! :-)

Is this issue still open? 


I believe @HargovindArora has already submitted a PR. We'll close this issue once his PR is approved and merged.
Repository: Cloud-CV/EvalAI
Issue: Error in creating challenge when host team contains more than 1 members
Root cause: Validation Gap

**Issue**
This line [here](https://github.com/Cloud-CV/EvalAI/blob/5be29063e0f88549eedacd9bcbe42bae2336557e/apps/challenges/views.py#L900) throws an  exception when creating a challenge with more than one host team members.

**Steps to Reproduce:**
1. Create a challenge host team
2. Add two members in the team (Total team size:3)
3. Now try creating a challenge by uploading the zip file.
4. Challenge should not get created.

**Fix:**
Use `filter` instead of `get` and loop over all the returned email id's.


@RishabhJain2018 I would like to work on this! :)


@HargovindArora Sure ! But please keep in mind that we have to fix it asap.  

@RishabhJain2018 Okay. I submit it to you ASAP

Hey @RishabhJain2018 please brief me that how can I reproduce this error! 

I have updated the issue description. Please take a look again.

@RishabhJain2018 It worked for me. The challenge got created


Can you please upload the screenshot of your host team showing all its members.

![screenshot from 2018-06-14 14-19-41](https://user-images.githubusercontent.com/22341493/41401664-0a40e45c-6fde-11e8-838a-e598ce0d6143.png)

@RishabhJain2018 Have a look!

@HargovindArora Yes, you are right. It is working. Hence, I am closing this.
Repository: Cloud-CV/EvalAI
Issue: Show proper error message when Yaml file config has errors
Root cause: Validation Gap

### Issue:
Proper error isn't displayed when yaml config for challenge creation isn't correct.

### Culprit Url:
`/api/challenges/challenge/challenge_host_team/{challenge_host_team_pk}/zip_upload/`

### Error:
ScannerError

### Solution
Log the error & display message as:
`Error in creating challenge. Please check the yaml configuration!`
For reference: https://www.programcreek.com/python/example/100007/yaml.scanner.ScannerError

Is this issue open?

No @sachinmukherjee 
@calenrobinette is working on it.

Ok
Repository: Cloud-CV/EvalAI
Issue: Add support for editing the challenge start and end date
GitHub labels: enhancement

GSOC-2019

medium-difficulty

frontend

new-feature

priority-high

**Current Scenario:**
Currently, a challenge host can't edit the challenge start date and end date. This can only be done by EvalAI admin.

**Deliverables:**
Add functionality to edit the challenge start date and end date on frontend.
Add a check to make sure that challenge end date is always after challenge start date.

Use this(https://indrimuska.github.io/angular-moment-picker/) package for adding date-time support.

I am interested in solving this issue how can I go about the same?

Can you help me out?
I am really interested to contribute to CloudCV.

Hi, @pujanm I am sorry that I missed this message. Thanks for your interest in CloudCV.

For the above issue, you can check how we have implemented the challenge details edit section. We will be following the same approach as that. Please let me know if you have further doubts. I'll be happy to help.

I was trying to edit the challenge overview it is giving this `/api/challenges/challenge_host_team/undefined/challenge/3` there is an undefined in place of `challengeHostId`. Is there any reason or I might have done some mistake while making this challenge?
@RishabhJain2018 

@RishabhJain2018 is this issue taken ?

I think @pujanm is still working on this issue.
>/api/challenges/challenge_host_team/undefined/challenge/3 there is an undefined in place of challengeHostId

I am not sure why it is showing this behaviour to you. Can you please try to debug it?

Yes.

@RishabhJain2018 I would like to work on it.

Sure!

Since @Ayukha isn't active these days, would you like to work on this @nikhilmudholkar?

@RishabhJain2018  I am working on it. And I'll open a PR soon.

Yaa i would like to work on it but it seems that someone is already working on this issue

@Ayukha Can you please let us know ETA on this? This is now becoming a priority thing for us

@RishabhJain2018 If @Ayukha isn't active these days, then I can work on this issue.

@RishabhJain2018  I'll try to fix it by tonight. Actually, I'm struggling in using the package suggested.
Repository: Cloud-CV/EvalAI
Issue: Display the created time in the zip configuration admin
GitHub labels: enhancement

easy_to_fix

backend

good first issue

Add `created_at` in the `list_display` tuple.
Reference link: https://github.com/Cloud-CV/EvalAI/blob/master/apps/challenges/admin.py#L61

I want to work on this issue.

Sure go ahead @sachinmukherjee .
Moreover, are you able to set up the dev environment?

Yes, I have successfully setup the development environment.
Repository: Cloud-CV/EvalAI
Issue: Prevent users from participating in the past challenges using the API.
Root cause: Validation Gap

# Current Behaviour

You can participate in any past challenge using the end-point `/api/challenges/challenge/{}/participant_team/{}` which is not the desired behavior.

# Expected Behaviour

We should place a check in the  view to see if it's a request to participate in a past challenge and then block it.
Repository: Cloud-CV/EvalAI
Issue: Add a check to stop filtering the disabled challenges under Hosted Challenges
Root cause: Validation Gap

**Current Scenario:**

Currently all the challenges which are hosted by the host are filtered under `Hosted Challenges` section.
Refer here: https://github.com/Cloud-CV/EvalAI/blob/master/apps/challenges/views.py#L84

**Deliverables:**

We want to modify the behaviour by not filtering the deleted challenges by the challenge host.

**How to Fix:**

- [ ] Modify the query to `challenge = Challenge.objects.filter(creator=challenge_host_team, is_disabled=False)`.
- [ ] Add the corresponding test case.
Repository: Cloud-CV/EvalAI
Issue: Change team name dialog box should disable submit button when no team name is entered
Root cause: Validation Gap


- The Expected Behavior
The change team name dialog box submit button should be enabled when a team name is entered.
![team name](https://user-images.githubusercontent.com/19318218/41724258-cb857914-758a-11e8-8e2b-ccfa9b4d6ecd.png)

- The Observed Behaviour
 At present even though no team name is entered it successfully updates the team name.
![screenshot from 2018-06-21 06-20-40](https://user-images.githubusercontent.com/19318218/41724285-dadf207c-758a-11e8-909f-186ff488adb3.png)


@sachinmukherjee Thanks for catching the bug.
 Would you like to work on it ?

Yes I would like to work on this.

Sure go ahead!
Repository: Cloud-CV/EvalAI
Issue: Futile error message pops up while submission
Root cause: Validation Gap


<img width="338" alt="screen shot 2018-06-29 at 1 11 14 am" src="https://user-images.githubusercontent.com/12206047/42070962-64af4354-7b76-11e8-8e80-dd681e469ec8.png">

This keeps popping up between consecutive submissions. Submissions are submitted and results are calculated successfully, so do not know why this pops up.
Repository: Cloud-CV/EvalAI
Issue: Use Yarn instead of Gulp
Root cause: Unimplemented Feature Gap

It's time to move away from Gulp. We would like to use Yarn as a package manager instead of gulp.
See https://bower.io/blog/2017/how-to-migrate-away-from-bower/ for more details. 

I would like to solve this issue!!!

@rakesh-1601: Please take one issue at a time. You can take it once you finish the other issue that you are working on right now. 

Can I work on this issue?

This issue is invalid since we are planning to move with https://github.com/Cloud-CV/EvalAI-ngx