zhiminy/EvalEng huggingface dataset:
Repository: Cloud-CV/EvalAI
Issue: Add constraints for setting challenge phase start and end date
Root cause: Validation Gap

## Current Observation

Challenge hosts can set the challenge phase start and end date to anything irrespective of the challenge start and end date.

## Deliverables

Add the following checks in proper places in the codebase (i.e some/all of these things would go in the ChallengePhase `save()` method)
- [ ] Challenge phase start datetime >= Challenge start datetime
- [ ] Challenge phase start datetime < Challenge end datetime
- [ ] Challenge phase end datetime > Challenge start datetime
- [ ] Challenge phase end datetime <= Challenge end datetime
- [ ] Challenge phase start datetime < Challenge phase end datetime 
- [ ] Add check if Challenge start datetime < Challenge end datetime

## Goal
- [ ] Can you open a new PR using this PR (https://github.com/Cloud-CV/EvalAI/pull/2020) and verify that it behaves as expected in all relevant scenarios before we proceed with merging?

@deshraj Can I take this?

sure @isht3 

@isht3 Are you working on it? If not, I can pick it up. 🌯 

@deshraj I am a beginner and would like to take it up as my first issue. Can you please guide me through this? 

Hi! @mehak16163 How familiar are you with Django? 😄 

I think I would be able to find the relevant files and make some code changes to it. 
But if required to build from the start I wouldn't be able to do that. 

@mehak16163 https://github.com/Cloud-CV/EvalAI/blob/ca4b12a0619d5ccf0fed8b5d04cbe07cf6b3c7a7/apps/challenges/models.py#L107 This is the model mentioned above. I suggest setting up and running the project and submitting a few challenges so you could be familiar with how the project workflow is. Let me know if you need any more help.

@isht3 can you please take lead on this issue and solve it? Please feel free to assign it to yourself. 

@deshraj @RishabhJain2018 @isht3 Hey! The following line, returns the end date of challenge or end date of a phase? https://github.com/Cloud-CV/EvalAI/blob/2a8670a4d1e037329c38ebc7d1b8a49c4ebb2188/apps/challenges/models.py#L148

@gautamjajoo It returns the end date of the Challenge Phase. Thanks for noticing, could you change it?

@isht3 Yeah! I will change it. 

@RishabhJain2018  Can I take this up?

Sure!

Is this open or the PR needs to be merged


Hi @GauravJain98, the PR needs to be merged for this. Please take up any other issue.
Repository: Cloud-CV/EvalAI
Issue: Improve dataset_split selection logic in submission worker
Root cause: Validation Gap

After integrating #1560, there can be multiple dataset_splits with the same name and codename and the code at  

https://github.com/Cloud-CV/EvalAI/blob/master/scripts/workers/submission_worker.py#L335

will fail as it will return `MultipleObjectsReturned` exception in the case where there are two or more dataset splits which have the same values of codename and split name.
Repository: Cloud-CV/EvalAI
Issue: Docker build: cannot access frontend content
Root cause: Configuration Error

I'm trying to reproduce the [Docker build instructions](https://github.com/Cloud-CV/EvalAI#using-virtual-environment) in an Ubuntu 16.04 system. The containers are being built correctly, I can connect to the backend (port 8000) and see the "Hello world", but when I connect to the frontend (port 8888) I always get an 404 error, the container is indeed active and responding but I can't access the web content, is there some configuration I'm missing?

```
git clone https://github.com/Cloud-CV/EvalAI.git evalai && cd evalai
cp settings/dev.sample.py settings/dev.py
docker-compose -f docker-compose.dev.yml up -d --build
```


@schomatis can you please paste the screenshot of the error that you are facing? 

Hi @deshraj, thanks for the fast reply, it's not an error in itself, meaning that the `docker-compose` successfully builds and runs the containers and I can access them with `wget`, but when trying to access the frontend through port 8888 I always get a 404 response (for any url), when I would expect to find some of the content from the [frontend views](https://github.com/Cloud-CV/EvalAI/tree/master/frontend/src/views/web), but I may be misunderstanding where the web content should be accessed.

```
$ wget localhost:8888
--2018-03-15 15:25:26--  http://localhost:8888/
Resolving localhost (localhost)... 127.0.0.1
Connecting to localhost (localhost)|127.0.0.1|:8888... connected.
HTTP request sent, awaiting response... 404 Not Found
2018-03-15 15:25:26 ERROR 404: Not Found.
```

@schomatis did you try opening it in your browser? Please let me know if the issue persists on the browser. 

@deshraj many people are facing this problem as per communication in gitter channel. 

@deshraj Yes, the issue persists on my browser. Both Firefox and Chrome failed to retrieve the web page.

I have a similar problem. I'm getting this on Chrome (this is the source, of course):

```
<!DOCTYPE html>
  | <html lang="en">
  | <head>
  | <meta charset="utf-8">
  | <title>Error</title>
  | </head>
  | <body>
  | <pre>Cannot GET /</pre>
  | </body>
  | </html>
 


```

Thanks for reporting. I will test this on my machine and see if the problem persists. 

In the docker file for nodejs, the command line for gulp is "gulp connect", whereas in the instructions for using virtualenv the command line is "gulp dev:runserver". I tried changing the gulp command line in the docker file to dev:runserver and it seems to be working. Could that be the problem?

@jheguia Yeah it seems like that is the cause of this issue. I am trying to reproduce the error.
Repository: Cloud-CV/EvalAI
Issue: Grammatical errors in E-mail Address confirmation e-mail

Hi! I'm new to programming and am still relying on manual testing.

I signed up for EvalAI and here is the e-mail I received:
![screen shot 2018-03-16 at 2 22 07 pm](https://user-images.githubusercontent.com/27714199/37545406-efe9f840-2925-11e8-8f88-01d8edc18e5e.png)

I'd be happy to draft up a new confirmation message that's less impersonal and reads better to native speakers.

@tashachin: That would be great. We would love to see a better confirmation email. Feel free to share once you have the new message. :) 

I'd like to fork the repo and update the code directly. In which file would I find the confirmation message? (I haven't worked with an email API before, so unfortunately it's not very intuitive for me to know where to look.)

You can find all the email related templates here: https://github.com/Cloud-CV/EvalAI/tree/master/apps/accounts/templates/account
Repository: Cloud-CV/EvalAI
Issue: Update Mapillary logo on main page
GitHub labels: enhancement

GSOC-2019

easy_to_fix

frontend

good first issue

The current Mapillary logo on the main page has to be updated with the one given below:

![mapillary](https://user-images.githubusercontent.com/2945708/37746707-e30b6872-2d51-11e8-8abc-6db410116809.png)

The current logo seems to be squished. Please paste the screenshot of the updated home page while submitting a pull request. 

Edit: instead of using Mapillary's logo, we would like to update it with Mapillary Research's logo shown below.

![mapillary_research](https://user-images.githubusercontent.com/2945708/37758078-57e90b12-2d85-11e8-9629-0af484194435.png)


@deshraj Can I pick this issue up?

@geekayush Sure go ahead!

@geekayush Can I work on this issue, if you are not working??

@gautamjajoo It seems @geekayush is inactive these days, so you can take this up :). Feel free to open a PR once it's done.
Repository: Cloud-CV/EvalAI
Issue: Add Random Number Generator Challenge zip file and documentation
Root cause: Documentation Deficiency

I have created a new sample challenge named `Lucky Challenge 2018` which we will use as a base challenge so that other challenge hosts can take inspiration from this. 

## What this challenge is about?

This challenge is a simple challenge which generates a random number between 1 and 100 for each entry and whosoever is lucky gets a good random number and reaches the top of the leaderboard. More details can be found in the evaluation script present in the challenge configuration file : [challenge_config.zip](https://github.com/Cloud-CV/EvalAI/files/1836558/challenge_config.zip)

## Deliverables 

- [x] Upload the challenge configuration zip file in the `examples/` directory (challenge configuration file: 
[challenge_config.zip](https://github.com/Cloud-CV/EvalAI/files/1836558/challenge_config.zip))
- [x] Add the challenge to the seed script so that this challenge gets created once we run the seed command `python manage.py seed`
- [x] Add documentation for walking through creating this Challenge (if you need help with writing documentation for this, feel free to comment on this issue thread)

Let us know if you have suggestions for this task. 


Can I start working on this?


@psushaanth23 sure, go ahead. 

In trying to reuse the attached file, the [example in the repo](https://github.com/Cloud-CV/EvalAI/blob/master/examples/example1/test_zip_file.zip), or even a simplified [VQA example](https://github.com/Cloud-CV/EvalAI-Examples/tree/master/vqa_challenge)  on the the main website (evalai.cloudcv.org), I get the error...
```
There are more than one YAML files in zip folder!
```
Are there any alternate challenge examples?

@ezavesky it looks this happened with you:

1. You unzipped the file
2. Made the changes to the challenge configuration files
3. When you tried to zip with your MacOSX compress tool it added the `__MacOSX` directory which adds up a bunch of extra files (and yml files) which you have to avoid. See this link for debugging: https://stackoverflow.com/questions/10924236/mac-zip-compress-without-macosx-folder

This will solve your issue. 

@deshraj Thank you for the quick note and patience. I confirm that the spurious directory was the issue above.

Great. Good to hear that it got resolved. 

Is this issue still open?


This issue is already fixed. Hence closing it.
Repository: Cloud-CV/EvalAI
Issue: Add file submission validation for different challenges
Root cause: Unimplemented Feature Gap

## Current Scenario

Currently, we allow the participants to make submissions in `json` and zip `format`. We would like to add support for challenge specific upload formats. 

## Deliverables

- [ ] Add a new ArrayField named `allowed_files` in `Challenges` model with default value set as `zip` and `json`
- [ ] Modify the challenge creation API using zip file to add support for allowed_file_extensions in the YAML config file. 
- [ ] Update the documentation for challenge creation using zip with respect to the above deliverable
- [ ] Add a file validator whenever a submission is made to check if the uploaded file type is supported by the challenge (see https://stackoverflow.com/questions/3648421/only-accept-a-certain-file-type-in-filefield-server-side/27081099 for more details) 

I would like to work on this, if that's ok.

Yes it's still open. You can work on it. :+1:

Hey, since there has been no activity for this issue in a while, I would like to take this issue. Please assign.

@party98 Please feel to work on this issue. Let us know if you have doubts. 

Thank you Deshraj. I’ll try to do some work on it this weekend. I’ve been busier than normal at work for the past month.
On Sep 11, 2018, 8:01 PM -0500, Deshraj Yadav <notifications@github.com>, wrote:
> @party98 Please feel to work on this issue. Let us know if you have doubts.
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub, or mute the thread.


is the issue still open?

Yeah this issue is still open. If @bryanfree66  is not working on it, please feel free to take it. 

@deshraj  @RishabhJain2018 Can I take this Up?

sure! you can work on this.

@deshraj  @RishabhJain2018 Can I work on this issue?

@vkartik97  I am working on this feature. Please take up another issue. 

@Ayukha Aren't you working on #1705 ? I also wanted to work on this or #1705 

@RishabhJain2018 @deshraj Can I work on this ?

@yashdusing 
As you have been assigned #1913, can I take this up?

@vkartik97 sure !

Thanks @yashdusing :)

@Ayukha, You have also been assigned #1705, so maybe I should work on this?

Can I work on this issue?

I think no one is working on this. Hence, you can work on this @namish800 :)

@RishabhJain2018 I am new to this project can you tell me how to proceed with it?


@deshraj @RishabhJain2018 Can you point me towards the file which validates the submissions.

@RishabhJain2018 @deshraj Can I also work on this issue?

@RishabhJain2018 @deshraj Done with all the deliverables, Can I make a PR for this issue?

@deshraj @RishabhJain2018 Please review my PR #2130 

I am a newbie I would like to work on it if it is still open

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


This issue has been automatically closed as there is no further activity. Thank you for your contributions.
Repository: Cloud-CV/EvalAI
Issue: Notify EvalAI Admin upon any contact message
GitHub labels: enhancement

backend

priority-high

The EvalAI admin should get a notification via email as well as a slack message whenever any user submits contact query. 

@RishabhJain2018 @deshraj Can I take up this issue?


@isht3 I think Rishabh has already solved this one. Feel free to  take up some other issue. 

@deshraj Okay!

@isht3 Can you please finish this issue? It seems to be very trivial for you. ;) 

@RishabhJain2018  Is this issue still open?

@RishabhJain2018 I could extend my PR to write about both of them. So, is this issue still open ?

This is already implemented, hence closing this.
Repository: Cloud-CV/EvalAI
Issue: Change the pagination of view all submissions in challenge host dashboard.
Root cause: Unimplemented Feature Gap

**Description:** The pagination under the `View All Submissions` tab for challenge host must have a default pagination of 100 entries. You can also make the pagination custom and letting the user decide out of choices 10,25,50,100.

**Deliverables:**

- [ ] The default pagination for `View All Submissions` under challenge host must have value 100.

- [ ]  A challenge host should be able to decide the pagination by himself using 10, 25, 50, 100 values. 

@live-wire I am closing this issue here. Can you please open a issue regarding this in EvalAI-ngx as we will need this feature.
Repository: Cloud-CV/EvalAI
Issue: Add feature to export limited columns of submission list
Root cause: Unimplemented Feature Gap

## Current Scenario

Currently when the challenge host export the submissions, all the fields are added to it by default. Having the functionality of allowing challenge host to choose which fields he would like to export would be nice. 

## Deliverables

- [x] Add the feature to select the fields on the frontend UI which challenge host wants to export
- [x] Update the export API in the backend to accept the fields to be exported in the form of query params.
Repository: Cloud-CV/EvalAI
Issue: Add embed leaderboard button 
Root cause: Unimplemented Feature Gap

## Current Scenario

Currently, most of the challenge organizers maintain their own leaderboard when the challenge ends. For example, VQA 2017 challenge (http://visualqa.org/challenge_2017.html) manages their own leaderboard manually. We can make it better by having an embed leaderboard button on the leaderboard page. 

## Deliverables

- [ ] Add a button `Embed leaderboard` on the leaderboard of a challenge
- [ ] Make sure that the leaderboard is only shown when embedded. Navigation bar and other things are skipped when shown on a different webpage.


Hi can I work on this.

Sure! You can work on this.

@live-wire I am closing this issue here. Can you please open an issue for the same in EvalAI-ngx as we will need this functionality there.
Repository: Cloud-CV/EvalAI
Issue: Allow and block particular email domains for a challenge
Root cause: Unimplemented Feature Gap

Currently, anyone can participate in a challenge. It would be great to have a functionality to allow/block only particular email domains to participate in a challenge as specified by the challenge host. 

## Deliverables

- [x] Add a field to Challenge model to store the `allowed_email_domains` (default=None, blank=True)
- [x] Add a field to Challenge model to store the `blocked_emails_domains` (default=None, blank=True)
- [ ] Update the challenge creation API to add support to incorporate `allowed_email_domains` and `blocked_email_domains`
- [ ] Update the Challenge creation using zip file documentation to add support for `allowed_email_domains`   and `blocked_email_domains`
- [ ] Update the challenge participation API to only allow `allowed_email_domains` and block `blocked_email_domains`

@deshraj @RishabhJain2018 Can I work on this?

Yeah sure. Assigning this to you. 

@deshraj for the `allowed_email_domains` and the other field I should be using `ArrayField`, yes?

Yes, you are right. 

@deshraj The last part of this issue is still remaining. I'll do it ASAP.

You mean the last 3 parts, right? 

Please add all of these remaining things in a single Pull request.
Repository: Cloud-CV/EvalAI
Issue: Limit number of concurrent submissions for a participant
Root cause: Unimplemented Feature Gap

## Current Scenario

Currently, a participant can make any number of submissions until he reaches the limit of doing max submissions per day. We would like to add another constraint where a participant can do at max 3  submissions and if he wants to do more then he has to wait until the status of the previously done submissions gets changed to `failed` or `canceled` or `finished`. This will help us to make sure that the participant cannot abuse the system by just making an unlimited number of submissions if the limit of doing max submissions per day is quite large. 

## Deliverables

- [ ] Add a new field `max_concurrent_submissions_allowed` integer field with default value set as 3.
- [ ] Add a constraint that 

```
if max_submissions_per_day < max_concurrent_submissions_allowed:
    set max_concurrent_submissions_allowed = max_submissions_per_day
```
Note: This would require to override the `save()` method I guess.
- [ ] Update the submission API to check:

```
if (number_of_submissions_with_running_status + number_of_submissions_with_submitted_status + number_of_submissions_with_submitting_status) == max_concurrent_submissions_allowed:
    # Don't allow the participant to do another submission
else:
    # Proceed and allow him to make submission
```

- [ ] When the participant is exceeding the limit then show a message like this:
`You have X submissions that are already being processed. Please wait for sometime to make another submission.`

@deshraj @RishabhJain2018 Can I start working on this?

@isht3 sure, go ahead. Assigning this to you.
Repository: Cloud-CV/EvalAI
Issue: Solve inconsistencies in the response handling for the front-end and back-end.
Root cause: Interface Contract Mismatch

# Current Scenario
Currently, there's a lot of inconsistencies in handling the various response messages on the front-end and on the back-end. 

* Error responses on controllers don't take in the response argument. `onError: function()` --> `onError: function(response)`
* Responses are not properly displayed. E.g `['Server error']` Which is vague and the user gets no feedback.
* Some responses are displayed in a list, which is a strike for the UI.

# Deliverables
* Try to make the success and the error responses from the back-end to be consistent with the Django and all-auth type responses.
* Make the success and error message display in the front-end to be clean and consistent all throughout the application.


@isht3 Can you please tell me in which files we need to make changes?

@sachinmukherjee https://github.com/Cloud-CV/EvalAI/blob/d86a3924ee8506b6c70223b53f7fe954f87cc8fe/frontend/src/js/controllers/challengeCtrl.js#L1030 In this line you can see that the error pretty much reveals nothing useful the user. Ideally, it should be changed to something useful regarding what's causing the error. Also, the responses sent from the backend aren't displayed and formatted properly. You could also look into that.

Hi I am newcomer and i want to work on this issue ! 

@nagpalm7 Yes, definitely. You can take this issue up!
Repository: Cloud-CV/EvalAI
Issue: KeyError: 'evaluation_script'
Root cause: Validation Gap

When you try to crate a challenge with a random zip configuration file, the application throws 500 which should be handled at the backend. 

```
KeyError: 'evaluation_script'
(5 additional frame(s) were not displayed)
...
  File "rest_framework/views.py", line 477, in dispatch
    response = self.handle_exception(exc)
  File "rest_framework/views.py", line 437, in handle_exception
    self.raise_uncaught_exception(exc)
  File "rest_framework/views.py", line 474, in dispatch
    response = handler(request, *args, **kwargs)
  File "rest_framework/decorators.py", line 52, in handler
    return func(*args, **kwargs)
  File "challenges/views.py", line 483, in create_challenge_using_zip_file
    evaluation_script = yaml_file_data['evaluation_script']

KeyError: 'evaluation_script'
```
Repository: Cloud-CV/EvalAI
Issue: Installation Error on UBUNTU 16.04
Root cause: Environment Incompatibility

When EvalAI is installated on UBUNTU 16.04, The webpage looks like below. 


![](https://files.gitter.im/Cloud-CV/EvalAI/Its0/Screenshot-from-2018-04-05-16-32-24.png)


![](https://files.gitter.im/Cloud-CV/EvalAI/MJq2/Screenshot-from-2018-04-05-16-32-15.png)


@gautamjajoo is there any error log in console/terminal ? If any, paste here.

And can you try frontend setup using updated `package.json` from PR #1639 and let me know the status

@spyshiv I have taken the latest pull and the frontend is the same.
Here's the terminal window


![](https://files.gitter.im/sridharjajoo/yp1W/Screenshot-from-2018-04-10-01-01-15.png)


![](https://files.gitter.im/sridharjajoo/yp1W/Screenshot-from-2018-04-10-01-01-27.png)


@gautamjajoo @ezavesky It seems like the error is fixed in #1639. After getting the latest changes, update your nodejs and npm to the latest version to get it to work. I am using node version 8.11.1 and npm 5.6.0 and it is working fine. Please let me know if this error is still not fixed. 

@deshraj @spyshiv I am using node version 8.11.1 and npm 5.6.0, but I am facing the following issue while executing  ```npm install```

![image](https://user-images.githubusercontent.com/24366008/38541551-0997fda0-3cbd-11e8-9f2b-c75719f031ef.png)


@spyshiv @deshraj I think so #1645 will solve this error.

Yes @gautamjajoo. Just 5 more minutes and then error will be resolved. :) 

Fixed in #1645