zhiminy/EvalEng huggingface dataset:
Repository: Cloud-CV/evalai-cli
Issue: Upcoming Command not working
Root cause: Documentation Deficiency

```sh
$ evalai challenges upcoming
```
Doesn't work . 
Instead the function that should be invoked is named `future`. [REF.](https://github.com/Cloud-CV/evalai-cli/blob/5a2f67e60ed1e3b146a1081059c0ee66768fce76/evalai/challenges.py#L99) 
```sh
$ evalai challenges future
```

@RishabhJain2018 @isht3 this is just a `typo` kind of issue. let me know if i can propose a quick fix. 😄 

@inishchith Yup, definitely a typo. Please feel free to start working on a fix! Thanks a lot!

@isht3 replacing `future` ( function name ) with upcoming with `upcoming` would work just fine, i feel. 
There are many files with `future` reference, so changing every occurrence isn't a good idea because even the  API `urls` has a `/future` reference .
What do you think?

@inishchith I feel that the keyword future is more compatible and consistent across all of EvalAI and EvalAI CLI. But I definitely agree that `upcoming` sounds more verbose when it comes to the command. It's definitely up for discussion. @RishabhJain2018 What do you think?

@RishabhJain2018 Any suggestions? 😄
Repository: Cloud-CV/evalai-cli
Issue: Multiple tokens
Root cause: Unimplemented Feature Gap

- [ ] Create a file which will have multiple tokens so that one can switch from host to participant creating some sort of a login kind of feature.
Label - enhancement

@RishabhJain2018   Can I take up this issue?

@RishabhJain2018  Can I take this Up?

@Ayukha  sure

@Ayukha Are you still working on this? If not, @darthgera123 @RishabhJain2018 Can I work on it?

Is anyone working on this issue?
I would like to submit a PR.
Repository: Cloud-CV/evalai-cli
Issue: Show error message while submitting when user doesn't participate in a challenge.
Root cause: Validation Gap

Error message like `you have not participated in the challenge` is not shown when a user tries to submit to a challenge without participating in it.

Hi Rishabh how can I help you ?
whats your issue with the code, code you specify?

@pydevsg Hey there! Have you set up the development environment of the CLI? If yes, try running the submission command without participating in a challenge and we'll get a very vague response, we need to change it to something a bit more verbose.

@RishabhJain2018 Is this the error message
``404 Client Error: Not Found for url: http://localhost:8000/api/jobs/submission/1 ``

@RishabhJain2018 @isht3 Can I take this issue up if it still needs work?

yeah, sure. but since you're working on another issue, it is open for others to work on.

Hi @RishabhJain2018 ,
Is the issue still open, I would Like to work on it??
Repository: Cloud-CV/evalai-cli
Issue: Add command for creating a challenge
Root cause: Unimplemented Feature Gap

The command should be like `evalai challenge create --file=<path_to_zip_file>`
Please feel free to ask if you have doubts.

Can I work on this issue?

sure! go ahead.

Hi, I am a beginner with CloudCV. I wanted to know how I could have proceeded with this issue in case I had it assigned to me? And if this issue still needs to be addressed, I'd like to contribute.  @RishabhJain2018 

Hi, @sinAshish Welcome to CloudCV! In order to start with the issue please take a look at the code in `challenges.py` for creating commands. You might also want to take a look at the EvalAI codebase for the API.

Hi! :) What's the status on this issue? I would like to work on this if it's free. @RishabhJain2018 @sinAshish 
I'm new to CloudCV

Hi @KhalidRmb 
Welcome to ClouCV!
Go though the Challenge Creation API in EvalAI for to further work on this issue :)
Repository: Cloud-CV/evalai-cli
Issue: Add command to display the already set token on CLI
Root cause: Unimplemented Feature Gap

The command should be `evalai gettoken`

The output should be the token which is set at `~/.evalai/token.json file`.

I'd like to work on this

Sure @nileshprasad137 You can work on this.

@nileshprasad137 Just a heads up, we'll need it urgently in a couple of hours or so :). Thanks for the quick response though. 

@RishabhJain2018 should i display the json {"token":"ksjhkja..."} or just the token string "ksjhkja"?

Thanks a lot for the quick PR. You should display just the token.
Repository: Cloud-CV/evalai-cli
Issue: Remove red background from all the error messages and make the text red.
GitHub labels: enhancement

help wanted

good first issue

priority-high

GSOC-2019

The current displayed error messages are: (Notice the red background and white font color)
<img width="586" alt="screenshot 2019-02-07 at 10 56 37 pm" src="https://user-images.githubusercontent.com/12206047/52458172-caf2ab00-2b2b-11e9-9c31-92f8345c39ac.png">

Instead, the same text should be displayed as
<img width="552" alt="screenshot 2019-02-07 at 10 57 18 pm" src="https://user-images.githubusercontent.com/12206047/52458187-e9f13d00-2b2b-11e9-9ee0-ddb247cb71b0.png">




@nileshprasad137 Would you like to fix this too? Meanwhile, I'll review that PR.

Sure , I'll fix this too!

Thanks!
Repository: Cloud-CV/evalai-cli
Issue: Make copyright year dynamic
GitHub labels: enhancement

help wanted

good first issue

GSOC-2019

### Observed Behaviour
The copyright year in the footer of docs landing page is outdated - 2018

### Expected Behaviour
Make the copyright year dynamic.

@RishabhJain2018 

I would like to work on this issue

@nikhilmudholkar Sure! You can work on this issue.

@gautamjajooI have submitted the PR. Please review it. Also, the Travis build tests are failing for some reason. All I have done was changing a single line in index.html. How can that lead to the failing of test cases? Can you please help me understand this?

@gautamjajoo @nikhilmudholkar did issue is resolved or not? if not let me know I m interested in solving this issue?


@prasadgujar Sure! Go ahead.
Repository: Cloud-CV/evalai-cli
Issue: Add copy to clipboard feature in command
Root cause: Unimplemented Feature Gap

### Deliverables
- [x] Add support for copying to clipboard for all the commands listed on the docs page. 
- [x] Add an icon for copying the text.
- [x] Show a successful message ` Command copied to clipboard`.

@RishabhJain2018 

@gautamjajoo 
I’d like to take this up

Sure! you can work on this.
Repository: Cloud-CV/evalai-cli
Issue: Add a command for logging in to evalai using cli
Root cause: Unimplemented Feature Gap

## Deliverables

- [ ] Add a command `evalai login <username>` which will prompt for the `password`.
- [ ] Once the password is entered, then the command will create a auth_token for the user by using the EvalAI login API, & saves it in the `token.json` file at `~/.evalai/token.json` in the user's system.

I'd like to work on this!

Sure! @nileshprasad137 You can work on this.

User should not be allowed to see the password while typing, right?

yeah
Repository: Cloud-CV/evalai-cli
Issue: Fix bug while copying command
Root cause: Validation Gap

## Bug:

Currently, when the command `evalai set_token <auth_token>` is copied to clipboard, then the content which is copied is `evalai set_token &lt;auth_token&gt;`.

## Expected Behaviour

The copied content should be `evalai set_token <auth_token>`

@RishabhJain2018 can i work on this?


@RishabhJain2018 Also, when the command `evalai push <image:tag> -p 4` is copied , the content is `evalai push &lt;image:tag&gt; -p 4 `.

Sure @inishchith.
Repository: Cloud-CV/evalai-cli
Issue: Remove hard-coded token file name.
Root cause: Architectural Constraint

We need to figure out a better way to save the token file name.
One way could be to have a `config.json` that saves these values and is loaded at startup. This will allow us to not only save custom config file names but also custom locations.
We can prompt the user at startup to enter these values if they are missing, providing the current values as defaults (similar to how ssh keys are generated).

Solved. #41
Repository: Cloud-CV/evalai-cli
Issue: Add unit tests for download_file command
Root cause: Documentation Deficiency


Can I work on this issue?

Hi @RishabhJain2018 Can i work on this issue?

Sure @pushkalkatara 

@RishabhJain2018 I am unable to login into the CLI using the ID: host and Password; password. It says invalid credentials. What might be the issue?

I have written the tests but unable to run them due to no login. Here's the commit - https://github.com/pushkalkatara/evalai-cli/commit/f4b511750ff24a8c673b0460cb0f470b2a2f41d5

> @RishabhJain2018 I am unable to login into the CLI using the ID: host and Password; password. It says invalid credentials. What might be the issue?

Please check that you're running EvalAI locally as well and host url is set to `localhost`.

yes host url is set to localhost:8888 and the EvalAI is running locally. Still when i login using credentials ID: host and password: password, it just exits without any message. I think it should give invalid credentials or login successful.
```
(venv) (base)  ~/cv/dev/evalai-cli: evalai login 
username: host
Enter password: 
```

The host URL should be set to `http://localhost:8000` as CLI interacts with the API.

The pytests worked, but still, login as a host or a participant is not working.

@RishabhJain2018 I solved the issue and created a pull request. Please review the PR #173 .
Repository: Cloud-CV/evalai-cli
Issue: [docs] Documentation on usage of download_file
Root cause: Documentation Deficiency

https://evalai-cli.cloudcv.org/ does not contain documentation on how to use the download_file CLI command.

please, can you tell me what is the command exactly?


Hi @Lakshyashishir2 Recently `download_file` command was implemented in evali-cli. You can find the code here - https://github.com/Cloud-CV/evalai-cli/blob/master/evalai/submissions.py#L185

You have to add documentation for the user in the README as well as in the docs describing on how to use this command. Also, there is an open PR with tests for your reference - https://github.com/Cloud-CV/evalai-cli/pull/173

@Lakshyashishir2 for the GCI task, use The sample command is `evalai download_file 'https://evalai.cloudcv.org/web/submission-files?bucket=<s3_bucket_name>&key=<file_path_in_bucket>' ` as the sample download link.
Repository: Cloud-CV/evalai-cli
Issue: [bug, #todo] hardcoded host url
Root cause: Architectural Constraint

Currently, the host URL is hardcoded, it should be fetched from the CLI. Also, mentioned as a TODO here - [download_file](https://github.com/Cloud-CV/evalai-cli/blob/a658e23a6c4c65ec10b017087de93d09bab08ab4/evalai/submissions.py#L191)
Repository: Cloud-CV/evalai-cli
Issue: Some CLI commands not working
Root cause: Documentation Deficiency

1. evalai challenge 1 participant 2
```
Usage: evalai challenge [OPTIONS] CHALLENGE COMMAND [ARGS]...

Error: No such command "participant".
```

2. evalai challenges upcoming
```
Usage: evalai challenges [OPTIONS] COMMAND [ARGS]...

Error: No such command "upcoming".
```

In the first part, are you trying to register for the challenge. You can try using ``participate`` instead of ``participant`` and in the second part I guess use the word ``future``(Ref: #104 ). 

Yes, I think then docs needs to be changed - `evalai-cli.cloudcv.org`
Repository: Cloud-CV/evalai-cli
Issue: download_file arguments - bucket and key
Root cause: Documentation Deficiency

Currently, the download_file CLI command requires the whole API URL to download the file.

- [ ] Example on how to get the URL.
- [ ] Documentation on how to use the URL in CLI. (passed with " ")

Discussion on how changing from URL to just bucket and key to download the file can be done. The current implementation fetches the API URL from user, parses the bucket and key, and again generates the URL to get a response from the server.
Repository: Cloud-CV/evalai-cli
Issue: calling evaluate() infinite times if submission id is NoneType
Root cause: Validation Gap

Faced this issue while trying nocaps on google colab and making a submission after initial iterations. Rather, user should be given an error that token is not set, or team is not created, or not participated in the challenge.

![nocaps](https://user-images.githubusercontent.com/21266230/68671963-94dcce80-0576-11ea-8427-3d2c9787628b.png)
Repository: Cloud-CV/evalai-cli
Issue: No code written under evalai-cli/evalai/urls.py
Root cause: Unimplemented Feature Gap

_Thanks for taking the time to report a bug in the evalai-cli project, or propose a new feature! Before creating a new issue, please do a quick search to check that it hasn't already been filed on the [issue tracker](https://github.com/Cloud-CV/evalai-cli/issues)._

If you'd like to report a bug, please include the following information:
- the expected behaviour
- the observed behaviour
- steps to reproduce the problem
- which operating system and/or browser you're using, if relevant
- (if possible) copies of the console/server logs

On the other hand, if you'd like to propose a new feature:
- please describe, as clearly as possible, what you'd like to see
- (if possible) include mockups/screenshots, if these would help convey the meaning better

Thanks!
Repository: Cloud-CV/evalai-cli
Issue: Many files need to be updated and added in evalai-cli/docs/
Root cause: Documentation Deficiency

_Thanks for taking the time to report a bug in the evalai-cli project, or propose a new feature! Before creating a new issue, please do a quick search to check that it hasn't already been filed on the [issue tracker](https://github.com/Cloud-CV/evalai-cli/issues)._

If you'd like to report a bug, please include the following information:
- the expected behaviour
- the observed behaviour
- steps to reproduce the problem
- which operating system and/or browser you're using, if relevant
- (if possible) copies of the console/server logs

On the other hand, if you'd like to propose a new feature:
- please describe, as clearly as possible, what you'd like to see
- (if possible) include mockups/screenshots, if these would help convey the meaning better

Thanks!


@ssaahhaajj Are you working on the same? 

Well, @ssaahhaajj we can also make this a task for GCI students.

I am not currently working on the same but definitely we can make it as a task for GCI students. Actually, I have applied for GCI mentor at cloudCV, so I added it for that purpose only. @super-beam @RishabhJain2018 

Great! @ssaahhaajj
Repository: Cloud-CV/evalai-cli
Issue: [Doc] : Readme.md - Description and CLI commands to be updated.

I think, the `readme.md ` is quite undescriptive in the aim of this repo(evalai-cli). To elaborate further, I would like to mention 2 changes in readme.md file.

**1. Description.** 
To a user who is unaware of EvalAI, the` readme` files makes little sense ( though I know, that user can go [here](https://evalai-cli.cloudcv.org) or search about EvalAI). Nonetheless, for a good first impression, `Readme` file should describe how Eval-cli  functions with  EvalAI. For example to directly upload training  codes on the platform in the form of docker images.

**2. CLI commands**
Though CLI commands are available [here](https://evalai-cli.cloudcv.org) , I think it's a good practice to include the commands with a brief explanation in the `readme` file. It would it more informative. 





A see a similar issue(#171) related to this one.
Repository: Cloud-CV/evalai-cli
Issue: [Docs] Custom backend configuration documentation

Currently, it's unclear on how to use custom backend for EvalAI. 
While running EvalAI locally (forked version) the user has to set the host to localhost:8000 but in the documentation its mentioned:
```
evalai host -sh https://evalapi.cloudcv.org
```
instead it should be 
```
evalai host -sh https://localhost:8000
```
Repository: Cloud-CV/evalai-cli
Issue: [GCI-2019] Pretty print CLI output on terminal.
Root cause: Unimplemented Feature Gap

Configure to print output in a pretty format on CLI. Currently, printing output is directly through click python library in plain text. Click supports [ClickUtils](http://click.palletsprojects.com/en/7.x/utils/) with functionalities such as ANSI Color and Pager Support. Student's task - 

1. Modify individual function output using ClickUtils, make a PR and attach a screenshot for the same. Look on commands available [here](https://evalai-cli.cloudcv.org/).
2.  Discuss the idea on how to manage long outputs considering the size of the terminal. (Like in the case of showing all the challenges.)

For queries, contact mentors at https://gitter.im/Cloud-CV/EvalAI

@pushkalkatara Take a look at my PR

#197 
Please take a look at my PR, the screenshot of the PR has been uploaded on the GCI website as required
Repository: Cloud-CV/evalai-cli
Issue: [Docs] readthedocs documentation build failed.
Root cause: Configuration Error

The readthedocs documentation for EvalAI-CLI shows build failing with the most recent build from 1 year and 5 months. The build was not configured correctly.
![evalai-cli-readthedocs](https://user-images.githubusercontent.com/37668193/70717147-c7afe900-1d13-11ea-9f96-3101b3726a78.PNG)
Also, the `Short URLs` on the [overview page](https://readthedocs.org/projects/evalai-cli/) are broken. 
These point to `evalai-cli.readthedocs.io` and `evalai-cli.rtfd.io`
The link to repository should be https://github.com/Cloud-CV/evalai-cli instead it is https://github.com/Cloud-CV
Repository: Cloud-CV/evalai-cli
Issue: [PyPI docs] Link to project homepage on PyPI broken

On the [evalai-cli PyPI page](https://pypi.org/project/evalai/), the link that `Homepage` under `Project Links` in left-bar points to is https://github.com/Cloud-CV/evalai_cli
![left-bar-evalai-pypi](https://user-images.githubusercontent.com/37668193/70718116-a223df00-1d15-11ea-8e97-61ad437629b3.PNG)

Following this leads to a 404 error as it should be https://github.com/Cloud-CV/evalai-cli
![evalai_cli-404](https://user-images.githubusercontent.com/37668193/70718006-7274d700-1d15-11ea-94c5-8f126fdd34e1.PNG)
Repository: Cloud-CV/evalai-cli
Issue: [Docs] No such command: upcoming

On the doc page: https://evalai-cli.cloudcv.org/
The third command in examples: `evalai challenges upcoming` raises an error.
```
C:\Users\nikochiko> evalai challenges upcoming
Usage: evalai challenges [OPTIONS] COMMAND [ARGS]...

Error: No such command "upcoming".
```
I think this should be changed to `evalai challenges future` as per https://github.com/Cloud-CV/evalai-cli/blob/master/evalai/challenges.py#L106
Repository: Cloud-CV/evalai-cli
Issue: [New feature] Add feature to reset_token (logout)
Root cause: Unimplemented Feature Gap

Currently, the CLI only has functionality to `set_token` and `get_token`. However, the `set_token` only accepts 40-character long strings. Therefore, once a token is added, there is no way to take that back expect to switch it with a different token. Also, there is functionality to `evalai login` but no `logout`.
Expected: There should be a command to log out. And also one for resetting token to a undefined value.

@RishabhJain2018 @vkartik97 @pushkalkatara Can I work on this task?
Repository: Cloud-CV/evalai-cli
Issue: [error message] get_token command gives out wrong message
Root cause: Documentation Deficiency

When JSON is not in the evalai path and get_token command is used, it throws an error 
```
"\nThe authentication token json file doesn't exist at the required path. "
                "Please download the file from the Profile section of the EvalAI webapp and "
                "place it at ~/.evalai/token.json or use evalai -t <token> to add it.\n\n",
```

In the second method, we should use `evalai set_token <token>` command rather than `evalai -t <token>`

SC - https://github.com/Cloud-CV/evalai-cli/blob/c74479845e9669fc02c1c3b3799ab3d1e8f4d903/evalai/get_token.py#L20
Repository: Cloud-CV/evalai-cli
Issue: Hardcoded AWS region
Root cause: Architectural Constraint

When making docker submissions the AWS region is hardcoded to `us-east-1`:

https://github.com/Cloud-CV/evalai-cli/blob/20fe1e61aa263aa8d16f93f67b24fefa6367c868/evalai/submissions.py#L124-L130

It remains possible to set arbitrary `AWS_REGION` values in eval-ai though:

https://github.com/Cloud-CV/EvalAI/blob/master/apps/challenges/utils.py#L66-L89

The issue specific to https://github.com/Cloud-CV/evalai-cli is:

1. How do you make a submission to a non `us-east-1` region?

A question that might need to be asked at https://github.com/Cloud-CV/EvalAI is:

2. If `challenge.use_host_credentials` is `True`, why are the challenge credentials used? My expectation was the host credentials referred to those stored in `os.environ`.
Repository: Cloud-CV/evalai-cli
Issue: [docs] Missing instructions for development setup on Windows
Root cause: Documentation Deficiency

**Current behavior:**
The `README.md` [Development Setup](https://github.com/Cloud-CV/evalai-cli#development-setup) section only provides instructions for setting up environment on Linux. (The `source` command is specific to Linux)
![cli-dev-setup](https://user-images.githubusercontent.com/37668193/71375285-b56f5e00-25e3-11ea-8c1c-4add14b66cba.PNG)

**Expected behavior:**
The instructions should include steps for Windows as well.

Note: I would like to work on this issue. 

PR #110 is already open for the same.

@gautamjajoo Thanks for informing! :+1:
Closing the issue now.
Repository: Cloud-CV/evalai-cli
Issue: [new feature] Create shell script to set up development environment
Root cause: Unimplemented Feature Gap

**Current behavior:**
The user has to go through a lot of steps to set up the development environment locally. This often leads to errors due to missed commands / unfulfilled dependencies on the users side or due to outdated documentation. 

**New functionality:**
A script could be created that does all the automation and sets up the development environment for the user. 
What this feature can do:
* Saves a lot of time for user in multiple ways (not having to install dependencies manually, or going through every step in the documentation)
* Reduces scope for errors
* The script can be put on the CI/CD pipeline to make sure that the development setup is functioning as expected.

Note: I would like to work on this if the maintainers approve. (I am also a GCI participant, so if possible I would like to have this added as a task). 
@RishabhJain2018 @vkartik97 @Ram81 Can I go ahead and submit a PR?
Repository: Cloud-CV/evalai-cli
Issue: Fix Progress bar for uploading a docker image.
Root cause: Algorithmic Error

**Current behavior**:
When pushing a docker image using evalai-cli, a new progress bar appears for every incremental progress in the uploading process (Can be seen in the terminal when uploading the image). 

![image](https://user-images.githubusercontent.com/31621523/71520794-e7363c80-28e3-11ea-906d-b0e6d848bb06.png)

**Expected**:
The task is to ensure that the progress bar is shown in a single line.
Repository: Cloud-CV/evalai-cli
Issue: Challenge submission error handling
Root cause: Validation Gap

If the challenge ID or phase does not exist on EvalAI server, the CLI should respond with necessary error, but the CLI crashes with the error as shown below - 

![img](https://user-images.githubusercontent.com/21266230/71539230-661c8b00-295f-11ea-8cf4-df1d44abcbd4.png)
Image Courtesy: GCI Student aaaakshat


@pushkalkatara I have fixed the error.

https://github.com/Cloud-CV/evalai-cli/pull/236