zhiminy/EvalEng huggingface dataset:
Repository: Cloud-CV/evalai-cli
Issue: [bug] Improper implementation of click causes errors while handling exceptions
Root cause: Validation Gap

Current behavior:
While displaying the errors, `click.style` is being called on the errors to make the colors consistent. However, this implementation is not correct as `click.style` does not support an `Exception` as an argument, hence it raises an error on its own. See this snap:
![cli-bug-error-formatting](https://user-images.githubusercontent.com/37668193/71540772-03ce8500-2975-11ea-8003-d28a3a32d4cd.PNG)

Expected:
The error message should be displayed. Like:
![cli-bug-working](https://user-images.githubusercontent.com/37668193/71540776-1e086300-2975-11ea-8786-b768b7f407bb.PNG)
<br>
Another option to still retain the styling and colors is to call `str()` on the errors first.
i.e. 
```
echo(style(str(exception), ... ))
```
Repository: Cloud-CV/evalai-cli
Issue: [enhancement] Define wrapper functions for click's echo function

## Current Implementation:
We are calling click's echo function directly each time we call it.

## The implementation I propose:
Instead of calling it directly, It sounds good to define wrapper functions like
```python
def print_error(message, bold=True):
	echo(
		style(
			message,
			bold=bold,
			fg="red"
		)
	)

def print_success(message, bold=True):
	echo(
		style(
			message,
			bold=bold,
			fg="green"
		)
	)
```
and call them to make the codes more descriptive.

Can I take this and solve along with [#243](https://github.com/Cloud-CV/evalai-cli/issues/243)?

@ShubhamPandey28 Sure :+1:
Repository: Cloud-CV/evalai-cli
Issue: Remove duplicated code

A lot of files have duplicated code.
Files and counts are as follows.
Count | Path
1 | evalai/utils/challenges.py
1 | evalai/utils/teams.py
11 | tests/test_challenges.py
2 | tests/test_requests.py
2 | tests/test_teams.py

Removing these duplicate codes and taking a look at them might help lessening the code and the unnecessary complexity.

Hi @amanex007 There are some more duplications, one is also open as a GCI task here https://codein.withgoogle.com/dashboard/tasks/4597587785547776/ . You can work on it if you're interested. Thanks for pointing out.
Repository: Cloud-CV/evalai-cli
Issue: [enhancement] Use different color for command usage examples within messages

#### Current Behaviour:
When a command is to be shown in a message as a usage example, the current way is to use backticks (`) to enclose it. This is hard to read as the rest of the message is of the same color.
![current-message](https://user-images.githubusercontent.com/37668193/71675237-59ef5e00-2da3-11ea-87d7-bb1901503b05.PNG)

#### Expected Behavior:
Use a different color for the commands and a different color for the rest of the message. This should be done by using a function to parse the string, fetch the substring within the backticks (`) and return a complete message as expected with the part in backticks being shown in a different color.
That is:
![proposed-behaviour](https://user-images.githubusercontent.com/37668193/71675388-ae92d900-2da3-11ea-8277-ea23d5a04c19.PNG)

As red, yellow, white and green are used for other types by convention, I think cyan (as shown above) can be a good choice.

I would like to do this if you alllow.

@nikochiko  Is it okay if I make a new interface.py in evalai/util containing the parsed echo function or can you please suggest me the way it is wanted.

@ShubhamPandey28 I think these changes should go either along with or after the enhancements proposed in https://github.com/Cloud-CV/evalai-cli/issues/234 as this can make use of the additions proposed there. So it would be better if that issue is resolved first, and then we come to this.

For example, I was thinking of having a syntax like:
```python
def error_style(message):  # from 234 !?
    return style(message, bold=True,  fg="red")

def command_style(command):  # from 234 !?
    return style(command, bold=True, fg="cyan")

def prettify_error_with_command(message):
    ## incorporate both above methods here ##

# Usage:
echo(prettify_error_with_command(MESSAGE))
```

The approach mentioned in issue#234 recommends directly echoing the messages, which would make it harder to use both side-by-side.

What do you think? @ShubhamPandey28 @takitsuse 
cc: @vkartik97 @RishabhJain2018 

@ShubhamPandey28 Yes, you are on the right track. Both changes should be made in the same file like interface.py 
I think the mentors can help with the naming here. I was thinking `formatting.py` or `text_styling.py`. Something that can give a more clear idea of what we are trying to achieve here.

For printing things in multiple colors, @nikochiko's edit on proposal #234 looks reasonable. Sounds good to me👍
Repository: Cloud-CV/evalai-cli
Issue: Unreachable Error handling
Root cause: Validation Gap

## The observed behavior
We have [some lines](https://github.com/Cloud-CV/evalai-cli/blob/master/evalai/utils/submissions.py#L274-L281) that handles `requests.exceptions.MissingSchema`, but unfortunately, these lines will be never called because [`submission_details_request`](https://github.com/Cloud-CV/evalai-cli/blob/cc9a93bc063ef7036e0b028c47fde73631d2f01c/evalai/utils/submissions.py#L219) has a `except` clause for `requests.exceptions.RequestException` which is a superset of `requests.exceptions.MissingSchema`.
This means, if a submission has no result, the program shows the unintended error message "Could not establish a connection to EvalAI." instead of the intended error message "The Submission is yet to be evaluated."

## The expected behavior
If a submission has no result, the program must show the intended error message "The Submission is yet to be evaluated.".
Repository: Cloud-CV/evalai-cli
Issue: Redesign get_challenge_count to accept both host and participant flags.
Root cause: Algorithmic Error

The function `get_challenge_count` called when either participant or host is `True` first checks for host and if host is false, checks for participant.

So after activating both flags, only the host challenges are printed. We have to redesign the `get_challenge_count` function to account for this (and possibly rename it too).

#24
Repository: Cloud-CV/evalai-cli
Issue: Code coverage ignores command functions

**Observed**:
Code coverage doesn't check command functions currently, so even if commands are not tested, code coverage will be 100%.

eg: `push` command in `submissions.py` is not tested, so code coverage of the file must not be 100 % but is 100% (see [here](https://coveralls.io/builds/28046756/source?filename=evalai%2Fsubmissions.py)).

Can confirm :+1: 
See this coveralls build: https://coveralls.io/jobs/57952941/source_files/1455465142 
I added two functions at the end one of them was a click command: https://coveralls.io/jobs/57952941/source_files/1455465142#L43

I think this is the problem: https://github.com/Cloud-CV/evalai-cli/blob/master/.coveragerc#L9

@takitsuse Nice Catch! 👌 

@nikochiko I thought the file was clandestine too but was too busy to check that out (thanks, school exam). I'll give it a try :+1: 

np! 😄
Repository: Cloud-CV/evalai-cli
Issue: Remove replication of code to get current host

The code to fetch the current host URL is repeated in these two places:
* https://github.com/Cloud-CV/evalai-cli/blob/master/evalai/set_host.py#L14
* https://github.com/Cloud-CV/evalai-cli/blob/master/evalai/utils/auth.py#L88-L100

The util method get_host_url can be used in the host command instead.


I would like to work on this issue

Sure!
Repository: Cloud-CV/evalai-cli
Issue: [docs] Add documentation for login command
Root cause: Documentation Deficiency

Feature to login with command `evalai login` was implemented here: https://github.com/Cloud-CV/evalai-cli/pull/138 
But the documentation lacks a reference to the login command. 

As for its location in the documentation, I think having it in the "**Get Started**" section would be most appropriate. Also, I think it should replace the `set_token` and `get_token` commands there. (Any feedback here?)
![Capture](https://user-images.githubusercontent.com/37668193/73157157-39969300-4106-11ea-8aa8-e9e9d7743499.PNG)

**Suggesting changes:**
- [ ]  Add documentation for `login` command and appropriate instructions (in Get Started section)
- [ ]  Move the documentation for `set_token` and `get_token` to another section (maybe a new section `Configure Token`)

Link to documentation: https://evalai-cli.cloudcv.org/

Can i work on this issue?

Hi @devil-cyber, Sure!

Thanks

On Mon, Sep 28, 2020, 7:48 PM Rishabh Jain <notifications@github.com> wrote:

> Hi @devil-cyber <https://github.com/devil-cyber>, Sure!
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/Cloud-CV/evalai-cli/issues/271#issuecomment-700037303>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AMPIV6VEIRCDDIAAFBSNIDTSICLM7ANCNFSM4KL4Y6VA>
> .
>
Repository: Cloud-CV/evalai-cli
Issue: Add supported python versions to documentation

Some python versions may not be supported by EvalAI-CLI.
The task is to install and test the app with various versions (`python 3.x`), test some functionality and check for any errors due to version, then update the readme correspondingly.

**Only check for versions yet to reach end-of-life (reference: https://devguide.python.org/#status-of-python-branches)**

_Labels: [docs, good-first-issue]_
Repository: Cloud-CV/evalai-cli
Issue: Add a progress bar for large submission file uploads
Root cause: Unimplemented Feature Gap

**Description**: 
Currently, the large submission file upload feature shows only a plain text and the user doesn't know of the progress of the file uploaded. We should add a progress bar so that the user knows the progress.

![Screenshot 2020-10-23 at 2 30 55 PM](https://user-images.githubusercontent.com/12206047/97040800-82b43000-153c-11eb-83b6-255b069f353b.png)



Fixed in #291 

@RishabhJain2018 we can close this issue
Repository: Cloud-CV/evalai-cli
Issue: Request: Make dependency version constraints softer.
Root cause: Architectural Constraint

I was recently bumping versions of common libraries in one of my [old repositories](https://github.com/nocaps-org/updown-baseline), that used EvalAI CLI v1.3.7, PyTorch v1.1.0, AllenNLP v0.8.4

I decided to bump PyTorch to v1.7.0 and AllenNLP to v1.0+ (the most recent versions right now). But I am facing a dependency deadlock between EvalAI and AllenNLP:

![image](https://user-images.githubusercontent.com/10494087/102941935-f3ff5900-4481-11eb-825b-73009dd72d2c.png)

I wonder if you could make boto3 version requirement a bit softer. This may hinder a bunch of old PyTorch codebases to use newer versions of PyTorch, because AllenNLP versions have a strict dependency on PyTorch.

In general, could you review all your requirements and see if they can be made softer (e.g. tqdm)? For example, the major version could remain a hard requirement, but minor versions could be soft.
Repository: Cloud-CV/evalai-cli
Issue: Add tests for get_user_access_token method

### Description

Add tests for `get_user_access_token` method in [`evalai/utils/auth.py`](https://github.com/Cloud-CV/evalai-cli/blob/master/evalai/utils/auth.py#L19).

@Ram81 I would like to work on this issue. But I am a little unfamiliar with the working of this method.
From what I see, this method takes the auth token as input and returns the same token again as output?

Can you help me with this?

@ShauryaAg the method accepts a expiring auth token as a parameter and makes API call to evalai backend to get an Access Token. Expiring auth tokens expire when user logs out but the access token can be used to make api calls irrespective of login state

@Ram81 sorry about the confusion, my local clone was not up-to-date.

I would like to take this issue.

@ShauryaAg Sure, go ahead.

@Ram81 The test for `get_user_access_token` are already written in [test_get_token.py](https://github.com/Cloud-CV/evalai-cli/blob/master/tests/test_get_token.py). As `get_token` reads from the same file (`~/.evalai/token.json`) as set during `login` (which calls `utils.auth.get_user_auth_token_by_login` and `utils.auth.get_user_access_token`).
Repository: Cloud-CV/evalai-cli
Issue: Keyerror on logging errors when found in making a call to api
Root cause: Validation Gap

When an error is received while making a request to the backend, we catch it, and if it is present in the EVALAI_ERROR_CODES, we log it using the lines:
```
if response.status_code in EVALAI_ERROR_CODES:
    validate_token(response.json())
    echo(
        style(
            "\nError: {}\n"
            "\nUse `evalai challenges` to fetch the active challenges.\n"
            "\nUse `evalai challenge CHALLENGE phases` to fetch the "
            "active phases.\n".format(response.json()["error"]),
            fg="red",
            bold=True,
        )
    )
```
But response.json() does not have the key `error` in cases. For e.g., I made a request using an expired token and got an error. So when I logged the entire json.response(), it was something like
```
{'detail': 'Given token not valid for any token type', 'code': 'token_not_valid', 'messages': [{'token_class': 'RefreshToken', 'token_type': 'refresh', 'message': 'Token is invalid or expired'}]}
```
This does not contains a key named error

Hi @burnerlee, I run in the same error as you. I'm using a fresh install of v1.3.12. 

@burnerlee nice catch, this error response is from jwt token API that we added recently. If you are interested you can pick this issue and send a PR to fix it.

@Ram81 yeah sure, I'll make a PR for this

@Ram81, should I do something like, checking if error key is present in the response, if it does, use the same value, if it doesn't then use the detail key for getting the error?

@burnerlee yes

Oh okay, it is only this jwt error log which causes this error, or there may be some other errors too? Which need to be handled 😅

For now its just the JWT API which has a different error response format. Rest of our APIs are already handled

thank you so much ❤️
Repository: Cloud-CV/evalai-cli
Issue: Fix format of submission meta attributes saved on making a new submission
Root cause: Interface Contract Mismatch

Currently, when we make a new submission along with metadata through the cli, only the name of field and value are stored in the database. This is inconsistent with the case when a submission is made through the frontend. The metadata saved on creating a new submission/editing a previous one saves the entire details of the submission meta attributes including the options and description of the field.
Repository: Cloud-CV/evalai-cli
Issue: Add command to download all submissions of a challenge
Root cause: Unimplemented Feature Gap

### Description

We need a cli command that will allow users to download all submissions of a challenge. The command should handle two cases:

1. If a user is host of a challenge download all submissions made to the challenge for a phase
2. If a user is a participant download submissions only made by the team of user to the challenge for a phase.

Required backend APIs are already available for this feature [here](https://github.com/Cloud-CV/EvalAI/blob/master/apps/challenges/views.py#L1669).

@Ram81 may I pick up this issue and make a quick PR?

@burnerlee sure, go ahead

@Ram81 in what file format should the submissions be downloaded?


@Ram81 ping

@Ram81 ping

@burnerlee we should give two options `csv` and `json`. Sorry about the late reply

Not a problem at all😅 I'll make a PR to integrate this😀 thanks

@Ram81 also where do I save the files download by the user? I am planning to make a new folder at project root level named `submissions`. When the user runs the command through the cli to download submissions. by default the submissions would be downloaded in csv format and stored in the `submissions` folder with a name of format `submissions_<time_string_at_command_run>` . To download a json file the user can use the --json flag. 

Please suggest changes in this approach if needed

@Ram81 ping
Repository: Cloud-CV/evalai-cli
Issue: Proposal for creation of challenges through zip using cli.
Root cause: Unimplemented Feature Gap

Creating challenge through zip file by the means of cli.
The API already exists for the same on Evali Backend.
The input requirements are host team and we already have the functionality to create and view teams through cli.


@Ram81, Can we have this functionality in the cli, or is it left intentionally?

@savish28 we already support creation of challenge using github. We don't plan to support it using cli.

Closing this issue.
Repository: Cloud-CV/evalai-cli
Issue: AttributeError: 'Response' object has no attribute 'get
Root cause: Interface Contract Mismatch

_Thanks for taking the time to report a bug in the evalai-cli project, or propose a new feature! Before creating a new issue, please do a quick search to check that it hasn't already been filed on the [issue tracker](https://github.com/Cloud-CV/evalai-cli/issues)._

If you'd like to report a bug, please include the following information:
- the expected behaviour
- the observed behaviour
- steps to reproduce the problem
- which operating system and/or browser you're using, if relevant
- (if possible) copies of the console/server logs

On the other hand, if you'd like to propose a new feature:
- please describe, as clearly as possible, what you'd like to see
- (if possible) include mockups/screenshots, if these would help convey the meaning better

Thanks!

When I submit results, I get this error
![image](https://user-images.githubusercontent.com/30313569/130184011-560bd8b6-42d2-4c89-b3e2-6db7f6b633ac.png)




@Ram81 Can you please look into this?

@RishabhJain2018 I tried reproducing this issue but I wasn't able to. I'll send a PR once I am able to replicate the issue. I have a idea of what is going wrong here just need to verify it once.

I try to submit results today, the error  is not occurring. Thank you.
I get this information about LVIS 2021 Challenge
![image](https://user-images.githubusercontent.com/30313569/131066146-4c8f97c2-fe62-4254-8800-c2e88f09af11.png)
How can I know the value about my submit？




Hey @Bo396543018 , The worker was down, it is up now and your submission should be done evaluating soon!

@RishabhJain2018  Thank you for your inform~ The status of my submission is running now.
Repository: Cloud-CV/evalai-cli
Issue: Submission metadata appears to be dropped when submitting large files
Root cause: Interface Contract Mismatch

When I try to submit a file along with submission metadata using the `--large` flag with EvalAI-CLI, only the `default_submission_meta_attributes` are properly assigned and not the additional `submission_meta_attributes`. The problem might be that the function [`upload_file_using_presigned_url`](https://github.com/Cloud-CV/evalai-cli/blob/7deab41e9e1d84519ce70c4dfa96f99365366fcb/evalai/utils/common.py#L162) does not currently accept and handle those additional attributes but [`make_submission`](https://github.com/Cloud-CV/evalai-cli/blob/7deab41e9e1d84519ce70c4dfa96f99365366fcb/evalai/utils/submissions.py#L69) does.
Repository: Cloud-CV/evalai-cli
Issue: submission error
Root cause: Environment Incompatibility

I got an error when submit results.

 $evalai challenge 1719 phase 3392  submit --file ./sub.parquet --large 


Error: {'detail': 'Given token not valid for any token type', 'code': 'token_not_valid', 'messages': [{'token_class': 'RefreshToken', 'token_type': 'refresh', 'message': 'Token is invalid or expired'}]}

@kingwmk I am facing a similar error, can you please tell me how did you resolve this? thanks!

Hi @kingwmk @jeromepatel , please update your CLI version to latest one and update the token in your CLI with the latest token from your EvalAI profile to resolve this issue.
Repository: Cloud-CV/evalai-cli
Issue: Fix error on get all challenges
Root cause: Interface Contract Mismatch

### Description

Currently, `evalai challenges` is failing on both staging and production. On calling `evalai challenges`, we get the following error:
```
Error: None
```

Note that other usual commands are working.

The task is to figure out the issue why this is happening and fix it.

### Pre-requisites/Benefits
This issue essentially requires background on a few things:
- A high-level understanding of the [evalai-cli](https://github.com/Cloud-CV/evalai-cli/).
- An understanding of [EvalAI challenges Django model](https://github.com/Cloud-CV/EvalAI/apps/challenges).

### Approach
- Set up `evalai-cli` using the installation described [here](https://github.com/Cloud-CV/evalai-cli#development-setup).
- Reproduce the issue for both production and staging. This will require you to go over the repository to understand how to use EvalAI CLI for production and staging.
Steps for staging:
```
(habitat) lawn-128-61-54-75:habitat-challenge gunjanchhablani$ export EVALAI_CLI_ENVIRONMENT="PRODUCTION"
(habitat) lawn-128-61-54-75:habitat-challenge gunjanchhablani$ export EVALAI_API_URL=https://staging.eval.ai/
(habitat) lawn-128-61-54-75:habitat-challenge gunjanchhablani$ evalai push rearrange_submission:latest --phase march-dev-748
```
- Find the problem area in the code and open a draft PR with descriptive comments and appropriate fix.

Please reach out to us in case you face any issues.

Hey I came here from GSoC and was checking this issue and playing around with EvalAI web service and evalai-cli. I noticed that there is this PR https://github.com/Cloud-CV/evalai-cli/pull/341 that seems related to this, the fix I wrote is similar to that. I was thinking on doing a PR for it, but wanted to be sure because of the other PR that is created.

hey @gchhablani 
The second link of this [message](https://github.com/Cloud-CV/evalai-cli/issues/342#issue-1622157780) is 404     [this one](https://github.com/Cloud-CV/EvalAI/apps/challenges)
Kindly provide an alternate link
Thanks!
Repository: Cloud-CV/evalai-cli
Issue: 404 Client Error: Not Found for url: https://eval.ai/api/challenges/challenge/all
Root cause: External Dependency Breakage

I run `evalai challenges`
and got `404 Client Error: Not Found for url: https://eval.ai/api/challenges/challenge/all`.
What is the problem?
Thanks!


Is the endpoint changed?

I'm having the same issue. @RishabhJain2018 any thoughts? My token was expired, but I've hit refresh on my token and `evalai set_token <new_token>` already. Still the same error.

Hi, I'm new to the project and would like to work on this issue. Is it still open?