Repository: mlcommons/inference
Issue: Submission infra
Root cause: Documentation Deficiency

I've tried to test the new submission infra by uploading an encrypted tarball. I've received a confirmation by email with a submission id, but the next steps are not clear.

I suggest the confirmation email to include the following:
- Submitter id (submitter name as well?)
- Submission timestamp
- File name and its secure hash (e.g. `md5sum`)
- Link to check (http://submissions-ui.mlcommons.org/index is only for uploading?)

The recommended `openssl` command outputs:
```
$ openssl enc -e -aes256 -pass ...
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
$ openssl version
OpenSSL 1.1.1f  31 Mar 2020
```

Another feature request. We use private GitHub repositories to prepare our submissions (similar to as I described [here](https://github.com/mlcommons/inference_results_v1.1/blob/main/README.md)). GitHub lets us easily download a snapshot of a branch as a `zip` file. Un`zip`ping and re`tar`ring is rather tedious. Can the infra support `zip` files as well please?

> I've tried to test the new submission infra by uploading an encrypted tarball. I've received a confirmation by email with a submission id, but the next steps are not clear.
> 
> I suggest the confirmation email to include the following:
> 
> * Submitter id (submitter name as well?)
> * Submission timestamp
> * File name and its secure hash (e.g. `md5sum`)
> * Link to check (http://submissions-ui.mlcommons.org/index is only for uploading?)

No problem in general, I'll add it. The submission link is http://submissions-ui.mlcommons.org/submission in case you don't have it. You can also get there by clicking in the 'submissions' link in the upper left corner

<br><br>

> The recommended `openssl` command outputs:
> 
> ```
> $ openssl enc -e -aes256 -pass ...
> *** WARNING : deprecated key derivation used.
> Using -iter or -pbkdf2 would be better.
> $ openssl version
> OpenSSL 1.1.1f  31 Mar 2020
> ```
Just adding -pbkdf2 silences the warning. However let me test the flow of the Interface to see if this works fine

<br><br>



> Another feature request. We use private GitHub repositories to prepare our submissions (similar to as I described [here](https://github.com/mlcommons/inference_results_v1.1/blob/main/README.md)). GitHub lets us easily download a snapshot of a branch as a `zip` file. Un`zip`ping and re`tar`ring is rather tedious. Can the infra support `zip` files as well please?

I think this can be integrate as well

> No problem in general, I'll add it. 

Thank you @pgmpablo157321!

> The submission link is http://submissions-ui.mlcommons.org/submission in case you don't have it.
> You can also get there by clicking in the 'submissions' link in the upper left corner

Hmm, I didn't realise that the logo was a link to http://submissions-ui.mlcommons.org/index and "Submissions" was a link to http://submissions-ui.mlcommons.org/submission. It just appeared as "MLC Submissions"...

When I tried to verify my test submission, I received an email with only:

> The submission checker detected some errors for submission with id: `<id>`

How can I view the actual log?

Entering a wrong password still displays on a green field:
> Submission checker scheduled to run for submission with id: `<id>`
> The results will be send to your email.

The email correctly says:
> Couldn't decrypt the tarball with your results for submission with id: `<id>`

But I'd suggest to tell the user straightaway that the password is invalid.

Updates:

- Updated successful upload email with suggested information
- New way of using encryption to avoid deprecated method `openssl enc -e -aes256 -pass pass:<password> -pbkdf2 -in <tarfile_name>.gz -out <encrypted_tarfile_name>.gz`
- Accept encrypted .zip and .gz
- Return the logs containing the submission checker output in case an error occurs
- Send the email to To: field instead of Bcc: field (requested by @ashwin)
- Change MLPerf ID to Submitter ID in UI (requested by @ashwin)

> Entering a wrong password still displays on a green field:
> 
> > Submission checker scheduled to run for submission with id: `<id>`
> > The results will be send to your email.
> 
> The email correctly says:
> 
> > Couldn't decrypt the tarball with your results for submission with id: `<id>`
> 
> But I'd suggest to tell the user straightaway that the password is invalid.

I think this can't be avoided, since the UI only schedules the submission_checker to run

Thank you @pgmpablo157321!

I've tried again with a GitHub snapshot encrypted using the new command. This time (`6970748a-71ed-484f-b18f-ad8387a6f456`), even though the contents was exactly the same, I got no errors.

Unfortunately, `summary.csv` was empty (although it did contain two rows, like the number of entries I submitted). I think it would be good to return the log whether the checker run with or without errors.

Ok, I'll add the log to the email

> I think it would be good to return the log whether the checker run with or without errors.

Update: I made those changes could you try running the checker again

Hmm, doesn't look successful to me:
```
The submission checker ran succesfully for submission id: 6970748a-71ed-484f-b18f-ad8387a6f456

The log output of the submission checker was:

 Traceback (most recent call last):
  File "/workspace/submission-checker.py", line 1592, in <module>
    sys.exit(main())
  File "/workspace/submission-checker.py", line 1566, in main
    results = check_results_dir(config, args.submitter, args.skip_compliance, csv, args.debug)
  File "/workspace/submission-checker.py", line 1230, in check_results_dir
    config.set_type(system_type)
  File "/workspace/submission-checker.py", line 667, in set_type
    self.optional = self.base["optional-scenarios-edge"]
KeyError: 'optional-scenarios-edge'
```

Added some extra checks for this:
> Hmm, doesn't look successful to me:
> 
> ```
> The submission checker ran succesfully for submission id: 6970748a-71ed-484f-b18f-ad8387a6f456
> 
> The log output of the submission checker was:
> 
>  Traceback (most recent call last):
>   File "/workspace/submission-checker.py", line 1592, in <module>
>     sys.exit(main())
>   File "/workspace/submission-checker.py", line 1566, in main
>     results = check_results_dir(config, args.submitter, args.skip_compliance, csv, args.debug)
>   File "/workspace/submission-checker.py", line 1230, in check_results_dir
>     config.set_type(system_type)
>   File "/workspace/submission-checker.py", line 667, in set_type
>     self.optional = self.base["optional-scenarios-edge"]
> KeyError: 'optional-scenarios-edge'
> ```

``` 
assert "ERROR" not in log
assert log.stderr.endswith("SUMMARY: submission looks OK\n")
assert not log.stderr.startswith("Traceback (most recent call last):\n")
```

Thanks for the many useful improvements @pgmpablo157321.

A couple of other ideas:
- Can a submitter view all submitted archives in one place?
- The support email address at the page footer is still `mlperf.infra@gmail`.
- In the submission confirmation email, could you please put `Organization` at the top, instead of sandwiching it between `Submitter id` and `Submission id`? Maybe call it `Submitter organization`?

I also have a suggestion on the order of columns to make it easier to check the submitted results at a glance:
- `Accuracy`
- `Performance` (currently called `Result`)
- `Power`
- `Inferred` (Boolean)
- `Compliance` (Boolean or the number of tests?)
- `Errors` (Boolean or the number of errors?)

The `Version` column is not very informative (`v2.0`), but perhaps it can be moved to the front?


@pgmpablo157321 I've done another test submission (`1f805e2a-3fa0-4b39-89f6-d28eb45c680f`), which has returned some errors. From the log I can see it is because the checker is run twice: "for version 2.0" and "for version 1.1". The test submission fails "for version 1.1" because of MultiStream and rng seeds. Any reason to run "for version 1.1"?

> @pgmpablo157321 I've done another test submission (`1f805e2a-3fa0-4b39-89f6-d28eb45c680f`), which has returned some errors. From the log I can see it is because the checker is run twice: "for version 2.0" and "for version 1.1". The test submission fails "for version 1.1" because of MultiStream and rng seeds. Any reason to run "for version 1.1"?

@psyhtest  This is only until today, that the test submission period ends. This was made this way so that submitters that haven’t got their 2.0 results could test the interface with the results of the previous round. But obviously those results won't be included