Repository: mlcommons/inference
Issue: Big mlperf_log_accuracy.json is generated when running TEST01 with MultiStream mode

WG need to decide how to push this big file to github. What we got here is 5GB of data for a single file. And similar sized file will also be generated for other test if MultiStream scenario needs to be run.

@bitfort Didn't you figure out how to push big files to GitHub for MLPerf Training? What did you do?

If it comes down to the deadline, commit the md5sum of the file and we'll fix it in post.

> If it comes down to the deadline, commit the md5sum of the file and we'll fix it in post.

This log is for auditing, so we still have one week to figure it out.

It shouldn't be necessary to push huge log files for audit. George is working on how to manage this.

Hi Aaron,

We're aware of this issue. You can either reduce accuracy_log_probability or I have a script that I will package with the next release that can be used to truncate the file. The inference results should be mostly duplicates given the limited performance sample count so most of it would be redundant.

Should we maybe gzip them ?

I'm fine with that as well

I recommend we suggest everyone do it the same way - and that way be to truncate the file to fixed length. Even zipped, it's going to be hundreds of megabytes.

got it. I think normal github repo has a 100GB limit. There is git lfs but we'd need to set this up and eventually pay.
Maybe a combination ? As long its in the script ...

Hi @guschmue / @DilipSequeira, for the TEST01 Part IV "Truncate logs to reduce accuracy log file size in preparation for uploading  bash ./truncate_log.sh <accuracy_log_file> <number of samples>" what should the number of samples to use?


@georgelyuan? 

Assuming one line per sample, I think 10000 samples would be reasonable.
For SSD-Large, 1000 is fine given the size of the returned results.

Hi,

TEST03 also has big accuracy logs for ssd-large. To reduce the size I used
the truncate_log.sh on all accuracy logs in TEST01 and TEST03. I hope it is
o.k.
Otherwise we'll have ~400M audit folder (just for closed. This would be
even a bigger issue for NVIDIA/Google and Intel...

Best,
Itay


On Tue, Oct 15, 2019 at 2:46 PM georgelyuan <notifications@github.com>
wrote:

> Assuming one line per sample, I think 10000 samples would be reasonable.
> For SSD-Large, 1000 is fine given the size of the returned results.
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/mlperf/inference/issues/508?email_source=notifications&email_token=ABTRU3HP2B46ASXYQNNX4UTQOY22ZA5CNFSM4I7QPKA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBKKOFQ#issuecomment-542418710>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABTRU3FCR5IBZQJTNFTNYPLQOY22ZANCNFSM4I7QPKAQ>
> .
>


Hi Itay,

Accuracy logs should be the same size as for submission. Is that not the case?

I get much bigger files. This email thread is about big accuracy logs and
you also have a script to reduce the size  truncate_log.sh
Am I missing something?


On Thu, Oct 17, 2019 at 9:53 AM georgelyuan <notifications@github.com>
wrote:

> Hi Itay,
>
> Accuracy logs should be the same size as for submission. Is that not the
> case?
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/mlperf/inference/issues/508?email_source=notifications&email_token=ABTRU3HX7YCPYYAIU7RDYJLQPCJ63A5CNFSM4I7QPKA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBQZHXY#issuecomment-543265759>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABTRU3GRFLDKNAHVDNYSVULQPCJ63ANCNFSM4I7QPKAQ>
> .
>


> I get much bigger files. This email thread is about big accuracy logs and you also have a script to reduce the size truncate_log.sh Am I missing something?
> […](#)
> On Thu, Oct 17, 2019 at 9:53 AM georgelyuan ***@***.***> wrote: Hi Itay, Accuracy logs should be the same size as for submission. Is that not the case? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#508?email_source=notifications&email_token=ABTRU3HX7YCPYYAIU7RDYJLQPCJ63A5CNFSM4I7QPKA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBQZHXY#issuecomment-543265759>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABTRU3GRFLDKNAHVDNYSVULQPCJ63ANCNFSM4I7QPKAQ> .

If MultiStream.samples_per_query remains the same as your submission, then the accuracy log of your accuracy run of TEST03 should be the same as the one in your submission. If you find the big log after your performance run of TEST03, this means you have run the test with TEST01 audit.config.

I know, for Test03 they are the same size. Still seems unnecessary to me
but as you wish I can push them. Note that the audits multiply the size of
the total git by more than 2 - is that what we want?

On Thu, Oct 17, 2019 at 11:32 AM AARON ZHONG <notifications@github.com>
wrote:

> I get much bigger files. This email thread is about big accuracy logs and
> you also have a script to reduce the size truncate_log.sh Am I missing
> something?
> … <#m_-4444235873797806669_>
> On Thu, Oct 17, 2019 at 9:53 AM georgelyuan *@*.***> wrote: Hi Itay,
> Accuracy logs should be the same size as for submission. Is that not the
> case? — You are receiving this because you commented. Reply to this email
> directly, view it on GitHub <#508
> <https://github.com/mlperf/inference/issues/508>?email_source=notifications&email_token=ABTRU3HX7YCPYYAIU7RDYJLQPCJ63A5CNFSM4I7QPKA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBQZHXY#issuecomment-543265759>,
> or unsubscribe
> https://github.com/notifications/unsubscribe-auth/ABTRU3GRFLDKNAHVDNYSVULQPCJ63ANCNFSM4I7QPKAQ
> .
>
> If MultiStream.samples_per_query remains the same as your submission, then
> the accuracy log of your accuracy run of TEST03 should be the same as the
> one in your submission. If you find the big log after your performance run
> of TEST03, this means you have run the test with TEST01 audit.config.
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/mlperf/inference/issues/508?email_source=notifications&email_token=ABTRU3CHBMOIAMOXYBUQCSTQPCVUHA5CNFSM4I7QPKA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBRC24Q#issuecomment-543305074>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABTRU3HHYWI4UPBSWHVMZPDQPCVUHANCNFSM4I7QPKAQ>
> .
>


@guschmue Would this be an issue for our repo?

> I get much bigger files. This email thread is about big accuracy logs and you also have a script to reduce the size truncate_log.sh Am I missing something?
> […](#)
> On Thu, Oct 17, 2019 at 9:53 AM georgelyuan ***@***.***> wrote: Hi Itay, Accuracy logs should be the same size as for submission. Is that not the case? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#508?email_source=notifications&email_token=ABTRU3HX7YCPYYAIU7RDYJLQPCJ63A5CNFSM4I7QPKA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBQZHXY#issuecomment-543265759>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABTRU3GRFLDKNAHVDNYSVULQPCJ63ANCNFSM4I7QPKAQ> .

Truncating for TEST01 is fine, but I didn't think it would be necessary for TEST03.