Repository: mlcommons/inference
Issue: GPT-J implementation is giving segmentation fault on Nvidia GPU
Root cause: Resource Mishandling

I'm able to successfully run the gpt-j model using the shared pretrained model on CPU. the below result is just for a single input.

```
Results

{'rouge1': 32.7869, 'rouge2': 6.7797, 'rougeL': 22.9508, 'rougeLsum': 29.5082, 'gen_len': 212, 'gen_num': 1}
```

But on Nvidia GPU (4090 with 24GB memory) I'm getting segmentation fault as follows.

```
Run Directory: /home/arjun/CM/repos/ctuning@mlcommons-ck/cm-mlops/script/get-dataset-cnndm

CMD: cd '/home/arjun/CM/repos/local/cache/e15bfdcfabe545b8/inference/language/gpt-j' &&  /usr/bin/python3 main.py --model-path=/home/arjun/checkpoint-final --dataset-path=/home/arjun/CM/repos/local/cache/31b8797c52f043c1/install/cnn_eval.json --scenario Offline  --mlperf_conf '/home/arjun/CM/repos/local/cache/e15bfdcfabe545b8/inference/mlperf.conf' --user_conf '/home/arjun/CM/repos/ctuning@mlcommons-ck/cm-mlops/script/generate-mlperf-inference-user-conf/tmp/3cbe44644bca4f84aaf7a07322c6cc28.conf' --gpu

2023-06-01 20:46:57.721822: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-06-01 20:46:57.773481: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
Loading PyTorch model...
Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:08<00:00,  2.75s/it]
Casting models to GPU...
Constructing QSL
Encoding Samples
Number of Samples in query_samples :  8
./run.sh: line 32: 310967 Segmentation fault      (core dumped) /usr/bin/python3 main.py --model-path=/home/arjun/checkpoint-final --dataset-path=/home/arjun/CM/repos/local/cache/31b8797c52f043c1/install/cnn_eval.json --scenario Offline --mlperf_conf '/home/arjun/CM/repos/local/cache/e15bfdcfabe545b8/inference/mlperf.conf' --user_conf '/home/arjun/CM/repos/ctuning@mlcommons-ck/cm-mlops/script/generate-mlperf-inference-user-conf/tmp/3cbe44644bca4f84aaf7a07322c6cc28.conf' --gpu
```



@nvzhihanj is this the issue you had mentioned as fixed? Actually I was using the privately shared checkpoint model - could it be the reason?

@arjunsuresh seems like you already picked up the latest changes I pushed, and this seems like a new bug.
My guess is that the device RAM is probably not enough to keep running the model - can you monitor the ram usage using `nvidia-smi` and see if that's the issue?

Also not sure why TF is called here, isn't the script pytorch only?

I hit a similar issue - try chaging the `--max_example` might help

Thank you @nvzhihanj for your reply. It seems my issue is with the GPU memory.

![Screenshot from 2023-06-03 22-24-38](https://github.com/mlcommons/inference/assets/4791823/98417ddb-f6e4-4115-97f5-b7897a4abee8)


@nvzhihanj can you help with some guidance on GPU configuration.

@psyhtest please provide some details about your setup and the issue observed.

@psyhtest @mrasquinha-g the torch implementation of GPT-J causal model is quite sub-optimal, and below is the required GPU memory for running:
FP16, beam_size=4, BS=1: 37 GB
FP32, beam_size=4 BS=1: 78 GB

To save memory, you can either turn off KV cache, or use CPU memory. But note these will slow down the inference.

Thank you, @nvzhihanj. 

> To save memory, you can either turn off KV cache, or use CPU memory. 

Can you please advise how to do that?

We have two A5000 GPUs with 24G VRAM connected via NVLink (i.e. 48G in total). Is there a way to run inference in FP16 on one GPU but use all available VRAM?

So far, we've managed to evaluate a hundred samples in FP16 using 2 beams. I guess the number of beams forms part of the workload definition, i.e. using only 2 beams would not be MLPerf-compliant?

@psyhtest should be fine for open division right?

@arjunsuresh For Open should be fine, but we are aiming at Closed :)

I guess we should discuss that during today's LLM meeting. It's strange that bert-99.9 is not in edge category but gpt-j is as this can change the definition of edge systems. Also we need to see if multistream is applicable for gpt-j. 

> It's strange that bert-99.9 is not in edge category but gpt-j 

Yes, rather bizarre that GPT-J is even considered for Edge.

Especially if you look at the distribution of latencies from a sample run of the reference implementation on the A5000 GPU (with 2 beams, which barely fits 24G):
- Min: 2.0 s
- Mean: 5.2 s
- 90th percentile: 8.2 s
- 99th percentile: 9.2 s

<details><pre>
================================================
MLPerf Results Summary
================================================
SUT name : PySUT
Scenario : SingleStream
Mode     : PerformanceOnly
90th percentile latency (ns) : 8183313239
Result is : VALID
  Min duration satisfied : Yes
  Min queries satisfied : Yes
  Early stopping satisfied: Yes
Early Stopping Result:
 * Processed at least 64 queries (117).
 * Would discard 3 highest latency queries.
 * Early stopping 90th percentile estimate: 8797132675
 * Not enough queries processed for 99th percentile
 early stopping estimate (would need to process at
 least 662 total queries).
================================================
Additional Stats
================================================
QPS w/ loadgen overhead         : 0.19
QPS w/o loadgen overhead        : 0.19
Min latency (ns)                : 1971737790
Max latency (ns)                : 9264584392
Mean latency (ns)               : 5183591908
50.00 percentile latency (ns)   : 4911315309
90.00 percentile latency (ns)   : 8183313239
95.00 percentile latency (ns)   : 8537941592
97.00 percentile latency (ns)   : 8797132675
99.00 percentile latency (ns)   : 9018680756
99.90 percentile latency (ns)   : 9264584392
</pre></details>

Actually it'll be good to put some constraints on what 'Edge category' is like we have for 'datacenter' [here](https://github.com/mlcommons/inference_policies/blob/master/inference_rules.adoc). We also face questions from new submitters regarding if their system can be submitted under edge category and currently we can only say "yes" for any system.  

@psyhtest Currently the plan is to restrict gpt-j to datacenter only unless some submitter wants it under edge.

Just completed a full accuracy run with beam_size=1 and float16 dtype.
```
Completed :  13365

No warnings encountered during test.

No errors encountered during test.
Test Done!
Destroying SUT...
Destroying QSL...
Finished destroying SUT.
Finished destroying QSL.



No warnings encountered during test.

No errors encountered during test.

* cm run script "run accuracy mlperf _cnndm"
  * cm run script "get python3"
  * cm run script "get mlcommons inference src"
  * cm run script "get dataset cnndm _validation"
  * cm run script "get generic-python-lib _package.rouge_score"
/usr/bin/python3 '/home/arjun/CM/repos/local/cache/e15bfdcfabe545b8/inference/language/gpt-j/evaluation.py' --mlperf-accuracy-file '/home/arjun/CM/repos/ctuning@mlcommons-ck/cm-mlops/script/app-mlperf-inference-reference/valid_results/default-reference-gpu-pytorch-v2.0.1-default_config/gptj/offline/accuracy/mlperf_log_accuracy.json' --dataset-file '/home/arjun/CM/repos/local/cache/31b8797c52f043c1/install/cnn_eval.json'
2023-06-16 21:43:00.225968: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-06-16 21:43:00.241528: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
[nltk_data] Downloading package punkt to /home/arjun/nltk_data...
[nltk_data]   Package punkt is already up-to-date!
Constructing QSL
Encoding Samples

Results

{'rouge1': 42.9971, 'rouge2': 18.9708, 'rougeL': 29.0505, 'rougeLsum': 40.2615, 'gen_len': 4579050, 'gen_num': 13368}
Finished destroying QSL.
```

Performance for fp16 using beam_size=1 on Nvidia RTX 4090.

```
performance_sample_count : 13368

No warnings encountered during test.

No errors encountered during test.
Test Done!
Destroying SUT...
Destroying QSL...
Finished destroying SUT.
Finished destroying QSL.


================================================
MLPerf Results Summary
================================================
SUT name : PySUT
Scenario : Offline
Mode     : PerformanceOnly
Samples per second: 0.528369
Result is : VALID
  Min duration satisfied : Yes
  Min queries satisfied : Yes
  Early stopping satisfied: Yes

================================================
Additional Stats
================================================
Min latency (ns)                : 1403561400
Max latency (ns)                : 46512988762266
Mean latency (ns)               : 23293487807394
50.00 percentile latency (ns)   : 23318857310064
90.00 percentile latency (ns)   : 41845403612236
95.00 percentile latency (ns)   : 44169782612269
97.00 percentile latency (ns)   : 45093922824745
99.00 percentile latency (ns)   : 46017808189866
99.90 percentile latency (ns)   : 46460808026037

```

> Performance for fp16 using beam_size=1 on Nvidia RTX 4090.
> Max latency (ns)                : 46512,988,762,266

Is that ~13 hours for 24K samples in Offline? Wow. Is Single Stream about the same or noticeably slower?

@arjunsuresh Comparing your run against the [reference](https://github.com/mlcommons/inference/tree/master/language/gpt-j) and our run:

METRIC | FP32, beams=4 | FP16, beams=1| BF16, beams=2
-|-|-|-
`rouge1` | 42.9865 | 42.9971 | 43.1493
`rouge2` | 20.1235 | 18.9708 | 19.7645
`rougeL` | 29.9881 | 29.0505 | 29.7605
`rougeLsum` | N/A | N/A | 40.312
`gen_len` | N/A | N/A | 4170034

In FP16/BF16, `rouge1` is even higher than the reference. Do we know which metric(s) will be the main one (or is it all three?) and accuracy threshold(s) (if applicable)?

@psyhtest That was for float16 - just updated this. 

"In FP16/BF16, rouge1 is even higher than the reference. Do we know which metric(s) will be the main one (or is it all three?) and accuracy threshold(s) (if applicable)?"

Current plan is to use all the 3 metrics and also the gen_len. That is, for the 99 percentile model, submissions must reach 99 percentage of all the 3 rouge reference values and the submitted gen_len must also reach 99 percentage of the reference gen_len. 

13 hours is for the entire 24756 samples in Offline scenario - about 0.5 QPS. SingleStream is also having similar performance.

```
cm run script --tags=run,mlperf,inference,generate-run-cmds,_performance-only \
--submitter="cTuning" --model=gptj --backend=pytorch --device=cuda --scenario=SingleStream \
--precision=float16 --target_latency=2000 --quiet -v  --env.GPTJ_BEAM_SIZE=1  --execution_mode=valid
```


```

================================================
MLPerf Results Summary
================================================
SUT name : PySUT
Scenario : SingleStream
Mode     : PerformanceOnly
90th percentile latency (ns) : 3342048833
Result is : VALID
  Min duration satisfied : Yes
  Min queries satisfied : Yes
  Early stopping satisfied: Yes
Early Stopping Result:
 * Processed at least 64 queries (662).
 * Would discard 47 highest latency queries.
 * Early stopping 90th percentile estimate: 3510888322
 * Early stopping 99th percentile estimate: 3943392517

================================================
Additional Stats
================================================
QPS w/ loadgen overhead         : 0.46
QPS w/o loadgen overhead        : 0.46

Min latency (ns)                : 854873973
Max latency (ns)                : 3943392517
Mean latency (ns)               : 2156036046
50.00 percentile latency (ns)   : 2002272680
90.00 percentile latency (ns)   : 3342048833
95.00 percentile latency (ns)   : 3590949091
97.00 percentile latency (ns)   : 3652569696
99.00 percentile latency (ns)   : 3860585143
99.90 percentile latency (ns)   : 3943392517

================================================
Test Parameters Used
================================================
samples_per_query : 1
target_qps : 0.5
target_latency (ns): 0
max_async_queries : 1
min_duration (ms): 600000
max_duration (ms): 0
min_query_count : 662
max_query_count : 702
qsl_rng_seed : 10003631887983097364
sample_index_rng_seed : 17183018601990103738
schedule_rng_seed : 12134888396634371638
accuracy_log_rng_seed : 0
accuracy_log_probability : 0
accuracy_log_sampling_target : 0
print_timestamps : 0
performance_issue_unique : 0
performance_issue_same : 0
performance_issue_same_index : 0
performance_sample_count : 13368

No warnings encountered during test.

No errors encountered during test.

```

Meanwhile it might be useful to have GPTJ in edge category as many people might be interested to run LLM in a constrained environment. Even multistream makes sense for GPT-J right? 

Even with early stopping multistreaming needs minimum of 662 queries of 8 samples each. 662* 8 = 5296 samples and at 0.5QPS this will take about 10592 seconds or ~3 hours which I believe is reasonable. Below is the result for a short run. 
```
================================================
MLPerf Results Summary
================================================
SUT name : PySUT
Scenario : MultiStream
Mode     : PerformanceOnly
99th percentile latency (ns) : 22397560128
Result is : INVALID
  Min duration satisfied : Yes
  Min queries satisfied : Yes
  Early stopping satisfied: NO
Recommendations:
Early Stopping Result:
 * Only processed 15 queries.
 * Need to process at least 662 queries for early stopping.

================================================
Additional Stats
================================================
Per-query latency:  
Min latency (ns)                : 14024285711
Max latency (ns)                : 22397560128
Mean latency (ns)               : 17536886962
50.00 percentile latency (ns)   : 17094408897
90.00 percentile latency (ns)   : 19653829747
95.00 percentile latency (ns)   : 22397560128
97.00 percentile latency (ns)   : 22397560128
99.00 percentile latency (ns)   : 22397560128
99.90 percentile latency (ns)   : 22397560128
```

> Current plan is to use all the 3 metrics and also the gen_len. That is, for the 99 percentile model, submissions must reach 99 percentage of all the 3 rouge reference values and the submitted gen_len must also reach 99 percentage of the reference gen_len.

Thank you, @arjunsuresh. As no reference `gen_len` is provided, we should ask to share it. At the moment, I conclude that only 99% of `rouge2` is not reachable with 1-2 beams.

> Meanwhile it might be useful to have GPTJ in edge category as many people might be interested to run LLM in a constrained environment.

For Edge servers, probably. For Edge appliances, probably not; even BERT is quite tough there.

> Even multistream makes sense for GPT-J right?

Not sure I see a use case for that.

The issue of allowing GPT-J in the edge category was discussed at the 6/20 working group meeting. Resolution is to allow the benchmark in "Edge category". Multistream scenario is not appropriate for this benchmark and will be disallowed. 

@pgmpablo157321 Please make the appropriate submission checker changes.

Updating the first column from the [meeting notes](https://docs.google.com/document/d/1QRQWufJvzwepThDsLmj-ebkLXFkVVdma2wo56Uyjo0o/edit?pli=1).

METRIC | FP32 / `beams=4` | FP16 / `beams=1` | BF16 / `beams=2`
-|-|-|-
`rouge1` | 42.9865 | 42.9971 | 43.1493
`rouge2` | 20.1235 | 18.9708 | 19.7645
`rougeL` | 29.9881 | 29.0505 | 29.7605
`rougeLsum` | 40.1658 | N/A | 40.312
`gen_len` | 4,016,878 | N/A | 4,170,034

With for BF16 / `beams=2`, `gen_len` is 3.8% higher than for FP32 / `beams=4`. Should we consider double-sided margins e.g. +- 5%?

@nvzhihanj Thank you for [this link](https://huggingface.co/docs/transformers/v4.30.0/en/main_classes/text_generation#transformers.GenerationConfig.use_cache) to disable cache which made the model to run on a 24 GB GPU with beam_size=4. But it is slower than running on CPU with cache on. I guess doing a singlestream run on CPUs and then inferring the offline number is the only option for those without a better GPU. Or use quantization. 

@arjunsuresh  Yes, KVCache boosts the runtime by a lot so this is sort of expected.
Another options is to parallelize the inference. I saw some hints in this file: https://github.com/huggingface/transformers/blob/fabe17a726bbf6081cfbcc975d8ac451a81f3e2d/src/transformers/models/gptj/modeling_gptj.py#L752 but I haven't tried it yet. 

Thank you @nvzhihanj for that. Unfortunately, I have only a single GPU but this can be useful for @psyhtest 

@psyhtest Shall I close this issue?

FYI. Here is the result I got from A100 FP16 with the default beams=4.

(llm) [root@node044 gpt-j]# python evaluation.py --mlperf-accuracy-file ./build/logs/mlperf_log_accuracy.json --dataset-file ./data/cnn_eval.json --verbose |& tee log.output_FP16_eval.txt
[nltk_data] Downloading package punkt to /root/nltk_data...
[nltk_data]   Package punkt is already up-to-date!
Constructing QSL
Encoding Samples

Results

{'rouge1': 42.9633, 'rouge2': 20.1007, 'rougeL': 29.9595, 'rougeLsum': 40.1322, 'gen_len': 4019478, 'gen_num': 13368}
Finished destroying QSL.


I don't know why the test ran on 92k samples. and the --max_examples don't work for me. the below run was with it, the default value 13368 got overwritten.  Posted it just for having more data points to reference.

```
================================================
MLPerf Results Summary
================================================
SUT name : PySUT
Scenario : Offline
Mode     : PerformanceOnly
Samples per second: 0.154288
Result is : VALID
  Min duration satisfied : Yes
  Min queries satisfied : Yes
  Early stopping satisfied: Yes

================================================
Additional Stats
================================================
Min latency (ns)                : 4450036978
Max latency (ns)                : 598881808261269
Mean latency (ns)               : 99515234474199
50.00 percentile latency (ns)   : 299053883014596
90.00 percentile latency (ns)   : 538693329919518
95.00 percentile latency (ns)   : 568788823032549
97.00 percentile latency (ns)   : 580831834365537
99.00 percentile latency (ns)   : 592866666054421
99.90 percentile latency (ns)   : 598278125688941

================================================
Test Parameters Used
================================================
samples_per_query : 92400
target_qps : 140
target_latency (ns): 0
max_async_queries : 1
min_duration (ms): 600000
max_duration (ms): 0
min_query_count : 1
max_query_count : 0
qsl_rng_seed : 10003631887983097364
sample_index_rng_seed : 17183018601990103738
schedule_rng_seed : 12134888396634371638
accuracy_log_rng_seed : 0
accuracy_log_probability : 0
accuracy_log_sampling_target : 0
print_timestamps : 0
performance_issue_unique : 0
performance_issue_same : 0
performance_issue_same_index : 0
performance_sample_count : 10

No warnings encountered during test.

No errors encountered during test.
``` 

@hanyunfan That's because you have target_qps as 140. You can try with target_qps=1.Then, it'll run for minimum number of queries in offline scenario, which is 24576.