Repository: PaddlePaddle/Paddle

Issue #40381: The BERT model testing application is disabled. What is the current status of this model?
Created: 2022-03-09T12:20:50Z
In commit: 4fe465c the application for running BERT model was switched off for CPU.
Following application stops to build:
/repos/paddle_paddle/build/paddle/fluid/inference/tests/api/test_analyzer_bert
Previously, there were reports of major drops in model performance:
#37075
#40339
More or less from the middle of last year, a slow and sometimes abrupt drop in the performance of this model can be observed. For sure, the results obtained at this time are below the performance recorded for the Release 2.2 version.
Question
In this regard, I have a question as to whether this model has permanently dropped out of attention or was it turned off inadvertently?
Mentioned decrease is visible in the chart:

Ultimately, we went down to the following results:

The above data was recorded with the following parameters:
CLX processor:
            Name: Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
           Model: 85
          Family: 6
         Sockets: 2
Cores per socket: 20
Threads per core: 2
     CPU MHx min: 1000.0000
     CPU MHx max: 3900.0000
       cache L1d: 32K
       cache L1i: 32K
        cache L2: 1024K
        cache L3: 28160K

CPX processor:
Turbo: OFF
            Name: Intel(R) d on(R) Gold 6348H CPU @ 2.30GHz
           Model: 85
          Family: 6
         Sockets: 4
Cores per socket: 24
Threads per core: 2
     CPU MHx min: 1000.0000
     CPU MHx max: 4200.0000
       cache L1d: 32K
       cache L1i: 32K
        cache L2: 1024K
        cache L3: 33792K

SKX processor
Turbo: OFF
            Name: Intel(R) Xeon(R) Platinum 8180 CPU @ 2.50GHz
           Model: 85
          Family: 6
         Sockets: 2
Cores per socket: 28
Threads per core: 2
     CPU MHx min: 1000.0000
     CPU MHx max: 3800.0000
       cache L1d: 32K
       cache L1i: 32K
        cache L2: 1024K
        cache L3: 39424K

CMAKE parameters:
-DCMAKE_BUILD_TYPE=Release -DWITH_GPU=OFF -DWITH_MKLDNN=ON -DWITH_TESTING=ON -DWITH_PROFILER=OFF -DWITH_STYLE_CHECK=OFF -DON_INFER=ON -DWITH_INFERENCE_API_TEST=ON -DPY_VERSION=3.6

Test run command:
/repos/paddle_paddle/build/paddle/fluid/inference/tests/api/test_analyzer_bert --gtest_filter=Analyzer_bert.profile --cpu_num_threads=1 --repeat=1000 -use_mkldnn=true --infer_data=/repos/paddle_paddle/build/third_party/inference_demo/bert_emb128/data.txt  --batch_size=1    --infer_model=/repos/paddle_paddle/build/third_party/inference_demo/bert_emb128/model

For the attention of @jczaja @wozna

Duplicate issue #37075: BERT performance drop
Created: 2021-11-09T12:28:46Z
Commit b9fdd3b seems to be responsible for the decline in the BERT's performance. Depending on the machine on which the tests were performed, the decrease ranges from 2.66% to 3.21%. Below are the screenshots of the sheets with the results of consecutive commits together with the fps results for three processors
CLX processor:
            Name: Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
           Model: 85
          Family: 6
         Sockets: 2
Cores per socket: 20
Threads per core: 2
     CPU MHx min: 1000.0000
     CPU MHx max: 3900.0000
       cache L1d: 32K
       cache L1i: 32K
        cache L2: 1024K
        cache L3: 28160K

results:

CPX processor:
Turbo: OFF
            Name: Intel(R) d on(R) Gold 6348H CPU @ 2.30GHz
           Model: 85
          Family: 6
         Sockets: 4
Cores per socket: 24
Threads per core: 2
     CPU MHx min: 1000.0000
     CPU MHx max: 4200.0000
       cache L1d: 32K
       cache L1i: 32K
        cache L2: 1024K
        cache L3: 33792K

results:

SKX processor
Turbo: OFF
            Name: Intel(R) Xeon(R) Platinum 8180 CPU @ 2.50GHz
           Model: 85
          Family: 6
         Sockets: 2
Cores per socket: 28
Threads per core: 2
     CPU MHx min: 1000.0000
     CPU MHx max: 3800.0000
       cache L1d: 32K
       cache L1i: 32K
        cache L2: 1024K
        cache L3: 39424K

results:

CMAKE parameters:
-DCMAKE_BUILD_TYPE=Release -DWITH_GPU=OFF -DWITH_MKLDNN=ON -DWITH_TESTING=ON -DWITH_PROFILER=OFF -DWITH_STYLE_CHECK=OFF -DON_INFER=ON -DWITH_INFERENCE_API_TEST=ON -DPY_VERSION=3.6

Test run command:
/repos/paddle_paddle/build/paddle/fluid/inference/tests/api/test_analyzer_bert --gtest_filter=Analyzer_bert.profile --cpu_num_threads=1 --repeat=1000 -use_mkldnn=true --infer_data=/repos/paddle_paddle/build/third_party/inference_demo/bert_emb128/data.txt  --batch_size=1    --infer_model=/repos/paddle_paddle/build/third_party/inference_demo/bert_emb128/model

Closing comment (2022-05-24T07:29:52Z): notes from 5/20 meeting (same as #40381 )
BERT may have been disabled accidently during a restructure. @jiangjiajun is confirming this and bring BERT back accordingly.