Repository: eclipse-openj9/openj9

Issue #15017: IBM Semeru 18.0.1 crashes in libj9gc29.so
Created: 2022-05-06T12:55:32Z
Java -version output
openjdk version "18.0.1" 2022-04-19
IBM Semeru Runtime Open Edition 18.0.1.0 (build 18.0.1+10)
Eclipse OpenJ9 VM 18.0.1.0 (build openj9-0.32.0, JRE 18 Linux amd64-64-Bit Compressed References 20220422_62 (JIT enabled, AOT enabled)
OpenJ9   - 9a84ec34e
OMR      - ab24b6666
JCL      - b577b89345e based on jdk-18.0.1+10)

Summary of problem
We observed crashes in some of our test applications when executed with IBM Semeru 18.0.1. Those tests are running together with our JVMTI agent.
I observed the following message twice so far:
Unhandled exception

Type=Segmentation error vmState=0x00000000
J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001

Handler1=00007F4FB0AF3290 Handler2=00007F4FB0851150 InaccessibleAddress=0000000000000000
RDI=00007F4FAC05FB10 RSI=0000000000000007 RAX=0000000000000000 RBX=00000000F00D1E90
RCX=0000000000000000 RDX=0000000099669966 R8=00007F4F2FFFE5F0 R9=00007F4F2FFFE5F0
R10=0000000000000006 R11=0000000000000000 R12=00007F4F2FFFE200 R13=00007F4F2FFFE030
R14=00007F4FAA051340 R15=00007F4E74002C48
RIP=00007F4FA9D23BBD GS=0000 FS=0000 RSP=00007F4F2FFFE030
EFlags=0000000000010246 CS=0033 RBP=00007F4F2FFFE440 ERR=0000000000000004
TRAPNO=000000000000000E OLDMASK=0000000000001000 CR2=0000000000000000
xmm0 0000003000000020 (f: 32.000000, d: 1.018558e-312)
xmm1 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm2 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm3 000000004153c21e (f: 1096008192.000000, d: 5.415000e-315)
xmm4 000000003eccccd5 (f: 1053609152.000000, d: 5.205521e-315)
xmm5 0000000039347648 (f: 959739456.000000, d: 4.741743e-315)
xmm6 000000004141ecbc (f: 1094839424.000000, d: 5.409226e-315)
xmm7 000000003d884dc2 (f: 1032342976.000000, d: 5.100452e-315)
xmm8 242d643a00080e00 (f: 527872.000000, d: 2.021873e-134)
xmm9 2a3d032500180900 (f: 1575168.000000, d: 3.162448e-105)
xmm10 2eec225f00110700 (f: 1115904.000000, d: 1.158582e-82)
xmm11 3ada3f4600190500 (f: 1639680.000000, d: 3.392366e-25)
xmm12 3b3b0af000100700 (f: 1050368.000000, d: 2.236922e-23)
xmm13 491e6e48001a0500 (f: 1705216.000000, d: 1.696573e+44)
xmm14 4dff21d5000c0500 (f: 787712.000000, d: 5.245758e+67)
xmm15 4ff688cb000e0d00 (f: 920832.000000, d: 1.630812e+77)
Module=/home/jenkins/agent/oa-master/agent/test/java/build/tmp/caches/jdk-ibm-semeru_64-18.0.1.10.0/lib/default/libj9gc29.so
Module_base_address=00007F4FA9B8A000
Target=2_90_20220422_62 (Linux 5.11.0-36-generic)
CPU=amd64 (40 logical CPUs) (0x3ef38ec000 RAM)
----------- Stack Backtrace -----------
_Z27concurrentStackSlotIteratorP8J9JavaVMPP8J9ObjectPvP16J9StackWalkStatePKv+0x39d (0x00007F4FA9D23BBD [libj9gc29.so+0x199bbd])
jitWalkFrame+0x16b (0x00007F4FAA9A4D2B [libj9jit29.so+0x951d2b])
jitWalkStackFrames+0xddc (0x00007F4FAA9A611C [libj9jit29.so+0x95311c])
walkStackFrames+0xb3 (0x00007F4FB0B33483 [libj9vm29.so+0x80483])
_ZN28GC_VMThreadStackSlotIterator9scanSlotsEP10J9VMThreadS1_PvPFvP8J9JavaVMPP8J9ObjectS2_P16J9StackWalkStatePKvEbb+0x76 (0x00007F4FA9BCC8A6 [libj9gc29.so+0x428a6])
_ZN28MM_ConcurrentMarkingDelegate15scanThreadRootsEP18MM_EnvironmentBase+0x1ac (0x00007F4FA9D241BC [libj9gc29.so+0x19a1bc])
_ZN15MM_ConcurrentGC10scanThreadEP18MM_EnvironmentBase+0x61 (0x00007F4FA9CE6301 [libj9gc29.so+0x15c301])
memorySubSpaceAsyncCallbackHandler+0x6d (0x00007F4FA9CB325D [libj9gc29.so+0x12925d])
dispatchAsyncEvents+0xc7 (0x00007F4FB0AF4887 [libj9vm29.so+0x41887])
javaCheckAsyncMessages+0xa8 (0x00007F4FB0ACADE8 [libj9vm29.so+0x17de8])
old_slow_jitCheckAsyncMessages+0x85 (0x00007F4FAA991475 [libj9jit29.so+0x93e475])
 (0x00007F4FAA9A89F6 [libj9jit29.so+0x9559f6])

The small diagnostic files for those 2 crashes are attached. If necessary I may be able to provide core dumps as well.
A third crash did not create any diagnostic files and only printed the following:
Thread 000000000038AF00 structures scan: slot 00007F73941A8CF8 has bad value FFFFFFFFFFFFFFFF, iterator state 2
java: ../../../../../../openj9/runtime/gc_glue_java/ScavengerRootScanner.hpp:123: virtual void MM_ScavengerRootScanner

Not sure if this is related or a separate issue.
Diagnostic files
diagnostic_files.zip

Duplicate issue #15063: LambdaLoadTest_special_J9_5m_9_FAILED Segmentation error vmState=0x00020002
Created: 2022-05-16T12:11:46Z
Failure link
From an internal build job/Test_openjdk17_j9_special.system_x86-64_linux_testList_2/197/(cent7x64-1-7):
java version "17.0.4" 2022-07-19
IBM Semeru Runtime Certified Edition 17.0.4+2 (build 17.0.4+2)
Eclipse OpenJ9 VM 17.0.4+2 (build master-19c3dc498, JRE 17 Linux amd64-64-Bit Compressed References 20220514_149 (JIT enabled, AOT enabled)
OpenJ9   - 19c3dc498
OMR      - c945d253d
JCL      - 32611aab1 based on jdk-17.0.4+2)

Rerun in Grinder - Change TARGET to run only the failed test targets.
Optional info
Failure output (captured from console output)
[2022-05-14T18:11:27.295Z] variation: Mode187
[2022-05-14T18:11:27.295Z] JVM_OPTIONS:  -Xaggressive -Xgcpolicy:gencon -Xjit -Xnocompressedrefs

[2022-05-14T18:13:29.208Z] LT  stderr Unhandled exception
[2022-05-14T18:13:29.208Z] LT  stderr Type=Segmentation error vmState=0x00020002
[2022-05-14T18:13:29.208Z] LT  stderr J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
[2022-05-14T18:13:29.208Z] LT  stderr Handler1=00007F0DE50882A0 Handler2=00007F0DE4DE41A0 InaccessibleAddress=0000000000000000
[2022-05-14T18:13:29.208Z] LT  stderr RDI=00007F0DE004A9E0 RSI=0000000000000007 RAX=0000000000000000 RBX=00007F0D5D9A1A50
[2022-05-14T18:13:29.208Z] LT  stderr RCX=0000000000000000 RDX=0000000099669966 R8=00007F0D081BB850 R9=00007F0D080F8A80
[2022-05-14T18:13:29.208Z] LT  stderr R10=0000000000000006 R11=000000000000000A R12=00007F0DD8A433C0 R13=00007F0DE0A91318
[2022-05-14T18:13:29.208Z] LT  stderr R14=00007F0D080F88C0 R15=0000000000000000
[2022-05-14T18:13:29.208Z] LT  stderr RIP=00007F0DD871F1A9 GS=0000 FS=0000 RSP=00007F0D080F88B0
[2022-05-14T18:13:29.208Z] LT  stderr EFlags=0000000000010246 CS=0033 RBP=00007F0D080F8E00 ERR=0000000000000004
[2022-05-14T18:13:29.208Z] LT  stderr TRAPNO=000000000000000E OLDMASK=0000000000000000 CR2=0000000000000000
[2022-05-14T18:13:29.208Z] LT  stderr xmm0 0000003000000020 (f: 32.000000, d: 1.018558e-312)
[2022-05-14T18:13:29.208Z] LT  stderr xmm1 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-05-14T18:13:29.208Z] LT  stderr xmm2 7375706300757063 (f: 7696483.000000, d: 1.498998e+248)
[2022-05-14T18:13:29.208Z] LT  stderr xmm3 000000003686e057 (f: 914808896.000000, d: 4.519757e-315)
[2022-05-14T18:13:29.208Z] LT  stderr xmm4 000000003d37672a (f: 1027041088.000000, d: 5.074257e-315)
[2022-05-14T18:13:29.208Z] LT  stderr xmm5 006e006538e3f3ba (f: 954463168.000000, d: 1.335113e-306)
[2022-05-14T18:13:29.208Z] LT  stderr xmm6 000000003dbd4264 (f: 1035813504.000000, d: 5.117599e-315)
[2022-05-14T18:13:29.208Z] LT  stderr xmm7 0000000042c80000 (f: 1120403456.000000, d: 5.535529e-315)
[2022-05-14T18:13:29.208Z] LT  stderr xmm8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-05-14T18:13:29.208Z] LT  stderr xmm9 0000000041040000 (f: 1090781184.000000, d: 5.389175e-315)
[2022-05-14T18:13:29.208Z] LT  stderr xmm10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-05-14T18:13:29.208Z] LT  stderr xmm11 0000ff0000000000 (f: 0.000000, d: 1.385239e-309)
[2022-05-14T18:13:29.208Z] LT  stderr xmm12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-05-14T18:13:29.208Z] LT  stderr xmm13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-05-14T18:13:29.208Z] LT  stderr xmm14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-05-14T18:13:29.208Z] LT  stderr xmm15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-05-14T18:13:29.208Z] LT  stderr Module=/home/jenkins/workspace/Test_openjdk17_j9_special.system_x86-64_linux_testList_2/openjdkbinary/j2sdk-image/lib/default/libj9gc_full29.so
[2022-05-14T18:13:29.208Z] LT  stderr Module_base_address=00007F0DD857D000
[2022-05-14T18:13:29.208Z] LT  stderr Target=2_90_20220514_149 (Linux 3.10.0-1160.62.1.el7.x86_64)
[2022-05-14T18:13:29.208Z] LT  stderr CPU=amd64 (4 logical CPUs) (0x1e8d05000 RAM)
[2022-05-14T18:13:29.208Z] LT  stderr ----------- Stack Backtrace -----------
[2022-05-14T18:13:29.209Z] LT  stderr _ZN26MM_MarkingSchemeRootMarker11doStackSlotEPP8J9ObjectPvPKv+0x389 (0x00007F0DD871F1A9 [libj9gc_full29.so+0x1a21a9])
[2022-05-14T18:13:29.209Z] LT  stderr jitWalkFrame+0x16b (0x00007F0DD93C709B [libj9jit29.so+0x98209b])
[2022-05-14T18:13:29.209Z] LT  stderr jitWalkStackFrames+0xddc (0x00007F0DD93C848C [libj9jit29.so+0x98348c])
[2022-05-14T18:13:29.209Z] LT  stderr walkStackFrames+0xb3 (0x00007F0DE50C8533 [libj9vm29.so+0x80533])
[2022-05-14T18:13:29.209Z] LT  stderr _ZN28GC_VMThreadStackSlotIterator9scanSlotsEP10J9VMThreadS1_PvPFvP8J9JavaVMPP8J9ObjectS2_P16J9StackWalkStatePKvEbb+0x76 (0x00007F0DD85BF116 [libj9gc_full29.so+0x42116])
[2022-05-14T18:13:29.209Z] LT  stderr _ZN14MM_RootScanner13scanOneThreadEP18MM_EnvironmentBaseP10J9VMThreadPv+0xfd (0x00007F0DD85B75FD [libj9gc_full29.so+0x3a5fd])
[2022-05-14T18:13:29.209Z] LT  stderr _ZN14MM_RootScanner11scanThreadsEP18MM_EnvironmentBase+0xbf (0x00007F0DD85B633F [libj9gc_full29.so+0x3933f])
[2022-05-14T18:13:29.209Z] LT  stderr _ZN14MM_RootScanner9scanRootsEP18MM_EnvironmentBase+0x42 (0x00007F0DD85B8E92 [libj9gc_full29.so+0x3be92])
[2022-05-14T18:13:29.209Z] LT  stderr _ZN18MM_MarkingDelegate9scanRootsEP18MM_EnvironmentBaseb+0x14b (0x00007F0DD8719DBB [libj9gc_full29.so+0x19cdbb])
[2022-05-14T18:13:29.209Z] LT  stderr _ZN19MM_ParallelMarkTask3runEP18MM_EnvironmentBase+0x79 (0x00007F0DD873EC69 [libj9gc_full29.so+0x1c1c69])
[2022-05-14T18:13:29.209Z] LT  stderr _ZN21MM_ParallelDispatcher3runEP18MM_EnvironmentBaseP7MM_Taskm+0x9e (0x00007F0DD86AB84E [libj9gc_full29.so+0x12e84e])
[2022-05-14T18:13:29.209Z] LT  stderr _ZN19MM_ParallelGlobalGC7markAllEP18MM_EnvironmentBaseb+0x135 (0x00007F0DD86C6ED5 [libj9gc_full29.so+0x149ed5])
[2022-05-14T18:13:29.209Z] LT  stderr _ZN19MM_ParallelGlobalGC24mainThreadGarbageCollectEP18MM_EnvironmentBaseP22MM_AllocateDescriptionbb+0x1a0 (0x00007F0DD86C78A0 [libj9gc_full29.so+0x14a8a0])
[2022-05-14T18:13:29.209Z] LT  stderr _ZN15MM_ConcurrentGC22internalGarbageCollectEP18MM_EnvironmentBaseP17MM_MemorySubSpaceP22MM_AllocateDescription+0x26 (0x00007F0DD8748AC6 [libj9gc_full29.so+0x1cbac6])
[2022-05-14T18:13:29.209Z] LT  stderr _ZN12MM_Collector14garbageCollectEP18MM_EnvironmentBaseP17MM_MemorySubSpaceP22MM_AllocateDescriptionjP28MM_ObjectAllocationInterfaceS3_P20MM_AllocationContext+0xb6 (0x00007F0DD8688C76 [libj9gc_full29.so+0x10bc76])
[2022-05-14T18:13:29.209Z] LT  stderr _ZN17MM_MemorySubSpace20systemGarbageCollectEP18MM_EnvironmentBasej+0x124 (0x00007F0DD86A54F4 [libj9gc_full29.so+0x1284f4])
[2022-05-14T18:13:29.209Z] LT  stderr _ZN17MM_MemorySubSpace20systemGarbageCollectEP18MM_EnvironmentBasej+0x2c (0x00007F0DD86A53FC [libj9gc_full29.so+0x1283fc])
[2022-05-14T18:13:29.209Z] LT  stderr _ZN17MM_MemorySubSpace20systemGarbageCollectEP18MM_EnvironmentBasej+0x2c (0x00007F0DD86A53FC [libj9gc_full29.so+0x1283fc])
[2022-05-14T18:13:29.209Z] LT  stderr j9gc_modron_global_collect_with_overrides+0x38 (0x00007F0DD85AB4A8 [libj9gc_full29.so+0x2e4a8])
[2022-05-14T18:13:29.209Z] LT  stderr JVM_GC_Impl+0x6a (0x00007F0DD7BB0EEA [libjclse29.so+0x61eea])
[2022-05-14T18:13:29.209Z] LT  stderr  (0x00007F0D422A906C [<unknown>+0x0])
[2022-05-14T18:13:29.209Z] LT  stderr ---------------------------------------
[2022-05-14T18:13:29.209Z] LT  stderr JVMDUMP039I Processing dump event "gpf", detail "" at 2022/05/14 11:13:27 - please wait.

[2022-05-14T19:18:30.694Z] STF 12:18:29.512 - =====================   R E S U L T S   =====================
[2022-05-14T19:18:30.694Z] STF 12:18:29.512 - Stage results:
[2022-05-14T19:18:30.694Z] STF 12:18:29.512 -   setUp:     pass
[2022-05-14T19:18:30.694Z] STF 12:18:29.512 -   execute:  *fail*
[2022-05-14T19:18:30.694Z] STF 12:18:29.512 -   teardown:  pass
[2022-05-14T19:18:30.694Z] STF 12:18:29.512 - 
[2022-05-14T19:18:30.694Z] STF 12:18:29.512 - Overall result: **FAILED**
[2022-05-14T19:18:30.694Z] 
[2022-05-14T19:18:30.694Z] LambdaLoadTest_special_J9_5m_9_FAILED

50x grinder - https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/23749/
fyi @dmitripivkine

Closing comment (2022-05-16T19:43:19Z): The crash occur due bad O-slot in JIT frame. O-slot points to mid-object:
Checking THREAD STACKS...  <gc check (1): from debugger: THREAD STACKS: slot 7f0de0232a00(7f0d081bb850) -> 7f0d5d9a1a50: class pointer is null>

<7f0de0232a00> JIT frame: bp = 0x00007F0CEB99F1F8, pc = 0x00007F0D425CDFE3, unwindSP = 0x00007F0CEB99F1A0, cp = 0x00007F0CE0010120, arg0EA = 0x00007F0CEB99F200, jitInfo = 0x00007F0D090496B8
<7f0de0232a00> 	Method: net/adoptopenjdk/test/lambda/TestLambdaRecursive.testDeepRecursiveLambda()V !j9method 0x00007F0CE00103C0
<7f0de0232a00> 	Bytecode index = 2, inlineDepth = 0, PC offset = 0x0000000000000443
<7f0de0232a00> 	stackMap=0x00007F0D090499C4, slots=I16(0x0001) parmBaseOffset=I16(0x0010), parmSlots=U16(0x0000), localBaseOffset=I16(0xFFE8)
<7f0de0232a00> 	Described JIT temps starting at 0x00007F0CEB99F1E0 for IDATA(0x0000000000000003) slots
<7f0de0232a00> 		I-Slot: : t2[0x00007F0CEB99F1E0] = 0x00007F0D5DC71BC0
<7f0de0232a00> 		I-Slot: : t1[0x00007F0CEB99F1E8] = 0x00007F0D425F2D9F
<7f0de0232a00> 		I-Slot: : t0[0x00007F0CEB99F1F0] = 0x0000000000000206
<7f0de0232a00> 	JIT-RegisterMap = UDATA(0x0000000000000D04)
<7f0de0232a00> 		JIT-RegisterMap-I-Slot[0x00007F0D081BB840] = UDATA(0x00007F0DD01F07C8) (jit_rax)
<7f0de0232a00> 		JIT-RegisterMap-I-Slot[0x00007F0D081BB848] = UDATA(0x00007F0D5D9A1A38) (jit_rbx)
<7f0de0232a00> 		JIT-RegisterMap-O-Slot[0x00007F0D081BB850] = 0x00007F0D5D9A1A50 (jit_rcx) <--------------
<7f0de0232a00> 		JIT-RegisterMap-I-Slot[0x00007F0D081BB858] = UDATA(0x00007F0DD98A94C0) (jit_rdx)
<7f0de0232a00> 		JIT-RegisterMap-I-Slot[0x00007F0D081BB860] = UDATA(0x0000000000000000) (jit_rdi)
<7f0de0232a00> 		JIT-RegisterMap-I-Slot[0x00007F0D081BB868] = UDATA(0x00007F0CEB99F210) (jit_rsi)
<7f0de0232a00> 		JIT-RegisterMap-I-Slot[0x00007F0D081BB870] = UDATA(0x00007F0D081BB8E0) (jit_rbp)
<7f0de0232a00> 		JIT-RegisterMap-I-Slot[0x00007F0D081BB878] = UDATA(0x00007F0DD869D8F3) (jit_rsp)
<7f0de0232a00> 		JIT-RegisterMap-O-Slot[0x00007F0D081BB880] = 0x00007F0DD01F07C8 (jit_r8)
<7f0de0232a00> 		JIT-RegisterMap-I-Slot[0x00007F0D081BB888] = UDATA(0x00007F0DD01F0798) (jit_r9)
<7f0de0232a00> 		JIT-RegisterMap-O-Slot[0x00007F0D081BB890] = 0x00007F0DD01F0798 (jit_r10)
<7f0de0232a00> 		JIT-RegisterMap-O-Slot[0x00007F0D081BB898] = 0x00007F0D5D9A1A18 (jit_r11)
<7f0de0232a00> 		JIT-RegisterMap-I-Slot[0x00007F0D081BB8A0] = UDATA(0x00007F0DD01F07E0) (jit_r12)
<7f0de0232a00> 		JIT-RegisterMap-I-Slot[0x00007F0D081BB8A8] = UDATA(0x00007F0DD01F07E8) (jit_r13)
<7f0de0232a00> 		JIT-RegisterMap-I-Slot[0x00007F0D081BB8B0] = UDATA(0x00007F0DE0080D40) (jit_r14)
<7f0de0232a00> 		JIT-RegisterMap-I-Slot[0x00007F0D081BB8B8] = UDATA(0x00007F0D081BBB48) (jit_r15)
<7f0de0232a00> 	JIT-Frame-RegisterMap[0x00007F0CEB99F1C8] = UDATA(0x00007F0DD86BC37B) (jit_rbx)
<7f0de0232a00> 	JIT-Frame-RegisterMap[0x00007F0CEB99F1D0] = UDATA(0x00007F0D081BB8E8) (jit_r9)
<7f0de0232a00> 	JIT-Frame-RegisterMap[0x00007F0D081BB890] = UDATA(0x00007F0DD01F0798) (jit_r10)
<7f0de0232a00> 	JIT-Frame-RegisterMap[0x00007F0D081BB898] = UDATA(0x00007F0D5D9A1A18) (jit_r11)
<7f0de0232a00> 	JIT-Frame-RegisterMap[0x00007F0D081BB8A0] = UDATA(0x00007F0DD01F07E0) (jit_r12)
<7f0de0232a00> 	JIT-Frame-RegisterMap[0x00007F0D081BB8A8] = UDATA(0x00007F0DD01F07E8) (jit_r13)
<7f0de0232a00> 	JIT-Frame-RegisterMap[0x00007F0D081BB8B0] = UDATA(0x00007F0DE0080D40) (jit_r14)
<7f0de0232a00> 	JIT-Frame-RegisterMap[0x00007F0D081BB8B8] = UDATA(0x00007F0D081BBB48) (jit_r15)

0x7F0D5D9A1A30 :  0000000000000000 00007f0de012c200 [ ................ ]
0x7F0D5D9A1A40 :  0000000000000002 00007f0d5d9a1a50 [ ........P..].... ]
0x7F0D5D9A1A50 :  0000000000000031 00007f0de0130f00 [ 1............... ] <---
0x7F0D5D9A1A60 :  00007f0d5d9a1a78 0000003300000001 [ x..]........3... ]
0x7F0D5D9A1A70 :  0000000000000000 00007f0de012c200 [ ................ ]

> !j9object 0x7F0D5D9A1A38
!J9IndexableObject 0x00007F0D5D9A1A38 {
    struct J9Class* clazz = !j9arrayclass 0x7F0DE012C200   // [B
    Object flags = 0x00000000;
    U_32 size = 0x00000002;
	[0] =  49, 0x31
	[1] =   0, 0x00
}

This failure looks similar to #15017
@0xdaryl FYI