Repository: microsoft/msquic

Issue #2684: Shutdown hang on XDP with Nth Alloc failure
Created: 2022-05-06T22:18:37Z
Describe the bug
A connection gets stuck in shutdown during the Nth allocation failure tests on XDP. This causes RegistrationClose to hang, waiting for the connection to get unstuck.
Affected OS

 All
 Windows Server 2022
 Windows 11
 Windows Insider Preview (specify affected build below)
 Ubuntu
 Debian
 Other (specify below)

Additional OS information
No response
MsQuic version
main
Steps taken to reproduce bug
Run Nth alloc failture tests.
Expected behavior
Tests pass.
Actual outcome
Test failure with watchdog timeout.
Additional details
No response

Duplicate issue #2763: Shared EC Hang on Shutdown
Created: 2022-05-27T14:27:20Z
Describe the bug
Found from: https://dev.azure.com/ms/msquic/_build/results?buildId=316196&view=results
Looking at quic.log from the failure, I see that the NthAllocFail test failed with a watchdog timeout. From the logs, I could see that we end up hanging when we call RegistrationClose. Looking at the dump, the main thread is in the following callstack:
01 000000ec`0c58edc0 00007ffa`6d4dfca0 KERNELBASE!WaitForSingleObjectEx+0x8e
02 000000ec`0c58ee60 00007ffa`6d4df402 msquic!QuicWorkerUninitialize+0x80 [D:\a\1\msquic\src\core\worker.c @ 158] 
03 000000ec`0c58eeb0 00007ffa`6d4b41de msquic!QuicWorkerPoolUninitialize+0x52 [D:\a\1\msquic\src\core\worker.c @ 824] 
04 000000ec`0c58eef0 00007ff6`66bb1fa9 msquic!MsQuicRegistrationClose+0x12e [D:\a\1\msquic\src\core\registration.c @ 249] 
05 000000ec`0c58ef40 00007ff6`66be27f5 msquictest!MsQuicRegistration::~MsQuicRegistration+0x89 [D:\a\1\msquic\src\inc\msquic.hpp @ 342] 
06 000000ec`0c58ef90 00007ff6`66a78202 msquictest!QuicTestNthAllocFail+0x8b5 [D:\a\1\msquic\src\test\lib\DataTest.cpp @ 2330] 
07 000000ec`0c58f4c0 00007ff6`66c3d5ed msquictest!Misc_NthAllocFail_Test::TestBody+0x1d2 [D:\a\1\msquic\src\test\bin\quic_gtest.cpp @ 1594] 
08 000000ec`0c58f5e0 00007ff6`66c1695d msquictest!testing::internal::HandleExceptionsInMethodIfSupported<testing::Test,void>+0xad [D:\a\1\msquic\submodules\googletest\googletest\src\gtest.cc @ 2657] 
09 000000ec`0c58f6c0 00007ff6`66c1765e msquictest!testing::Test::Run+0xcd [D:\a\1\msquic\submodules\googletest\googletest\src\gtest.cc @ 2681] 
0a 000000ec`0c58f730 00007ff6`66c18079 msquictest!testing::TestInfo::Run+0x1ae [D:\a\1\msquic\submodules\googletest\googletest\src\gtest.cc @ 2856] 
0b 000000ec`0c58f7f0 00007ff6`66c1f1ba msquictest!testing::TestSuite::Run+0x1d9 [D:\a\1\msquic\submodules\googletest\googletest\src\gtest.cc @ 3014] 
0c 000000ec`0c58f8c0 00007ff6`66c3d8ae msquictest!testing::internal::UnitTestImpl::RunAllTests+0x53a [D:\a\1\msquic\submodules\googletest\googletest\src\gtest.cc @ 5876] 
0d 000000ec`0c58fac0 00007ff6`66c189f9 msquictest!testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl,bool>+0xbe [D:\a\1\msquic\submodules\googletest\googletest\src\gtest.cc @ 2657] 
0e 000000ec`0c58fba0 00007ff6`66b3e803 msquictest!testing::UnitTest::Run+0x169 [D:\a\1\msquic\submodules\googletest\googletest\src\gtest.cc @ 5449] 
0f 000000ec`0c58fc50 00007ff6`66a7c404 msquictest!RUN_ALL_TESTS+0x13 [D:\a\1\msquic\submodules\googletest\googletest\include\gtest\gtest.h @ 2293] 
10 000000ec`0c58fc80 00007ff6`66cbe069 msquictest!main+0x184 [D:\a\1\msquic\src\test\bin\quic_gtest.cpp @ 1913] 
11 000000ec`0c58fcf0 00007ff6`66cbdf8e msquictest!invoke_main+0x39 [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl @ 79] 
12 000000ec`0c58fd40 00007ff6`66cbde4e msquictest!__scrt_common_main_seh+0x12e [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl @ 288] 
13 000000ec`0c58fdb0 00007ff6`66cbe0de msquictest!__scrt_common_main+0xe [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl @ 331] 
14 000000ec`0c58fde0 00007ffa`a5fb4ed0 msquictest!mainCRTStartup+0xe [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_main.cpp @ 17] 
15 000000ec`0c58fe10 00007ffa`a6cce39b kernel32!BaseThreadInitThunk+0x10
16 000000ec`0c58fe40 00000000`00000000 ntdll!RtlUserThreadStart+0x2b

That is waiting on the Worker->Done event.
_IRQL_requires_max_(PASSIVE_LEVEL)
void
QuicWorkerUninitialize(
    _In_ QUIC_WORKER* Worker
    )
{
    QuicTraceEvent(
        WorkerCleanup,
        "[wrkr][%p] Cleaning up",
        Worker);

    //
    // Clean up the worker execution context.
    //
    Worker->Enabled = FALSE;
    if (Worker->ExecutionContext.Context) {
        QuicWorkerThreadWake(Worker);
        CxPlatEventWaitForever(Worker->Done);    <====== Blocked Here
This is supposed to be set in the worker loop:
BOOLEAN
QuicWorkerLoop(
    _Inout_ struct CXPLAT_EXECUTION_CONTEXT* Context,
    _Inout_ uint64_t* TimeNow,
    _In_ CXPLAT_THREAD_ID ThreadID
    )
{
    QUIC_WORKER* Worker = (QUIC_WORKER*)Context->Context;

    if (!Worker->Enabled) {
        QuicWorkerLoopCleanup(Worker);
        CxPlatEventSet(Worker->Done);         <=== Should be set here
This worker loop is supposed to be executed by the Worker->ExecutionContext.CxPlatContext platform worker. The really weird thing is that it doesn't have the Worker->ExecutionContext in its ExecutionContexts list.
0:000> ?? &Worker->ExecutionContext
struct CXPLAT_EXECUTION_CONTEXT * 0x000002b6`fe3fe9b0
   +0x000 Entry            : CXPLAT_SLIST_ENTRY
   +0x008 Context          : 0x000002b6`fe3fe9b0 Void
   +0x010 CxPlatContext    : 0x000002b6`fe3f0900 Void   <==== Pointer to CXPLAT_WORKER
   +0x018 Callback         : 0x00007ffa`6d4df980     unsigned char  msquic!QuicWorkerLoop+0
   +0x020 NextTimeUs       : 0xffffffff`ffffffff
   +0x028 Ready            : 0x1 ''

0:000> ?? (CXPLAT_WORKER*)0x2b6fe3f0900
struct CXPLAT_WORKER * 0x000002b6`fe3f0900
   +0x000 Running          : 0x1 ''
   +0x008 WakeEvent        : 0x00000000`00000234 Void
   +0x010 Thread           : 0x00000000`00000238 Void
   +0x018 ThreadId         : 0x7a4
   +0x020 DatapathEC       : 0x000002b6`fe3f1010 Void
   +0x028 ExecutionContexts : 0x000002b6`fe3f7410 CXPLAT_SLIST_ENTRY     <==== List that is supposed to contain &Worker->ExecutionContext
   +0x030 ECsReady         : 0 ''
   +0x038 ECsReadyTime     : 0xffffffff`ffffffff

0:000> dx -r1 (*((msquic!CXPLAT_SLIST_ENTRY *)0x2b6fe3f7410))
(*((msquic!CXPLAT_SLIST_ENTRY *)0x2b6fe3f7410))                 [Type: CXPLAT_SLIST_ENTRY]
    [+0x000] Next             : 0x0 [Type: CXPLAT_SLIST_ENTRY *]          <=== No next, means only 1 entry in the list

Affected OS

 All
 Windows Server 2022
 Windows 11
 Windows Insider Preview (specify affected build below)
 Ubuntu
 Debian
 Other (specify below)

Additional OS information
Not specific to an OS necessarily, but looks to be an issue with SharedEC in general.
MsQuic version
main
Steps taken to reproduce bug
Run tests on a SharedEC (or XDP) build.
Expected behavior
Test should succeed
Actual outcome
The tests hang

Closing comment (2022-06-03T23:20:45Z): I think this is related to #2684 Or that is related to this.