Hello All,
I suspect that the belief that this problem is due to a locking issue with Microsoft code is probably misplaced. Indeed, Microsoft might not be in a position to address the problem.
Pat's analysis assumes that there is one "pool tracker table" (PoolTrackTable), but this is not the case on multiprocessor systems: they have one table per processor and PoolTrackTable is just the table for processor 0.
19: kd> dq nt!PoolTrackTable l 1
fffff803`d6efe9e0 ffffce0f`c2602000
19: kd> dq nt!ExPoolTagTables l 1a
fffff803`d6efa9e0 ffffce0f`c2602000 ffffbd80`3ac41000
fffff803`d6efa9f0 ffffbd80`3ad00000 ffffbd80`3ad71000
fffff803`d6efaa00 ffffbd80`3ae40000 ffffbd80`3aea1000
fffff803`d6efaa10 ffffbd80`3af3d000 ffffbd80`3b000000
fffff803`d6efaa20 ffffbd80`3b069000 ffffbd80`3b105000
fffff803`d6efaa30 ffffbd80`3b19b000 ffffbd80`3b235000
fffff803`d6efaa40 ffffbd80`3b2d0000 ffffbd80`3b366000
fffff803`d6efaa50 ffffbd80`3b400000 ffffbd80`3b496000
fffff803`d6efaa60 ffffbd80`3b540000 ffffbd80`3b600000
fffff803`d6efaa70 ffffbd80`3b680000 ffffbd80`3b700000
fffff803`d6efaa80 ffffbd80`3b796000 ffffbd80`3b830000
fffff803`d6efaa90 ffffbd80`3b8c6000 ffffbd80`3b960000
fffff803`d6efaaa0 00000000`00000000 00000000`00000000
The "one table per processor" design removes the need for the lock management suggested by Pat.
If there is no room in a per-processor table for a new entry, then an "expansion" table (shared by all processors) is used; this table is adequately protected by locks. As far as I can tell, this expansion table never shrinks (or is removed); the two full dumps available in this thread indicate that the expansion table has not been used (PoolTrackTableExpansion == 0).
The values that Pat has observed in _POOL_TRACKER_TABLE NonPagedBytes fields (e.g. fffffffffdff47d0 and ffffffffffff9e20) are plausibly correct; there is no requirement to free a pool allocation on the same processor from which it was allocated. Treating these NonPagedBytes values as signed integers and summing across all processors gives reasonable results.
In the two full dumps, the tag being tracked as CM64. The !poolused debugger extension performs the necessary summation of the individual processor table values.
5: kd> !poolused 1 CM64
........................
Sorting by Tag
NonPaged Paged
Tag Allocs Frees Diff Used Allocs Frees Diff Used
CM64 2829193 2829191 2 480 0 0 0 0
The instruction that "causes" the exception uses the r14 register, which is not saved in the _KTRAP_FRAME created by KiGeneralProtectionFault; disassembly of ExpPoolTrackerChargeEntry shows that r14 must contain either 0x8 (offset of NonPagedBytes) or 0x20 (offset of PagedBytes). The "Circular Kernel Context Logger" indicates that no interrupt has plausibly taken place between entry into the ExpPoolTrackerChargeEntry routine and the exception. In some of the minidumps, a _CONTEXT structure with valid r14 value at time of exception can be found, showing that r14 contains an expected value.
nt!ExpPoolTrackerChargeEntry+0x40:
lock xadd qword ptr [r14+r8],rbp
The !pte command on an address formed by the r8 value plus an assumed r14 value of 0x8 or 0x20, shows that the address writen by the xadd command is valid and writeable:
5: kd> !pte @r8+8
VA ffff8081a7b9bd58
PXE at FFFFB65B2D96C808 PPE at FFFFB65B2D901030 PDE at FFFFB65B202069E8 PTE at FFFFB64040D3DCD8
contains 0A00000125A0F863 contains 0A00000125A10863 contains 0A0000013AC11863 contains 8A000001262E8963
pfn 125a0f ---DA--KWEV pfn 125a10 ---DA--KWEV pfn 13ac11 ---DA--KWEV pfn 1262e8 -G-DA--KW-V
The "mystery" (to me) is what caused the general-protection exception.
According to "Intel 64 and IA-32 Architectures | Software Developer’s Manual", the only reason for a general-protection exception when executing the xadd instruction is "If the memory address is in a non-canonical form". I think that the !pte command demonstrates that this is not the case.
The limited range of models and CPU types on which the problem has been observed suggests that there is a hardware dependency, so the obligation to resolve the problem does not rest solely on Microsoft.
Gary