RegOpenKeyTransacted returns 6801 for SOFTWARE/HKCU, while SYSTEM and .DEFAULT succeed

Hello @S Sa ,

Thank you for the detailed description.

6801 (ERROR_RM_NOT_ACTIVE) and 0xC0190005 (STATUS_RM_NOT_ACTIVE) mean the transaction resource manager (RM) is "not started or was shut down due to an error" (System Error Codes 6000–8199, MS-ERREF NTSTATUS). This points to the RM of the affected hives rather than to the transaction itself.

1. Diagnostic error vs. hive-specific issue

Your results already point to a hive-specific issue rather than an error in the diagnostic. In my tests, mistakes in transaction usage return different codes, never 6801:

• an inactive transaction (committed, rolled back, or timed out) returns 6701;
• a NULL or invalid transaction handle returns 6700;
• a closed handle or a non-transaction handle returns 6;
• a handle without TRANSACTION_ENLIST returns 5.

These codes are listed in System Error Codes 6000–8199, and RegOpenKeyTransacted requires "a handle to an active transaction".

To confirm, use one transaction to open HKLM\SYSTEM, then HKLM\SOFTWARE, then HKLM\SYSTEM again, and call GetTransactionInformation after the failing call. If the transaction is still active, the third open succeeds, and the transaction can still be committed, then your API usage is correct and the failure is specific to the hive. Testing HKCU with a new local user can confirm whether the original profile hive is affected. I'd avoid HKLM\SAM, HKLM\BCD00000000 and hives loaded with RegLoadAppKey as test targets, because they return 6801 on transacted open even on healthy systems.

For context, Transactional Registry relies on KTM and CLFS logs (About KTM), which "can be corrupted" (Log Types). Each hive has its own RM log files next to the hive file, for example SOFTWARE{GUID}.TM.blf and SOFTWARE{GUID}.TMContainer*.regtrans-ms in C:\Windows\System32\config, and NTUSER.DAT{GUID}.TM* in the user profile.

On a lab machine with build 26200.9457, I reproduced your exact symptom by corrupting the first log container of a test hive while leaving its .blf intact:

• ordinary opens succeed;
• transacted operations return 6801 / 0xC0190005, with or without elevation, from both 64-bit and 32-bit processes;
• the error persists across reloads;
• other hives in the same transaction keep working.

This per-hive behavior is not documented.

2. Event logs, providers, and diagnostics

As you noted, 6801 is an API return code. In my repro, no event was written to the System or Application logs, so the failure has to be captured with tracing. The following tools captured it:

• Process Monitor shows RegOpenKey on the affected key with Result 0xC0190005. The hive's {GUID}.TM.blf and .regtrans-ms files are opened and read without file-level errors but are never written to, whereas a healthy hive's RM files are written. Process Monitor also supports boot-time logging, which can be used to observe hives loaded at startup.
• WPR with the built-in "Registry I/O activity" profile (wpr -start Registry, Built-in Recording Profiles) records the registry Open event for the key with status 0xC0190005.
• The ETW provider Microsoft-Windows-Kernel-Registry records event ID 2 with Status=0xC0190005 and the key name. Its event schema is not publicly documented.

logman start TxRReg -p Microsoft-Windows-Kernel-Registry 0xFFFFFFFFFFFFFFFF 0xFF -o C:\Temp\kreg.etl -ets
(run your diagnostic)
logman stop TxRReg -ets
tracerpt C:\Temp\kreg.etl -o C:\Temp\kreg.xml -of XML -y

Comparing the size and timestamps of the SOFTWARE{GUID}.* and NTUSER.DAT{GUID}.* files with a healthy machine can also help confirm whether the same cause applies on your system.

3. fsutil resource setautoreset

Based on the documentation and my testing, it only affects the volume's default resource manager, not the registry hive resource managers involved here, and I'm not aware of any documentation that applies it to this symptom.

The command applies to the volume's default Transactional Resource Manager (fsutil resource), and "each NTFS volume is a resource manager" (Transactional NTFS). The article that recommends it addresses Windows Update/TxF failures, not registry hives (Troubleshoot error 0x80071A91).

I tested this on a separate NTFS volume. Running the command and remounting the volume did reset the volume's RM (its identifier changed), but a hive on that volume with a corrupted RM log still returned 6801. So I wouldn't expect it to resolve this issue.

If you can share the relevant Process Monitor or ETW entries, I'm happy to help confirm the cause.

Hope this helps. If you found my response helpful or informative, I would greatly appreciate it if you could follow this guide so others can find it.

Thank you.