Windows desktop capture/streaming app becomes unresponsive

Hello,

I am developing a Windows desktop communication application with real-time screen sharing. The application captures the entire monitor and streams it to another user.

The app uses DXGI Desktop Duplication / Windows Graphics Capture and a GPU-based capture/encoding pipeline. I am investigating an issue that occurs when a heavy game is active in the foreground and the GPU is under 99–100% load.

Observed behavior:

When the game is active in the foreground, my application window may stop updating and Windows may mark it as “Not Responding”.

The screen capture / streaming pipeline can become unstable or appear to stall.

If I Alt+Tab to my application or to another window, the application immediately “wakes up” and capture/telemetry starts updating again.

If I limit the game FPS so GPU usage drops to around 96–97%, the capture/streaming pipeline works much better.

A separate lightweight DXGI probe process can still receive frames in the same scenario, so I am trying to determine whether this is related to scheduling, UI message pump starvation, DWM/GPU priority, WGC/DXGI behavior, or blocking inside my application pipeline.

Questions:

Is there any official Windows mechanism to prevent a desktop capture/streaming application from being starved when a fullscreen or borderless game is active and the GPU is fully loaded?

What is the recommended architecture for keeping a desktop communication app responsive while running real-time monitor capture and GPU encoding in the background?

Are there official best practices for:

DXGI Desktop Duplication under heavy foreground GPU load;

Windows Graphics Capture under heavy foreground GPU load;

DWM scheduling and foreground application priority;

MMCSS for capture/encoding/network threads;

Direct3D GPU thread priority;

avoiding the “Not Responding” state when the app window is not foreground?

Does Windows provide any supported way to reserve a small amount of GPU scheduling time for a background real-time capture/streaming process?

Are there known limitations when capturing a monitor while a game is using Independent Flip, Hardware Composed Independent Flip, MPO, VRR/G-Sync, or fullscreen optimizations?

Environment:

OS: Windows 11

GPU: NVIDIA RTX 3080

Capture target: full monitor capture

APIs tested: DXGI Desktop Duplication and Windows Graphics Capture

The issue is most visible when the foreground game drives GPU utilization to 99–100%

The issue becomes much less severe when the game is FPS-capped and leaves a few percent of GPU headroom

I am not looking for undocumented hooks or driver-level methods. I want to understand the officially supported Windows approach for building a low-latency real-time screen sharing application that remains responsive under heavy foreground game GPU load.

Thank you.