How can a Windows desktop app reliably locate and highlight controls in other applications for step-by-step guidance?

perry jones 0 Reputation points
2026-10-09T00:18:01.2366667+00:00

I'm developing a Windows desktop application called Not My Grandkid, designed to help older adults and people who are uncomfortable using computers.

Our goal is to provide an interactive assistant that can guide users through tasks on their actual computer, both online and offline.

This is intended to work across supported Windows desktop versions, not exclusively Windows 11.

What the application needs to do

When someone asks for help performing a task, the software should:

  1. Understand what the user wants to accomplish.

Identify the correct application, window, button, menu, or control.

Locate that control on the user's actual screen.

Visually highlight the control and explain what to do through voice and text.

Wait for the user to perform the action.

Confirm the result and continue guiding the user through the remaining steps.

The objective is real-time, interactive assistance across Windows and third-party applications, rather than simply displaying instructions.

Current development status

We have implemented portions of the voice guidance and visual overlay systems.

We've also worked on locating and highlighting Windows taskbar controls, including handling monitor selection and screen coordinates.

However, we have not yet achieved reliable end-to-end guidance through actual application controls.

Where we need technical guidance

Is Microsoft UI Automation the appropriate foundation for identifying controls across different Windows versions and desktop applications?

How can we reliably locate and highlight controls in Win32, WPF, WinUI, and other application frameworks?

What approaches are recommended when an application does not expose its controls through UI Automation?

How should we manage DPI scaling, multiple monitors, moving windows, and dynamically changing interfaces?

How can the application reliably detect that the user completed a step before proceeding?

Are there Microsoft-supported accessibility APIs, development frameworks, or reference implementations suitable for building this kind of interactive desktop assistant?

What limitations should we anticipate when supporting different Windows versions?

We are not trying to take control away from users. Our goal is to provide accessible, interactive guidance while allowing users to perform actions themselves.

We're looking for technical direction from developers familiar with Windows accessibility, UI Automation, desktop application integration, and assistive software.I'm developing a Windows desktop application called Not My Grandkid, designed to help older adults and people who are uncomfortable using computers.

Our goal is to provide an interactive assistant that can guide users through tasks on their actual computer, both online and offline.

This is intended to work across supported Windows desktop versions, not exclusively Windows 11.

What the application needs to do

When someone asks for help performing a task, the software should:

Understand what the user wants to accomplish.

Identify the correct application, window, button, menu, or control.

Locate that control on the user's actual screen.

Visually highlight the control and explain what to do through voice and text.

Wait for the user to perform the action.

Confirm the result and continue guiding the user through the remaining steps.

The objective is real-time, interactive assistance across Windows and third-party applications, rather than simply displaying instructions.

Current development status

We have implemented portions of the voice guidance and visual overlay systems.

We've also worked on locating and highlighting Windows taskbar controls, including handling monitor selection and screen coordinates.

However, we have not yet achieved reliable end-to-end guidance through actual application controls.

Where we need technical guidance

Is Microsoft UI Automation the appropriate foundation for identifying controls across different Windows versions and desktop applications?

How can we reliably locate and highlight controls in Win32, WPF, WinUI, and other application frameworks?

What approaches are recommended when an application does not expose its controls through UI Automation?

How should we manage DPI scaling, multiple monitors, moving windows, and dynamically changing interfaces?

How can the application reliably detect that the user completed a step before proceeding?

Are there Microsoft-supported accessibility APIs, development frameworks, or reference implementations suitable for building this kind of interactive desktop assistant?

What limitations should we anticipate when supporting different Windows versions?

We are not trying to take control away from users. Our goal is to provide accessible, interactive guidance while allowing users to perform actions themselves.

We're looking for technical direction from developers familiar with Windows accessibility, UI Automation, desktop application integration, and assistive software.

Windows development | WinUI
0 comments

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Gatlin Le (WICLOUD CORPORATION) 985 Reputation points Microsoft External Staff Moderator
    2026-10-09T02:36:29.3933333+00:00

    Hi @perry jones ,

    You've already built the voice guidance, the overlay, and the taskbar targeting, so extending this to application controls is mostly about making each step reliable.

    Yes, Microsoft UI Automation (UIA) is the appropriate primary foundation for this assistant. It provides a common accessibility model across desktop frameworks, but coverage depends on what each application's controls expose. Reliable guidance therefore needs workflows with selectors and completion conditions validated for the applications you support. Microsoft's UI Automation client guide covers that client model.

    For locating application controls, start from the intended window's HWND using ElementFromHandle, then search within the relevant container. Combine AutomationId, ControlType, and parent context, and handle multiple matches explicitly. AutomationId is optional and is not guaranteed to survive application updates; Name can be localized. Keep selectors specific to the supported app and version, rather than assuming one tree shape across Win32, WPF, and WinUI. See obtaining elements and element property definitions.

    For highlighting, keep BoundingRectangle in physical screen pixels until it reaches the rendering boundary. Verify the overlay's PerMonitorV2 configuration in its application manifest; Per-Monitor V2 requires Windows 10 version 1703 or later. When drawing inside WinUI XAML, translate the screen rectangle into the XAML host's client coordinate space, then convert the resulting physical offsets and sizes using that root's current XamlRoot.RasterizationScale. Native positioning APIs that expect physical pixels should receive physical values. Dividing absolute desktop coordinates by one global scale factor is insufficient for mixed-DPI monitors.

    Refresh the bounds after movement, resizing, scrolling, or DPI changes, and reacquire elements after the UI replaces them. Hide the highlight while a target is unavailable. For virtualized lists, the target may not yet exist as a full UIA element; in your guidance-only design, prompt the user to reveal it and then resolve it again. Microsoft's virtualization documentation explains this limitation.

    For step completion, use UIA events as triggers to check the expected post-condition. Record the initial state and subscribe before displaying or speaking the prompt. After an event, re-read the expected result: for example, the intended dialog opened or the checkbox reached the requested ToggleState. A click, focus change, or disappearing control alone does not establish task completion. Keep bounded polling for missing events, because providers do not raise every possible event. Run UIA on a dedicated MTA worker that does not own windows, add and remove handlers on that same worker, and reject callbacks belonging to an earlier step. Microsoft's threading guidance covers the underlying requirements.

    When a control lacks a native UIA provider, check whether it exposes MSAA through UIA's LegacyIAccessible bridge. SetWinEventHook can supplement movement and window notifications, but it does not create missing control identities or completion semantics. Filter hooks to the relevant target. For this design, I recommend registering WINEVENT_OUTOFCONTEXT hooks on a separate thread with a message loop, then queueing refresh requests to the UIA worker. If neither accessibility interface supplies usable information, an app-specific integration or visual recognition is a separate implementation choice. Treat visual matches as uncertain and retain a user-confirmed fallback when completion cannot be verified.

    Elevated applications need separate consideration. uiAccess has signing, installation-location, and launch-context requirements; adding the manifest flag alone does not grant unrestricted access. In particular, Microsoft's assistive-technology security guidance notes that a UIAccess application launched by a non-admin user cannot access high-integrity UI. These privileges also do not grant access to system-integrity UI.

    For development, Accessibility Insights or Inspect can establish what a target actually exposes. Microsoft also documents winapp ui inspect. The UI Automation document client sample demonstrates cross-process access through TextPattern; it is a component example, not a complete guidance assistant. For Windows coverage, validate each supported app/OS combination and check the Windows App SDK support matrix, including edition and servicing channel, separately from API availability.

    For the next investigation, please choose one application control that currently fails and share the app/version, Windows build, elevation status, and its inspector properties alongside your resolver's result. Please indicate whether the failure is locating the element, drawing the highlight, or detecting completion. That comparison will help distinguish an accessibility-provider limitation from a problem in resolver logic, coordinate conversion, or completion-condition logic.

    If you found my explanation above was truly helpful or informative to you, I would greatly appreciate it if you could follow this guidance so others with the same issue could benefit as well.

    Was this answer helpful?

    0 comments

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.